The (real) arrival of spatially-enabled big data analysis

Major advances in Esri technology make 2016 an exciting year for utilities interested in harnessing big data analysis. Brian Bell, Esri Canada’s utilities industry manager, drills down on how enhancements and strengthened partnerships have brought powerful capabilities to the ArcGIS platform that will change the way utilities operate.

Mark my words, when, in some distant future, we look back on the year 2016, it will be remembered as the year The Tragically Hip made their cross-country farewell tour, our Olympic athletes brought home 22 medals from Rio and spatially-enabled big data management and analysis truly arrived.

For years, the GIS and mapping community, along with major Business Intelligence solution providers, have been talking about big data analysis. To date, ArcGIS has supported big data initiatives in a number of different ways, ranging from storage and retrieval through to online analytical processing. But often, the act of actually implementing these initiatives come in the form of incredibly complex architectures, software pairings and long project timelines. This year marks the release of some major technology advancements in ArcGIS that will drastically improve utility big data analysis with the merging of real-time and geospatial capabilities.

Tackling this topic could easily become an opus, so in an effort to be brief, here’s what’s new and what it means to utilities:

1. ArcGIS Data Store

ArcGIS Data Store is a new part of ArcGIS Server with 10.4; it's built right into the installer as an option. It’s an optimized spatiotemporal Big Data repository that’s pre-configured to act as a storage container for ArcGIS Feature Services in a relational and tile-cached format. And it's built to do it with high volume, real-time observational data sources when it's paired with the GeoEvent Extension for ArcGIS Server. This means that with ArcGIS Data Store, you can have a reliable container that can store and retrieve spatial, temporal and statistical data in real-time from high-volume observational sources, such as Smart Meter and SCADA systems. This will allow utilities to start pairing big data-generating systems (such as those I just mentioned and others) with the rest of their GIS data and network, opening the door for a whole new paradigm of utility-grade analysis.

And did I mention it manages all of this through back-end caching and web services to support dynamic 2D and 3D presentation? This has the potential to change the game for how utilities think about GIS and big data.

ArcGIS Data Store is a component of ArcGIS Server, accessed and configured directly within your web GIS portal. Data is stored, accessed and administered through ArcGIS Web Services that are optimized for fast, large-volume retrieval, making it easy for data and analysis to support your other GIS-centric workflows.

2. True big-data geoanalytics toolsets are here to complement a couple that have been around for a while

Esri announced two new parts of the ArcGIS platform that will be entering the fold in the second half of 2016: GeoAnalytics and Insights for ArcGIS. GeoAnalytics will bring large-scale fast batch analysis to ArcGIS Server, and, if you want to, you'll even be able to use Esri’s ArcGIS cloud infrastructure to support CPU-intensive processes. Insights for ArcGIS provides a means to work with these data sources and others in a web application that’s intuitive, extremely powerful and directly integrated with the rest of the ArcGIS platform, which will enable utilities to bring spatial analytics from the backroom to the executive suite.

Insights for ArcGIS performs spatial and statistical analysis through multiple information panes, which are implicitly connected to one another, producing multiple levels of insight into your data with a single click.

In addition to these recent releases, a popular framework for big data analysis that we've supported for several years are GIS Tools for Hadoop (http://esri.github.io/gis-tools-for-hadoop/). This toolset enables you to leverage the power of ArcGIS directly within Hadoop, which many utilities already have and are using to varying degrees.

3. Powerful partnerships are making it easier

Anyone who's worked with Esri technology knows one of our key strengths is our partners. This is no exception in the big data and real-time scene. Among many others, we've made great strides with OSIsoft and SAP, making the utilization of big data and real-time alongside GIS accessible as a core capability between our systems. Among other major workflows supported by ArcGIS and SAP, we jointly released support for connecting to SAP HANA as a Query Layer source, which enables SAP-managed, in-memory datasets to be visualized and analyzed across the ArcGIS platform, right out of the box. With OSIsoft, we’ve matured our joint offering for integrating the PI architecture with ArcGIS to create the PI Integrator for Esri ArcGIS, which has now been implemented successfully at many utilities in North America, adding a crucial level of spatial analysis capabilities to the incredibly powerful OSIsoft PI system.

An ArcGIS Operations Dashboard being fed in real time from OSIsoft’s PI Server. The PI Coresight advanced analysis widget is also shown in the upper middle of the screen, providing time-series analysis alongside real-time visualization.

These are just a few of the ways that GIS, big data and real-time capabilities have been converging in 2016. I think that in a few years, we’ll look back to 2016 as the year when it all started to make sense, and our technology really began helping utilities address their real-time big data analysis needs. If you'd like to learn more about what we're doing in this space, or to discuss some of your current challenges, feel free to contact us.