Why is the Big Data movement so important for GIS?

The term "Big Data" means different things to different people, but if you’ve used the Internet lately, then by default, you have probably used Big Data technology. But what does Big Data really mean and how do you know when to use it for your purposes? Is it a technology, a framework or an application? What does Big Data mean for general business and the GIS business in particular? Read this blog post to find out why Big Data is so important.

I recently spoke at a Big Data workshop that was organized by the Canadian Institute of Geomatics. There were speakers from across Canada and Europe and plenty of attendees with lots of good questions, ideas and perspectives. Workshop speakers covered the gamut of geospatial-related Big Data topics from policy, academic, technical, commercial and public utility aspects. So, was there a “eureka moment” at the workshop? Well not quite, but there were certainly some interesting concepts put forward.

There was consensus among the speakers that Big Data in geospatial applications can be used in a wide variety of applications including health care, environment, demographics, public safety, climate change, smart cities, image processing and public utilities. Here are several examples of Esri Big Data Use Cases. Many of the speakers indicated that these days, technology is not a significant barrier. However, despite the strides made to bring Big Data to the masses, the technology can’t be considered yet as a given. There are still technical concerns that need to be addressed in the development of any Big Data application.

The major area of commonality among the speakers was analytics. Big Data analytics are what’s really driving Big Data into the forefront of business, application and technology adoption. Before today’s Big Data techniques and technology were available, “tricks” such as data reduction, data smoothing, sample decreases and period of study decreases were used to make it possible for the technology at that time to obtain a scientifically defensible result within a reasonable amount of time.

This Alberta Wildfires Public Information App is a good example of a Big Data application. It displays real-time geotagged content from social media sites including Twitter, YouTube and Flickr, and was very useful in obtaining live updates on what was happening on the ground in Fort McMurray during the wildfires that started in May.

Today, computer storage, processing and networking capabilities have improved to an extent that it’s now possible to process every collected event, do some analysis and still get an answer quickly. For example, there are applications today that are continually collecting the locations of thousands of city taxis in real time, determining the optimal locations for their next potential fare pickup based on statistical analysis and dispatching available taxis to these locations. These Big Data systems are improving the efficiency of the city’s taxi systems, increasing customer satisfaction through obtaining a taxi quickly and reducing wasteful taxi traffic on the city streets. It’s a win for everyone.

One of the unique aspects of Big Data for GIS applications is that GIS needs to be measurement based. Both the value(s) collected and the positions collected are generally considered as almost exact. Therefore, all of the processing in GIS Big Data applications are considered precise numbers, and analysis is performed on these values within the tolerance specifications of the measurement.

Also, GIS Big Data applications require a lot of data, and the majority of this data is collected by sensors, packaged for transport and then sent to a Big Data storage facility. The Internet of Things (IoT) development is helping to make the collection of sensor data easier to implement. Many international standards are being developed and used for collecting data from both stationary sensors and moving sensors. Stationary sensors include water depth gauges, traffic cameras and air temperature sensors. Moving sensors include GPS-enabled position sensors, on-car video cameras and even mobile phones.

However, once the sensor data has been collected and sent to the host Big Data system, it needs to be quickly evaluated for atypical anomalies. This can be done using the ArcGIS GeoEvent Extension for Server software package. Examples of anomalies could be things like snow plows that have stopped moving in unexpected places, people out on parole who go to locations where they’re not supposed to be or simply the detection of a lightning strike. Further investigation or closer monitoring might be required when these anomalies are detected. However, if no anomaly is detected, then the sensor data needs to be stored in a Big Data store so that each event or reading can be found and further processed or analyzed as necessary.

Lots of technology has been developed over the years for implementing Big Data applications. Sensor technology has improved significantly so the proliferations of sensors will become common place. Wired and wireless networks will be put into place to collect sensor readings and bigger and faster computers will be installed for processing this data. However, don’t forget that eventually, a human needs to monitor for anomalies and notify others as per standard procedures. 

Now that you know this, is your organization ready for a Big Data application? If so, there are out-of-the-box (OOTB) software and services such as ArcGIS Online and ArcGIS for Server that can help you implement a Big Data system to improve your organization’s performance or increase its efficiency.