Automate Image Analysis with GeoAI and Deep Learning Webinar Q&A

We’re following up on our webinar ‘Automate Image Analysis with GeoAI and Deep Learning Workflows’ by answering your questions covering Tool and Processing Requirements, Model Training and Advanced Deep Learning

Machine learning and deep learning and GeoAI, oh my! These topics may seem intimidating, but ArcGIS Pro makes automating imagery tasks a breeze!

During our recent webinar, Jeff Petillion and I took the fear out of the unknown and demonstrated some of the machine learning and deep learning imagery applications in ArcGIS Pro, from image classification to training a custom model to using pretrained deep learning models for common tasks. If you missed it, check out the webinar recording to see what we covered.

Now that the curtain has been drawn, we are answering some of your questions to make leveraging GeoAI even easier.

Tool and Processing Requirements

Q: How do you install the deep learning frameworks in ArcGIS Pro?

A: The deep learning libraries/frameworks can be downloaded from the Deep Learning Libraries Installer GitHub page. The page also describes the other requirements for using the deep learning tools, including which license extensions you need and the system requirements for optimal performance. Check out the short video below for a visual guide to installing the deep learning frameworks on your computer for use in ArcGIS Pro.

VIDEO: Install Deep Learning Libraries in ArcGIS Pro

Q: How can we tell if our computer is capable of running the deep learning tools?

A: Most of the packages used in the deep learning tools can be run on any machine configuration, however some require a GPU, which results in better computing performance all around. The blog post, Deep Learning with ArcGIS Pro Tips & Tricks, goes into the details of all steps to get started with deep learning in ArcGIS Pro, including how to verify if you have a compatible GPU on your machine.

Q: Are there any data or processing limitations for machine learning and deep learning tools?

A: In general, the limitations for using image classification and deep learning tools are computer processing power, rather than inherent tool limitations on the amount of data. For general processing requirements in ArcGIS Pro please see the ArcGIS Pro system requirements help page. ArcGIS Pro also offers enhanced performance with GPU processing, accessible through the geoprocessing environments setting. For running training and inferencing deep learning tools in ArcGIS Pro it is recommended to have 8GB VRAM. If you are only performing inferencing (ex. detection or classification with a pretrained model), 4GB is the minimum required VRAM, but 8GB is recommended. If you don’t have the required 4–8GB VRAM, you can run deep learning tools on the CPU, though the processing time will be longer. See the Deep Learning Frameworks GitHub page for more information on the system requirements for using the deep learning tools in ArcGIS Pro.

Q: Can the image classification wizard be run using the GPU? Similar to how deep learning models (e.g., "Classify Pixels Using Deep Learning") can be run on the GPU with the tool's environment settings?

A: There is currently no option to run the image classification wizard or their equivalent geoprocessing tools using the GPU, only the deep learning tools and some tools in the Spatial Analyst toolbox. However, if you are looking for improved performance, the image classification geoprocessing tools do have a Parallel Processing factor option where the operation can be spread across multiple CPU cores.

Q: Are there credit costs when using the machine learning and deep learning models and tools?

A: It depends on whether you are using the tools in ArcGIS Pro or ArcGIS Online.

In ArcGIS Pro, the machine learning and deep learning tools are available with the image analyst toolbox and do not consume credits to use. The pretrained models available from the ArcGIS Living Atlas can be downloaded for free and do not cost credits to use with the ArcGIS Pro deep learning geoprocessing tools.

However, deep learning tools are also available for use in ArcGIS Online and those do consume service credits to run, just like the other analysis tools in ArcGIS Online. For example, downloading a pretrained deep learning model from the Living Atlas is free but using it to inference a hosted dataset in the ArcGIS Online Map Viewer would consume service credits. To use the Deep Learning Tools in ArcGIS Online, customers must have the ArcGIS Image for ArcGIS Online user type extension to access the deep learning tools in the Analysis tab. When using the analysis tools in ArcGIS Online, you can estimate the credits that a process will consume before running it.

Estimate credit use for the deep learning analysis tools in ArcGIS Online.

Model Training

Q: Can you run the image classification on an image that doesn’t have training data?

A: When using the Image Classification Wizard users have the option to generate an Output Classifier Definition File (.ecd file) during the Classify step. This .ecd file allows for subsequent classifications to be performed on spectrally similar rasters without having to train new data samples. Use the Classify Raster (Spatial Analyst) tool with the .ecd file as an input to perform this type of image classification.

Q: What is data augmentation and when should it be used?

A: Data augmentation is the process of artificially increasing the amount of training data you have by transforming your existing training data to create additional image chips. This is a great way to improve model performance when you are limited by the amount of available training data (for example, limited field samples, small area etc.).

Data augmentation can be performed with the Export Training Data for Deep Learning geoprocessing tool by setting the optional rotation parameter. There are additional data augmentation options that can be set using the prepare_data() function in the arcgis.learn module of the ArcGIS API for Python.

Q: What is the best size/shape to use when drawing polygons around object training samples to train an object detection model?

A: When collecting training samples for an object detection model, your polygon or bounding box should be as tight as possible to the object you are trying to detect, while still including all pixels of the object. A couple of pixels around the object is fine but the better the fit, the clearer the object delineation is to the computer. The choice of shape (polygon, rectangle, circle) will depend on the general shape of your object; palm trees may be best detected as circles while houses may need hand-drawn polygons to represent their shape.

Object training polygons should be drawn tightly around the object you are trying to detect.

Q: Can you tune the hyperparameters of deep learning models in ArcGIS Pro?

A: Yes! The hyperparameters of the deep learning tools include batch size and learning rate. These parameters can be set in the geoprocessing tool window or the tool arguments in python.  Hyperparameter tuning is often a time-intensive task that involves trial and error to find the optimal values for your model.

Q: How does image resolution impact the quality of the training data and the precision of the overall model?

A: A good rule of thumb is that if you (a human, I hope) can easily identify the objects or classes in the image, then a computer can be trained to do the same. Just as with us, the higher the image resolution, the easier it is for the computer to distinguish between objects and classes, but it also depends on what we are trying to identify. Small objects like boats will benefit from higher resolution imagery, while some larger objects like lakes can easily be detected at lower resolutions. The appropriate image resolution for your task will depend on what you are trying to accomplish and what imagery you have available.

The best resolution will depend on what you are trying to detect.

Q: How do we know when to use supervised or unsupervised classification methods?

A: A general guideline between supervised or unsupervised classification methods is as follows:

Supervised Image Classification

• Use supervised classification when you have or can create a set of representative training samples that are accurately labeled with known classes.
• Suitable when you have prior knowledge about the classes present in the image and want to classify pixels into these predefined classes.
• Requires the user to manually define training samples or collect ground truth data for training.

Unsupervised Image Classification

• Use unsupervised classification when you have limited or no prior knowledge of the classes in the image.
• Suitable for exploratory analysis or when you want to identify patterns or groupings within the image data.
• Does not require labeled training samples. Instead, it automatically groups pixels into clusters based on their spectral characteristics.

It's also worth noting that in some cases, a combination of both supervised and unsupervised approaches might be beneficial. For instance, you can use an unsupervised classification initially to identify clusters and then use those clusters as training data for supervised classification to refine the classification results.

Q: What image formats can you use as inputs for machine and deep learning tools?

A: The machine learning and deep learning tools in ArcGIS Pro can be used with any Esri-supported raster format, multiple rasters in a mosaic dataset, an image service, a folder of images or a feature class with image attachments.

Note that when using your model to inference new imagery, the imagery format (including number of bands, bit depth, etc.) should match the image format of the data the model was trained on. You may experience sub-optimal results or errors when using data that differs from the original training data format. If you are using a pretrained deep learning model, you can find the recommended imagery format in the guides and the descriptions available for each ‎Deep Learning Package on their item page of the ArcGIS Living Atlas.
 

The recommended imagery formats for each pretrained model are available on the model’s item page.

Q: Can you use multiple input types to improve segmentation of similar objects in Image Classification?

A: Segmentation can be performed in the Image Classification Wizard or with the Segment Mean Shift (Spatial Analyst) tool using a single Esri-supported raster as input. Although there is no option to use multiple raster inputs, there is an option to specify the band indexes, allowing users to select the three bands that offer the most differentiation between the features of interest.

Advanced Deep Learning

Q: Can the deep learning tools be run from Python?

A: All the deep learning geoprocessing tools have python equivalents in the ArcPy library. Example scripts can be found at the bottom of the documentation for each individual tool, ex: Train Deep Learning Model.

Additionally, the ArcGIS API for Python has even more deep learning functionality in the arcgis.learn module. You can combine tools from the ArcPy and ArcGIS API for Python libraries to write processing scripts in notebooks or other IDEs. For more information on the ArcGIS API for Python and the arcgis.learn module, please see the following blog post, Deep learning models in arcgis.learn.

Q: Is it possible to import an external model into ArcGIS Pro for use in the deep learning tools?

A: The only models that are available for use with the deep learning tools in ArcGIS Pro are listed on the deep learning models tool reference page. However, you can use the ArcGIS API for Python to access external models for use with the arcgis.learn module using Model Extension. Model Extension is a general-purpose wrapper for any object detection and pixel classification model (whether you found it or created it yourself) on top of the existing ArcGIS framework. Follow this guide to add a new model using Model Extension.

Q: Can you fine tune an existing (or pretrained) model with new training data?

A: Yes! The Train Deep Learning Model tool has a parameter called ‘Pre-trained Model’ under the Advanced section that allows you to load an existing pretrained model (whether trained by you or downloaded from the Living Atlas) to use as a base when training the new model. This process is called transfer learning, a technique in deep learning where an existing model is re-used for another similar task. Check out the Repurposing Deep Learning Models using Transfer Learning in ArcGIS blog post for more information on using transfer learning in ArcGIS.

A pretrained model can be loaded in the advanced parameters of the Train Deep Learning model tool.

That covers your key questions about working with GeoAI and imagery, from installing the deep learning libraries to tuning hyperparameters. Stay tuned for more GeoAI content.