Skip To Content

Deep learning in ArcGIS Pro

Mit der Image Analyst-Lizenz verfügbar.

ArcGIS Pro allows you to use statistical or machine learning classification methods to classify remote sensing imagery. Deep learning is a type of machine learning that relies on multiple layers of nonlinear processing for feature identification and pattern recognition described in a model. Deep learning models can be integrated with ArcGIS Pro for object detection and image classification. Training samples of features or objects of interest are generated in ArcGIS Pro with classification and deep learning tools. These training samples are used to train the model using a third-party deep learning framework by a data scientist or image scientist, and the model definition file is used to run the inference geoprocessing tools in ArcGIS Pro to either extract specific features or classify the pixels in the imagery. Model definition files can be used multiple times as inputs to the geoprocessing tools, making it easy to assess multiple images over different locations and time periods, after the model has been trained.

Deep learning workflow

Workflow item 1

Create training samples with the Training Samples Manager and use the Export Training Data for Deep Learning tool in ArcGIS Pro to prepare the data for the data scientist.

Workflow item 2

The data scientist uses the training data to develop models using a third-party deep learning framework.

Workflow item 3

The data scientist delivers a deep learning package, consisting of the model definition file and the trained model and/or Python script, back to the ArcGIS Pro processing and analysis environment.

Workflow item 4

Run the Detect Objects Using Deep Learning tool or the Classify Pixels Using Deep Learning tool to generate an output.

workflow item 5

You may leverage the ArcGIS API for Python to train the model with a deep learning framework.

Features and capabilities

Deep learning tools in ArcGIS Pro enable you to use more than the standard machine learning classification techniques.

  • Use convolutional neural networks for image classification.
  • Use a deep learning model to either classify image pixels or detect objects such as airplanes, trees, vehicles, water bodies, and oil well pads.
  • Integrate external deep learning model frameworks, including TensorFlow, CNTK, PyTorch and Keras.
  • Use a model definition file multiple times to detect change over time or detect objects in different areas of interest.
  • Generate a polygon feature class showing detected objects to be used for additional analysis or workflows.
  • The deep learning inference tools can use the CPU for distributed processing, or use the powerful GPU on each server node if available.

Get started with deep learning

The creation and export of training samples are done within ArcGIS Pro by a knowledgeable image analyst because they require strong knowledge of image classification workflows. The deep learning model is trained outside of ArcGIS Pro using a deep learning framework, by a data scientist or someone experienced in training deep learning models. Once the model is trained, you use an Esri model definition file (.emd) to run geoprocessing tools to detect or classify features in ArcGIS Pro. In some cases, all three steps may be completed by one analyst that has experience with deep learning models and ArcGIS Pro image classification.

You will need to install the proper deep learning framework Python API a cloned Python environment; otherwise, an error will occur when you add the Esri Model Definition file to the deep learning geoprocessing tools. Obtain the appropriate framework information from the data scientist, and refer to the framework's documentation for help with installation. For an example, see Set up TensorFlow deep learning framework for ArcGIS.

Additionally, the deep learning model framework must be installed on the machine where ArcGIS Pro is installed in order to run the inference tools because the inference logic is built into the framework Python API and is required for computation.

  1. Create and export training samples.
    1. Use the Training Samples Manager to select or create a classification schema.
    2. Create training site samples for the class categories or features of interest. Save the training sample file.
    3. Run the Export Training Data for Deep Learning geoprocessing tool to convert the source imagery and training samples into deep learning training data. The source imagery may be an image service, mosaic dataset, or raster dataset. The output of the tool is image chips or samples containing training sites to be used to train the deep learning model. An additional output of the tool is a template .emd file to be populated by the data scientist.
  2. Train the deep learning model.
    1. Use the image chips generated in ArcGIS Pro to train a model using a deep learning framework such as TensorFlow, CNTK, PyTorch or Keras.
    2. Update the .emd file with the name of the deep learning framework, the model configuration, and other optional properties. For information on required and optional parameters in the .emd file, see Esri model definition file and Detect Object Using Deep Learning.
  3. Run the inference geoprocessing tools in ArcGIS Pro.
    1. Use the Detect Objects Using Deep Learningor the Classify Pixels Using Deep Learning geoprocessing tools to process your imagery. If the trained model incorporated custom Python raster functions with additional variables, such as padding or confidence threshold for fine-tuning the sensitivity, these variables will appear on the geoprocessing tool's dialog box for user input. The data type, such as string or numeric, is specified in the Python raster function. Ideally, additional inference parameters should be limited to two.

      Beim Eingabemodell kann es sich um eine JSON-Datei mit der Esri-Modelldefinition (.emd) oder um eine JSON-Zeichenfolge handeln. Eine JSON-Zeichenfolge ist nützlich, wenn Sie dieses Werkzeug auf dem Server verwenden, sodass Sie die JSON-Zeichenfolge einfügen können, statt die .emd-Datei hochzuladen.

      The output of the Detect Objects Using Deep Learning tool is a feature class showing the objects detected by the model, and the Classify Pixels Using Deep Learning tool outputs a classified raster.

Esri model definition file

The .emd file is a JSON file that describes the trained deep learning model. It contains model definition parameters that are required to run the inference tools, and it should be modified by the data scientist that trained the model. There are required and optional parameters in the file as described in the table below.

Once the .emd file is completed and verified, it can be used in inferencing multiple times, as long as the input imagery comes from the same sensor as the original model input, and the classes or objects being detected are the same. For example, an .emd file that was defined with a model to detect oil well pads using Sentinel-2 satellite imagery can be used to detect oil well pads across multiple areas of interest and multiple dates using Sentinel-2 imagery.

Model definition file parameterExplanation

Framework

The name of a deep learning framework used to train your model.

Three deep learning frameworks are supported:

  • TensorFlow
  • CNTK
  • Keras
  • PyTorch

InferenceFunction

ModelConfiguration

The name of the model configuration.

The model configuration defines the model inputs and outputs, the inferencing logic, and the assumptions made about the model inputs and outputs. There are existing open source deep learning workflows that define standard input and output configuration and inferencing logic. ArcGIS supports a set of predefined configurations:

TensorFlow

CNTK

Keras

PyTorch

If you used one of the predefined configurations, type the name of the configuration in the .emd file. If you trained your deep learning model using a custom configuration, you will need to describe the inputs and outputs in the .emd file or in the custom Python file, in full.

ModelFile

The path to a trained deep learning model file. Many model framework file formats are based on Protocol Buffers, so the trained deep learning model is a .pb file, but other file formats are supported.

ModelType

The type of model. The type can be one of the following:

  • ImageClassification—For classifying pixels
  • ObjectDetection—For detecting objects or features

InferenceFunction

(Optional)

The path of the inference function.

An inference function understands the trained model data file and provides the inferencing logic. There are six inference functions that are supported in the ArcGIS Pro deep learning geoprocessing tools:

  • Detect Objects for TensorFlow
  • Classify Pixels for Tensor Flow
  • Detect Objects for CNTK
  • Classify Pixels for CNTK
  • Detect Objects for Keras
  • Detect Objects for PyTorch
If you used one of the inference functions above, there is no need to specify it in the .emd file. If your model is trained using a deep learning model configuration that is not yet supported, or it requires special inferencing logic, a custom inference function (a Python module) is required together with the trained model. In this case, set InferenceFunction to the Python module path. An inference Python module file can be located anywhere accessible by ArcGIS Pro.

ImageHeight

(Optional)

The number of rows in the image being classified or processed.

ImageWidth

(Optional)

The number of columns in the image being classified or processed.

ExtractBands

(Optional)

The band indexes or band names to extract from the input imagery.

Classes

(Optional)

Information about the output class categories or objects.

Below is an example of a model definition file (.emd) that uses a standard model configuration:

{
    "Framework": "TensorFlow",
    "ModelConfiguration": "ObjectDetectionAPI",
    "ModelFile":"C:\\ModelFolder\\ObjectDetection\\tree_detection.pb",
    "ModelType":"ObjectionDetection",
    "ImageHeight":850,
    "ImageWidth":850,
    "ExtractBands":[0,1,2],
    "Classes" : [
      {
        "Value": 0,
        "Name": "Tree",
        "Color": [0, 255, 0]
      }
    ]
}

Verwandte Themen