Deep learning in ArcGIS Pro

Available with Image Analyst license.

ArcGIS Pro allows you to use statistical or machine learning classification methods to classify remote-sensing imagery. Deep learning is a type of machine learning that relies on multiple layers of nonlinear processing for feature identification and pattern recognition described in a model. Deep learning models can be integrated with ArcGIS Pro for object detection, object classification, and image classification.

Training samples of features or objects of interest are generated in ArcGIS Pro with classification and deep learning tools. These training samples are used to train a deep learning model using a geoprocessing tool, and the resulting model definition file or deep learning model package (.dlpk) is used to run the inference geoprocessing tools to extract specific object locations, classify or label the objects, or classify the pixels in the imagery. You can also train the model outside of ArcGIS Pro using an external third-party framework and use a model definition file to run the inference geoprocessing tools. Model definition files and .dlpk packages can be used multiple times as inputs to the geoprocessing tools, allowing you to assess multiple images over different locations and time periods, after the model has been trained.

The workflow is represented in the diagram below.

Step	Description
	Create training samples in the Label Objects for Deep Learning pane, and use the Export Training Data For Deep Learning tool to convert the samples into deep learning training data.
	Use the Train Deep Learning Model tool to train a model using PyTorch, or train the model outside of ArcGIS Pro using a supported third-party deep learning framework.
	Use the trained model to run the Detect Objects Using Deep Learning tool, the Classify Pixels Using Deep Learning tool, or the Classify Objects Using Deep Learning tool to generate an output.

Features and capabilities

Deep learning tools in ArcGIS Pro allow you to use more than the standard machine learning classification techniques.

• Use convolutional neural networks or deep learning models to detect objects, classify objects, or classify image pixels.
• Integrate external deep learning model frameworks, such as TensorFlow, PyTorch, and Keras.
• Use a model definition file multiple times to detect change over time or detect objects in different areas of interest.
• Generate a polygon feature class showing the location of detected objects to be used for additional analysis or workflows.
• Take advantage of GPU processing as well as using CPU for distributed processing.

Get started with deep learning

The creation and export of training samples are done in ArcGIS Pro using the standard training sample generation tools. The deep learning model can be trained with the PyTorch framework using the Train Deep Learning Model tool, or it can be trained outside of ArcGIS Pro using another deep learning framework. Once the model is trained, use an Esri model definition file (.emd) to run geoprocessing tools to detect or classify features in your imagery.

You must install the deep learning framework Python packages; otherwise, an error occurs when you add the Esri model definition file to the deep learning geoprocessing tools. For information about how to install these packages, see Install deep learning frameworks for ArcGIS.

1. Create and export training samples.
   1. Use the Label Objects for Deep Learning pane or the Training Samples Manager to select or create a classification schema.
   2. Create training site samples for the class categories or features of interest. Save the training sample file.
   3. Run the Export Training Data for Deep Learning geoprocessing tool to convert the source imagery and training samples to deep learning training data. The source imagery can be an image service, a mosaic dataset, a raster dataset, or a folder of rasters. The output of the tool is image chips or samples containing training sites to be used to train the deep learning model. An additional output of the tool is a template .emd file that must be populated.
2. Train the deep learning model.
   1. Use the Train Deep Learning Model tool to train a deep learning model using the image chips generated in the previous step.
3. Run the inference geoprocessing tools in ArcGIS Pro.
   1. Use the Detect Objects Using Deep Learning, Classify Pixels Using Deep Learning, or Classify Objects Using Deep Learning geoprocessing tool to process your imagery. If the trained model incorporated custom Python raster functions with additional variables, such as padding or a confidence threshold for fine-tuning the sensitivity, these variables appear on the geoprocessing tool dialog box for user input. The data type, such as string or numeric, is specified in the Python raster function. Ideally, additional inference parameters should be limited to two.

      The Esri model definition parameter value can be an Esri model definition JSON file (.emd) or a JSON string. A JSON string is useful when this tool is used on the server so you can paste the JSON string, rather than upload the .emd file.

      The output of the Detect Objects Using Deep Learning tool is a feature class showing the objects detected by the model. The Classify Pixels Using Deep Learning tool outputs a classified raster. The Classify Objects Using Deep Learning tool requires a feature class and imagery as the input datasets, and the result is a feature class in which each object within each feature is labelled with a class name.

For information about requirements for running this tool and issues you may encounter, see Deep Learning in ArcGIS Pro Frequently Asked Questions.

Esri model definition file

The .emd file is a JSON file that describes the trained deep learning model. It contains model definition parameters that are required to run the inference tools, and it should be modified by the data scientist who trained the model. There are required and optional parameters in the file as described in the table below.

Once the .emd file is completed and verified, it can be used in inferencing multiple times, as long as the input imagery is from the same sensor as the original model input, and the classes or objects being detected are the same. For example, an .emd file that was defined with a model to detect oil well pads using Sentinel-2 satellite imagery can be used to detect oil well pads across multiple areas of interest and multiple dates using Sentinel-2 imagery.

Some parameters are used by all the inference tools; these are listed in the table below. Some parameters are only used with specific tools, such as the CropSizeFixed and the BlackenAroundFeature parameters, which are only used by the Classify Objects Using Deep Learning tool.

Model definition file parameter	Explanation
Framework	The name of a deep learning framework used to train your model. The following deep learning frameworks are supported: TensorFlow Keras PyTorch If your model is trained using a deep learning framework that is not listed, a custom inference function (a Python module) is required with the trained model, and you must set InferenceFunction to the Python module path.
ModelConfiguration	The name of the model configuration. The model configuration defines the model inputs and outputs, the inferencing logic, and the assumptions made about the model inputs and outputs. Existing open source deep learning workflows define standard input and output configuration and inferencing logic. ArcGIS supports the following set of predefined configurations: TensorFlow ObjectDetectionAPI DeepLab Keras MaskRCNN If you used one of the predefined configurations, type the name of the configuration in the .emd file. If you trained your deep learning model using a custom configuration, you must describe the inputs and outputs in full in the .emd file or in the custom Python file.
ModelType	The type of model. ImageClassification—For classifying pixels ObjectDetection—For detecting objects or features Object Classification—For classifying objects and features
ModelFile	The path to a trained deep learning model file. The file format depends on the model framework. For example, in TensorFlow, the model file is a .pb file.
Description	Provide information about the model. Model information can include anything to describe the model you have trained. Examples include the model number and name, time of model creation, performance accuracy, and more.
InferenceFunction (Optional)	The path of the inference function. An inference function understands the trained model data file and provides the inferencing logic. There are six inference functions that are supported in the ArcGIS Pro deep learning geoprocessing tools: Detect Objects for TensorFlow Classify Pixels for Tensor Flow Detect Objects for Keras Detect Objects for PyTorch Classify Objects for Pytorch If you used one of the inference functions above, there is no need to specify it in the .emd file. If your model is trained using a deep learning model configuration that is not yet supported, or it requires special inferencing logic, a custom inference function (a Python module) is required with the trained model. In this case, set InferenceFunction to the Python module path. An inference Python module file can be located anywhere ArcGIS Pro can access.
SensorName (Optional)	The name of the sensor used to collect the imagery from which training samples were generated.
RasterCount (Optional)	The number of rasters used to generate the training samples.
BandList (Optional)	The list of bands used in the source imagery
ImageHeight (Optional)	The number of rows in the image being classified or processed.
ImageWidth (Optional)	The number of columns in the image being classified or processed.
ExtractBands (Optional)	The band indexes or band names to extract from the input imagery.
Classes (Optional)	Information about the output class categories or objects.
DataRange (Optional)	The range of data values if scaling or normalization was done in preprocessing.
ModelPadding (Optional)	The amount of padding to add to the input imagery for inferencing.
BatchSize (Optional)	The number of training samples to be used in each iteration of the model.
PerProcessGPUMemoryFraction (Optional)	The fraction of GPU memory to allocate for each iteration in the model. The default is 0.95, or 95 percent.
MetaDataMode (Optional)	The format of the metadata labels used for the image chips.
ImageSpaceUsed (Optional)	The type of reference system used to train the model. MAP_SPACE PIXEL_SPACE
WellKnownBandNames (Optional)	The names given to each input band, in order of band index. Bands can then be referenced by these names in other tools.
AllTileStats	The statistics of each band in the training data.

The following is an example of a model definition file (.emd) that uses a standard model configuration:

{
    "Framework": "TensorFlow",
    "ModelConfiguration": "ObjectDetectionAPI",
    "ModelFile":"C:\\ModelFolder\\ObjectDetection\\tree_detection.pb",
    "ModelType":"ObjectionDetection",
    "ImageHeight":850,
    "ImageWidth":850,
    "ExtractBands":[0,1,2],
    "Classes" : [
      {
        "Value": 0,
        "Name": "Tree",
        "Color": [0, 255, 0]
      }
    ]
}

The following is an example of a model definition file (.emd) with more optional parameters in the configuration:

{
    "Framework": "PyTorch",
    "ModelConfiguration": "FasterRCNN",
    "ModelFile":"C:\\ModelFolder\\ObjectDetection\\river_detection.pb",
    "ModelType":"ObjectionDetection",
				"Description":"This is a river detection model for  imagery",
    "ImageHeight":448,
    "ImageWidth":448,
    "ExtractBands":[0,1,2,3],
				"DataRange":[0.1, 1.0],
				"ModelPadding":64,
				"BatchSize":8,
				"PerProcessGPUMemoryFraction":0.8,
				"MetaDataMode" : "PASCAL_VOC_rectangles",
				"ImageSpaceUsed" : "MAP_SPACE",
    "Classes" : [
      {
        "Value": 1,
        "Name": "River",
        "Color": [0, 255, 0]
      }
    ],
				"InputRastersProps" : {
						"RasterCount" : 1,
						"SensorName" : "Landsat 8",
						"BandNames" : [
								"Red",
								"Green",
								"Blue",
								"NearInfrared"
						]
				},
				"AllTilesStats" : [
      {
      		"BandName" : "Red",
      		"Min" : 1,
      		"Max" : 60419,
      		"Mean" : 7669.720049855654,
      		"StdDev" : 1512.7546387966217
      },
      {
      		"BandName" : "Green",
      		"Min" : 1,
      		"Max" : 50452,
      		"Mean" : 8771.2498195125681,
      		"StdDev" : 1429.1063589515179
      },
      {
      		"BandName" : "Blue",
      		"Min" : 1,
      		"Max" : 47305,
      		"Mean" : 9306.0475897744163,
      		"StdDev" : 1429.380049936676
      },
      {
      		"BandName" : "NearInfrared",
      		"Min" : 1,
      		"Max" : 60185,
      		"Mean" : 17881.499184561973,
      		"StdDev" : 5550.4055277121679
      }
    ],
}

Deep learning model package

A deep learning model package (.dlpk) contains the files and data required to run deep learning inferencing tools for object detection or image classification. The package can be uploaded to your portal as a DLPK item and used as the input to deep learning raster analysis tools.

Deep learning model packages must contain an Esri model definition file (.emd) and a trained model file. The trained model file extension depends on the framework you used to train the model. For example, if you trained your model using TensorFlow, the model file will be a .pb file, while a model trained using Keras will generate an .h5 file. Depending on the model framework and options you used to train your model, you may need to include a Python Raster Function (.py) or additional files. You can include multiple trained model files in a single deep learning model package.

Most packages can be opened in any version of ArcGIS Pro. By default, the contents of a package are stored in the <User Documents>\ArcGIS\Packages folder. You can change this location in the Share and download options. Functionality in the package that is not supported at the version of ArcGIS Pro being used to consume the package is not available.

To view or edit the properties of a .dlpk package, or to add or remove files from your .dlpk package, right-click the .dlpk package in the Catalog pane and click Properties.

Properties include the following information:

• Input—The .emd file, trained model file, and any additional files that may be required to run the inferencing tools.
• Framework—The deep learning framework used to train the model.
• ModelConfiguration—The type of model training performed (object detection, pixel classification, or feature classification).
• Description—A description of the model. This is optional and editable.
• Summary—A brief summary of the model. This is optional and editable.
• Tags—Any tags used to identify the package. This is useful for .dlpk package items stored on your portal.

Any property that is edited in the Properties window is updated when you click OK. If the .dlpk package item is being accessed from your portal in the Catalog pane, the portal item is updated.

For information about how to create a .dlpk package, see Share a deep learning model package.