Deep learning in ArcGIS Pro

Disponible avec une licence Image Analyst.

ArcGIS Pro allows you to use statistical or machine learning classification methods to classify remote-sensing imagery. Deep learning is a type of machine learning that relies on multiple layers of nonlinear processing for feature identification and pattern recognition described in a model. Deep learning models can be integrated with ArcGIS Pro for object detection, object classification, and image classification.

Training samples of features or objects of interest are generated in ArcGIS Pro with classification and deep learning tools. These training samples are used to train a deep learning model using a geoprocessing tool, and the resulting model definition file or deep learning model package (.dlpk) is used to run the inference geoprocessing tools to extract specific object locations, classify or label the objects, or classify the pixels in the imagery. You can also train the model outside of ArcGIS Pro using an external third-party framework and use a model definition file to run the inference geoprocessing tools. Model definition files and .dlpk packages can be used multiple times as inputs to the geoprocessing tools, allowing you to assess multiple images over different locations and time periods, after the model has been trained.

The workflow is represented in the diagram below.

Step	Description
	Create training samples in the Label Objects for Deep Learning pane, and use the Export Training Data For Deep Learning tool to convert the samples into deep learning training data.
	Use the Train Deep Learning Model tool to train a model using PyTorch, or train the model outside of ArcGIS Pro using a supported third-party deep learning framework.
	Use the trained model to run the Detect Objects Using Deep Learning tool, the Classify Pixels Using Deep Learning tool, or the Classify Objects Using Deep Learning tool to generate an output.

Features and capabilities

Deep learning tools in ArcGIS Pro allow you to use more than the standard machine learning classification techniques.

• Use convolutional neural networks or deep learning models to detect objects, classify objects, or classify image pixels.
• Integrate external deep learning model frameworks, such as TensorFlow, PyTorch, and Keras.
• Use a model definition file multiple times to detect change over time or detect objects in different areas of interest.
• Generate a polygon feature class showing the location of detected objects to be used for additional analysis or workflows.
• Take advantage of GPU processing as well as using CPU for distributed processing.

Get started with deep learning

The creation and export of training samples are done in ArcGIS Pro using the standard training sample generation tools. The deep learning model can be trained with the PyTorch framework using the Train Deep Learning Model tool, or it can be trained outside of ArcGIS Pro using another deep learning framework. Once the model is trained, use an Esri model definition file (.emd) to run geoprocessing tools to detect or classify features in your imagery.

You must install the deep learning framework Python packages; otherwise, an error occurs when you add the Esri model definition file to the deep learning geoprocessing tools. For information about how to install these packages, see Install deep learning frameworks for ArcGIS.

1. Create and export training samples.
   1. Use the Label Objects for Deep Learning pane or the Training Samples Manager to select or create a classification schema.
   2. Create training site samples for the class categories or features of interest. Save the training sample file.
   3. Run the Export Training Data for Deep Learning geoprocessing tool to convert the source imagery and training samples to deep learning training data. The source imagery can be an image service, a mosaic dataset, a raster dataset, or a folder of rasters. The output of the tool is image chips or samples containing training sites to be used to train the deep learning model. An additional output of the tool is a template .emd file that must be populated.
2. Train the deep learning model.
   1. Use the Train Deep Learning Model tool to train a deep learning model using the image chips generated in the previous step.
3. Run the inference geoprocessing tools in ArcGIS Pro.
   1. Use the Detect Objects Using Deep Learning, Classify Pixels Using Deep Learning, or Classify Objects Using Deep Learning geoprocessing tool to process your imagery. If the trained model incorporated custom Python raster functions with additional variables, such as padding or a confidence threshold for fine-tuning the sensitivity, these variables appear on the geoprocessing tool dialog box for user input. The data type, such as string or numeric, is specified in the Python raster function. Ideally, additional inference parameters should be limited to two.

      La valeur du paramètre Esri model definition (Définition du modèle Esri) peut être un fichier JSON de définition de modèle Esri (.emd) ou une chaîne JSON. Une chaîne JSON est utile lorsque cet outil est utilisé sur le serveur de sorte qu’il est possible de coller la chaîne JSON au lieu de télécharger le fichier .emd.

      The output of the Detect Objects Using Deep Learning tool is a feature class showing the objects detected by the model. The Classify Pixels Using Deep Learning tool outputs a classified raster. The Classify Objects Using Deep Learning tool requires a feature class and imagery as the input datasets, and the result is a feature class in which each object within each feature is labelled with a class name.

For information about requirements for running this tool and issues you may encounter, see Deep Learning in ArcGIS Pro Frequently Asked Questions.

Esri model definition file

The .emd file is a JSON file that describes the trained deep learning model. It contains model definition parameters that are required to run the inference tools, and it should be modified by the data scientist who trained the model. There are required and optional parameters in the file as described in the table below.

Once the .emd file is completed and verified, it can be used in inferencing multiple times, as long as the input imagery is from the same sensor as the original model input, and the classes or objects being detected are the same. For example, an .emd file that was defined with a model to detect oil well pads using Sentinel-2 satellite imagery can be used to detect oil well pads across multiple areas of interest and multiple dates using Sentinel-2 imagery.

Some parameters are used by all the inference tools; these are listed in the table below. Some parameters are only used with specific tools, such as the CropSizeFixed and the BlackenAroundFeature parameters, which are only used by the Classify Objects Using Deep Learning tool.

Model definition file parameter	Explanation
Framework	The name of a deep learning framework used to train your model. The following deep learning frameworks are supported: TensorFlow Keras PyTorch If your model is trained using a deep learning framework that is not listed, a custom inference function (a Python module) is required with the trained model, and you must set InferenceFunction to the Python module path.
ModelConfiguration	The name of the model configuration. The model configuration defines the model inputs and outputs, the inferencing logic, and the assumptions made about the model inputs and outputs. Existing open source deep learning workflows define standard input and output configuration and inferencing logic. ArcGIS supports the following set of predefined configurations: TensorFlow ObjectDetectionAPI DeepLab Keras MaskRCNN If you used one of the predefined configurations, type the name of the configuration in the .emd file. If you trained your deep learning model using a custom configuration, you must describe the inputs and outputs in full in the .emd file or in the custom Python file.
ModelType	The type of model. ImageClassification—For classifying pixels ObjectDetection—For detecting objects or features Object Classification—For classifying objects and features
ModelFile	The path to a trained deep learning model file. The file format depends on the model framework. For example, in TensorFlow, the model file is a .pb file.
Description	Provide information about the model. Model information can include anything to describe the model you have trained. Examples include the model number and name, time of model creation, performance accuracy, and more.
InferenceFunction (Optional)	The path of the inference function. An inference function understands the trained model data file and provides the inferencing logic. There are six inference functions that are supported in the ArcGIS Pro deep learning geoprocessing tools: Detect Objects for TensorFlow Classify Pixels for Tensor Flow Detect Objects for Keras Detect Objects for PyTorch Classify Objects for Pytorch If you used one of the inference functions above, there is no need to specify it in the .emd file. If your model is trained using a deep learning model configuration that is not yet supported, or it requires special inferencing logic, a custom inference function (a Python module) is required with the trained model. In this case, set InferenceFunction to the Python module path. An inference Python module file can be located anywhere ArcGIS Pro can access.
SensorName (Optional)	The name of the sensor used to collect the imagery from which training samples were generated.
RasterCount (Optional)	The number of rasters used to generate the training samples.
BandList (Optional)	The list of bands used in the source imagery
ImageHeight (Optional)	The number of rows in the image being classified or processed.
ImageWidth (Optional)	The number of columns in the image being classified or processed.
ExtractBands (Optional)	The band indexes or band names to extract from the input imagery.
Classes (Optional)	Information about the output class categories or objects.
DataRange (Optional)	The range of data values if scaling or normalization was done in preprocessing.
ModelPadding (Optional)	The amount of padding to add to the input imagery for inferencing.
BatchSize (Optional)	The number of training samples to be used in each iteration of the model.
PerProcessGPUMemoryFraction (Optional)	The fraction of GPU memory to allocate for each iteration in the model. The default is 0.95, or 95 percent.
MetaDataMode (Optional)	The format of the metadata labels used for the image chips.
ImageSpaceUsed (Optional)	The type of reference system used to train the model. MAP_SPACE PIXEL_SPACE
WellKnownBandNames (Optional)	The names given to each input band, in order of band index. Bands can then be referenced by these names in other tools.
AllTileStats	The statistics of each band in the training data.

The following is an example of a model definition file (.emd) that uses a standard model configuration:

{
    "Framework": "TensorFlow",
    "ModelConfiguration": "ObjectDetectionAPI",
    "ModelFile":"C:\\ModelFolder\\ObjectDetection\\tree_detection.pb",
    "ModelType":"ObjectionDetection",
    "ImageHeight":850,
    "ImageWidth":850,
    "ExtractBands":[0,1,2],
    "Classes" : [
      {
        "Value": 0,
        "Name": "Tree",
        "Color": [0, 255, 0]
      }
    ]
}

The following is an example of a model definition file (.emd) with more optional parameters in the configuration:

{
    "Framework": "PyTorch",
    "ModelConfiguration": "FasterRCNN",
    "ModelFile":"C:\\ModelFolder\\ObjectDetection\\river_detection.pb",
    "ModelType":"ObjectionDetection",
				"Description":"This is a river detection model for  imagery",
    "ImageHeight":448,
    "ImageWidth":448,
    "ExtractBands":[0,1,2,3],
				"DataRange":[0.1, 1.0],
				"ModelPadding":64,
				"BatchSize":8,
				"PerProcessGPUMemoryFraction":0.8,
				"MetaDataMode" : "PASCAL_VOC_rectangles",
				"ImageSpaceUsed" : "MAP_SPACE",
    "Classes" : [
      {
        "Value": 1,
        "Name": "River",
        "Color": [0, 255, 0]
      }
    ],
				"InputRastersProps" : {
						"RasterCount" : 1,
						"SensorName" : "Landsat 8",
						"BandNames" : [
								"Red",
								"Green",
								"Blue",
								"NearInfrared"
						]
				},
				"AllTilesStats" : [
      {
      		"BandName" : "Red",
      		"Min" : 1,
      		"Max" : 60419,
      		"Mean" : 7669.720049855654,
      		"StdDev" : 1512.7546387966217
      },
      {
      		"BandName" : "Green",
      		"Min" : 1,
      		"Max" : 50452,
      		"Mean" : 8771.2498195125681,
      		"StdDev" : 1429.1063589515179
      },
      {
      		"BandName" : "Blue",
      		"Min" : 1,
      		"Max" : 47305,
      		"Mean" : 9306.0475897744163,
      		"StdDev" : 1429.380049936676
      },
      {
      		"BandName" : "NearInfrared",
      		"Min" : 1,
      		"Max" : 60185,
      		"Mean" : 17881.499184561973,
      		"StdDev" : 5550.4055277121679
      }
    ],
}

Deep learning model package

Un paquetage de modèle d’apprentissage profond (.dlpk) contient les fichiers et les données nécessaires à l’exécution des outils d’inférence d’apprentissage profond pour la détection des objets ou la classification des images. Il est possible de charger le paquetage sur le portail comme élément DLPK et de l’utiliser comme entrée des outils d’analyse raster pour apprentissage profond.

Les paquetages de modèle d’apprentissage profond doivent contenir un fichier de définition de modèle Esri (.emd) et un fichier de modèle entraîné. L’extension du fichier de modèle entraîné est fonction de la structure d’entraînement utilisée pour le modèle. Par exemple, un modèle entraîné avec TensorFlow génère un fichier portant l’extension .pb, alors qu’un modèle entraîné avec Keras produit un fichier portant l’extension .h5. Selon la structure et les options utilisées pour entraîner le modèle, il peut être nécessaire d’ajouter une fonction raster Python (.py) ou des fichiers supplémentaires. Il est possible d’inclure plusieurs fichiers de modèle entraîné dans un seul et même paquetage de modèle d’apprentissage profond.

La plupart des paquetages peuvent être ouverts avec n’importe quelle version de ArcGIS Pro. Par défaut, le contenu d’un paquetage est stocké dans le dossier <User Documents>\ArcGIS\Packages. Vous pouvez changer cet emplacement dans les options de partage et de téléchargement. Les fonctionnalités du paquetage qui ne sont pas prises en charge par la version ArcGIS Pro utilisée pour exécuter le paquetage ne sont pas disponibles.

To view or edit the properties of a .dlpk package, or to add or remove files from your .dlpk package, right-click the .dlpk package in the Catalog pane and click Properties.

Properties include the following information:

• Input—The .emd file, trained model file, and any additional files that may be required to run the inferencing tools.
• Framework—The deep learning framework used to train the model.
• ModelConfiguration—The type of model training performed (object detection, pixel classification, or feature classification).
• Description—A description of the model. This is optional and editable.
• Summary—A brief summary of the model. This is optional and editable.
• Tags—Any tags used to identify the package. This is useful for .dlpk package items stored on your portal.

Any property that is edited in the Properties window is updated when you click OK. If the .dlpk package item is being accessed from your portal in the Catalog pane, the portal item is updated.

For information about how to create a .dlpk package, see Share a deep learning model package.