Deep learning in ArcGIS Pro

Disponible con licencia de Image Analyst.

ArcGIS Pro allows you to use statistical or machine learning classification methods to classify remote-sensing imagery. Deep learning is a type of machine learning that relies on multiple layers of nonlinear processing for feature identification and pattern recognition described in a model. Deep learning models can be integrated with ArcGIS Pro for object detection, object classification, and image classification.

Training samples of features or objects of interest are generated in ArcGIS Pro with classification and deep learning tools. These training samples are used to train a deep learning model using a geoprocessing tool, and the resulting model definition file or deep learning model package (.dlpk) is used to run the inference geoprocessing tools to extract specific object locations, classify or label the objects, or classify the pixels in the imagery. You can also train the model outside of ArcGIS Pro using an external third-party framework and use a model definition file to run the inference geoprocessing tools. Model definition files and .dlpk packages can be used multiple times as inputs to the geoprocessing tools, allowing you to assess multiple images over different locations and time periods, after the model has been trained.

The workflow is represented in the diagram below.

Step	Description
	Create training samples in the Label Objects for Deep Learning pane, and use the Export Training Data For Deep Learning tool to convert the samples into deep learning training data.
	Use the Train Deep Learning Model tool to train a model using PyTorch, or train the model outside of ArcGIS Pro using a supported third-party deep learning framework.
	Use the trained model to run the Detect Objects Using Deep Learning tool, the Classify Pixels Using Deep Learning tool, or the Classify Objects Using Deep Learning tool to generate an output.

Features and capabilities

Deep learning tools in ArcGIS Pro allow you to use more than the standard machine learning classification techniques.

• Use convolutional neural networks or deep learning models to detect objects, classify objects, or classify image pixels.
• Integrate external deep learning model frameworks, such as TensorFlow, PyTorch, and Keras.
• Use a model definition file multiple times to detect change over time or detect objects in different areas of interest.
• Generate a polygon feature class showing the location of detected objects to be used for additional analysis or workflows.
• Take advantage of GPU processing as well as using CPU for distributed processing.

Get started with deep learning

The creation and export of training samples are done in ArcGIS Pro using the standard training sample generation tools. The deep learning model can be trained with the PyTorch framework using the Train Deep Learning Model tool, or it can be trained outside of ArcGIS Pro using another deep learning framework. Once the model is trained, use an Esri model definition file (.emd) to run geoprocessing tools to detect or classify features in your imagery.

You must install the deep learning framework Python packages; otherwise, an error occurs when you add the Esri model definition file to the deep learning geoprocessing tools. For information about how to install these packages, see Install deep learning frameworks for ArcGIS.

1. Create and export training samples.
   1. Use the Label Objects for Deep Learning pane or the Training Samples Manager to select or create a classification schema.
   2. Create training site samples for the class categories or features of interest. Save the training sample file.
   3. Run the Export Training Data for Deep Learning geoprocessing tool to convert the source imagery and training samples to deep learning training data. The source imagery can be an image service, a mosaic dataset, a raster dataset, or a folder of rasters. The output of the tool is image chips or samples containing training sites to be used to train the deep learning model. An additional output of the tool is a template .emd file that must be populated.
2. Train the deep learning model.
   1. Use the Train Deep Learning Model tool to train a deep learning model using the image chips generated in the previous step.
3. Run the inference geoprocessing tools in ArcGIS Pro.
   1. Use the Detect Objects Using Deep Learning, Classify Pixels Using Deep Learning, or Classify Objects Using Deep Learning geoprocessing tool to process your imagery. If the trained model incorporated custom Python raster functions with additional variables, such as padding or a confidence threshold for fine-tuning the sensitivity, these variables appear on the geoprocessing tool dialog box for user input. The data type, such as string or numeric, is specified in the Python raster function. Ideally, additional inference parameters should be limited to two.

      El valor del parámetro definición de modelo de Esri puede ser un archivo JSON de definición de modelo de Esri (.emd) o una cadena de caracteres JSON. Una cadena de caracteres JSON es útil cuando esta herramienta se utiliza en el servidor para pegar la cadena de caracteres JSON, en lugar de cargar el archivo .emd.

      The output of the Detect Objects Using Deep Learning tool is a feature class showing the objects detected by the model. The Classify Pixels Using Deep Learning tool outputs a classified raster. The Classify Objects Using Deep Learning tool requires a feature class and imagery as the input datasets, and the result is a feature class in which each object within each feature is labelled with a class name.

For information about requirements for running this tool and issues you may encounter, see Deep Learning in ArcGIS Pro Frequently Asked Questions.

Esri model definition file

The .emd file is a JSON file that describes the trained deep learning model. It contains model definition parameters that are required to run the inference tools, and it should be modified by the data scientist who trained the model. There are required and optional parameters in the file as described in the table below.

Once the .emd file is completed and verified, it can be used in inferencing multiple times, as long as the input imagery is from the same sensor as the original model input, and the classes or objects being detected are the same. For example, an .emd file that was defined with a model to detect oil well pads using Sentinel-2 satellite imagery can be used to detect oil well pads across multiple areas of interest and multiple dates using Sentinel-2 imagery.

Some parameters are used by all the inference tools; these are listed in the table below. Some parameters are only used with specific tools, such as the CropSizeFixed and the BlackenAroundFeature parameters, which are only used by the Classify Objects Using Deep Learning tool.

Model definition file parameter	Explanation
Framework	The name of a deep learning framework used to train your model. The following deep learning frameworks are supported: TensorFlow Keras PyTorch If your model is trained using a deep learning framework that is not listed, a custom inference function (a Python module) is required with the trained model, and you must set InferenceFunction to the Python module path.
ModelConfiguration	The name of the model configuration. The model configuration defines the model inputs and outputs, the inferencing logic, and the assumptions made about the model inputs and outputs. Existing open source deep learning workflows define standard input and output configuration and inferencing logic. ArcGIS supports the following set of predefined configurations: TensorFlow ObjectDetectionAPI DeepLab Keras MaskRCNN If you used one of the predefined configurations, type the name of the configuration in the .emd file. If you trained your deep learning model using a custom configuration, you must describe the inputs and outputs in full in the .emd file or in the custom Python file.
ModelType	The type of model. ImageClassification—For classifying pixels ObjectDetection—For detecting objects or features Object Classification—For classifying objects and features
ModelFile	The path to a trained deep learning model file. The file format depends on the model framework. For example, in TensorFlow, the model file is a .pb file.
Description	Provide information about the model. Model information can include anything to describe the model you have trained. Examples include the model number and name, time of model creation, performance accuracy, and more.
InferenceFunction (Optional)	The path of the inference function. An inference function understands the trained model data file and provides the inferencing logic. There are six inference functions that are supported in the ArcGIS Pro deep learning geoprocessing tools: Detect Objects for TensorFlow Classify Pixels for Tensor Flow Detect Objects for Keras Detect Objects for PyTorch Classify Objects for Pytorch If you used one of the inference functions above, there is no need to specify it in the .emd file. If your model is trained using a deep learning model configuration that is not yet supported, or it requires special inferencing logic, a custom inference function (a Python module) is required with the trained model. In this case, set InferenceFunction to the Python module path. An inference Python module file can be located anywhere ArcGIS Pro can access.
SensorName (Optional)	The name of the sensor used to collect the imagery from which training samples were generated.
RasterCount (Optional)	The number of rasters used to generate the training samples.
BandList (Optional)	The list of bands used in the source imagery
ImageHeight (Optional)	The number of rows in the image being classified or processed.
ImageWidth (Optional)	The number of columns in the image being classified or processed.
ExtractBands (Optional)	The band indexes or band names to extract from the input imagery.
Classes (Optional)	Information about the output class categories or objects.
DataRange (Optional)	The range of data values if scaling or normalization was done in preprocessing.
ModelPadding (Optional)	The amount of padding to add to the input imagery for inferencing.
BatchSize (Optional)	The number of training samples to be used in each iteration of the model.
PerProcessGPUMemoryFraction (Optional)	The fraction of GPU memory to allocate for each iteration in the model. The default is 0.95, or 95 percent.
MetaDataMode (Optional)	The format of the metadata labels used for the image chips.
ImageSpaceUsed (Optional)	The type of reference system used to train the model. MAP_SPACE PIXEL_SPACE
WellKnownBandNames (Optional)	The names given to each input band, in order of band index. Bands can then be referenced by these names in other tools.
AllTileStats	The statistics of each band in the training data.

The following is an example of a model definition file (.emd) that uses a standard model configuration:

{
    "Framework": "TensorFlow",
    "ModelConfiguration": "ObjectDetectionAPI",
    "ModelFile":"C:\\ModelFolder\\ObjectDetection\\tree_detection.pb",
    "ModelType":"ObjectionDetection",
    "ImageHeight":850,
    "ImageWidth":850,
    "ExtractBands":[0,1,2],
    "Classes" : [
      {
        "Value": 0,
        "Name": "Tree",
        "Color": [0, 255, 0]
      }
    ]
}

The following is an example of a model definition file (.emd) with more optional parameters in the configuration:

{
    "Framework": "PyTorch",
    "ModelConfiguration": "FasterRCNN",
    "ModelFile":"C:\\ModelFolder\\ObjectDetection\\river_detection.pb",
    "ModelType":"ObjectionDetection",
				"Description":"This is a river detection model for  imagery",
    "ImageHeight":448,
    "ImageWidth":448,
    "ExtractBands":[0,1,2,3],
				"DataRange":[0.1, 1.0],
				"ModelPadding":64,
				"BatchSize":8,
				"PerProcessGPUMemoryFraction":0.8,
				"MetaDataMode" : "PASCAL_VOC_rectangles",
				"ImageSpaceUsed" : "MAP_SPACE",
    "Classes" : [
      {
        "Value": 1,
        "Name": "River",
        "Color": [0, 255, 0]
      }
    ],
				"InputRastersProps" : {
						"RasterCount" : 1,
						"SensorName" : "Landsat 8",
						"BandNames" : [
								"Red",
								"Green",
								"Blue",
								"NearInfrared"
						]
				},
				"AllTilesStats" : [
      {
      		"BandName" : "Red",
      		"Min" : 1,
      		"Max" : 60419,
      		"Mean" : 7669.720049855654,
      		"StdDev" : 1512.7546387966217
      },
      {
      		"BandName" : "Green",
      		"Min" : 1,
      		"Max" : 50452,
      		"Mean" : 8771.2498195125681,
      		"StdDev" : 1429.1063589515179
      },
      {
      		"BandName" : "Blue",
      		"Min" : 1,
      		"Max" : 47305,
      		"Mean" : 9306.0475897744163,
      		"StdDev" : 1429.380049936676
      },
      {
      		"BandName" : "NearInfrared",
      		"Min" : 1,
      		"Max" : 60185,
      		"Mean" : 17881.499184561973,
      		"StdDev" : 5550.4055277121679
      }
    ],
}

Deep learning model package

Un paquete de modelo de aprendizaje profundo (.dlpk) contiene los archivos y datos necesarios para ejecutar herramientas de inferencia de aprendizaje profundo para la detección de objetos o la clasificación de imágenes. El paquete se puede cargar en su portal como un elemento DLPK y utilizarse como entrada para las herramientas de análisis de ráster de aprendizaje profundo.

Los paquetes de modelos de aprendizaje profundo deben contener un archivo de definición de modelo de Esri (.emd) y un archivo de modelo entrenado. La extensión de archivo del modelo entrenado depende del marco que se usó para entrenar al modelo. Por ejemplo, si entrenó su modelo mediante TensorFlow, el archivo del modelo es un archivo .pb, mientras que un modelo entrenado con Keras genera un archivo .h5. En función del marco del modelo y de las opciones que haya usado para entrenar el modelo, es posible que necesite incluir una función ráster de Python (.py) o archivos adicionales. Puede incluir varios archivos de modelo entrenados en un mismo paquete de modelo de aprendizaje profundo.

La mayoría de los paquetes se pueden abrir con cualquier versión de ArcGIS Pro. De forma predeterminada, el contenido de un paquete se almacena en la carpeta <User Documents>\ArcGIS\Packages. Puede cambiar esta ubicación en las Opciones de descarga y uso compartido. La funcionalidad del paquete que no sea compatible con la versión de ArcGIS Pro que se utiliza para consumir el paquete no estará disponible.

To view or edit the properties of a .dlpk package, or to add or remove files from your .dlpk package, right-click the .dlpk package in the Catalog pane and click Properties.

Properties include the following information:

• Input—The .emd file, trained model file, and any additional files that may be required to run the inferencing tools.
• Framework—The deep learning framework used to train the model.
• ModelConfiguration—The type of model training performed (object detection, pixel classification, or feature classification).
• Description—A description of the model. This is optional and editable.
• Summary—A brief summary of the model. This is optional and editable.
• Tags—Any tags used to identify the package. This is useful for .dlpk package items stored on your portal.

Any property that is edited in the Properties window is updated when you click OK. If the .dlpk package item is being accessed from your portal in the Catalog pane, the portal item is updated.

For information about how to create a .dlpk package, see Share a deep learning model package.