Skip To Content

Detect Objects Using Deep Learning

Available with Image Analyst license.

Summary

Runs a trained deep learning model on an input raster to produce a feature class containing the objects it finds. The features can be bounding boxes or polygons around the objects found, or points at the centers of the objects.

The trained deep learning model package consists of an Esri model definition (.emd) JSON file. It contains the path to the Python raster function to be called to process each raster tile, and the path to the trained binary deep learning model file created from third-party training software such as TensorFlow or CNTK.

Usage

  • You need to install the proper deep learning framework Python API (TensorFlow or CNTK) into the ArcGIS Pro Python environment; otherwise, an error will occur when you add the Esri model definition file to the tool. The appropriate framework information should be provided by the person who created the Esri model definition file.

  • For information about modifying or cloning the Python environment, see Python Package Manager.

  • This tool calls a third-party deep learning Python API (such as TensorFlow or CNTK) and uses the specified Python raster function to process each raster tile.

  • You can also write your own custom Python modules by following examples and instructions on the Esri Python raster function GitHub pageAnatomy of a Python raster function.

  • The Model Definition parameter can be an Esri model definition (.emd) JSON file or a JSON string. A JSON string is useful when this tool is used on the server, because you can paste the JSON string rather than upload the .emd file.

  • See the sample below for the .emd file.

    {
        "Framework":"CNTK",
        "ModelConfiguration":"FasterRCNN",
        "InferenceFunction":"C:\\deep-learning\\PyRfx\\ObjectDetector.py",
        "ModelFile":".\\CoconutTreeDetection.model",
        "ExtractBands" : [0,1,2],
    
        "Classes" : [
        {
            "Value" : 0,
            "Name" : "CoconutTree",
            "Color" : [0, 255, 0]
        }
        ]
    }
  • Use the Non Maximum Suppression parameter to identify and remove duplicate features from the object detection.

  • For more information about deep learning, see Deep Learning in ArcGIS Pro.

Syntax

DetectObjectsUsingDeepLearning (in_raster, out_detected_objects, in_model_definition, {arguments}, {run_nms}, {confidence_score_field}, {class_value_field}, {max_overlap_ratio})
ParameterExplanationData Type
in_raster

Input image used to detect objects.

Raster Dataset; Raster Layer; Mosaic Layer; Image Service; MapServer; Map Server Layer; Internet Tiled Layer
out_detected_objects

The output feature class that contains geometries circling the object or objects detected in the input image.

Feature Class
in_model_definition

The in_model_definition parameter can be an Esri model definition (.emd) JSON file or a JSON string. A JSON string is useful when this tool is used on the server, because you can paste the JSON string rather than upload the .emd file.

Contains the path to the deep learning binary model file, the path to the Python raster function to be used, and other parameters such as preferred tile size or padding.

File; String
arguments
[arguments,...]
(Optional)

The function arguments are defined in the Python raster function class. This is where you list additional deep learning parameters and arguments for experiments and refinement, such as a confidence threshold for adjusting the sensitivity. The names of the arguments are populated by the tool from reading the Python module.

Value Table
run_nms
(Optional)

Performs non-maximum suppression, where duplicate objects are identified and the duplicate feature with a lower confidence value is removed.

  • NO_NMSAll objects that are detected will be in the output feature class. This is the default.
  • NMSRemoves duplicate objects that are detected.
Boolean
confidence_score_field
(Optional)

The field in the feature class that contains the confidence scores as output by the object detection method.

This parameter is required when you use the NMS keyword for the run_nms parameter.

Field
class_value_field
(Optional)

The class value field in the input feature class. If not specified, the tool will use the standard class value fields Classvalue and Value. If these fields do not exist, all features will be treated as the same object class.

Field
max_overlap_ratio
(Optional)

The maximum overlap ratio for two overlapping features, which is defined as the ratio of intersection area over union area. The default is 0.

Double

Code sample

DetectObjectsUsingDeepLearning example 1 (Python window)

This example creates a feature class based on object detection.

# Import system modules
import arcpy
from arcpy.ia import *

# Check out the ArcGIS Image Analyst extension license
arcpy.CheckOutExtension("ImageAnalyst")

DetectObjectsUsingDeepLearning("c:/detectobjects/moncton_seg.tif", 
     "c:/detectobjects/moncton_seg.shp", "c:/detectobjects/moncton.emd")
DetectObjectsUsingDeepLearning example 2 (stand-alone script)

This example creates a feature class based on object detection.

# Import system modules
import arcpy
from arcpy.ia import *

"""
Usage: DetectObjectsUsingDeepLearning( in_raster, out_detected_objects, 
       in_model_definition, {arguments})
"""

# Set local variables
in_raster = "c:/classifydata/moncton_seg.tif"
out_detected_objects = "c:/detectobjects/moncton.shp"
in_model_definition = "c:/detectobjects/moncton_sig.emd"


# Check out the ArcGIS Image Analyst extension license
arcpy.CheckOutExtension("ImageAnalyst")

# Execute 
DetectObjectsUsingDeepLearning( in_raster, out_detected_objects, 
in_model_definition, "padding 0;nms_threshold 0.2;conf_threshold 0")

Licensing information

  • Basic: Requires Image Analyst
  • Standard: Requires Image Analyst
  • Advanced: Requires Image Analyst

Related topics