Review labels and training data

Available with Image Analyst license.

Available with Spatial Analyst license.

The quality of the training data is important when training a deep learning model. ArcGIS Pro includes tools that allow users to review, approve, reject, and update your labels or exported training data. These tools can perform both a manual review that is visual and an automated review based on quantitative assessments.

Label tab

Once the objects have been labeled, a manual quality assurance (QA) review can be performed to check for labeling errors. The Label tab has tools to review, approve, reject, and edit labels. Two types of users are involved in the QA process:

• Reviewers—These reviewers can only approve or reject labels.
• Editors—These editors can approve, reject, and edit labels.

The tools on the Label tab can be used by both reviewers and editors and are described in the following table:

Tool	Description
Load training samples	Open an existing training sample feature class for review.
Save	Save the edits to the current feature class.
Save As	Save the edits as a new feature class.
Approve	Approve the selected labels.
Reject	Reject the selected labels.
Delete	Delete the selected labels.
Edit	Edit the shape of the selected training sample.

The lower part of the pane displays all records in the labels feature class, shown as a table. Each label has a class name, the current status, and any issues that may have been reported as follows:

• Class Name—Lists each label by its class names. Each class name can be modified, and new class names can be added.
• Status—Shows the current status of the labels in the review process. By default, the status of all labels is set to Pending. A reviewer can change the status to Approved or Rejected.
• Issue—Lists the reason why a label was rejected during the review process. When a label is rejected, the reviewer selects the reason why the label was rejected. Choices are Wrong label and Poor shape.

Note:

When one or more rows are selected in the table, the map zooms in to the label so that it is easier to review. This is particularly helpful when multiple labels are selected for review.

Data tab

Once the objects have been labeled and the training data has been exported, a QA review can be performed. The Data tab provides tools for a quick visual review of the errors in the labels and exported image chips. The gallery displays the class name and a bounding box for each image chip. At this stage, reviewers evaluate the quality of the exported training data to ensure that it meets the project's needs. Some issues that can occur include image chips that are too big or too small, missing labels, partial labels, or a lack of overlap. When a thumbnail of an image chip is selected, it is displayed at full size in the Preview window below the gallery.

Note:

Currently the Data tab only supports training data from object detection.

The tools on the Data tab are described in the following table:

Tool	Description
Load training data	Load training data from a folder.
Class	Select the class to filter the training data.
Playback controls	The navigation and playback controls allow you to move through the exported image chips along with their corresponding labels. Only 20 image chips are displayed in the gallery at once, but the navigation tools control which image chips to view. The Play button displays all the image chips in sequence. The Pause button pauses the playback. The Speed drop-down list controls the play speed. Available speed options are 0.5x, 1x, 2x, and 3x. For manual reviews, use the First , Previous , Next , or Last buttons to navigate to a set of images in the list.
Jump to	Navigate to a specific image. Provide the image number in the text box, and click the Jump to button.

Auto Review tab

For large datasets, manual review can be time consuming and inefficient. The tools on the Auto Review tab can be used to quickly assess the quality of the labels by comparing them with small ground truth data.

Detect objects labels

The Auto Review tab automatically evaluates object detection labels using ground truth data. The tool generates a set of accuracy metrics, including Common Objects in Context (COCO), mean Average Precision (mAP), and a Precision x Recall curve, to provide an assessment of the data's quality. Using the Intersection over a Union (IoU) ratio, it identifies and counts true and false positives and negatives, outputting a detailed report with all the key metrics needed to evaluate the dataset.

The following table lists the parameters associated with the Auto Review tab for object detection:

Parameter	Description
Labels	The polygon feature class containing the objects detected from the Detect Objects Using Deep Learning tool.
Ground Truth Features	The polygon feature class containing ground truth data.
Output Accuracy Table	The output accuracy table.
Output Accuracy Report	The name of the output accuracy report. The report is a PDF document containing accuracy metrics and charts.
Detected Class Value Field	The field in the detected objects feature class that contains the class values or class names. If no field name is provided, a Classvalue or Value field will be used. If these fields do not exist, all records will be identified as belonging to one class. The class values or class names must match those in the ground reference feature class exactly.
Ground Truth Class Value Field	The field in the ground truth feature class that contains the class values. If no field name is provided, a Classvalue or Value field will be used. If these fields do not exist, all records will be identified as belonging to one class. The class values or class names must match those in the detected objects feature class exactly.
Minimum intersection over Union (IoU)	The IoU ratio that will be used as a threshold to evaluate the accuracy of the object detection model. The numerator is the area of overlap between the predicted bounding box and the ground reference bounding box. The denominator is the area of union or the area encompassed by both bounding boxes. The IoU ranges from 0 to 1.
Mask Features	A polygon feature class that delineates the area or areas where accuracy will be computed. Only the features that intersect the mask will be assessed for accuracy.

Pixel classification labels

The Auto Review tab automatically evaluates pixel classification labels using ground truth data. The tool computes a confusion matrix based on errors of omission and commission and the IoU score. The accuracy is computed between the labels and the ground truth data; the labels can be either a feature class or a raster layer.

The following table lists the parameters associated with the Auto Review tab for pixel classification:

Parameter	Description
Labels	The input classified raster.
Ground Truth Data	The input classification image or other thematic GIS reference data. The input can be a raster or feature class. Typical data is a classification image of a single band of integer data type. If using polygons as input, only use those that are not used as training samples. They can also be GIS land-cover data in shapefile or feature class format.
Output Confusion Matrix	The output file name of the confusion matrix in table format. The format of the table is determined by the output location and path. By default, the output will be a geodatabase table. If the path is not in a geodatabase, specify the .dbf extension to save it in dBASE format.
Number of Random Points	The total number of random points that will be generated. The actual number may exceed but never fall below this number, depending on sampling strategy and number of classes. The default number of randomly generated points is 500.
Sampling Strategy	Specifies the sampling scheme that will be used. Stratified random—Randomly distributed points will be created in each class in which each class has a number of points proportional to its relative area. Equalized stratified random—Randomly distributed points will be created in each class in which each class has the same number of points. Random—Randomly distributed points will be created throughout the image.
Minimum Point Distance	The minimum distance between the reference points.