Evaluate Point Cloud Classification Model (3D Analyst)

Summary

Evaluates the quality of one or more point cloud classification models using a well-classified point cloud as a baseline for comparing the classification results obtained from each model.

Usage

• To obtain the best evaluation results, the reference point cloud must be well classified for the objects of interest that are being evaluated.

• All input models being evaluated must have the same classification codes. If the reference point cloud does not have class codes whose values or meaning match the classes in the models being evaluated, use the Point Cloud Class Remap parameter to designate the reference point cloud's classification with the codes in the trained models.

• The PointCNN deep learning architecture is used to create the point cloud classification model. This architecture may yield slightly different results when processing the same set of input data. You can specify the same model as an input more than once to evaluate the consistency of the classification result from the point cloud. See PointCNN: Convolution On X-Transformed Points for more information on the potential for variability in the classification results.

• The evaluation process will create several output files in the Target Folder parameter value. The name of each file will begin with the text specified in the Base Name parameter value.

  • <base name>_ModelStatistics.csv—A table listing each model's overall accuracy, precision, recall, and F1 score.
  • <base name>_ClassCodeStatistics.csv—A table summarizing the evaluation results for each class code's accuracy, precision, recall, and F1 score.
  • <base name>_ConfusionMatrices.csv—A table summarizing how each point from each class code in the reference point cloud was classified by the input models. In addition to the number of true positives and false positives, this table also identifies the classes that the false positives were occurring in.
  • <base name>_ConfusionMatrix_<model number>.png—An image showing a chart that provides an easy to read depiction of the confusion matrix for a given model.

• The Reference Surface parameter is a required input when using a classification model that was trained with reference height information. The raster provided for this parameter is used to derive the relative height attribute for each overlapping point. The attribute is calculated by taking the z-value of each point and subtracting the height obtained from the raster through bilinear interpolation. This information helps differentiate objects associated with classes that have a distinctive range of relative heights from the raster surface, which allows them to be differentiated more readily. When the model is trained with relative height information, the raster surface that is provided as input for classification should be based on the same type of points. If the raster that was used for incorporating relative height during training was based on ground classified points, provide the same type of raster for this tool. For LAS datasets representing outdoor surveys, the most typical source of relative height information will be ground classified points from the same point cloud. A raster surface can be generated from the ground classified points by filtering the LAS dataset from its layer properties and using the LAS Dataset To Raster tool. If the tool is used in Python, the Make LAS Dataset Layer can be used to filter for the desired points prior to creating the raster. A ground surface can also be generated from a point cloud scene layer using the Point Cloud To Raster tool. Raster surfaces that are not sourced from the input point cloud can also be used, but you must ensure that the z-values in the raster correspond appropriately with the z-values in the point cloud.

• When using a classification model that was trained with certain classes that were excluded from the model, use the Excluded Class Codes parameter ensure that those same points are omitted from the set of points that are evaluated by the model. This means the point cloud being classified or evaluated should have the excluded points classified prior to running this tool. The advantage of excluding classes that do not provide a useful context for inferring the classification of objects of interest is that it can improve the speed of the training and inferencing process by reducing the number of points that are evaluated. For example, building classified points are usually immaterial to training a classification model for objects such as traffic lights, power lines, or various railroad assets. Building points can also be classified using the Classify LAS Building tool. If points with class 6, which represents buildings, were excluded when training the model, you must classify building points in the point clouds that will use this model to classify other objects of interest.

Parameters

Dialog

Label	Explanation	Data Type
Input Model Definition	The point cloud classification models and batch sizes that will be used during the evaluation process.	Value Table
Reference Point Cloud	The point cloud that will be used to evaluate the classification models.	LAS Dataset Layer; File
Target Folder	The directory that will store the files which summarize the evaluation results.	Folder
Base Name	The file name prefix that will be used for each of the output files summarizing the evaluation results.	String
Processing Boundary (Optional)	The polygon feature that delineates the portions of the reference point cloud that will be used for evaluating the classification models.	Feature Layer
Point Cloud Class Remapping (Optional)	The class codes from the reference point cloud must match the class codes in the models being evaluated. When the class codes do not match, use this parameter to associate the differing class codes in the point cloud with the classes that are supported in the models being evaluated.	Value Table
Reference Surface (Optional)	The raster surface that will be used to provide relative height values for each point in the point cloud data. Points that do not overlap with the raster will be omitted from the analysis.	Raster Layer
Excluded Class Codes (Optional)	The class codes that will be excluded from processing. Any value in the range of 0 to 255 can be specified.	Long

Derived Output

Label	Explanation	Data Type
Output Confusion Matrices	The CSV format table storing the confusion matrix for each class code in each input model.	Text File
Output Model Statistics	The CSV format table that summarizes the overall statistics of the input models.	Text File
Output Class Code Statistics	The CSV format table that summarizes the statistics for each class code in each input model.	Text File

Python

arcpy.ddd.EvaluatePointCloudClassificationModel(in_trained_model, in_point_cloud, target_folder, base_name, {boundary}, {class_remap}, {reference_height}, {excluded_class_codes})

Name	Explanation	Data Type
in_trained_model [in_trained_model,...]	The point cloud classification models and batch sizes that will be used during the evaluation process.	Value Table
in_point_cloud	The point cloud that will be used to evaluate the classification models.	LAS Dataset Layer; File
target_folder	The directory that will store the files which summarize the evaluation results.	Folder
base_name	The file name prefix that will be used for each of the output files summarizing the evaluation results.	String
boundary (Optional)	The polygon feature that delineates the portions of the reference point cloud that will be used for evaluating the classification models.	Feature Layer
class_remap [class_remap,...] (Optional)	The class codes from the reference point cloud must match the class codes in the models being evaluated. When the class codes do not match, use this parameter to associate the differing class codes in the point cloud with the classes that are supported in the models being evaluated.	Value Table
reference_height (Optional)	The raster surface that will be used to provide relative height values for each point in the point cloud data. Points that do not overlap with the raster will be omitted from the analysis.	Raster Layer
excluded_class_codes [excluded_class_codes,...] (Optional)	The class codes that will be excluded from processing. Any value in the range of 0 to 255 can be specified.	Long

Derived Output

Name	Explanation	Data Type
out_confusion_matrices	The CSV format table storing the confusion matrix for each class code in each input model.	Text File
out_model_statistics	The CSV format table that summarizes the overall statistics of the input models.	Text File
out_class_code_statistics	The CSV format table that summarizes the statistics for each class code in each input model.	Text File

Code sample

EvaluatePointCloudUsingTrainedModel example (Python window)

The following sample demonstrates the use of this tool in the Python window.

import arcpy
arcpy.env.workspace = 'C:/data'
arcpy.ddd.EvaluatePointCloudUsingTrainedModel(
        ['Transmission_Power_Lines.dlpk', 'Distribution_Power_Lines.dlpk'], 
        'Classified_Power_Lines.lasd', 'D:/Evaluate_PointCNN_Models', 
        'Power_Line_Results_', 'test_boundary.shp', [[18, 14], [20, 14]])

Environments

Extent, Processor Type, GPU ID, Output Coordinate System, Geographic Transformations

Licensing information

• Basic: Requires 3D Analyst
• Standard: Requires 3D Analyst
• Advanced: Requires 3D Analyst