The Hot Spot Analysis Comparison tool compares two hot spot analysis result layers and calculates their similarity and association. The similarity and association between the hot spot result layers is determined by comparing the significance level categories (99% hot, 95% hot, 90% hot, not significant, 90% cold, 95% cold, and 99% cold) between corresponding features (and their neighbors) in both input layers.
The tool calculates a global similarity and global kappa value to measure the overall similarity and association between the hot spot results. Local versions of the similarity and kappa values are also calculated for each pair of corresponding features. This allows you to map the comparisons to explore areas that have higher or lower similarity or association than the global values. The output features also include charts and custom symbology that highlight areas where the hot spot results are most dissimilar and summarize the significance level pairs of all corresponding features.
The input hot spot result layers must be the output features of the Hot Spot Analysis (Getis-Ord Gi*) or Optimized Hot Spot Analysis tools. Every feature in each result must be paired with a single corresponding feature of the other result so that their significance level categories can be compared. If the features of the two input hot spot results do not spatially align (such as polygons that do not have the same borders), the two feature layers will be intersected before the analysis, and the comparisons will be made on the feature intersections.
Similarity and association
The similarity of the hot spot results is the degree to which the hot spots, cold spots, and nonsignificant areas of both hot spot results spatially align, and the association (or dependence) between the results is the degree of statistical dependence between the underlying hot spot analysis variables. The distinction is subtle, but it is important because it is common for two hot spot results to be highly similar (many corresponding features have the same significance level) but still have little association or dependence. This is shown in the two hot spot result layers in the image below.
Because 23 of the 25 polygons in each result match categories, the two results are highly similar. However, because 24 of the 25 polygons in each result are cold spots, at least 23 matching polygons would be expected even if the underlying hot spot results are independent and unrelated. This means that despite a matching category for almost every polygon, there is no evidence that the results are statistically associated. It can be concluded that both variables are almost entirely cold spots with a single isolated hot spot, but there is no evidence of a relationship or association between them.
In the two hot spot results in the image below, 23 of the 25 polygons also match, so their similarity is the same as the first set of results. However, 22 of the matches are for cold spots, and one match is for a hot spot. With only two hot spots in each result, it is unlikely that the hot spots would align so closely by chance. This is evidence of an underlying relationship and association between the results. While the relationship is not necessarily causal, you may be able to influence the values of one result by changing the values of the other. For example, if one hot spot result represents hot and cold spots of infant mortality and the other represents hot and cold spots of pollution, a strong association between the variables suggests that infant mortality may decrease by decreasing pollution levels. For another example, if the two hot spot results represent pollution levels in successive years, the association can be interpreted as movement of pollution levels to the north and east.
The similarity between the hot spot results is measured by a similarity value between 0 and 1. If many corresponding features in both results have the same significance level, the value will be close to 1, and if many corresponding features do not have matching significance levels, the value will be close to 0. The association is measured by a kappa value: strongly associated results will have kappa values close to 1, and unassociated (independent) results will have kappa values close to 0 (or slightly negative). The kappa value is a rescaled version of the similarity value that accounts for spatial clustering and category frequencies in order to isolate the statistical association between the hot spot results. For reference, the first set of hot spot results above has a kappa value approximately equal to 0, and the second set of results has a kappa value approximately equal to 0.6.
Exclude nonsignificant features
When the hot spot results are dominated by a single category such as in the examples above, it is most common for it to be the nonsignificant category. However, if the nonsignificant features are not of research interest, you may not want the similarity and kappa values to simply reflect the abundance of nonsignificant areas in both results. To prevent this, you can use the Exclude Nonsignificant Features parameter to exclude any pair of corresponding features from the comparisons if both hot spot results are not statistically significant. If excluded, the tool calculates conditional similarity and kappa values that compare only the statistically significant hot and cold spots to accurately reflect their similarity and association. The counts and overall proportions of the significance level categories impact the similarity and kappa values, so consider the outcome before excluding large numbers of nonsignificant features.
Fuzzy similarity
When comparing two corresponding features, the result can be more than simple binary (yes or no) of whether the features have the same significance level category. The calculations of similarity and association use fuzzy set membership to allow partial matches between corresponding features based on significance level similarity and spatial neighborhoods. For example, 99% hot spots can be considered perfect matches to other 99% hot spots, partial matches to 95% hot spots, and complete mismatches to 99% cold spots. Two corresponding features can also be considered partial matches based on distance similarity if the features do not have the same significance level, but their neighboring features do. The overall similarity between any two corresponding features is their categorical similarity multiplied by the distance similarity. See the Spatial fuzzy kappa section below for details about the calculations.
Category similarity
There are seven possible significance level categories in each hot spot result. The categories have a natural ordering from 99% hot to 99% cold, and some categories are more similar to each other than others. Category similarity weights allow you to define how similar you consider the different significance level categories to be. Each combination of results (for example, 90% cold versus 95% hot) must have a category weight between 0 and 1 indicating their similarity. Combinations with weights equal to 1 are considered exact matches, and combinations with weights equal to 0 are considered completely dissimilar. Values between 0 and 1 indicate degrees of partial similarity between the categories. The weights must be symmetric; for example, the weight between 99% hot and 95% hot must be equal to the weight between 95% hot and 99% hot.
If two categories have a similarity weight equal to 1, the calculations of similarity and association will treat them as if they are the same category, so you can use the weights to combine different categories. For example, to perform the two hot spot analyses at a 95% confidence level, you can combine the 90% cold, not significant, and 90% hot categories using a weight equal to 1 for all combinations of the categories. The similarity values and kappa values will treat 90% cold and 90% hot categories as if they are not significant. Additionally, if you exclude nonsignificant features, any categories combined with the not significant category will also be excluded.
You can also reverse hot and cold relationships by giving large weights between hot and cold spots. Reversing relationships is recommended when the hot spot results have a negative relationship, such as cold spots of median income aligning with hot spots of diabetes.
Caution:
Category similarity weights only affect the calculation of the similarity and kappa values. Even if significance level categories are combined using similarity weights, the message tables, output layer symbology, and charts will treat them as separate categories. See the Tool outputs section below for more information.
The category similarity weights are specified using the Similarity Weighting Method parameter. The following options are available:
- Fuzzy weights—Similarity weights are fuzzy (nonbinary) and determined by the closeness of significance levels. All hot spots are completely dissimilar to all cold spots and nonsignificant features (and vice versa). The weights between 90%, 95%, and 99% hot and cold spots are determined by ratios of critical values of upper one-sided rejection regions of the normal distribution; for example, the weight between 95% hot and 99% hot is 1.645/2.33 = 0.71. See the first image in the Weight matrix pop-out section below for all other weights between the categories. This is the default.
- Exact significance level matching—Features must have the same significance level to be considered similar. For example, 99% hot spots will be considered completely dissimilar to 95% and 90% hot spots.
- Combine 90%, 95%, and 99% significant—Features that are 90%, 95%, and 99% hot spots will be considered perfectly similar to each other, and all features that are 90%, 95%, and 99% cold spots will be considered perfectly similar to each other. This option treats all features at or above 90% significance as being the same (statistically significant) and all features below 90% confidence as being the same (nonsignificant).
- Combine 95% and 99% significant—Features that are 95% and 99% hot spots will be considered perfectly similar, and features that are 95% and 99% cold spots will be considered perfectly similar. For example, 90% hot and 90% cold spots will be considered completely dissimilar to higher significance levels. This option treats all features at or above 95% significance as being the same (statistically significant) and all features below 95% significance as being the same (nonsignificant).
- Use only 99% significant—Only features that are 99% hot (or cold) spots will be considered perfectly similar to each other. This option treats all features below 99% significance as being nonsignificant.
- Reverse hot and cold relationships—The default fuzzy weights will be used, but hot spots of the first hot spot result will be considered similar to the cold spots of the second hot spot result. For example, 99% hot spots in one result will be considered perfectly similar to 99% cold spots in the other result and partially similar to 95% and 90% cold spots in the other result.
- Get weights from table—Weights defined by fields of a table layer will be used. The table is provided in the Input Weights Table parameter and must contain CATEGORY1, CATEGORY2, and WEIGHT fields. Provide the significance level categories of the pair (the Gi_Bin field values of the input layers) in the category fields and the similarity weight between them in the weight field. For example, the [-3, -2, 0.6] row assigns the similarity weight value 0.6 to the 99% cold versus 95% cold combination. If a combination is not provided in the table, the weight is assumed to be 0. The table can be exported from the weight matrix pop-out.
- Custom weights—Custom similarity weight values provided in the Category Similarity Weights parameter will be used.
Weight matrix pop-out
The Category Similarity Weights parameter allows you to interactively view and edit the weights using a weight matrix pop-out. The displayed weights update as you choose different options of the Similarity Weighting Method parameter, so you can see the weights associated with each option and make any edits. To open the pop-out, click the Custom button next to the parameter.
To assign a custom weight between a significance level combination, click the associated cell, type the weight value between 0 and 1, and press Enter. To keep the weights symmetric, you can only edit cells in the lower left half of the matrix, and the weight will be mirrored to the equivalent cell on the upper right. The following image shows an example of custom weights that use exact significance level matching with reversed hot and cold relationships (for example, 95% hot is perfectly similar to 95% cold and completely dissimilar to all other significance levels):
After providing the weights, click OK or click outside the pop-out to apply the weights. If any weights were altered, the Similarity Weighting Method parameter value will change to Custom weights. You can also click Cancel or the Close button to close the pop-out and not apply the changes.
The Export button opens a browse dialog box that allows you to save the weight values as a table so that they can be reused later with the Get weights from table option. To reuse custom weights in the future, it is recommended that you create the weight file using the weights matrix pop-out; then use the weights table for future comparisons.
Distance similarity
In addition to categorical similarity, distance similarity allows partial matches when the features do not have the same significance level but other features in their neighborhood do have matching significance levels. Because hot spot analysis is a spatial method that uses local neighborhoods, the significance level of each feature is a characterization of the values of the feature and its closest neighbors, not just of the feature. In this sense, if any neighboring features are similar, they should contribute somewhat to the similarity of their neighbors.
The Number of Neighbors parameter specifies the number of additional neighboring features that will be used in the comparisons, and partial similarity is incorporated using a distance weight based on the ordering of the neighbors. The feature receives a distance weight equal to 1, and the weights decrease consistently for each additional neighbor using the following formula:
The rank in the formula is the order of the neighbors and ranges from 0 (for the feature being compared) up to the number of neighbors (for the farthest neighbor). For example, with four neighbors (five, including the feature being compared), the following five distance weights are used: 5/5 (1), 4/5 (0.8), 3/5 (0.6), 2/5 (0.4), and 1/5 (0.2).
Note:
For polygons and lines, Euclidean distances between centroids are used to determine the order of closest neighbors. If the output spatial reference is a geographic coordinate system, chordal distances between centroids are used. The ordering of the neighbors (rather than raw distances) is used for distance weights to maintain the same expected similarity value for all features, even if their neighbors have different distances from the feature being compared.
Tool outputs
The results of the comparisons are returned through geoprocessing messages, a group layer of the output features, and charts.
Geoprocessing messages
The messages display information about global comparisons between the hot spot results. The messages display the following information:
- Similarity Value—A value between 0 and 1 measuring the overall similarity between the hot spot result layers. The value can be interpreted as a fuzzy probability that any pair of corresponding features have the same significance level category. The value is equal to the average of all local similarity values.
- Expected Similarity Value—The expected value of the similarity under the assumption that the two hot spot result layers are unassociated (independent). If the similarity value is larger than its expected value, this suggests an underlying dependence between the two maps. The value is mostly informational and is used to scale the similarity value when calculating the kappa value. The value is equal to the average of the local expected similarity values.
- Spatial Fuzzy Kappa—A measure of the association between the hot spot analysis variables that is calculated by scaling the similarity value by its expected value. Hot spot results that are perfectly associated will have the value 1, and unassociated (independent) results will have a value close to 0. Negative values indicate a negative relationship between the hot spot analysis variables. While the value has no lower bound, the values are rarely less than -3 in practice. There are no strict rules for interpreting kappa values, but common recommendations are to interpret values above 0.8 as almost perfect association, values between 0.6 and 0.8 as strong association, values between 0.4 and 0.6 as moderate association, values between 0.2 and 0.4 as fair association, values between 0 and 0.2 as slight association, and negative values as no association (or negative association for large negative values).
- Number of Nonsignificant Features—The number of hot spot significance level pairs in which both features are not statistically significant.
Note:
If nonsignificant features are excluded, they will not be included in the similarity, expected similarity, or spatial fuzzy kappa calculations. The labels will change to Conditional Similarity Value, Conditional Expected Similarity Value, Conditional Spatial Fuzzy Kappa, and Number of Excluded Nonsignificant Features to indicate that the values are conditioned on statistically significant features.
- The Categorical Weights Table message table displays the category weights between each hot spot significance level pair. For example, the image below displays the categorical weight table for the default categorical similarity weighting method:
- The Hot Spot Significance Level Pair (Counts) message table display counts of each hot spot significance level pair. For example, in the image below, the value 440 in the first row and second column means that 440 feature pairs were 99% cold in the first hot spot result and 95% cold in the second result. The row and column totals in the margins indicate the total counts of each significance level among each hot spot result.
- The Hot Spot Significance Level Pair (Percentages) message table displays the same information as the counts table, but the counts are converted to percentages of the row total. For example, in the image below, the cell that displayed 440 in the image above now displays 5.57 (440/7904 = 0.0557). This table is especially useful when the two hot spot results represent the same variable measured at different times. In this case, the tables allow you to see how the categories transitioned in the time between the measurements. For example, the image below shows that among the features that were 99% cold spots in the first result, 89.26 percent stayed as 99% cold spots, 5.57 percent changed to 95% cold spots, and so on.
Output features and group layer
The output features will be the intersections of the input hot spot result layers and will contain fields summarizing the local similarity and association for each pair of corresponding features. The feature class will have the following fields:
- Hot Spot 1 Input Value (GI_BIN_1)—An integer representing the significance level category of the feature from the first hot spot result. The values range from -3 (99% cold) to 3 (99% hot). The field will be of type long.
- Hot Spot 2 Input Value (GI_BIN_2)—An integer representing the significance level category of the feature from the second hot spot result. The field will be of type long.
- Hot Spot 1 Significance Level (GI_SIG_1)—The significance level category of the feature from the first hot spot result. The possible values are: Cold 99%, Cold 95%, Cold 90%, Not Significant, Hot 90%, Hot 95%, and Hot 99%. The field will be of type text.
- Hot Spot 2 Significance Level (GI_SIG_2)—The significance level category of the feature from the second hot spot result. The field will be of type text.
- Similarity Value (SIM_VALUE)—The local similarity value of the feature pair. The value will be between 0 and 1. The field will be of type double.
- Expected Similarity Value (EXP_SIM)—The expected value of the similarity of the feature pair. The value will be between 0 and 1. The field will be of type double.
- Spatial Fuzzy Kappa (KAPPA)—The spatial fuzzy kappa value of the feature pair. The field will be of type double.
- Significance Level Combinations (CAT_PAIR)—The combination of significance level categories of the hot spot results. This field is used as the basis for the two charts below. The field will be of type text.
When the tool is run in a map, three layers will be added to a group layer that allow you to explore the similarity, association, and significance level pairs spatially. The first layer displays the similarity values classified into five equal intervals between 0 and 1, and lower similarity values are in darker colors to emphasize the areas that are most dissimilar. The second layer displays the spatial fuzzy kappa values symbolized with equal intervals and six classes. The third layer displays each significance level combination with custom symbology to identify features where one input hot spot result was a statistically significant hot spot and the other was a statistically significant cold spot (in the custom symbology, 90%, 95%, and 99% significance is not distinguished in order to reduce the number of combinations). By default, the first layer is enabled and the last two are disabled.
Charts
The final layer comes with two charts to further investigate the significance level combinations between the results. These charts display the same information as the tables in the messages, but the charts are colored by the counts and percentages for ease of interpretation. You can also use selections between the charts and map to, for example, select all features that were 99% hot spots in one result and 99% cold spots in the other result, indicating the largest possible differences.
The Hot Spot Significance Level Pair Counts heat chart displays counts of each significance level combination with deeper shades of blue for higher counts. For example, in the image below, the pairs with the largest counts were 99% cold to 99% cold (top-left), nonsignificant to nonsignificant (middle), and 99% hot to 99% hot (bottom-right).
The Hot Spot 2 Level Counts within Hot Spot 1 Level Categories bar chart displays stacked horizontal bars to visualize the counts of each significance level category of the second hot spot result within the categories of the first result. For example, in the image below, the vast majority of 99% hot and cold spots were also significant hot and cold spots (the top and bottom bars are mostly blue and red, respectively). However, among nonsignificant features in the first result, there were more matching hot spots than cold spots in the second result (the middle bar has more red than blue). If the two hot spot result layers represent temperatures measured at different times, this may indicate a general warming of the study area between the measurement times.
Spatial fuzzy kappa
The association between the hot spot result layers is measured by a kappa value that quantifies the similarity of the results compared to how similar you would expect them to be if the two results were independent. The similarity value can be high due to large numbers of particular categories and spatial clustering of the categories. The kappa value corrects for category frequencies and spatial clustering to more accurately measure the underlying association between the hot spot result layers.
The kappa value is calculated by rescaling the similarity value by its expected value according to the following formula:
If the hot spot result layers are perfectly similar (similarity value equal to 1), the kappa value will also be equal to 1, indicating perfect association. If the similarity value is equal to its expected value, the kappa value will be 0, indicating that the results are unassociated and independent. If the similarity value is less than the expected value, the kappa value will be negative, indicating that there is a negative association between the hot spot results.
Kappa statistics were originally developed to test the consistency and reliability of raters using a Likert scale (Cohen 1960). The first version of the kappa statistic corrected for category frequencies (some Likert ratings are more common than others) but assumed that each rating was independent. Enhancements were made in the early 2000s to incorporate categorical and distance similarity for comparing categorical rasters (Hagen 2003, 235-249) (Hagen-Zanker, Straatman, and Uljee 2005, 769-785) (Hagen-Zanker 2009, 61-73) (Dou et. al. 2007, 726-734). However, these enhancements still assumed that the categories do not spatially cluster, which is not true for hot spot analysis results and most other spatial categorical variables. The Hot Spot Analysis Comparison tool enhances the kappa statistic to be a spatial fuzzy kappa statistic that accounts for categorical clustering (autocorrelation) of the significance level categories within each hot spot result.
Similarity value calculation
Local similarity values are calculated for each pair of corresponding features in the hot spot analysis results. The global similarity value is the average of all local similarity values.
The similarity value for the feature pair will be equal to 1 when the corresponding features each have the same significance level category (or have categories that have been combined by similarity weights). The similarity value will be equal to 0 when all neighbors of the first result have completely dissimilar significance level categories as all neighbors of the second result (for example, all hot spots in the first result and all cold spots in the second result). All other situations will result in similarity values between 0 and 1.
For each pair of features, the similarity value involves calculating two directional similarity values and taking the smaller of the two. The first directional similarity is the similarity from the first result to the second, and the second directional similarity is from the second result to the first. The calculation for each entails comparing the category of the feature of one result to the corresponding feature of the other result and each of its neighbors. For the corresponding feature and each neighbor, the category weight is multiplied by the distance weight, and the largest result is the directional similarity value.
For example, the image below shows two hot spot results, A and B. A and B each have three features: one hot spot (red), cold spot (blue), and nonsignificant feature (light gray). The largest polygons are the first feature pair, the smallest polygons are the second feature pair, and the medium polygons are the third feature pair. The centroids of the polygons are shown to help determine which polygons are closer than others; the first polygon is slightly closer to the second polygon than it is to the third polygon.
For this example, assume that the category weight between matching categories (hot to hot, cold to cold, and nonsignificant to nonsignificant) is 1, the weight between hot and cold spots is 0, and nonsignificant features have weight 1/2 with both hot and cold spots.
The following table shows the category weights, distance weights, and similarity of the directional similarity from the result A to result B. The similarity value in the last column is calculated by multiplying the distance weight and category weight:
| Combination | Distance weight | Category weight | Similarity |
|---|---|---|---|
A1 to B1 | 1 (corresponding feature) | 0 (cold to hot) | 0 |
A1 to B2 | 2/3 (first neighbor) | 1/2 (cold to nonsignificant) | 1/3 = 0.33 |
A1 to B3 | 1/3 (second neighbor) | 1 (cold to cold) | 1/3 = 0.33 |
The largest similarity from result A to result B is 0.33, which occurs for two neighbor combinations. The following table shows the directional similarity from result B to result A.
| Combination | Distance weight | Category weight | Similarity |
|---|---|---|---|
B1 to A1 | 1 (corresponding feature) | 0 (hot to cold) | 0 |
B1 to A2 | 2/3 (first neighbor) | 1 (hot to hot) | 2/3 = 0.67 |
B1to A3 | 1/3 (second neighbor) | 1/2 (hot to nonsignificant) | 1/6 = 0.17 |
The largest similarity from result B to result A is 0.67.
The local similarity value for the feature pair is the smaller of the two directional similarities (A to B and B to A), so the similarity value of the first feature pair is 0.33. The same procedure is also used to calculate the similarity value for the second and third feature pairs, and for this example, both have similarity values equal to 0.5. The global similarity value is the average of the similarity values of all feature pairs, and for this example, the global similarity value is 4/9 = 0.44.
If nonsignificant features are excluded, their similarity values will not calculated, and they will not be included in the average for the global similarity value; however, they will still be used as neighbors when calculating the similarity value of features that are not excluded.
Note:
This example used hot spot results with only three feature pairs and three significance level categories to reduce the number of combinations. However, at least 20 feature pairs are required to use the tool, and category weights must be provided between all seven significance level categories.
Expected similarity value calculation
For each feature, the calculation of the expected similarity value uses the same procedure as the similarity value; however, the feature of the first result is paired with random features in the second result rather than its corresponding feature.
By comparing random neighborhoods, the expected value accounts for category frequency (more common categories are more likely to be randomly chosen) and category clustering within the neighborhoods (the random neighborhoods are likely to contain clusters of features with similar significance level categories). The similarity value from each random pairing is a single estimate of the similarity value under the assumption that the two hot spot results are independent. To calculate the expected similarity value of a feature, each feature of the first result is paired with many random neighbors, and the random similarity values are averaged. The Number of Permutations parameter specifies the number of random pairings for each feature. Larger numbers of permutations will increase the run time of the tool and increase the precision of the expected similarity and kappa values.
The global expected similarity value is the average of the expected similarity values of all feature pairs. If nonsignificant features are excluded, the excluded features will never be chosen as random neighbors, and their expected similarity value is not calculated; however they can still be included as neighbors of the randomly selected features.
Note:
The global expected similarity value is an unbiased estimate of the true global expected value under the assumption of independence between the two results. However, the variance of the global expected value is not the same as the variance of the global similarity value due to correlations between overlapping neighborhoods. This means that traditional rank-based permutation p-values for the global similarity value are not valid for this procedure. Enhancing the methodology to support significance testing is an area of active research.
Best practices and limitations
Consider the following when using the tool:
- The choice of category similarity weights and whether to exclude nonsignificant features should be made based on the questions that you want to answer by performing the comparisons. You should not choose values and options only to maximize or minimize the similarity or association between the hot spot results. For example, while you can use category similarity weights to combine the 99% hot and 90% cold categories, the comparison will likely not answer a meaningful question, unless there is some reason to believe that 99% hot spots in one result should be considered similar to 90% cold spots in the other result. Similarly, excluding or including nonsignificant features should be determined by whether the nonsignificant areas represent areas of research interest.
- If either of the input hot spot result layers contains overlapping polygons, the overlaps will be intersected into new features. This can cause similarity values to not equal 1 even for result layers with identical significance level categories. The XY Tolerance environment can be used to remove unintended overlaps, such as geocoding errors. It is recommended that you review the number of features in the output features to determine if there are more intersections than expected.
- If the two hot spot results are polygons of different sizes, the intersection will subdivide large polygons into many smaller polygons. This changes the counts of the significance level categories and affects the similarity and association. There must be at least 20 feature intersections to use the tool.
- Changing the order of input hot spot results will not affect the similarity values, but the expected similarity and kappa values will change slightly due to randomness in permutations. The axes of the message tables and charts will also reverse, which will make it easier to interpret in some cases. Because the messages and charts display counts of the significance level categories of the second hot spot result within categories of the first result, you can instead display the categories of the first result within categories of the second result by reversing the order of the input layers.
References
Cohen, Jacob. 1960. "A coefficient of agreement for nominal scales." Educational and Psychological Measurement. 20:1, 37-46. https://doi.org/10.1177/001316446002000104.
Dou, Weibei, Yuan Ren, Qian Wu, Su Ruan, Yanping Chen, Daniel Bloyet, and Jean-Marc Constans. 2007. "Fuzzy kappa for the agreement measure of fuzzy classifications." Neurocomputing. 70, 726-734. https://dx.doi.org/10.1016/j.neucom.2006.10.007.
Hagen, Alex. 2003. "Fuzzy set approach to assessing similarity of categorical maps." International Journal of Geographical Information Science. 17:3, 235-249. https://doi.org/10.1080/13658810210157822.
Hagen-Zanker, Alex, Bas Straatman, and Inge Uljee. 2005. "Further developments of a fuzzy set map comparison approach." International Journal of Geographical Information Science. 19:7, 769-785. https://doi.org/10.1080/13658810500072137.
Hagen-Zanker, Alex. 2009. "An improved Fuzzy Kappa statistic that accounts for spatial autocorrelation." International Journal of Geographical Information Science. 23:1, 61-73. https://doi.org/10.1080/13658810802570317.