Calculate the consistency ratio for pairwise comparison

The Assign Weights by Pairwise Comparison tool calculates the relative weights for a set of input variables by comparing them in pairs. Diagnostic functionality is provided to ensure that the comparison evaluations among all pairs is logically consistent. The tool calculates the Consistency Ratio (CR) to help you improve the consistency of your evaluations. For example, your pairwise evaluations will be inconsistent if you rank variable A to be twice as important as variable B, rank variable B to be twice as important as variable C, and rank variable C to be the same importance as variable A. That is, if A > B, and B > C, then C cannot be equal to A.

Keeping the CR within acceptable limits helps prevent inconsistent rankings. According to Saaty (2000):

• If the CR <= 0.1, the consistency of the pairwise comparison is acceptable.
• If the CR > 0.1, the comparison evaluations should be adjusted to improve consistency.

Consistency ratio calculation process

The steps for calculating CR are as follows:

• The Consistency Index (CI) is calculated.
• The Random Index (RI) is determined based on the number of variables.
• The CR is calculated by dividing the CI by RI.

The following comparison matrix will be used in an example demonstrating how the CR is calculated for five variables.

Table 1: A comparison matrix with derived weights.

Variable	Solar_Gain	Aspect	Elevation	Dist_to_Road	Dist_to_Elect	Weights
Solar_Gain	1	3	2	2	0.333	0.235
Aspect	0.333	1	0.5	1	0.2	0.089
Elevation	0.5	2	1	1	0.5	0.136
Dist_to_Road	0.5	1	1	1	5	0.247
Dist_to_Elect	3	5	2	0.2	1	0.294

The series of equations described below are used in the calculations.

Compute the CI

The CI is calculated using the following formula:

Where:

• λ (the principal eigenvalue) is the average of the λ_i for each variable
• n is the number of input variables

The λ_i value for each variable is calculated using the following formula:

Where:

• w_i denotes the derived weights for each input variable (the Weights column in Table 1)
• SW_i is the weighted sum for each variable i

The weighted sum for each variable is calculated using the following formula:

Where:

• a_ij is the value from the original pairwise comparison matrix where i refers to the row and j refers to the column
• wj are the derived weights of the corresponding variables

For example, the SW for the Solar_Gain variable row is calculated as follows:

The λ_i for the Solar_Gain variable is calculated as follows:

Repeat the calculations for each row to find the λ_i value for each variable.

Next, average the λ_i values to obtain λ.

In this example, λ is 6.462 as specified in the table below.

Table 2: Calculation for the Weighted Sum (SW) and the λ_i for each variable.

Variable	Weights(wi)	Weighted Sum(SWi)	λi = SWi / wi
Solar_Gain	0.235	1.366	5.812
Aspect	0.089	0.541	6.079
Elevation	0.136	0.826	6.070
Dist_to_Road	0.247	2.060	8.338
Dist_to_Elect	0.294	1.765	6.005
Average	NA	NA	6.462

With λ now computed, CI can be calculated using the formula provided above.

Since the number of variables in this example is five, the value for CI is calculated as follows:

CI = (6.462 - 5)/(5 - 1) = 0.365

Calculate the RI

The RI is a predefined statistical value that provides a benchmark for computing the CR for the pairwise comparisons. Developed by Saaty (2000), the RI values are derived from a Monte Carlo simulation for each matrix size. The derived RI values for each matrix size are shown in the table below. The matrix sizes are identified in the top row.

Matrix size	2	3	4	5	6	7	8	9
RI	0	0.58	0.90	1.12	1.24	1.32	1.41	1.45

The RI value is 0 for the two-variable comparison, because the comparison is inherently consistent. The RI values increase with the number of input variables.

In this example, the number of variables is five, so the corresponding RI value is 1.12.

Calculate the CR

The CR is calculated by dividing the CI by the RI. This CR value shows how consistent the pairwise comparisons are.

In this example, the CR is calculated as follows:

CR = 0.365/1.12 = 0.326

Since the CR value is larger than 0.1, you should modify the comparison evaluations to improve the consistency.

Improve consistency

If the comparisons of the input variables are determined to be inconsistent, the three pairs that contribute the most to the inconsistency are highlighted. The highlighted pairs should be adjusted to improve consistency.

The workflow below is used to determine the three most inconsistent pairs.

Construct a pairwise consistency matrix

Construct a matrix that will be used in the pairwise consistency evaluation (Saaty, 2003). This matrix is referred to as ε_ij in the following formula:

Where:

• a_ij is the value from the original pairwise comparison matrix, i refers to the row and j refers to the column.
• w_j and w_i are the derived weights for variable j and i.

Calculate the values

Once the matrix is constructed, the three largest values in the matrix ε_ij are identified. The corresponding row and column variables are considered inconsistent pairs.

In this comparison example, the CR value is larger than 0.1 (0.326). To identify the inconsistent pairs, the ε_ij value for the Solar_Gain row and Aspect column cell is calculated as follows:

Repeat the calculation for all cells.

Determine the most inconsistent pairs

The ε_ij matrix for the comparison example above is constructed as follows:

The highest three values in the matrix are the comparisons that contribute most to the inconsistency. They are the comparison evaluations for variables Dist_to_Road and Dist_to_Elect, for variables Aspect and Dist_to_Road, and for variables Dist_to_Elect and Solar_Gain. The three pairs are highlighted in red on both sides of the sliders on the tool dialog box.

In the original comparison matrix, the variable Solar_Gain is evaluated as twice as important as Dist_to_Road (Solar_Gain > Dist_to_Road), and three times less important as Dist_to_Elect (Dist_to_Elect > Solar_Gain). Logically, Dist_to_Road should be less important than Dist_to_Elect (Dist_to_Elect > Dist_to_Road). However, in the original comparison, the evaluation incorrectly assigned Dist_to_Road a weight five times as important as Dist_to_Elect (Dist_to_Road > Dist_to_Elect). This is a main cause of the inconsistency.

To correct this, move the slider toward the Dist_to_Elect side, and review the CR again. For example, if you move the slider to position 5 on the Dist_to_Elect side, and click the Consistency button again, the new CR is 0.029, indicating that the overall comparison is now consistent.

References

Saaty, T. L. 2000. Fundamentals of Decision Making and Priority Theory with the Analytic Hierarchy Process. Pittsburgh, PA: RWS Publications

Saaty, T. L. 2003. "Decision-making with the AHP: Why is the principal eigenvector necessary". European Journal of Operational Research, 145(1), 85-91.