Back to Top

Automate reconcile and post operations for sync-enabled branch versioned data

In this topic

  1. Applicable scenario
  2. Code example

Portal administrators and version administrators can use Python scripting to automate a scheduled reconciliation of replica versions after field workers have synchronized edits to data included in branch versioning.

This workflow is applicable for managing replica versions that were created using a sync-enabled feature service with branch versioned data.

Learn more about working with offline maps with branch versioned data

Applicable scenario

The Python example in the next section uses the feature service ArcGIS REST API and ArcPy geoprocessing tools to reconcile and post replica versions with validation attribute rules in place. The following describes when you'd use the code sample and what the sample script does.

Note:

This script is an example of a workflow with a specific data setup. Your workflows may not follow this exact data model, but the concepts and script logic may still be used to customize an automated script for your organization. For example, you may not use validation rules, but you can alter the script to remove the evaluation of validation rules. Also, in some cases when using validation rules, you may want to reconcile, run validation, and post each version individually before running the evaluate operation to ensure the validation includes edits made by all other editors.

Prerequisite:

In this example, the data setup is as follows:

  • The feature service must be sync-enabled with branch versioned data with the Create a version for each downloaded map option selected , which requires ArcGIS Enterprise 10.8.1 or later.
  • Replicas have been created on the service by taking a map offline.
  • The version management server capability has been enabled on the service.
  • The credentials provided for the portal user in the script must be for a member of either the default portal Administrator role or a custom role that has the version management privilege granted to it.
  • In this example, the data also has validation attribute rules applied and the validation server capability enabled. This is not a requirement to work with replica versions, but it provides a way to ensure data integrity.

  1. Get a list of the replica versions for the feature service using the replicas resource in ArcGIS REST API.
  2. Use the versionsInfos resource in ArcGIS REST API to get the properties of the version. Using these properties, create a filter to get only the versions in which the evaluation date is null or the modified date is greater than the last evaluation date (meaning edits have been made since the last evaluation). Append the names of the versions that satisfy the filter to the listOfVersionsToValidate list that will be evaluated.
  3. For each version in the evaluation list, use the evaluate operation in ArcGIS REST API to evaluate the validation attribute rules on the service. If the evaluation is successful with no errors returned, the version is ready to reconcile and post. If the evaluation is successful but errors are returned, don't post the edits; generate a message so that the errors can be manually inspected and fixed.
  4. For versions that were successfully evaluated with no errors, run the Reconcile Versions tool with the option to post edits.
  5. Run the Reconcile Versions tool with the option to reconcile only (no post) on all replica versions.

Code example

The complete Python code example below completes the operations listed above. This script includes an option to generate a log file that captures the output of each completed operation and can be viewed after the script completes. 

# Import modules
import arcpy, traceback, urllib, json, urllib.request, urllib.parse, os, urllib.error, datetime

# Overwrite the reconcile log output each time the script is run
arcpy.env.overwriteOutput = True

# Script parameters
serviceName = "MyServiceName"
baseURL = "https://MyServer.MyDomain.com"
portalAdmin = "MyAdminUser"
portalAdminPwd = "MyAdmin.Password"
logFileScript = "C:/Logs/validateRecPostScriptLog.txt"
logfileOutputRecPost = 'C:/Logs/reconcile_log.txt' 

# Choose to output a log file for the script
outputScriptReport = True

# Define functions
def openURL(url, params=None):
    """This function used to open a URL and returns the json response"""
    try:
        request_params = {'f':'pjson'}
        if params:
            request_params.update(params)
        encodedParams = urllib.parse.urlencode(request_params)
        request = urllib.request.urlopen(url, encodedParams.encode('UTF-8'))
        response = request.read()
        json_response = json.loads(response)
        return json_response
    except:
        print (traceback.format_exc())

def versionInfo(versionOwner=""):
    """This function queries the versions owned by the versionOwner.
    It returns a list of dictionaries."""
    vmsUrlinfo = "{}/server/rest/services/{}/VersionManagementServer/versionInfos?&ownerFilter={}&includeHidden=&f=json&token={}".format(baseURL, serviceName, versionOwner, token)
    response = openURL(vmsUrlinfo)
    if response['success'] == True:
        versionsDict = response['versions']
        return versionsDict
    else:
        return("Unable to get version info")

def evaluateUrl(validationUrl, validate_params):
    """This function runs evaluate on the validation server
    It returns the json response."""
    evalJsonResp = openURL(validationUrl, validate_params)
    if evalJsonResp['success'] == False:
        return [False, evalJsonResp]
    else:
        return [True, evalJsonResp]

def generateMessage(msg, print_statement=True):
    """This function generates messages as the script runs. If print_statement
    is set to True, print to the screen. If outputScriptReport is set to true,
    write the message to the logfile"""
    if outputScriptReport == True:
        with open(logFileScript, 'a') as log_file:
            log_file.write(msg + "\n")
    if print_statement == True:
        print(msg)

def recPostVersions(versionList, post):
    """This function runs the Reconcile Versions GP tool to reconcile
    and optionally post to the feature service"""
    if post == True:
        postVersion = "POST"
    elif post == False:
        postVersion = "NO_POST"
    # Reconcile and post the replica versions 
    # This tool is set to abort if there are conflicts and detects conflicts by object
    arcpy.ReconcileVersions_management(featureService,
                                        'ALL_VERSIONS',
                                        'sde.DEFAULT',
                                         versionList,
                                        'NO_LOCK_ACQUIRED',
                                        'ABORT_CONFLICTS',
                                        'BY_OBJECT',
                                        'FAVOR_EDIT_VERSION',
                                        postVersion,
                                        'KEEP_VERSION',
                                        logfileOutputRecPost)
    generateMessage(arcpy.GetMessages()+"\n")
    
# Start execution
generateMessage('Starting Validation/Reconcile/Post Automation Script... {:%Y-%b-%d %H:%M:%S}\n'.format(datetime.datetime.now()))

# Sign in to ArcGIS Enterprise    
signIntoPortal = arcpy.SignInToPortal(baseURL+"/portal", portalAdmin, portalAdminPwd)
generateMessage("Signed into ArcGIS Enterprise {} as user {}".format(baseURL+"/portal", portalAdmin))

# Get the token returned by the SignIntoPortal arcpy function to use for making REST requests
token = signIntoPortal['token']

# Build the feature service URL
featureService = "{}/server/rest/services/{}/FeatureServer".format(baseURL, serviceName)

# Get a list of the replica versions from the REST endpoint
listOfRepVersions = []
replicaVersionsURL = featureService + "/replicas?returnVersion=true&f=pjson"
repVersionsJson = openURL(replicaVersionsURL, signIntoPortal)
for repVersion in repVersionsJson:
    versionName = repVersion['replicaVersion']
    listOfRepVersions.append(versionName)
    
# Create an empty list to append version names to validate
listOfVersionsToValidate = []

# Iterate through each version returned by the versionInfo() function to find 
# the versions that need to be validated that are also in the listOfRepVersions list
for version in versionInfo():
    print("")    
    # Parse the version info response, which is a python dictionary/json
    # If the version name is sde.DEFAULT, pass since we do not want to evaluate the default version
    if version['versionName'] == "sde.DEFAULT":
        pass
    # If the modifiedDate property is null, pass
    elif version['modifiedDate'] == "None":
        pass
    # If the evaluation date is null, append the version name to the list to listOfVersions to be evaluated
    elif version['evaluationDate'] == None:
        if version['versionName'] in listOfRepVersions:
            listOfVersionsToValidate.append(version['versionName'])
    # If the evaluation date is not null, but it has been modifed since the last evaluation, add it to the list to be validated
    elif version['evaluationDate'] != None and version['modifiedDate'] > version['evaluationDate']:
        if version['versionName'] in listOfRepVersions:
            listOfVersionsToValidate.append(version['versionName'])
    # If none of these conditions are met
    else:
        generateMessage("Version {} will not be validated.".format(version['versionName']))
            
# Validate versions
generateMessage('The following versions will be validated: {}\n'.format(listOfVersionsToValidate))

# Create lists to contain versions where the validation passed or failed
failEval = []
passEval = []

# For each version in the list of versions, build the json request needed to validate
for i in listOfVersionsToValidate:
    validate_params = { "gdbVersion": i,
             "sessionId": "",
             "evaluationArea": "",
             "changesInVersion": "true",
             "selection": "",
             "evaluationType": '["validationRules"]',
             "returnEdits": "true",
             "async": "false",
             "f": "pjson",
             "token": token
    }
    # Build the REST URL used to validate the service
    validationUrl = baseURL + "/server/rest/services/"+ serviceName +"/ValidationServer/evaluate"
    
    # Call the evalVersion() function to validate the version
    evalVersion = evaluateUrl(validationUrl, validate_params)
    
    # If the evaluate failed, append to the failEval list
    if evalVersion[0] == False:
        generateMessage("Evalution of version {} failed".format(i))
        generateMessage(str(evalVersion[1]))
        failEval.append(i)
        
    # If the evaluate passed, check to see if errors were returned    
    elif evalVersion[0] == True:
        # If errors are returned, only reconcile this version
        if evalVersion[1]['errorsIdentified'] != 0:
            generateMessage("{} Errors were identified in version {}.\nThe version will be reconciled but will not be posted.\n".format((str(evalVersion[1]['errorsIdentified'])),i))
            generateMessage(str(evalVersion[1]), False)
        # If errors were not found this version can be posted
        else:
            generateMessage("Evaluation of version {} passed with no errors identified.\nThis version will be reconciled and posted.\n".format(i))
            generateMessage(str(evalVersion[1]))
            passEval.append(i)

# The versions that passed validation should be reconciled/posted
generateMessage('\nThe following versions passed validation and will be reconciled and posted: {}\n'.format(passEval))

# Run recPostVersions on the list of versions that passed evaluation with the option to post
recPostVersions(passEval, True)

# Open the reconcile log file and append the results to our report
with open(logfileOutputRecPost, 'r') as openRecLog:
    generateMessage(openRecLog.read(), False)
    
# Run recPostVersions with the option to reconcile all replica versions, no post
recPostVersions(listOfRepVersions, False)

# Open the reconcile log file and append the results to our report
with open(logfileOutputRecPost, 'r') as openRecLog:
    generateMessage(openRecLog.read(), False)

# Script execution complete
generateMessage('Validate, Reconcile, & Post Script Complete.')

Feedback on this topic?

In this topic
  1. Applicable scenario
  2. Code example