Extending geoprocessing through Python modules

Nota:

This workflow requires conda, pip, setuptools, and wheel to be installed in your Python environment. These packages are shipped with the default ArcGIS Pro Python environment.

Python toolboxes and custom toolboxes can be distributed to other ArcGIS Pro users as Python packages by utilizing the setuptools library. The process for building and distributing these toolboxes starts with the creation of a Python module. For example purposes, we will use bar.py.

bar.py

Sample code for the bar Python module.

import os 
 
def hello(): 
    return 'Hello ' + os.getenv('username')

For distribution, the module bar is organized into a package named foo. For the package to be built and distributed, a specific directory structure must exist. A directory named foo must be created to store the bar module. Since distribution requires that the directory storing the bar module be within a parent directory, a directory named src is created to house the foo directory and the bar module. The directory structure will be as follows:

src
└───foo
     └  bar.py

For the bar module to initialize and execute specific code once it has been imported, it requires an __init__.py file. With the __init__.py file, the foo directory is treated as a package by Python and others will be able to access the bar module.

__init__.py

Sample code for __init__.py in foo.

from foo import bar

The directory structure will now be the following:

src
└───foo
     ├  __init__.py 
     └  bar.py

With the files and directory structure, the bar module can be imported with import foo and the foo.bar.hello() function can be executed.

The next step is to build a distribution package for the foo package, so that it can be installed into the Python site-packages directory. This way, the package can be shared and used by others. A distribution package can be created by writing a setup.py script.

setup.py

Sample code to create a setup.py script.

import os 
from setuptools import setup 
 
def read(fname): 
    return open(os.path.join(os.path.dirname(__file__), fname)).read() 
 
setup(name='foo', 
      version='1.0', 
      author='Esri', 
      description=("Example for extending geoprocessing through Python modules"), 
      long_description=read('Readme.txt'), 
      python_requires='~=3.3', 
      packages=['foo'], 
      )

The setup.py build script sets several properties, and directs the build utility to the package directory. The long description is stored in an accompanying Readme.txt file. The setup.py and Readme.txt files should be located in the src directory. For more information on using setup(), see Packaging and distributing projects. The directory structure will be as follows:

Src
├  setup.py
├  Readme.txt
└──foo
    ├  __init__.py 
    └  bar.py

Nota:

Additional __pycache__ folders may appear in the directories throughout this process. These are caches of compiled versions of modules generated by Python to speed up the loading of modules. To simplify the folder structure diagrams, they are not shown here.

With the directory structure in place, an installer can be built for the foo package by running the command below from within the src directory from the command prompt.

python setup.py sdist bdist_wheel

The sdist bdist_wheel command creates dist, build, and foo.egg-info folders in the src parent directory. In the dist folder, a wheel file named foo-1.0-py3-none-any.whl is created. A wheel file can be distributed directly with pip.

Installing wheel distributions with pip

Nota:

Modifying ArcGIS Pro's default Python environment (arcgispro-py3) may result in unintended consequences. It is recommended that you only modify a cloned environment.

Clone the default ArcGIS Pro environment and switch to the cloned environment using the Python Package Manager.

After cloning and switching environments, open the ArcGIS Pro Python Command Prompt to install the wheel file directly with pip. Pip will install the wheel file into the active environment. From the command prompt pointing to your src\dist directory, run the following:

pip install foo-1.0-py3-none-any.whl

Alternatively, the foo directory can also be copied directly from the build\lib directory to the %LocalAppData%\ESRI\conda\envs\<environment_name>\Lib\site-packages folder. If there are restrictions in place that prohibit running an installation, copying the foo directory from the build\lib directory to the site-packages directory will produce the same effect as installing it through pip, except that the package will not be recognized or managed by pip.

Once the foo module has been installed or copied into the site-packages directory, the structure will be the following:

site-packages
└──foo
    ├  __init__.py 
    └  bar.py

Distributing ArcGIS Protoolboxes as Python packages

This process can be further expanded to extend geoprocessing functionality with your own toolboxes that appear alongside ArcGIS Pro system toolboxes. Your toolboxes will appear and be accessible in all the ways that system toolboxes within ArcGIS Pro are; accessible through the Geoprocessing window, and from arcpy. In addition, this allows the custom toolbox module to take advantage of the well-established methodology that ArcGIS Pro system toolboxes have for message distribution, language-based help, and response to localized settings. The following is the standard directory structure:

Src
├  setup.py
├  Readme.txt
└──foo
    ├  __init__.py 
    ├  bar.py
    └──esri
        ├──arcpy
        ├──help
        │   └──gp
        │       ├──messages
        │       └──toolboxes
        └──toolboxes

Custom toolboxes (.tbx and .pyt) are placed in the esri\toolboxes directory along with any supporting .py files. The esri\help\gp directory is where the toolbox and tool metadata (.xml) for custom toolboxes and tools are stored. The naming convention for the toolbox is <toolbox alias>_toolbox.xml and for each tool is <toolname>_<toolbox alias>.xml. The esri\help\gp\messages directory is where any geoprocessing message (.xml) files are placed. These message files are used within the Python toolboxes for messages that need to be internationalized. The toolbox and tool labels categories override files that are located in the esri\help\gp\toolboxes  directory.

The entire process of extending geoprocessing through Python packages can be demonstrated through the creation of a Python toolbox named SamplePythonToolbox.

SamplePythonToolbox.pyt

Sample code to create a Python toolbox:

# -*- coding: utf-8 -*-

import arcpy
import os
import foo

class Toolbox(object):
    def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of the
        .pyt file)."""
        self.label = "Sample Python Toolbox"
        self.alias = "SamplePythonToolbox"

        # List of tool classes associated with this toolbox
        self.tools = [SampleTool]

class SampleTool(object):
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "Sample Tool"
        self.description = ""
        self.canRunInBackground = False

    def getParameterInfo(self):
        """Define parameter definitions"""
        parameters=[arcpy.Parameter(displayName='Msg', 
                                  name='msg',
                                  datatype='GPString',
                                  parameterType='Derived',
                                  direction='Output')
                                  ]
        return parameters

    def isLicensed(self):
        """Set whether tool is licensed to execute."""
        return True

    def updateParameters(self, parameters):
        """Modify the values and properties of parameters before internal
        validation is performed.  This method is called whenever a parameter
        has been changed."""
        return

    def updateMessages(self, parameters):
        """Modify the messages created by internal validation for each tool
        parameter.  This method is called after internal validation."""
        return

    def execute(self, parameters, messages):
        """The source code of the tool."""
        result = foo.bar.hello()
        messages.AddMessage(result + ", welcome to the sample tool")
        parameters[0].value = result
        return

Learn more about creating a Python toolbox

In the SamplePythonToolbox.pyt toolbox, the foo package has been imported and the execute method of the SampleTool class has called the hello()  function from the bar module. This is an effective way of distributing custom Python code as a package and exposing its functionality through ArcGIS Pro geoprocessing tools. Once the SamplePythonToolbox.pyt has been created, the side-panel help can be configured for the toolbox through the metadata edited in the Catalog pane's Edit Metadata menu.

After creating the metadata for the toolbox, the arcpy.gp.createtoolboxsupportfiles function can be run to generate the supporting .xml files found within the esri folder structure described above. The command can be used on any toolbox. Place the SamplePythonToolbox.pyt in a separate staging folder, then run arcpy.gp.createtoolboxsupportfiles(<path to .tbx or .pyt>).

Nota:

Make sure that an alias is set on the toolbox before running arcpy.gp.createtoolboxsupportfiles. The alias is needed to generate the side-panel help files.

Within the esri folder, create a toolboxes folder and place SamplePythonToolbox.pyt and its accompanying .xml files into it. The staging folder should now look like the following:

Staging
 └──esri
     ├──arcpy
     │   └  SamplePythonToolbox.py
     ├──help
     │   └──gp
     │       ├  SamplePythonToolbox_toolbox.xml
     │       ├  SampleTool_SamplePythonToolbox.xml
     │       ├──messages 
     │       └──toolboxes
     │           └  SamplePythonToolbox.xml
     └──toolboxes
         ├  SamplePythonToolbox.pyt
         ├  SamplePythonToolbox.pyt.xml
         └  SamplePythonToolbox.SampleTool.pyt.xml

Nota:

The messages folder is not generated by arcpy.gp.createtoolboxsupportfiles; it must be created manually to make the error messages used in the script tools or Python toolboxes localizable, but can be omitted if there is no plan for internationalization.

The esri folder must be copied over from the staging directory where it was created into the src\foo directory. The directory structure for the distribution will now be as follows:

Src
├  setup.py
├  Readme.txt
└──foo
    ├  __init__.py 
    ├  bar.py
    └──esri
        ├──arcpy
        │   └  SamplePythonToolbox.py
        ├──help
        │   └──gp
        │       ├  SamplePythonToolbox_toolbox.xml
        │       ├  SampleTool_SamplePythonToolbox.xml
        │       ├──messages
        │       └──toolboxes
        │           └  SamplePythonToolbox.xml
        └──toolboxes
            ├  SamplePythonToolbox.pyt
            ├  SamplePythonToolbox.pyt.xml
            └  SamplePythonToolbox.SampleTool.pyt.xml

To reflect these changes in our distribution, the setup.py file must be modified.

Modified setup.py

Sample code to include setup.py directory changes:

import os 
from setuptools import setup 
 
def read(fname): 
    return open(os.path.join(os.path.dirname(__file__), fname)).read() 
 
setup(name='foo', 
      version='2.0', 
      author='Esri', 
      description=("Example for extending geoprocessing through Python modules"), 
      long_description=read('Readme.txt'), 
      python_requires='~=3.3', 
      packages=['foo'], 
      package_data={'foo':['esri/toolboxes/*',  
                  'esri/arcpy/*', 'esri/help/gp/*',  
                  'esri/help/gp/toolboxes/*', 'esri/help/gp/messages/*'] 
                  }, 
      )

The new setup.py differs from the original by one line where the additional data found within the esri directory is added to the package. The version has also been incremented to 2.0. To update the distribution, remove the build, dist, and foo.edd-info folders from the src directory, and rerun python setup.py sdist bdist_wheel. To install, run the following from the command prompt pointing to your src\dist directory:

pip install --upgrade foo-2.0-py3-none-any.whl

Now when the builder for the foo package is executed and installed, the following directory structure will be created in the Python site-packages directory:

site-packages
└──foo
    ├  __init__.py 
    ├  bar.py
    └──esri
        ├──arcpy
        │   └  SamplePythonToolbox.py
        ├──help
        │   └──gp
        │       ├  SamplePythonToolbox_toolbox.xml
        │       ├  SampleTool_SamplePythonToolbox.xml
        │       ├──messages
        │       └──toolboxes
        │           └  SamplePythonToolbox.xml
        └──toolboxes
            ├  SamplePythonToolbox.pyt
            ├  SamplePythonToolbox.pyt.xml
            └  SamplePythonToolbox.SampleTool.pyt.xml

Nota:

The messages folder is only present if it contains files.

In ArcGIS Pro, the SamplePythonToolbox can now be searched for in the Geoprocessing pane and is located under the Toolboxes tab. The toolbox and tool can be accessed from arcpy as follows: arcpy.SamplePythonToolbox.SampleTool()

Using ArcGIS Pro and the Python package development process provided by the setuptools library, it is possible to build and install a package that extends geoprocessing with custom tools in custom toolboxes that can be viewed and executed from within the ArcGIS system toolboxes and arcpy. For English language distributions, this is all that is needed. For more on extending geoprocessing for distribution in languages other than English, see Internationalization of Python modules.