Derived Data Products

Owned by Diego Cesa de Meira (Unlicensed)
Sept 05, 2019
5 min read

Overview

In some cases, users require a dataset that is derived from one or more third-party datasets that are avaialble online as downloadable files (e.g., shapefile, file geodatabases, ore even CSVs). This scenario represents a significant number of the TC Smart Maps datasets, which are often derived from datasets that are downloadable from FGP or other government websites.

In these cases, the source datasets are processed in some way, including filtering, field removal or renaming, and joining with other datasets. Currently, @Lawrence Chong (Unlicensed) performs this processing for TC Smart Maps mostly using FME, and partially by hand.

Preparation for ArcGIS Enterprise

ArcGIS Pro is the Esri-preferred method for preparing data to be published to ArcGIS Enterprise. Using ArcGIS Pro, the data publisher can configure the presentation of the dataset (including symbology, labeling, fields, popups, etc.), and can also perform most if not all of the other preprocessing tasks.

Once the data publisher has prepared the dataset, they can publish the dataset to ArcGIS Enterprise.

Automating Data Updates

Once the dataset is published to ArcGIS Enterprise, many data updates can be automated using FME. FME seems to be a suitable candidate for pre-processing of TC Smart Maps layers for the following reasons:

  1. Lawrence is already using it extensively, and he already has a license.

  2. It is capable of updating existing ArcGIS Enterprise feature services.

  3. If necessary, additional functionality can be added to FME using Python scripting.

FME Workflows can be executed on FME Desktop, FME Server, or on FME Cloud (we've confirmed that, if the source data files are available online, FME workflows to update ArcGIS Enterprise services can be executed entirely within the FME Cloud environment).

Case Study: Populated Places

The Populated Places TC Smart Maps layer was used as a case study to demonstrate how ArcGIS Pro and FME can be used to create and maintain a TC Smart Maps layer.

The Populated Places layer uses the Canadian Geographical Names Database (CGNDB) as its source data. The CGNDB data are downloadable as a zipped shapefile from FGP.

  1. The shapefile was downloaded from FGP and unzipped into a working folder.

  2. A new project for Populated Places was created in ArcGIS Pro.

  3. The Populated Places layers were created using the CGNDB data and configured.

  4. The Populated Places layers were published to a feature service using ArcGIS Pro.

  5. An FME workspace was created to update the Populated Places layers. The FME workspace downloads the zipped shapefile from FGP, performs the necessary pre-processing on the data, and updates each of the existing Populated Places layers in EGIS.

  6. Once the FME workspace was validated, it was published to FME Cloud.

  7. An automation was created in FME Cloud to run the workspace each time the FTP directory containing the CGNDB files is updated (they are updated weekly).

Notes

A few notes on this approach:

  1. It requires that data stewards have a license for ArcGIS Pro. Lawrence has a license for ArcGIS Pro, as do many of the GIS power users within TC. Probably most or all of the users who would publish data would be licensed for ArcGIS Pro. As ArcGIS Desktop licenses are consolidated within EGIS, licenses should be available to any users who needs them.

  2. Using ArcGIS Pro to author the map services is the Esri-recommended workflow. This gives the data steward / data publisher precise control over how the services are configured.

  3. FME appears to be able to do all of the data transformations that are possible in ArcGIS Pro / ArcPy, as well as additional manipulations that are not possible in ArcGIS Pro / ArcPy.

  4. Once the service is created, site backup and restore will preserve the service configuration in the event of failure (e.g., catastrophic failure, accidental deletion).

  5. The ArcGIS Pro project that's used to define the map service should be placed under source control.

  6. The FME workspace should be integrated into Lawrence's FME existing workspace for updating TC Smart Maps.

  7. The ArcGIS Pro project (.aprx) should be placed under source control. Potentially, the layer files (.lyrx) and service drafts (.sd) files should also be placed under source control.

  8. The FME workspace should be placed under source control.

  9. Datasets should not be placed under source control. If it's necessary to store datasets, they should be stored on Azure blob storage or similar.

  10. FME cannot yet update portal item metadata. This would still need to be completed via an external process.

  11. To enable the workspace to run in FME Desktop, the NTv2 must be downloaded from the Canadian Geodetic Survey and referenced by FME Workbench (Tools > FME Options > Coordinate Systems > NAD83_to_CSRS > Edit > ... , then browse to the downloaded Ntv2_0.gsb file). This is because, in this particular case, the source data are represented in CSRS coordinates.

  12. To enable the workspace to run in FME Cloud, a couple of things had to be configured:

    1. A static IP address had to be created and assigned to the FME Cloud instance, then an inbound firewall rule had to be added to the portal virtual machine to allow inbound traffic from that IP address.

    2. The above-referenced Ntv2_0.gsb file had to be uploaded to the Resources/Data folder. In addition, the TC Smart Maps/FME/WorkbenchGridOverride_NAD83_to_CSRS.txt file (in the EGIS source repository) was uploaded to the Resources > Engine > CoordinateSystemGridOverrides folder.