User authors their own dataset

Overview

A client program has their own app or data collection program which results in a spatial output. This could be CSV with an address or lat/lon, a spatial database, or anything in between.

Static output

Client produces ad-hoc updates

If the user is already maintaining their data using ESRI tools then they should be responsible for publishing and maintaining their dataset on EGIS as well.

If the user creates exports from some other system, and that export is in suitable format (eg. CSV, etc.) then that data can be published to EGIS through the web interface.

Update existing content? There is an 'update data' button available on hosted feature services. I could upload a CSV with x,y information and publish as a hosted feature service. I could then manually update the CSV and use that button to update the feature service. This rewrites the entire dataset of the service and doesn't just append new records.

We should note that I tried to write over a tcc_not_geocoded.csv file. This file contains addresses to be geocoded. Updating the CSV content item did not effect the hosted feature service. Updating the feature service resulted in an error.

Error executing tool. Publish Portal Service Job ID: j573ba5fd63274e6f831dc2f19313af3f : ERROR 999999: Something unexpected caused the tool to fail. Please refer to the error help for potential solutions, or contact Esri Technical Support http://esriurl.com/support . Only CSVs with coordinates can be overwritten. Failed to execute (Publish Portal Service). Failed. Failed to execute (Publish Portal Service).

There are formats that we can update via the web, but a CSV that requires geocoding is not one of them.

Client produces regular or frequent updates

If a user produces updates either frequently, or consistently at regular intervals, then it may make sense to automate the publishing of that data to ensure the EGIS content is current.

We'll need to rely on TC network ETL processes to publish this data to EGIS.

Database

TC Hosted Database

Databases hosted in the TC network aren't readable by the EGIS platform in the cloud. This may not be true indefinitely, but until that changes we'll need to rely on TC network ETL processes to publish this data to EGIS.

Cloud hosted database

Databases in the cloud may be readable directly from EGIS. A layer could be described and published from ArcGIS Pro once, and the data should remain current. The client would be responsible for maintaining the metadata of the content item.

TC network ETL processes

Data on the TC network that can't reasonably be manually maintained by its owner will need to rely on scripts or FME workflows to publish to EGIS.

Onboarding At present, the EGIS team has no infrastructure on the TC network, and as such has no capacity to schedule regular data maintenance tasks. We can, however, work with each client individually to help them create ETLs in their infrastructure to keep their content current in EGIS.

Cloud ETL processes

Given the cloud first adoption strategy from TBS, the intent is to create ETL process that can run entirely in the cloud. Data in the cloud should be readable by these ETL processes, which is covered by the User needs 3rd party data use cases.