In progress
What is AIS feed?
The Department of National Defense provides global maritime vessel data for ship tracking through the AIS (Automatic Identification System) feed.
EGIS team has been working on make the AIS feed available for the entire department through the EGIS Portal.
The team receive the URL and credentials from Alicia Hogue on July 23rd, 2019 and three different proof-of-concepts were developed to publish data to EGIS Portal.
Python Script
This is a simple solution. After downloading csv from DND’s AIS data feed link, upload this csv to Portal as a service directly. Then periodically update the service through updating the underline GDB file by using the following script:
import arcpy
import os
arcpy.env.workspace = "c:/data"
arcpy.CopyFeatures_management("features.csv", os.path.join("outgdb.gdb", "myfeatures"), "POINT")
The frequency of updating the feature service directly from Portal is so high and the GDBs are kept in the temporal folder all the time which caused the C: drive full and crashed the ArcGIS Server system.
This solution is discarded.
FME
Two separated services were created using FME desktop to publish AIS data on EGIS Dev Portal.
DND_AIS_Features_Single: This service keeps all the vessel types under the same layer allowing the user to filter and refine what is presented on the map. The service is available on the fallowing link:
https://devweb.tcgis.ca/portal/home/item.html?id=311ef693e27f4bf0952256b05353212a
DND_AIS_Features_Multi: This services generates one layer for each vessel type allowing users turn on and off the relevant data set different symbology to each vessels. The service is available on the following link:
https://devweb.tcgis.ca/portal/home/item.html?id=b0939104421649d5bc7f04de774e3ec0
AIS is a live feed and the data must be refreshed constantly. To do so FME provides a tool called FME Cloud that uses triggers to run the workspace base on a time schedule or when other conditions are met. As the FME Cloud was not available for tasting, a simple task schedule was set in the local machine to test the data update.
For each service, multiple and single layers, two FME workspaces were created. One workspace to fetch the data for the first time, create the layers and publish the service in the portal. The second workspace should run in a schedule, it compares the current data with the new data and updates or deletes records accordingly.
FME drawbacks and cost:
- There is no feature on FME Desktop to set hosted layer properties and symbology. A Python script can be used to set configurations when publishing the service for the first time. The updates will not change the configuration.
- There is an annual fee for Desktop license of C$3000.00 plus the FME Cloud that starts at C$0.90 per hour.
GeoEvent Server
The GeoEvent Server approach currently was defined as the way to go for publishing AIS data since it can handle live and historical data.
A .Net script is being used to pull data from DND AIS feed URL directly. The script runs every 5 minutes triggered by the task scheduled on the server (EGIS-PLT-EVT). The polled records are processed, split into two CSV files, based on the data source (Maerospace and Costal), and the cleaned data is stored in the GeoDatabase of Azure SQL Server Service.
The GeoEvent Server consumes the two CSV files as input, massages the data in the GeoEvent Services and updates the data in the feature services published in ArcGIS Server, which are in turn consumed in Portal for ArcGIS.
eBIDA Integration
Azure Data Lake Storage Gen2 provided by the eBIDA team pulls data from the DND AIS feed URL, storing the data in raw format (CSV). The data is made available to eGIS team trough Microsoft Azure Storage Explorer, however to consume the real time data using the .Net script the Data Lake must have hierarchical infrastructure in the preview enabled. The eBIDA team is analyzing the impact of this change in the current Data Lake infrastructure.
The 10 TB historical data is stored in the data lake with the format of the .nmea.gz files. So extracting the files and then load the raw json files to GeoEvent Server Big Data Store is the ongoing procedure. Note, as of today, ESRI has not supported ADLS Gen 2 Data Lake yet.
Another challenge for the integration is to set a data layout that fits the eGIS needs. A prepossessing may be required. However, eBIDA project is at the proof-of-concept stage and the plans are to go live on the beginning of next fiscal year, April 1st. The final state aimed by both teams is to feed from the data lake and publish the feature layers on eGIS portal.
Current Services Available
Currently the AIS feed is publicly available on Sandbox in the following link: https://sbxweb.tcgis.ca/portal/home/webmap/viewer.html?webmap=9b657e410e654331b0bb084ee9567b50
On Pilot PROD:
https://egis-sige.tcgis.ca/portal/home/webmap/viewer.html?webmap=53e80e0743c142d6ae95aa5e54c3030f
During the sprint 29 this service will be moved to Pilot environment where it will be shared with entire organization.