...
The team receive the URL and credentials from Alicia Hogue on July 23rd, 2019 and two three different proof-of-concepts were developed to publish data to EGIS Portal.
Python Script
This is a simple solution. After downloading csv from DND’s AIS data feed link, upload this csv to Portal as a service directly. Then periodically update the service through updating the underline GDB file by using the following script:
import arcpy
import os
arcpy.env.workspace = "c:/data"
arcpy.CopyFeatures_management("features.csv", os.path.join("outgdb.gdb", "myfeatures"), "POINT")
The frequency of updating the feature service directly from Portal is so high and the GDBs (each is around 60MB) are kept in the temporal folder all the time which caused the C: drive full and crashed the ArcGIS Server system.
This solution is discarded.
FME
Two separated services were created using FME desktop to publish AIS data on EGIS Dev Portal.
...
The GeoEvent Server approach currently was defined as the way to go for publishing AIS data since it can handle live and historical data.
A .Net script is being used to pull and data from the DND AIS feed URL directly. The script runs every 5 minutes triggered by the task scheduled on the server (EGIS-PLT-EVT). The polled records are processed, splinted in split into two CSV files, based on the data source (Maerospace and Costal), and the cleaned data is stored in the GeoDatabase of Azure SQL Server Service.
The GeoEvent Server consumes the two CSV files as input, massages the data generating the featured services as the output. The services are published on the portal.
EBIDA Integration
The Data Lake provided by the EBIDA in the GeoEvent Services and updates the data in the feature services published in ArcGIS Server, which are in turn consumed in Portal for ArcGIS.
eBIDA Integration
Azure Data Lake Storage Gen2 provided by the eBIDA team pulls data from the DND sourceAIS feed URL, storing the data in raw format (CSV). The data is made available to EGIS eGIS team trough Microsoft Azure Storage Explorer, however to consume the real time data using the .Net script the Data Lake must have hierarchy call hierarchical infrastructure in the preview enabled. The EBIDA eBIDA team is analyzing the impact of this change in the current Data Lake infrastructure.
The 10 TB historical data is stored in the data lake with the format of the .nmea.gz files. So extracting the files and then load the raw json files to GeoEvent Server Big Data Store is the ongoing procedure. Note, as of today, ESRI has not supported ADLS Gen 2 Data Lake yet.
Another challenge for the integration is to set a data layout that fits the EGIS eGIS needs. A prepossessing may be required. However, EBIDA eBIDA project is on at the proof-of-concept stage and the plans are to go live on the beginning of next fiscal year, April 1st. The final state aimed by both teams is to feed from the data lake and publish the feature layers on EGIS eGIS portal.
Current Services Available
Currently the AIS feed is publicly available on Sandbox in the following link: https://sbxweb.tcgis.ca/portal/home/webmap/viewer.html?webmap=9b657e410e654331b0bb084ee9567b50
During the sprint 29 this service will be moved to Pilot environment where it will be shared with entire organization.On Pilot PROD:
https://egis-sige.tcgis.ca/portal/home/webmap/viewer.html?webmap=53e80e0743c142d6ae95aa5e54c3030f