Working with Azure Storage

Overview

Looking to a future where our maintenance scripts are run from the cloud, it will be required to work with Azure Storage. A few possible use cases:

Assuming no pipe from CLOUD back to TC, a client with an app in TC exports their data on a regular basis and loads that data into Azure Storage.
A client with an app running in Azure exports their data directly to azure storage.
The TC data lake is a form of data storage
A client would like has none-spatial static documents that they would like to attach to each spatial feature (eg. ACAP projects)

Reading from a BLOB

If the owner of the storage account provides us with a Shared Access Signature (SAS), then we can reference the blob as a URL like any other online resource.

Writing to a BLOB

Using the azure-storage module for python we have full control over reading, writing, and deleting content from all forms of Azure Storage. GitHub - Azure-Samples/azure-sdk-for-python-storage-blob-upload-download: How to upload and download blobs from Azure Blob Storage with Python

import datetime
from azure.storage.blob import BlockBlobService, PublicAccess
...
# Fill folder with PDFs a la current ACAP discussion
print("\nTry loading a folder of ACAP PDFs")
acap_path = "C:\\Users\\someone\\Downloads\\ACAP_Projects"
tooOld = datetime.datetime.utcnow() - datetime.timedelta(minutes=10)
for files in os.listdir(acap_path):
	print("\tuploading " + files)
	try:
		block_blob_service.create_blob_from_path('acap-demo/pdf', 
                                                          blob_name=files,file_path=os.path.join(acap_path,files), 
                                                          if_unmodified_since=tooOld)
	except Exception as e:
		print("\t\t still fresh? " + files)

Managing credentials

When run from an Azure Function context, a service principal can define access to resources. With that we can also read key vaults to find secrets (keys, SAS) to get access to storage held by our clients. https://docs.microsoft.com/en-ca/python/api/overview/azure/key-vault?toc=%2Fpython%2Fazure%2FTOC.json&view=azure-python

Development

You can emulate Azure Storage on your workstation. This is allows you to work offline, and to avoid repeated transfer costs while figuring things out. https://docs.microsoft.com/en-us/azure/storage/common/storage-use-emulator