There are different ways to quickly get Pilot back when fatal error happens on Pilot, without having clients wait long time for the errors to be corrected. Setting up disaster recovery system for Pilot is one way to do it. The key is to use the steps in https://www.esri.com/arcgis-blog/products/arcgis-enterprise/administration/migrate-to-a-new-machine-in-arcgis-enterprise-two/ to deploy a new system same as Pilot. Then add the new system as an endpoint in Azure Traffic Manager.
Below is the procedure to use a sandbox resource group e.g., TC-Sandbox-BXUeGIS-RGfor the DR system.
Step 1: create DR VMs. This is done on your local desktop with Powershell window.
Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope Process
cd C:\Work\EGIS\ARM\templates
.\deploy.ps1 -Subscription 0921cb70-a968-4860-a0dc-7f77ba3a35c4
-ResourceGroupName TC-Sandbox-BXUeGIS-RG
-TemplateFilePath .\tccloud-prod-template.json `
-ParametersFilePath .\parameters.json
Step 2: prepare for ESRI enterprise GIS software installation
Remotely logon to Web0 VM, using \egisadmin and password in KeyVault for sandbox.
change file D:\DSC\PowerShell DSC\addUsersToLocalGroups.ps1, with the following:
[ValidateLength(3, 60)]
[string] $roles = "web,por,gis,ds,bds,evt",
this should add evt and bds to the $roles parameter.
Run the command in Powershell:
cd ‘D:\DSC\PowerShell DSC’
.\addUsersToLocalGroups.ps1 -egis_env bxu
Modify D:\DSC\PowerShell DSC\STG-BaseDeployment-10.7.1-same-as-PLT.json, with the following changes:
a. Replace STG with BXU
b. Replace stg with bxu
c. Insert passwords and username
d. change DNS name to egis-sige.tcgis.ca, this will force egis-sige.tcgis.ca is recognized by the new DR system
e. Change the two paths of authorization file to D:\\Authorization_Files\\……, as the authorization files are not on the shared drive.
Change C:\Windows\System32\drivers\etc\host on each VM
[private ip address of web0] egis-sige.tcgis.ca
Step 3: Install ESRI Enterprise GIS Software
Logout Web0, and login back with cloud\arcgisservice account.
Run the command in Powershell:
Configure-ArcGIS
-ConfigurationParametersFile 'D:\DSC\PowerShell DSC\STG-BaseDeployment-10.7.1-same-as-PLT.json'
-Mode InstallLicenseConfigure -DebugSwitch
Make sure the federated server and the hosting server are valid in Portal for ArcGIS.
Restore backup from PLT to BXU:
Copy egis_backup_full.properties from PLT por0 to BXU por0 C:\scripts\Site Backup\portal.
Change SHARED_LOCATION= \\EGIS-BXU-POR0\d
Run command from por0 Powershell:
cd 'D:\DSC\PowerShell DSC'
PS C:\scripts\Site Backup\portal> ./RestoreSite –BackupPropertiesFile egis_backup_full.properties
Step 4: Install and configure GeoEvent Server
Work on the GeoEvent VMs.
Change D:\DSC\PowerShell DSC\STG_GISServer-GeoEvent_DataStore-10.7.1.json with the following:
a. Replace STG with BXU
b. Replace stg with bxu
c. Insert passwords and username
On EVT0 machine, run command from Powershell:
cd 'D:\DSC\PowerShell DSC'
Configure-ArcGIS
-ConfigurationParametersFile 'STG_GISServer-GeoEvent_DataStore-10.7.1.json'
-Mode InstallLicenseConfigure -DebugSwitch
Need to manually configure and apply ArcGIS Server/GeoEvent Server licenses, then create a new ArcGIS Server site as it has been failed. The create new ArcGIS Server site page is automatically loaded when navigating to https://localhost:6443/arcgis.
Install microsoft SQL Server 2012 Native Client from egis-bxu-ds0\installs\sqlncli.msi
Install ArcGIS Web Adaptor on web0 VM for realtime ArcGIS Server, from \\egis-bxu-ds0\installs
Copy C:\Work folder from PLT to BXU evt0.
TO DO: automate the following process with ArcGIS Server backup/restore:
Add task scheduler to run ais pull exe from C:\Work\GIS_Projects\DNDRealtimeDataFeeds\Release every 5 minutes.
Copy the files from the directories of old arcgis server to the new arcgis server: https://community.esri.com/thread/70787
You will have to copy all the resources from Pilot GIS0 to bxu GIS0. The best is to check all folders and subfolders in arcgisserver folder.
\arcgisserver\config-store\services
\arcgisserver\directories\arcgiscache
\arcgisserver\directories\arcgisoutput
\arcgisserver\directories\arcgissystem\arcgisinput
Copy all the .sd files from PILOT EVT under C:\Users\arcgisservice\AppData\Local\ESRI\Desktop10.6\Staging to BXU evt0 C:\Work\GIS_Projects\Staging.
Change ArcGIS GeoEvent Server and ArcGIS GeoEvent Gateway services to Automatic (Delay) and restart ArcGIS Server
Republish the services from ArcGIS Server, using “publish service“ button.
Import geoevent.config file from PLT to BXU evt0.
Note:
If after restore the contents from PLT procedure GIS server does not restart successfully, restart all the VMs helped to resolve the issues.
All the VMs are currently shutdown.
How to spin up DR system?
Restart all the VMs in TC-Sandbox-BXUeGIS-RG with the following order:
Restart egis-bxu-por0
Restart egis-bxu-gis0
Restart egis-bxu-ds0
Restart egis-bxu-evt0
Restart egis-bxu-web0
Restore the latest backup, using the following command from por0 Powershell:
cd 'D:\DSC\PowerShell DSC'
PS C:\scripts\Site Backup\portal> ./RestoreSite –BackupPropertiesFile egis_backup_full.properties
7. Enable PLT-DR from Traffic Manager egis-plt and manually disable plt if needed in order to fix the errors on plt while having DR of PLT running.