This page will record any unique incidents, issues and/or fixes required when the MDM subsystem fails.
Important :
when testing the Acceptance region, the MDM will be unavailable in Production and vice/versa
the MDM does not work well with multiple users doing simultaneous distributions
the MDM mailbox password is set to “never expire”, but auto updates every 80 days. This is controlled by the MDM - MAILBOX PASSWORD MAX AGE value on the General properties table
Indications that the Process is down
After 15-20 minutes, the process start date fails to populate and attempt numbers stay at zero
2. Failure emails are sent from
TC.DoNotReplyNAC-NepasrepondreCNA.TC@tc.gc.ca to TC.AAIR-RAINA.TC@tc.gc.ca
The failure reason pertains to , SSL Certificates, AutoDiscovery or MDM InboxPassword
Details of Causes
“Auto Discovery” and/or “SSL Certificate failure”
see PBI# 137134 fixed by SMGS Ticket #IM360871 (October 2021)
The MDM will not send out emails and the generic AAIR inbox begins receiving error emails (shown above)
This is a known recurrent error. Shared Services (SSC) needs to make an update on their end. When this occurs, Log a high-priority Orion ticket stating that the CAWIS-MDM is a “mission critical” application. Its also good practice to make mention of the error and ticket number on the IT Information Portal channel on Teams to help ensure the ticket gets attention in a timely manner
An Auto-discovery (CAWIS looking for address of exchange server) incident also occurred in Dec 2021, but somehow self corrected without notifying SSC or Service Desk - reason unknown
2. PASSWORD or Authentication errors
See Pbi# 202037 fixed by Orion Ticket #INC33024 (September 2022),
ORION Ticket #INC125250 (January 2023):
Having established that the issue is not related to SSL Certificates or AutoDiscovery
From the checklist at :
https://tcmarin.atlassian.net/wiki/spaces/CA/pages/1879802073
Check the log at \\ncrws548\scripts\CivAv\CAWIS-MDM\
Note: The log files are automatically rotated when they hit a certain size. You may need to look in the “logs” folder if the problem happened a while ago.Log an ORION Service desk ticket to the attention of SSC, to request that the password for the AAIR inbox be reset.
Information to include:
1. MDM is mission critical - the AAIR inbox is used to send Aviation safety information.
2. The error message in the email.
3. The error messages in the log.
Resetting the AAIR inbox is an interactive process.
An agent/LAN Admin will initiate process. In January 2023, this was completed via a Teams screen sharing session in which the agent initiated the process but provided control of the screen to the maintenance team member who performed the password change.
Once the password is changed the following changes need to be done in the CAWIS database. They should be done immediately after the database change.
After the database change is complete check the CAWIS-MDM log (step 1).
Check that the email account is not locked: Open a command prompt and type “net user DNRENAC /domain“. In the response, you should get something like below:
If the line “Account active” says “Locked” or “No”, contact the Service Desk for them to reactivate the account.
If there’s no logs being generated => the task isn’t running at all :
Contact the group responsible for the server for diagnostic or server restart
Repetitive messages => One of the threads is stuck:
You can wait for the deadlock-prevention timer (2hr) to kick in.
OR contact the group responsible for the server to kill the task.
If there’s a message like “Application is disabled due to database configuration.“
In the TR47 table, ensure the configuration called “MDM - APPLICATION ENABLE” is set to “true”
If there’s a message like “exchange server is unreachable.“ => The email account might be locked.
Open a command prompt and type “net user DNRENAC /domain“. In the response, you should get something like below:
If the line “Account active” says “Locked” or “No”, contact the Service Desk for them to reactivate the account.
The MDM inbox/account is called DNRENAC. Currently, it’s password is auto updated every 80 days, by the MDM process on the script server at \ncrws548\scripts\CivAv\CAWIS-MDM Again, log files in the “Log” directory will tell you exactly when the process began failing
The “password refresh process” updates password, the encryption key and the MDM - MAILBOX PASSWORD LAST CHECK DATE.
However, TC IM/IT can also allocate a new mailbox password if its believed that the password has been altered outside the MDM process,
Go to CAWIS MAIN MENU - Code table Admin - General Properties
Note : most fields (with the exception of “password” ) will be UPPERCASED when saved on the TR47 interface page. This would nullify any benefits of backing up up the old encrypted password at this point.
Null out the encryption value at MDM - MAILBOX PASSWORD SECRET. This needs to be done on the backend.
Also, store the previous password and encryption value at MDM BACKUP - LAST KNOWN PASSWORD ,MDM BACKUP - LAST SECRET ENCRYPTION
First, backup the existing password/encryption info
UPDATE TR47_CAWIS_GENERAL_PROPS A SET A.PROPERTY_VALUE =
(SELECT PROPERTY_VALUE FROM TR47_CAWIS_GENERAL_PROPS B
WHERE B.PROPERTY_NAME = 'MDM - MAILBOX PASSWORD' )
WHERE PROPERTY_NAME = 'MDM BACKUP - LAST KNOWN PASSWORD';
UPDATE TR47_CAWIS_GENERAL_PROPS SET PROPERTY_VALUE =
(SELECT PROPERTY_VALUE FROM TR47_CAWIS_GENERAL_PROPS B
WHERE B.PROPERTY_NAME = 'MDM - MAILBOX PASSWORD SECRET' )
WHERE PROPERTY_NAME = 'MDM BACKUP - LAST SECRET ENCRYPTION';
then, null out the encryption key
UPDATE TR47_CAWIS_GENERAL_PROPS SET PROPERTY_NAME = NULL
WHERE PROPERTY_NAME = 'MDM - MAILBOX PASSWORD SECRET';
Then, Enter the unencrypted password into the MDM - MAILBOX PASSWORD
Then, Change the value MDM - MAILBOX PASSWORD LAST CHECK DATE to SYSDATE + 2
Any backlog of distribution jobs may take 15-20 minutes to begin executing afterward
The MDM inbox password will auto-update normally by way of the MDM process a day or so afterward. A new encrypted password and encryption key will then appear on the General properties table.
3. Hidden ASCII characters in an Airworthiness Directive (AD) subject
Very often the CAW Inspectors will simply copy and paste the subject from the AD document into the data entry page when processing a new AD.
A symptom of this would be that the MDM refuses to process a “specific” AD, but allows processing of others.
This issue was addressed in TFS Bug 13919
However, if it should reoccur for any reason, simply copy the English and French AD subjects into notepad, remove any strange looking characters, then paste those subject lines back into the page and resave .
4. MDM Email box disabled / locked
Again this can only be fixed via an Orion ticket. The MDM password is in the the General properties table under the CAWIS code maintenance menu , if required. The Inbox being locked is likely a side effect of an invalid password on TR47
(see SMGS TICKETS IM325231, IM307333 Solution - Resolution Code: Access Control U Had to reset and unlock the mailbox account for user ID DNRENAC)
DNRENAC = (National Aircraft Certification – Distribution TC.CAWIS-SWIMN-DISTRIB-PROD.TC@TC.GC.CA)
5. Process locked
Distribution fails to Launch.
MDM runs 2 separate threads to send and process. The Sending thread exits without notifying the processing thread. The processing thread runs until hitting a “deadlock” counter (240 runs, approx 30 seconds = 2hrs) This is called a deadlock-prevention timer
Afterward the processing thread kills the process and begins the distribution fresh
So this type of incident is self correcting. However if the process does not begin after 2hrs, then we need to investigate other reasons.
Add Comment