CAWIS MDM - SWIMN MDC

System Profile

Technology Assessment

Environment Access Information

System Overview

<--->

Good To Know

A single emailing can contain up to 20,000+ emails.

MDM does the following steps to manage Mailbox size as the Mailbox cannot exceed its 2 gig size limit otherwise replies (bounce-back messages) will be lost:

• All processed messages are move to the MDMBounced folder and subsequently deleted at the end of the run

• Using BCC, MDM sends one message to 500 recipients every 60 seconds.  The MDM process continues until all emails have been sent.  Concurrently MDM is checking for and processing any bounce back responses as follows.  

  • The responses in the inbox are loaded into a list.

  • Each response in that list is processed and then moved from the Inbox to the MDMBounced folder.

  • When the end of the list is reached all messages in the MDMBounced folder are deleted, freeing up the space.

  • This entire process takes less than 15 seconds to complete.

  • If the MDM process is still running (as it continues to send out messages every 60 seconds until all message are sent) then this cleanup process restarts every 60 seconds.

How-To and Fixes

See also. for detailed MDM repair information and history: MDM OUTAGES - BSD - Civil Aviation - Confluence (atlassian.net)

e-mail notifications regarding expiration of the AAIR inbox password

See also

CAWIS DevOps task: 235611 CAWIS Airworthiness Directive (AD) Distribution
Bug 235611: CAWIS Airworthiness Directive (AD) Distribution - Boards (azure.com)

CAWIS DevOps task 240218 CAWIS AD Distribution MDM not functional
Bug 240218: CAWIS AD Distribution MDM not functional - Boards (azure.com)

For the purpose of earlier awareness and easier

See also “General Troubleshooting on this page

• All members of the maintenance team, including the product owner, have been added to an automated process that will send an email when the AAIR /RAINA (TC) password will expire within 10 days.
  
  
  
  Note: we had hoped to be able to provide the team distribution list (Team Avro Arrows). Unfortunately, the initial test email that was not distributed. Our contact, Robert Lang, also tested using the distribution list for the team he is a member list. That did not work either. He advised that the email is possibly being flagged as a “mass email” (spam) when the distribution list is used.
  
  For that reason, the contact information is listed individually.
  
  The current recipients are listed below.
  
  Team lead:
  alain.asgill@tc.gc.ca   ASGILLA
  
  Product owner:

• kelly,haynes@tc.gc.ca HAYNESA
  
  
  Developers:

• phuong.nguyen@tc.gc.ca  NGUYEPH

• riadh.said@tc.gc.ca     SAIDR  

• christopher.collins@tc.gc.ca  COLCHRI

• haoyuan.wang@tc.gc.ca   WANGHA  

• rajesh.shah@tc.gc.ca    SHAHR  

• hiren.amin@tc.gc.ca     AMINH  

• siamak.shirzad@tc.gc.ca SHIRZAS

• sufyan.maghur@tc.gc.ca  MAGHUSU

• noreen.dertinger@tc.gc.ca  DERTINN
  
  
  The process will send an email once everyday until the password is reset.
  
  Example:
  
  

  

AAIR inbox inbox access:

• alain.asgill@tc.gc.ca   ASGILLA

• phuong.nguyen@tc.gc.ca  NGUYEPH

• riadh.said@tc.gc.ca     SAIDR  

• christopher.collins@tc.gc.ca  COLCHRI

• haoyuan.wang@tc.gc.ca   WANGHA  

• rajesh.shah@tc.gc.ca    SHAHR  

• hiren.amin@tc.gc.ca     AMINH  

• siamak.shirzad@tc.gc.ca SHIRZAS

• sufyan.maghur@tc.gc.ca  MAGHUSU

• noreen.dertinger@tc.gc.ca  DERTINN
   have been granted full permission and send as to TC.AAIR-RAINA.TC@tc.gc.ca
  
  

• These permissions take min 24 hrs to apply at which time mailbox

  can be added using instructions located in Knowledge base

  article https://orion.tc.gc.ca/dwp/app/#/knowledge/KBA00002102/rkm
  
  The following users already had access to the inbox previously:

  

   

To fix the AAIR Password issue

1. Log an Orion ticket using the Orion “Report an issue” category:

In the description:

1. Request SSC to fix the SSC password issue.

2. Indicate that the AAIR inbox, which uses CAWIS MDM, is a mission critical aviation safety application.

3. Include the error message from the relevant AAIR inbox email

4. Include the error information from the log (see step 1 in “General troubleshooting”.)
   
   Notes:
   1) as non SSC staff we will not be able to log a ticket for SSC directly.
   
   2) per an email discussion with service desk, only BMC Administrators are able to open tickets that are prefixed with INC. We don’t have access to those tickets that begin with “INC”
   
   
   Reset of the password
   

• Resetting the AAIR inbox is an interactive process.

• An agent/LAN Admin will initiate process. In January 2023, this was completed via a Teams screen sharing session in which the agent initiated the process but provided control of the screen to the maintenance team member who performed the password change.

  Once the password changes need to be done in the CAWIS database. They should be done immediately after the database change. The steps are provide in the next section, “Updating TOAD”.

  After the database change is complete check the CAWIS-MDM log (step 1).

  Check that the email account is not locked: Open a command prompt and type “net user DNRENAC /domain“. In the response, you should get something like below:

  

  If the line “Account active” says “Locked” or “No”, contact the Service Desk for them to reactivate the account.

Updating the database

immediately following the successful reset of the AAIR inbox password, the CIVAV developer should make the following updates in TOAD). This is time sensitive.

Here’s the steps to fix the issue:

1. In the table TR47_CAWIS_GENERAL_PROPS:
   
   First, backup the existing password/encryption info

   UPDATE TR47_CAWIS_GENERAL_PROPS A SET A.PROPERTY_VALUE =
   (SELECT PROPERTY_VALUE FROM TR47_CAWIS_GENERAL_PROPS B
   WHERE B.PROPERTY_NAME = 'MDM - MAILBOX PASSWORD' )
   WHERE PROPERTY_NAME = 'MDM BACKUP - LAST KNOWN PASSWORD';

   UPDATE TR47_CAWIS_GENERAL_PROPS SET PROPERTY_VALUE =
   (SELECT PROPERTY_VALUE FROM TR47_CAWIS_GENERAL_PROPS B
   WHERE B.PROPERTY_NAME = 'MDM - MAILBOX PASSWORD SECRET' )
   WHERE PROPERTY_NAME = 'MDM BACKUP - LAST SECRET ENCRYPTION';

   then, null out the encryption key

   UPDATE TR47_CAWIS_GENERAL_PROPS SET PROPERTY_NAME = NULL
   WHERE PROPERTY_NAME = 'MDM - MAILBOX PASSWORD SECRET';
   
   Note : most fields (with the exception of “password” ) will be UPPERCASED when saved on the TR47 interface page in CAWIS. This would nullify any benefits of backing up up the old encrypted password at this point.

Clearing the value of “MDM - MAILBOX PASSWORD SECRET” is to tell MDM that the password is unencrypted. (Do not remove the row).

a. Set the value of “MDM - MAILBOX PASSWORD” to the new, unencrypted password.

b. Set the value of “MDM - MAILBOX PASSWORD LAST CHECK DATE to SYSDATE + 2”. This will ensure that MDM does the password change two days after the reset (you have to wait a day to do another password change)

• Any backlog of distribution jobs may take 15-20 minutes to begin executing afterward

• The MDM inbox password will auto-update normally by way of the MDM process a day or so afterward. A new encrypted password and encryption key will then appear on the General properties table.

• TC IM/IT can also allocate a new mailbox password if its believed that the password has been altered outside the MDM process,General Troubleshooting

See also the discussion in CAWIS DevOps task: 235611 CAWIS Airworthiness Directive (AD) Distribution
Bug 235611: CAWIS Airworthiness Directive (AD) Distribution - Boards (azure.com) on Jan 26 and 27, 2023.

Contact list for the contacts mentioned below:

1. Check the log at \\ncrws548\scripts\CivAv\CAWIS-MDM\.
   Note: The log files are automatically rotated when they hit a certain size. You may need to look in the “logs” folder if the problem happened a while ago.
   If there’s no logs being generated => the task isn’t running :

   1. Contact the group responsible for the server for diagnostic or server restart.
      Current update, Jan 2023 regarding who to contact:

      For support on the web application contact the web group.
      For a production issue with the web application, open an ORION ticket to the attention of the web server group so that they can work on and document the issue.
      
      

   2. If the process seems to be looping the same messages => One of the thread in stuck:

      1. You can wait for the deadlock-prevention timer (2hr) to kick in.

      2. OR contact the group responsible for the server to kill the task.
         
         Per Marcin on Jan 27, 2023: “the deadlock issues we had we back in 2020 and for those you need to contact Oracle dba group.”
         

   3. If there’s a message like “Application is disabled due to database configuration.“ => The application is disabled

      1. In the TR47 table in the CAWIS database, set the configuration called “MDM - APPLICATION ENABLE” to “true”.
         

   4. If there’s a message like “exchange server is unreachable.“ => The email account might be locked.

      1. Open a command prompt and type “net user DNRENAC /domain“. In the response, you should get something like below:

         

         If the line “Account active” says “Locked” or “No”, contact the Service Desk for them to reactivate the account. To-date, most recently in Jan 2023, Robert Lang and Muhammad Duhabei assisted with this.

• The MDM inbox/account is called DNRENAC. Currently, it’s password is auto updated every 80 days, by the MDM process on the script server at \\ncrws548\scripts\CivAv\CAWIS-MDM Again, log files in the “Log” directory will tell you exactly when the process began failing