In Transport Canada, we have seen more and more cases that backend services need to communication between services to fulfill business requirements in SOA or microservice architecture. One of challenges of these type of communication is to ensure data consistency while maintain services availability.

In order to make sure the data is eventually consistent between microservices even cross-platforms such as power apps and dotnet web app, one of most common approach is sending data asynchronously from one service to another service via events using Event Driven Architecture (EDA).

In this paper, we will show how to leverage EDA patten to implement transactional outbox pattern to provide data eventually consistent among the communication in .NET web App.

Let’s see how implementation works step by step:

Dotnet CAP Implementation Example:

Dependencies

We are using CAP library in this case. CAP is a library based on .net Standard, which is a solution to deal with distributed transactions, also has the function of EventBus, it is lightweight, easy to user, and efficient.

First, our TC implementation of CAP library DotNetCore.CAP.AzureServiceBus with Azure Service Bus supporting can be found under TC DevOps Nuget Repo. The current version is 2022.3.14.1 which supports DotNetCore.CAP.XXX with version 6.0.1.

CAP.AzureServiceBus is depending on CAP library which contains outbox and inbox pattern implementation inside.

Second, we need to DotNetCore.CAP.PostgreSql into dependencies. The current version is 6.0.1

After library referenced, we will show what we should do to configure CAP library.

Configuration

The services provided through CAP library should be registered in IServiceCollection in startup.cs through the IoC container for .NET core project. Basically, transport and storage configuration are required to use CAP.

User will first need to configure CAP related settings in appsettings file as follows. We are using PostgresSQL for local event storage and Azure Service Bus for transport.

"AzureServiceBus": {
    "ConnectionString": "#{ServiceBusConnectionString}#",
    "TopicName": "#{DefaultTopicName}#",
    "Subscription": "#{DefaultSubscriptionName}#",
    "SubscriptionToTopics": {
      "subscription1": "#{Topics1}#",
      "subscription2”: "#{Topics2}#"
    },
    "FailedRetryCount": 10,
    "FailedRetryInterval": 60
  },
  "ConnectionStrings": {
    "PostgreSql": "#{EDACAPConnectionStringPostgreSql}#"
  }

Please note, all connectionString and Topics are based on your instance of Service Bus implementation which can be located under your instance of Service Bus/Shared Access Policies.

After adding those settings into appsettings, we need to use those values in ConfigureServices in startup.cs during services configuration as following:

#region CAP implementation
  string connectString = Configuration.GetValue<string>("ConnectionStrings:PostgreSql");
  string? schema = Assembly.GetEntryAssembly()?.GetName().Name?.ToLower();
            services.AddCap(x =>
              {
                  x.UseAzureServiceBus(opt =>
                  {
                      opt.ConnectionString = Configuration.GetValue<string>("AzureServiceBus:ConnectionString");
                      opt.TopicPath = Configuration.GetValue<string>("AzureServiceBus:TopicName");
                      opt.SubscriptionToTopics = Configuration.GetSection("AzureServiceBus:SubscriptionToTopics")
                      .GetChildren().ToDictionary(x => x.Key, x => x.Value);
                  });
                  x.DefaultGroupName = Configuration.GetValue<string>("AzureServiceBus:Subscription");
                  x.FailedRetryCount = Configuration.GetValue<int>("AzureServiceBus:FailedRetryCount");
                  x.FailedRetryInterval = Configuration.GetValue<int>("AzureServiceBus:FailedRetryInterval");
                  x.UsePostgreSql(x => { x.ConnectionString = connectString; x.Schema = schema; });
              });

#endregion

Note: if you have an error in the line 3, please add using System.Reflection;

Publishing and Subscribing Messages

After we successfully configured settings and loaded related CAP services into IServicesCollection through ConfigureServices call, we can start to publish or subscribe messages.

Publishing

In order to publish message, we should first inject ICapPublisher service which has been added into IServicesCollection inside controllers or services where the places we want to publish messages. We use controller as example as below, but service can also work well with ICapPublisher.

After ICapPublisher was injected into controller, we can then simply call Publish or PublishAsync method to publish the event.

await _capPublisher.PublishAsync("Topics1", event);

Subscribing

Events published onto Azure Service Bus can be received by dotnet core app through CAP library in controller with the CapSubscribe attribute. Only two annotation need to be used to make subscribing working.

Message Storage

Selected CAP library supports OutBox and InBox Pattern on top of Event Driven out of box. The idea behind this pattern ensures that the message will be not lost if we lose the network, for example.

When Application starts, CAP library will automatically generate two tables named Published and Received under application schemas as default based on storage type we configured in startsup.

Although we are using PostgreSql in our example here, CAP library supports multiple database systems to store messages, such as SQL Server, MySQL, MongoDB, etc.

Message Transport

We are using Azure Service Bus as messaging transport provide in our example here. Selected CAP library also supports many third-party messaging technologies such as RabbitMQ, Kafka, Amazon SQS, etc. In addition, it also supports some custom configurations for transport methods such as default group. In addition, retry mechanism parameters can be configured for errors that may occur during transport such as FailedRetryInterval and FailedRetryCount we configured in Appsettings file.

Links

· Project Repository: EDASampleApi

· Cross platform EDA white paper: Power App integrate with .NET through EDA

· CAP library Page: https://cap.dotnetcore.xyz

· CAP library GitHub repository: GitHub - dotnetcore/CAP: Distributed transaction solution in micro-service base on eventually consistency, also an eventbus with Outbox pattern

Marine

EDA service with CAP library