Implementing First Registry Transaction as a Saga

Introduction

Saga is a design pattern for ensuring data consistency when implementing update transactions in microservices architectures. We are going to take a concrete scenario of creating vessel registry record, aka “First Registry” and see how saga pattern can be applied to it.

The objective is to get to understand the moving parts, and think of implementation, challenges and implications of using sagas. It will also make it easier to appreciate the utility of third party frameworks such as MassTransit to simplify saga implementation.

Current Architecture and First Registry Business Scenario

Post-MVP Vessel E-Registry architecture rests on the following microservices:

• Vessel Registry Service

• Vessel Detail Service

• Client (aka Contact) Service

• Work Management Service

• Document Management Service

• List of Value Service

Additional services will be added in the future. For example, a service for managing vessel registry history.

The objective of First Registry scenario is to take all information from a completed service request (work item), create a new vessel registry record and mark work item as completed. For simplicity, we are not going to discuss issuance of first registry certificate and updates to history information here.

Since vessel registry data is spread across 3 services - Client Service, Vessel Detail Service and Vessel Registry Service, we will need to create 3 records in their respective databases, and update 1 record in Work Management Service database to change the work item status. We will make an assumption that services would have to be updated in the following order:

1. Insert new record to client database through client service;

2. Insert new record to vessel detail database through vessel detail service;

3. Insert new record to vessel registry service through vessel registry service using new client Id and vessel detail Id for connecting the first two records to the vessel registry record;

4. Update status of the work item to “completed/done”.

Schemas of the services' databases evolve over time, so this assumption may or may not hold, however it should not affect our analysis of the problem as regardless of the order of updates the technical challenges we face will still remain the same.

Saga Essentials

When we have a monolith application and insert records in Contacts, Vessel Details and Vessel Registration tables and update Work Items table in its only database, it is simple to make sure that either all 4 tables are updated, or none at all. All we need is to use database transactions. Back to microservices reality where we have all 4 tables living in 4 separate databases and are behind 4 autonomous microservices, we still have to deliver the “all or nothing” behavior, which is so easily achieved in a monolith.

Sagas deliver a compromise between advantages of using microservices architecture and “the price” of infeasibility of building transactional behavior in a distributed system.

The compromise is that our transaction consistency will only be “eventual”. This assumption is what lets us continue the conversation about sagas. If “eventual consistency” is not good enough then Sagas is the wrong design pattern. So how can we achieve this eventual consistency?

In simple terms, the services exchange messages about their state. Sagas prescribe the order of updates, or when a record fails to get added in one service, an already created record is “rolled back” in another service. For example, if we have created records in Contact and Vessel Detail tables, but failed to create a record in Vessel Registration table for some reason, we must respond to the failure event and rollback the records from the first two tables, otherwise we would have left our tables in an inconsistent state.

Commonly we talk about sagas as having 2 flavors: choreography and orchestration. See the microservices.io for their detailed discussion.

Let’s list pros and cons of each:

1. Choreography Pros:

   1. Simple model: each service subscribes to events emitted by other services.

   2. Each service is “smarter” as it is aware of events from surrounding services, even though it does not care about the services themselves.

2. Choreography Cons:

   1. Saga logic (sequence of happy path calls and rollbacks) is spread across participating services making it hard to understand this logic: it is not stored in one place.

   2. As the number of saga participants grows, each existing service is obliged to subscribe to new participants' events, which increases “virtual coupling” between them via message formats. This conflicts with single responsibility principle in architecture. For example, in the First Registry scenario, the Work Management Service would need to subscribe to events from Vessel Registry service to participate in the “choreography”, which is a violation this principal.

   3. It is challenging to pass parameters between the services, specifically in a scenario such as First Registry, when we need one set of parameters to insert Contact record, another set of parameters to insert Vessel Detail record, and a third one - to insert the registry, while the saga participants are invoked sequentially.

3. Orchestration Pros:

   1. It makes it simpler to understand the intended course of happy path and rollback steps, and therefore to develop and maintain such sagas.

   2. It is easier to pass parameters, as they always go from the orchestrator down to the saga participants.

4. Orchestration Cons:

   1. There is a need of a new “orchestrator microservice”, which adds complexity to the system.

   2. The orchestrator risks growing into a “distributed monolith” over time with all the coordination logic that it contains, while saga participants remain “dumb” as they do little more than wrapping around data repositories.

   3. An orchestrator can be seen as a single point of failure in a saga transaction.

We are going to pick the orchestration-based saga implementation going forward, as it seems that concern 4a is not a major one, and concern 4b could be addressed by limiting saga orchestrator strictly to dispatcher type logic, and delegate all business rules fulfilment to saga participants. This would keep it lightweight and manageable even when we have many different sagas to orchestrate. Lastly, to address concern 4c we will look into architecture measures to ensure that saga context would not be lost in the event of the orchestrator’s crash.

Saga Resiliency

Theoretical possibility of a crash of an orchestrator or any of saga participants, and the need to recover from one is one factor that makes saga implementations complicated. Let’s picture the orchestrator-based design supporting First Registry transaction:

Open

There are few things we should emphasize about what’s shown on this diagram:

1. All operations between microservices and the orchestrator (implemented inside Vessel Registry Controller microservice) are sent via a message broker, or Azure Service Bus (ASB) to be more specific in our case. Events are published to topics, commands are listened on queues. Why use asynchronous messaging? Primary reason is its reliability. ASB topics and queues guarantee that each message will be delivered “at least once” to its consumer. HTTP calls do not provide such guarantee. Our saga progresses asynchronously as commands are sent over queues and get executed by services, which then publish events to topics for the orchestrator to consume them and decide on which next commands to issue to which service.

2. Each service should be idempotent (the same command should produce the same result at any time). Let us take any microservice, for example Contact Service, and imagine that it has received a message to add a contact record, and has successfully completed the insertion. Now, because message queues may send the same message more than once given their “at least once” guarantee, the next time the service receives the same “Add Contact” message it will execute the insertion again and create a duplicate record. This is both a data corruption situation and a violation of idempotence principle. One common way to counter this is implement an Idempotent Consumer pattern: each time a message arrives, it is logged to a database table, so that the next time when a duplicate message arrives, the service could compare it to one in the table and discard it, thereby achieving message deduplication and idempotence. The table storing messages will however need to be “groomed” regularly as the number of messages in it grows over time. This makes us consider using a background Deduplication Cleanup Job.

3. Let’s consider another scenario: Contact Service has received a message to add a contact, has succeeded inserting a new record, but crashed just before dispatching a ”Contact Added” event. While we hope this could happen rarely in practice, it still can, and is a pretty bad situation: our orchestrator will never “hear back” from the Contact Service about an outcome of this specific step even after the Contact Service gets restarted. In general with modern technology’s built-in high availability we can assume that our services will be able to recover from crashes within minutes, and perhaps even seconds. So the real issue here has to do with losing the “Contact Added” message we were about to send back to the orchestrator in the process. Here comes Transactional Outbox design pattern to the rescue. We create another table in each of our services' databases and name it MESSAGE_OUTBOX - or similarly. When we update or insert our business data record, we insert a corresponding message to the MESSAGE_OUTBOX table as a part of the same transaction. Then we have another background service job, let’s name it Message Relay Job, which reads this table and sends messages to the topic. This way even if we crash, we will still be able to send the right message once we are back up.

These three considerations apply to any of the saga participants despite the type of saga. They also apply to the orchestrator just as well as they do to other microservices. Now we can state that we have made our saga resilient, but we had to introduce extra complexity including 2 new background jobs. Not only that, but also the time that it takes to run through our saga has grown significantly compared to the good old monolith transaction.

User Experience Implications

Let us now assess the impact of saga-based transactions on the user experience. If a message relay job in each service checks for messages every 1/2 a second, then the total time it takes for all 4 services to run their tasks may exceed 1/2 x 4 x 2 = 4 seconds + the ASB time. Clearly, this puts our First Registry scenario in the class of long running API calls as far as our User Interface is concerned. To make sure the user experience is not degraded we have to resort to asynchronous request-reply pattern.

Here it is shown as it applies to our architecture:

Open

The implications of such API behavior to the UI are significant: instead of letting users wait for 1-2 seconds for a transaction to complete, we need to rethink the UI experience such that responses are returned quickly and users are visually notified once the tasks they submit get completed. User experience inspiration may be taken from the Azure Portal, where this style of UI behavior is standard.

A Closer Look at a Microservice

We saw how each microservice needs to have additional plumbing to participate in sagas. Schematically we can present each microservice as shown below:

Business logic is at its core, and adapters to external world are at its edges: database, ASB queues and topics, Web API. Let us take Vessel Detail Service as an example, and magnify the above picture. We will get a more involved structure, where only key classes are shown:

On the diagram:

1. The database only shows new tables used to support transactional outbox and message deduplication.

2. “Traditional” business logic is colored in green. This includes our standard web controllers, repositories and models.

3. Classes supporting commands are colored in purple. Here we have:

   1. AddVesselDetailCommand representing the actual command message and its parameters;

   2. CommandListenerJob - an implementation of IHostedService interface, a background job listening for commands over a message queue;

   3. MessageParser is responsible for converting messages into commands;

   4. MessageDeduplicator is responsible for de-duplication of incoming commands. It is using CommandRepository, which is also used by CommandRepositoryCleanupJob, which maintains the COMMAND_REGISTRY table.

4. Classes supporting messaging are colored in cyan. They include:

   1. VesselDetailAdded event model, which is created by EventFactory and saved to OutboxRepository.

   2. The already existing VesselDetailDBContext is colored cyan because it will need to update OutboxRepository and VesselRepository within one transaction.

   3. Lastly, EventRelayJob is working with OutBoxRepository to fetch and send the events to an ASB topic.

As you can see, these are only the core types supporting the functionality. Actual implementation will have more classes and also configuration. The good news is that this plumbing is the same for all microservices.

Vessel Registry Controller Microservice: Saga Orchestrator

This microservice is similar to other microservices in terms of its structure, but has additional features to handle saga orchestrations. Before we look at class diagram for the service, let’s come up with a formal definition of the First Registry saga. A good way to do this is using a Finite State Machine diagram:

As shown on the diagram, we have several “happy path” states on the top row, and a few compensatory states below. There are a few points that we can make about this diagram:

1. This diagram is a way to present the requirement to the saga explicitly. It makes certain assumptions about desired behavior of the system. For example, we assume that the “Adding Registry” state is a pivot state in the saga, meaning that if it succeeds then we are not rolling anything back. In other words, this diagram reflects an assumption that if “Adding Registry” results in success, and subsequent “Updating Work Item” fails, we leave the work item in an inconsistent state, which will need to be manually corrected, but we are leaving all three vessel registry records (contact, vessel detail and vessel registration) in a consistent state with each other. An alternative design of this diagram could require rolling back the entire saga from the point when “Updating Work Item” stage fails.

2. Despite all measures for enhancing saga resiliency that we talked about earlier, sagas can still fail and lead to data inconsistencies. In the “eventual consistency” concept we started from, the term “eventual” is ambiguous. Is it seconds, minutes or maybe days? If we establish thresholds for how quickly sagas must complete, then we have to account for their potential failures and data corruption they may cause. The likelihood of such failures is low, as it can be thought of as a multiplication of a low probability of failure by a low probability of rollback failure. And so it is not feasible to add additional rollback or retry layers, as it adds complexity. A more rational approach is to invest in being able to pinpoint quickly what specifically is failing, and then fix it. And there is hardly a better strategy than implementing strategic diagnostic logging to achieve this goal.

3. As the diagram above is an example of a classical finite state machine, the sagas and their states can be modeled using State design pattern.

Here is a variation of the previous diagram, which assumes that an orchestrator can issue some commands in parallel, such as “Add Client” and “Add Vessel Detail”.

It contains fewer states, looks simpler, and takes less time to complete, but this simplification needs to be balanced against adding complexity to the Orchestrator, which would now need to receive events back from two services - Contact Service and Vessel Detail Service in order to make a decision on which next transition to make.

One more note about resiliency before we look at the class diagram: in addition to the measures common to all data services we also need a way to persist states of our saga. Since our orchestrator is a single point of failure, we cannot let it crash and leave a running saga in an undetermined state. Assuming that high availability measures will be taken for the orchestrator on a platform level, we still need to ensure its data integrity so that it can pick up where it has left off at any time.

Let us now inspect the structure of the orchestrator microservice:

The classes on the diagram could be broken down as follows:

1. Classes colored green are responsible for handling web requests:

   1. WebController is the ingress point for the saga, it implements CreateFirstRegistry() method and exposes a status endpoint GetStatus(correlationId).

   2. The role of OrchestratorProxy is to validate and then send a command to the orchestrator to a message queue. This as shown on the sequence diagram above allows service to respond to the UI caller quickly with an HTTP 202 code or analogous way to indicate acceptance of request. From this point on the "saga movement" occurs only asynchronously.

2. Classes colored cyan are responsible for consuming events:

   1. The orchestrator must process events received from each of the microservices participating in sagas. This is one differentiator from a standard microservice, where consumption of events is strictly optional. This is the focus of EventListenerJob, which is shown on the diagram. Other classes supporting message deduplication and transactional outbox pattern are omitted. Once an event is received and parsed, it invokes Orchestrator to load the corresponding saga and respond to the event. This is the mechanism of saga transitioning through its states.

3. Classes colored purple are responsible for dispatching and managing commands:

   1. Here we have 2 background jobs: CommandListenerJob for listening to incoming commands like the one enqueued by OrchestrationProxy, and CommandRelayJob which dispatches commands to saga participants. Classes supporting repository access, command parsing and creation are not shown.

4. Classes colored orange represent core orchestrator components. One thing we should emphasize is the orchestrator is designed to handle multiple sagas for various business scenarios. Each “saga type” can be thought of as a set of classes representing states from a finite state machine diagram, like the one we have discussed earlier.

   1. Orchestrator acts as a controller. It drives SagaFactory, which creates new sagas given their names, allowing us to distinguish between each of the "saga types". We would use SagaFactory when initiating a new saga, for instance.

   2. Saga type along with SagaState and its derivatives represent State design pattern. A created Saga is an instance of a "saga type", and it also stores input data passed by initiating asynchronous command, such as "Create first registry", in its InputParameters field. Each concrete instance of SagaState "knows" how to handle incoming events of success or failure, which commands to dispatch, and which next concrete instance of SagaState to set on its parent Saga instance. In essence the finite state machine diagram shown earlier is implemented by the concrete instances of SagaState type.

   3. SagaRepository is responsible for persisting and retrieving saga instances along with their current state and original input data from durable storage (database).

5. Database:

   1. Since deduplication is needed for both incoming commands and events, we combine their registry into one table named INBOX_REGISTRY.

   2. In order to reliably dispatch commands to consumers we use the OUTBOX_REGISTRY table.

   3. SAGA_REGISTRY table is where saga instances are persisted.

As we mentioned above, we move through sagas as the Orchestrator consumes and handles events that it receives from participating microservices. Let us picture one instance of such transition:

On the diagram:

1. The Orchestrator receives an event, which was already deduplicated and parsed, and asks SagaRepository to find an existing saga instance by correlation Id. SagaRepository finds and constructs a specific type of Saga, let's say it is "Create First Registry", and loads it with its current state and original input parameters that it has received when it was initiated.

2. The Orchestrator asks the Saga instance to handle the event, and it delegates this to a ConcreteSagaState1. It is hardwired to execute a ConcreteCommand instance and change parent saga's state to ConcreteSagaState2. An example of such interaction could look as follows: the concrete state the saga is in could be "Adding Registry", and the event it is passed on could be "Vessel Registry Rejected". The "Adding Registry" state would dispatch a command named "Rollback Vessel Details" to Vessel Detail microservice and change its parent saga state to "Undoing Add Vessel Details".

3. “Dispatching a command” means saving it to repository. What happens next is both SagaRepository and CommandRepository are committed within the same transaction controlled by DbContext. This step guarantees that saga state and the outgoing command are consistent with each other. The diagram ends here, but what happens next is the CommandRelayJob reads the command from OUTBOX_TABLE and writes it to the message queue of the target microservice.

To recap on where we are at - the Orchestrator is a finite state machine, which knows how to listen to events, read and persist sagas in their interim state, and to dispatch commands to participating microservices. It should not grow into a monolith as long as all it does is plays event and command dispatcher role. What will change over time is the number of sagas, and composition of their states.

A Note on Isolation

Sagas are notoriously prone to isolation issues. When comparing them to database transactions, and using the popular ACID (Atomic, Consistent, Isolated, Durable) acronym, which is commonly used to describe database transaction qualities, the hardest one to achieve in sagas is the “I” letter - the isolation. Again, we can “get by” here due to less stringent concurrency requirements to sagas as compared to database transactions. The specific measures we may need to take to improve isolation qualities of our sagas are as follows:

1. Use correlation Ids everywhere: in messages and when accessing saga states. They are at the center of all interactions. Expect that the unique correlation Ids may be duplicated due to the “at least once” behavior of ASB queues and topics.

2. Use “pending” connotation on naming of saga states, as this simplifies making decisions about queueing parallel calls while a saga is in such “pending” state. An example of a state using the “pending” connotation is “Adding Registry” as opposed to “Added Registry”.

3. We may consider adding CorrelationId column to each of our tables storing business entities. This will make it easier to support idempotence and isolation when executing commands by the microservices.

Conclusions and Next Steps

We have taken an in-depth look at sagas composition and assessed the implications that using them brings to the overall application. Using a concrete example of First Registry business scenario we have shown how a saga-based system can work, which core components it needs to have and why. This should be sufficient to begin planning the work and start implementing sagas in our code.

At the same time, we can raise questions related to sagas and to the particular implementation approach described here, which need further analysis and answers:

1. A “monolith” version of First Registry transaction executed against a single database would take a fraction of a second to complete, while the same transaction under proposed design will take seconds to complete. Is there a way to safely “speed up” our sagas without impacting their robustness?

2. We have admitted a low but non-zero likelihood of data corruption resulting from failed sagas. There is normally no such concern about monolith databases. What is going to be our strategy and countermeasures?

3. The sheer number of classes, messages, queues, topics and background jobs involved in implementing sagas increase system complexity greatly, and bear a risk of making the system brittle. What can be done to simplify things?

The next logical step is looking at platforms such as MassTransit. MassTransit promises to abstract away the burden of reliable messaging between the services, as well as implementing saga orchestrations as state machines. This has a significant potential for reducing the amount of plumbing we would need to build ourselves.

The hope is that this discussion has helped us to deeper understand the underlying challenges presented by sagas, and will empower us to utilize the features that MassTransit offers in a pragmatic and rational manner.