CHAPTER 10 · ARCHITECTURE

Microservices II — Data, Events, Deployment & Migration

Chapter 9 established that every microservice owns its own database. This chapter deals with the consequences: how to query and transact across services that no longer share a schema (polyglot persistence and event-driven architecture), how to actually run a fleet of services in production (deployment patterns), and how to get from a working monolith to this architecture without betting the company on a risky rewrite (refactoring strategies). Same running example: the shopping application's Order, Customer, Product and Shipping services.

Polyglot persistence

Recall from Chapter 9: each service's data is private, accessible only through its API. A natural consequence is polyglot persistence — different services use whatever kind of database best fits their own workload, not one enterprise-wide choice. Order Service might use a relational database (transactions matter); Product Catalog might use a document store (flexible, evolving schema, read-heavy); a "customers who bought this also bought…" feature might use a graph database (relationship queries are its whole job).

🧠 Why this matters

Polyglot persistence gives you loose coupling and lets each service pick the best tool for its own job — genuinely valuable. But it means you've given up the one thing a single shared database quietly gave you for free: the ability to query and transact across all your data at once.

The two challenges

Challenge 1 — no cross-service transactions. Placing an order might need to atomically create the order and reserve customer credit. With one shared database, that's one ACID transaction. With Order DB and Customer DB as separate schemas (possibly separate database engines entirely), there is no single transaction that spans both. The traditional distributed-systems answer — two-phase commit (2PC) — technically works, but it's slow, and it requires every participant to be online and responsive at commit time, which defeats the whole point of loosely-coupled, independently-deployable services.

Challenge 2 — no cross-service joins. Suppose you need to generate an invoice for an order: it needs the order's line items (Order DB) and the customer's billing name and address (Customer DB). With one shared schema that's a single SQL join. Split across two databases, you cannot write SELECT * FROM Orders JOIN Customers ... — the databases might not even be the same technology.

❌ What you could do with one shared DB

SELECT o.OrderId, o.Total, o.LineItems,
       c.BillingName, c.BillingAddress
FROM   Orders o
JOIN   Customers c ON c.CustomerId = o.CustomerId
WHERE  o.OrderId = @orderId;

One query, one transaction, instantly consistent. This is exactly what you lose by splitting the schema.

✅ What you do instead, across services

var order = await _orderClient.GetOrderAsync(orderId);
var customer = await _customerClient.GetCustomerAsync(order.CustomerId);
var invoice = Invoice.From(order, customer); // compose in application code

Two remote calls instead of one join — slower, and now subject to the partial-failure handling from Chapter 9. For anything queried often, you wouldn't do this live at all — you'd maintain a materialised view (below).

These two challenges are not solved by "trying harder" — they require a genuinely different strategy, which is exactly what event-driven architecture provides.

Event-driven architecture for microservices

A microservice publishes an event whenever something notable happens — typically, when it updates one of its own business entities. Other microservices subscribe to the events they care about; when one receives an event, it updates its own entities, which may in turn publish more events. Chain enough of these together and you've implemented a business transaction that spans multiple services, one local step at a time — without ever needing a distributed transaction.

Worked example: checking credit when creating an order

This is the same "place order → reserve credit" transaction from above, now built as a chain of events instead of a single ACID transaction:

Figure: Order Service creates the order (NEW) and publishes OrderCreated. Customer Service reserves credit and publishes CreditReserved. Order Service consumes that event and moves the order to OPEN — no distributed transaction anywhere.

Try the full step-by-step flow, including the failure branch, in the demo below.

Materialised views

Events also solve the "no cross-service join" problem for data you query often: instead of joining live, a dedicated service subscribes to the relevant events from both sides and maintains a pre-joined, denormalised view. The slides' example: a Customer Order View Updater service listens to events from both Customer Service and Order Service, and keeps a "Customer Order View" table up to date.

Figure: a dedicated updater subscribes to both services' events and keeps a fast, pre-joined read model. Queries hit the view directly — no live cross-service calls needed.

✅ Benefits	❌ Drawbacks
Implements transactions spanning multiple services, with eventual consistency instead of a blocking distributed transaction.	The programming model is genuinely more complex than plain ACID — you must design and implement compensating actions to recover from a failure partway through (e.g. cancel the order if credit reservation fails).
Enables fast, pre-joined materialised views for cross-service queries.	Applications must tolerate temporarily inconsistent data — an in-flight transaction is visible partway through, and a materialised view can lag behind the source data.
Services stay loosely coupled — no shared schema, no distributed lock manager.	Subscribers must be able to detect and ignore duplicate events (most messaging systems guarantee "at least once" delivery, not "exactly once").

Achieving atomicity: the dual-write problem

All of the above assumes that publishing an event and updating the database happen atomically — together, or not at all. But look closely at what "publish an event" actually requires: Order Service must both insert a row into the ORDER table and publish an OrderCreated event. Those are naturally two separate operations against two separate systems (a database, and a message broker) — and that's the dual-write problem.

❌ The naive dual write

public async Task PlaceOrderAsync(Order order)
{
    await _db.Orders.AddAsync(order);
    await _db.SaveChangesAsync();                 // ① DB write commits...

    await _eventBus.PublishAsync(
        new OrderCreated(Guid.NewGuid(), DateTime.UtcNow,
                          order.Id, order.CustomerId, order.Total));
    // ② ...then the process crashes, or the broker is unreachable.
    // The order now exists — but Customer Service never hears about it.
}

If the crash happens between ① and ②, the database and the rest of the system permanently disagree. This is exactly the atomicity problem the slides warn about.

✅ The fix: make publishing follow from the commit, not a second step

public async Task PlaceOrderAsync(Order order)
{
    await _db.Orders.AddAsync(order);
    await _db.SaveChangesAsync();   // the ONLY write the app performs
    // publishing is handled outside the app — see below
}

Three real strategies achieve this, all avoiding a second, independently-failable write from the application itself.

Strategy	How it works
Event sourcing	Don't store current state at all — store the ordered sequence of events as the source of truth. "Insert a row" and "publish an event" become the same operation: appending to the event log. Current state is simply the replay of all events so far. Other services subscribe to that same log directly.
Transaction log tailing (a.k.a. Change Data Capture / CDC)	Keep a normal database and a normal single write. A separate process (e.g. Debezium) tails the database's own commit/transaction log and emits a message for every committed row change. The application never has to publish anything itself — "if it's committed, a change event follows" is guaranteed by the database's own durability, not by application code. Caveat: these are low-level row deltas (INSERT/UPDATE on a given table), which usually still need translating into meaningful domain events — CDC captures what changed in the tables, not what happened in the business.
Database triggers	A DB-side trigger fires on insert/update and, within the same local database transaction as the business write, inserts a row into an events table. Because both writes are one local ACID transaction, they succeed or fail together. Here the event-writing logic lives inside the database (in trigger code) rather than in the application — simple, but harder to test and version. A separate poller reads the events table and publishes to the broker.

💡 Tip — Transactional Outbox vs database triggers, kept straight

These two are related but not the same. In the Transactional Outbox pattern, the application writes the business row and an outbox row in one local ACID transaction; a separate message relay then reads the outbox table and publishes to the broker (polling it, or tailing it via CDC). With database triggers, a trigger inside the database populates that outbox row instead of application code. Same goal — one atomic local write, then a reliable publish afterwards — but the outbox pattern keeps the event-shaping logic in your (testable, versioned) application code, whereas triggers push it into the database. The outbox pattern is the more commonly recommended of the two.

🎓 Lecture view vs reality

For the exam: in event sourcing you don't store current state — the ordered event log is the single source of truth, and state is the replay of events. In practice, that does not mean current state can't exist. You almost always build projections (a.k.a. materialised/read views) that fold the events into current-state tables for fast queries — replaying from the beginning on every read would be far too slow. And to avoid replaying millions of events, systems periodically write a snapshot (the folded-up state as of event N) and then replay only the events after it. The event log stays the source of truth; projections and snapshots are the standard, expected read-side optimisations built from it.

A minimal event record type in C#

OrderEvents.cs

public abstract record DomainEvent(Guid EventId, DateTime OccurredAtUtc);

public sealed record OrderCreated(
    Guid EventId, DateTime OccurredAtUtc,
    Guid OrderId, Guid CustomerId, decimal Total)
    : DomainEvent(EventId, OccurredAtUtc);

public sealed record CreditReserved(
    Guid EventId, DateTime OccurredAtUtc,
    Guid OrderId, Guid CustomerId, decimal Amount)
    : DomainEvent(EventId, OccurredAtUtc);

public sealed record CreditReservationFailed(
    Guid EventId, DateTime OccurredAtUtc,
    Guid OrderId, Guid CustomerId, string Reason)
    : DomainEvent(EventId, OccurredAtUtc);

Records give value-based equality and immutability for free — exactly what an event should be: an immutable fact about something that already happened.

Try it: order/credit saga stepper

Place an order and step through the event chain one message at a time. Try both outcomes — including the failure branch and its compensating action.

Order status: — Customer credit: —

What to notice: every step is one local write plus one published event — never a transaction touching both services' databases at once. In the failure branch, there's no "undo" of a shared transaction to fall back on; Order Service must run its own explicit compensating action (cancel the order) in response to the CreditReservationFailed event.

🎓 Lecture view vs reality

What the demo shows is a choreographed saga: there is no coordinator — each service simply reacts to events and emits its own, and the "saga" is the emergent chain. The alternative is an orchestrated saga, where a central saga orchestrator explicitly tells each service what to do next and drives the compensations. Either way, a production saga must also handle three things the happy path hides: compensating actions that themselves fail (they need retries and must be idempotent); duplicate or redelivered messages (consumers must be idempotent, because messaging is usually at-least-once — see the "detect and ignore duplicate events" row above); and following one business transaction across many services (attach a correlation id to every event so the whole flow can be traced and reconciled). Chapter 11 takes both coordination styles apart side by side, and covers the idempotency and dead-letter machinery those three problems need.

Choosing a deployment strategy

Deploying a monolith means provisioning N servers and running M identical copies of one artifact on each — simple. Deploying a microservices application means doing that separately, for every service, each with its own scaling needs. Four patterns cover almost everything in production today.

Figure: same three service instances, three different isolation boundaries — none (shared host), a full VM each, or a lightweight container each.

Pattern	How it works	✅ Benefit	❌ Drawback
Multiple instances per host	Provision hosts, run several service instances directly on each (well-known port per instance) — the traditional deployment approach.	Efficient use of resources; fast to start; no extra virtualisation layer.	No resource isolation between instances (a runaway process can starve its neighbours); you manage ports and dependencies by hand.
Instance per VM	Package each service as a VM image (e.g. an AWS EC2 AMI); each running instance is a full VM launched from that image.	Strong isolation via the hypervisor; simple, well-understood resource limits; mature tooling.	Heavyweight — each VM boots a full OS (slow, minutes); wasteful — N copies of a full OS kernel for N instances.
Instance per container	Each instance runs in its own container — an OS-level virtualisation boundary (its own port and filesystem namespace, resource limits via cgroups) without a full guest OS. Docker is the dominant technology.	Starts in seconds, not minutes; much higher density than VMs; still gets real resource limits and isolation.	Needs an orchestrator (Kubernetes, etc.) at any real scale to schedule, heal and network containers.
Serverless	Package the service as code (e.g. a ZIP) and upload it to a platform like AWS Lambda, which runs however many instances are needed to handle load, and bills per invocation based on time and memory used.	Nobody — not you, not anyone on your team — manages servers, VMs, or containers at all. Scales to zero when idle.	Constrained to supported languages/runtimes; execution time limits; "cold start" latency on the first request after idling; harder to reason about long-running or stateful work.

Try it: deployment pattern comparator

Pick a deployment pattern and see how it lays out three service instances, plus a rough scorecard for isolation, density and startup speed.

What to notice: isolation and density trade off against each other for multi-instance-per-host vs VMs, while containers land in the sweet spot most teams pick today — and serverless removes the axis entirely by removing the "host" concept from your job.

Refactoring a monolith into microservices

Turning a working monolith into microservices is application modernisation, not a rewrite. Never do a "Big Bang" rewrite — freezing the existing product while a separate team rebuilds it from scratch as microservices sounds appealing and is extremely risky: you stop shipping value for months (or years), and you're rebuilding business rules nobody fully documented in the first place. Instead, refactor incrementally, running the evolving monolith and the new microservices side by side.

Strategy 1 — Stop digging

🌍 Real-world analogy

The Law of Holes: "if you find yourself in a hole, stop digging." If the monolith is already unmanageable, the first, cheapest move is simply to stop making it bigger.

All new functionality is built as a standalone microservice — nothing new is added to the monolith. The monolith still exists and still needs to be touched sometimes (usually via a thin anti-corruption layer — glue code that translates between the monolith's model and the new service's model, so the new service doesn't have to adopt the monolith's legacy assumptions).

Figure: the monolith stops growing; every new capability is born as its own service from day one.

Strategy 2 — Split frontend and backend

Most applications have a natural seam between presentation logic (rendering pages, handling UI requests) and business/data-access logic underneath. Split along that seam into two separate applications: a presentation-layer app, and a business-layer app exposing a coarse-grained API (one or more facades over the business logic). This alone roughly halves the size of "the monolith," even before any further decomposition.

Figure: two deployables instead of one; presentation now calls business logic remotely (REST/gRPC) instead of an in-process method call.

Strategy 3 — Extract services

Turn existing modules within the monolith into standalone microservices, one at a time. Each extraction shrinks the monolith a little more; repeat until it either disappears entirely or shrinks down to "just another service" small enough not to be a problem anymore. This is the same idea as Martin Fowler's Strangler Fig pattern — new services gradually grow up around the monolith and take over its responsibilities, the way a strangler fig grows around a host tree until the tree is no longer needed.

Figure: one module carved out at a time — each extraction shrinks the monolith and grows the service estate.

Picking which module to extract first: start with the module that either (a) changes most often — extracting it unblocks the most deployments — or (b) has the tightest resource-isolation need (like our CPU-hungry image-resizer from Chapter 9), since that's where you get payoff fastest. Avoid starting with the module most entangled with everything else — that one's hardest and riskiest to pull out first.

Try it: strangle the monolith

Click a module to extract it into its own service (Strategy 3, in action). Watch the anti-corruption "glue" line appear as each module leaves the monolith.

Extracted: 0 / 4 modules

What to notice: each extraction is independent and reversible in this demo — in real life it's a one-way door per module, which is exactly why you do it one module at a time instead of all at once.

Chapter 10 quiz

💡 Recap

Together, Chapter 9 and this chapter cover the full arc: why teams split a monolith (the Scale Cube's Y-axis), how the pieces talk (API Gateway, IPC, partial-failure handling, service discovery), how their data stays consistent without shared transactions (polyglot persistence, events), how to actually run the fleet (deployment patterns), and how to get there without a risky rewrite (incremental refactoring). For the architectural patterns microservices builds on top of — layered architecture, SOA, pub/sub, broker — see Chapter 8. Chapter 11 then works through the pattern catalogue these two chapters keep reaching for: API Gateway, Circuit Breaker, Saga, CQRS and Sidecar, plus the idempotency and dead-letter machinery that keeps them safe — and the GoF patterns living inside each individual service.