Microservices III — Patterns in Practice
Chapter 9 and Chapter 10 built the architecture: services that own their data, talk over the network, and stay consistent through events. This chapter is the pattern catalogue for actually surviving in that world — the five architectural patterns every microservices system reaches for (API Gateway, Circuit Breaker, Saga, CQRS, Sidecar), the two ways to coordinate a saga, the idempotency and dead-letter machinery that makes any of it safe under retries, and finally the ordinary GoF design patterns that live inside a single service.
The trade you made
Every pattern in this chapter exists to pay off one debt. When you moved from a monolith to a distributed system, you exchanged function calls for network calls — and a network call is not a slower function call. It is a different kind of thing, with three failure modes a method call simply does not have.
| What a function call gives you | What a network call gives you instead | The pattern that pays it back |
|---|---|---|
| Returns in nanoseconds, cost invisible | Latency — milliseconds at best, and it multiplies with every hop in a chain | API Gateway (aggregate many calls into one), CQRS (pre-compute the read) |
| Either runs or throws — the callee cannot be "absent" | Partial failure — the callee may be down, slow, or reachable-but-broken, and a slow dependency is more dangerous than a dead one | Circuit Breaker, timeouts, DLQs |
| One ACID transaction spans everything you touch | No shared transaction — each service commits locally, so "half done" is a state your system can really be in | Saga + idempotency |
| Called exactly once | At-least-once delivery — retries mean the same message can arrive twice, or ten times | Idempotency keys |
None of these patterns is decoration. Each one is a direct answer to a specific thing the network took away from you. If you can name which guarantee a pattern is replacing, you understand the pattern — and that framing is exactly what an exam question like "justify your choice of pattern" is looking for.
The five MSA architectural patterns
These five are the canonical set. The first two are about how requests flow, the middle two about how data behaves, and the last about where cross-cutting concerns live.
1. API Gateway
One front door for all clients: routes, authenticates, aggregates. A reverse proxy with brains.
2. Circuit Breaker
Stop calling a dependency that is clearly broken, so one service's failure doesn't take the whole system down with it.
3. Saga
A business transaction as a chain of local transactions, with compensating actions instead of rollback.
4. CQRS
Split the model you write through from the model you read through, because the two jobs want opposite things.
5. Sidecar
Attach a helper process alongside the service to take over logging, TLS, retries and telemetry — no library, no code change.
+ Idempotency & DLQs
Not on the "five" list, but a saga without them is a saga that corrupts data the first time the network hiccups.
1. API Gateway
The gateway is the single entry point for every client. It routes each request to the right downstream service, handles authentication once instead of five times, and often aggregates several service responses into the one payload a mobile screen actually needs.
Think of it as a reverse proxy with brains. A plain reverse proxy forwards /orders
to whichever box serves orders. A gateway also checks your ticket at the door, decides you're allowed in, calls
three departments on your behalf, and hands you one combined answer.
The full treatment — the client-to-microservice problem it solves, the benefits and drawbacks, and how it relates to service discovery — is in Chapter 9. What the new slides add is the shape of the code, so here it is in C#. This is the minimal "dumb proxy" half:
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHttpClient();
var app = builder.Build();
// The routing table: public prefix -> internal service base address.
var services = new Dictionary<string, string>
{
["order"] = "http://order-service:8080",
["inventory"] = "http://inventory-service:8080"
};
app.MapGet("/proxy/{serviceName}/{*path}", async (
string serviceName, string path,
HttpRequest request, IHttpClientFactory factory) =>
{
if (!services.TryGetValue(serviceName, out var baseUrl))
return Results.NotFound($"Unknown service '{serviceName}'.");
var target = $"{baseUrl}/{path}{request.QueryString}";
var client = factory.CreateClient();
client.Timeout = TimeSpan.FromSeconds(3); // never inherit the 100s default here
var response = await client.GetAsync(target);
var body = await response.Content.ReadAsStringAsync();
return Results.Content(body, "application/json", statusCode: (int)response.StatusCode);
});
app.Run();
The interesting half is aggregation — the part that actually earns the gateway its keep, because it turns a mobile client's three round trips over a slow radio link into one:
app.MapGet("/api/order-screen/{orderId}", async (string orderId, IHttpClientFactory factory) =>
{
var client = factory.CreateClient();
// Bound the WHOLE aggregation with one budget for the screen. Without this,
// the gateway will sit waiting on a dead dependency until HttpClient's
// 100-second default gives up — long after the user has left.
using var budget = new CancellationTokenSource(TimeSpan.FromSeconds(2));
var ct = budget.Token;
// Fan out in parallel — sequential awaits would add up the latencies.
var orderTask = client.GetFromJsonAsync<OrderDto>($"http://order-service:8080/orders/{orderId}", ct);
var shippingTask = client.GetFromJsonAsync<ShippingDto>($"http://shipping-service:8080/by-order/{orderId}", ct);
var reviewsTask = client.GetFromJsonAsync<ReviewDto[]>($"http://review-service:8080/by-order/{orderId}", ct);
await Task.WhenAll(orderTask, shippingTask, reviewsTask);
return Results.Ok(new OrderScreenDto(
Order: await orderTask,
Shipping: await shippingTask,
Reviews: await reviewsTask));
});
For the exam: the gateway is a single entry point that routes, authenticates and aggregates.
That is exactly right. Three things production adds. (1) Nobody hand-writes the routing
half. You configure YARP, Ocelot, Envoy, Kong or your cloud's API Gateway; you only write code for the
aggregation half, where real business shaping happens. (2) Fan out in parallel, and bound the whole
thing. The Task.WhenAll above matters — three sequential 200 ms awaits is a
600 ms screen; in parallel it is 200 ms. The CancellationTokenSource matters just as
much: an aggregating gateway is the single most likely place in your system to sit waiting on a dead
dependency, and an unbounded call inherits HttpClient's 100-second default. That is why gateways,
timeouts and circuit breakers always turn up together — and why a real
implementation also decides what to return when one of the three fails: usually the screen minus the
reviews, not a blank error page. (3) One gateway for every client is a compromise. A mobile app and an internal admin console
want very different payloads, so teams often run one gateway per client type — the Backend for
Frontend (BFF) pattern. Be aware, too, that the gateway is a single point of failure and a deployment
bottleneck if every team has to edit it; that is the standard critique to raise in an essay answer.
2. Circuit Breaker
If Service A calls Service B and B is down, A should stop trying. Every doomed call ties up a connection, a thread-pool slot and a caller who is now waiting — and when A's resources run out, A fails too, and whoever calls A fails next. That is cascading failure, and the circuit breaker exists to stop it at the first hop.
| State | What happens to a request | How you leave this state |
|---|---|---|
| Closed (healthy) | Calls flow normally; failures are counted. | Failure count crosses the threshold → Open. |
| Open (tripped) | Fails immediately, without touching the network. No connection, no wait, no thread held. | The cooldown period elapses → Half-Open. |
| Half-Open (probing) | Exactly one trial request is allowed through. | It succeeds → back to Closed. It fails → straight back to Open for another cooldown. |
Chapter 9 has the state diagram, a full C# implementation and an interactive simulator — work through those if the three states aren't yet automatic. What is worth adding here is a critique, because the version on the slides is a good example of how this pattern gets subtly wrong.
public static class CircuitBreaker
{
public static int Failures = 0; // shared by EVERY caller
public static int Threshold = 5;
public static bool IsOpen = false; // once true, true forever
}
async Task<string> CallExternalServiceAsync(HttpClient client)
{
if (CircuitBreaker.IsOpen)
throw new HttpRequestException("Service temporarily unavailable (503)");
try
{
return await client.GetStringAsync("http://unstable-service/api");
}
catch (Exception)
{
CircuitBreaker.Failures++;
if (CircuitBreaker.Failures >= CircuitBreaker.Threshold)
CircuitBreaker.IsOpen = true;
throw;
}
}
// One policy pipeline PER dependency, registered on that dependency's client.
builder.Services
.AddHttpClient<IPaymentClient, PaymentClient>(c =>
{
c.BaseAddress = new Uri("http://payment-service:8080");
c.Timeout = TimeSpan.FromSeconds(3);
})
.AddStandardResilienceHandler(); // retry + timeout + circuit breaker,
// scoped to THIS client only
Microsoft.Extensions.Http.Resilience (built on
Polly) rather than hand-rolling. It gives you per-dependency state, a real half-open probe, a rolling failure
rate window rather than a lifetime count, and retries with jitter — and it is configuration, not code
you have to maintain. Hand-write the breaker once to learn it (Chapter 9 does exactly that); use the library in
anything real.For the exam: Closed / Open / Half-Open, and the point is to stop a localised failure cascading. Say that and you have the marks. Two nuances worth a sentence if you have room. (1) A breaker on its own only makes failures fast — it doesn't make them invisible. The caller still has to decide what to show the user, so a breaker is usually paired with a fallback (cached data, a degraded response, "recommendations unavailable" instead of a blank page). (2) Breakers trip on rates over a window, not lifetime counts, and they only help for the dependency they wrap — which is why "one global breaker" is the classic beginner mistake.
3. Saga — distributed transactions without a distributed transaction
Each microservice owns its own database, so there is no BEGIN TRANSACTION that spans Order,
Payment and Inventory — and no global transaction manager to run one. The saga pattern's answer:
model the business process as a sequence of local transactions, each committed in its own
service, and when a later step fails, undo the earlier ones by running explicit
compensating transactions.
Booking a trip. Reserve the hotel — done, committed, real. Book the flight — sold out. There is no cosmic "undo" that un-reserves the hotel for you: you have to make a phone call and cancel it. That call is the compensating transaction, and noticing that it is a new business action rather than a rollback is the whole insight of the pattern.
A rollback erases history — after it, nothing ever happened. A compensation is a new transaction that makes up for an old one, and the record of both remains. Refunding a payment is not the same as never charging it: the customer saw the charge, the statement shows both lines, and some compensations are only partial (a cancellation fee) or impossible (the confirmation email has already been sent). Designing the compensations is usually harder than designing the happy path.
Choreography — event-driven, no boss
Each service does its local transaction and publishes an event; the next service is listening and reacts. Nobody is in charge, and the "saga" is simply the chain that emerges.
Orchestration — a conductor tells everyone what to do
An orchestrator — a dedicated service, or a component inside one of them — holds the saga's state and issues explicit commands: "charge this card", "reserve this stock". Participants just do as they are told and reply. It is the same idea as the orchestrator in SOA.
Deep dive: an orchestrator with a compensation stack, in C#
The trick that makes compensations manageable: as each step succeeds, push its undo onto a stack. If a later step fails, you simply unwind the stack — which automatically compensates in reverse order, and only for the steps that actually ran. Note what happens when an undo throws: it is dead-lettered rather than swallowed, and the remaining compensations still run.
public sealed class OrderSagaOrchestrator
{
private readonly IPaymentService _payments;
private readonly IInventoryService _inventory;
private readonly IOrderService _orders;
private readonly IDeadLetterQueue _deadLetters;
public OrderSagaOrchestrator(IPaymentService payments,
IInventoryService inventory,
IOrderService orders,
IDeadLetterQueue deadLetters)
{
_payments = payments; _inventory = inventory;
_orders = orders; _deadLetters = deadLetters;
}
public async Task<SagaResult> PlaceOrderAsync(OrderRequest request, CancellationToken ct = default)
{
var sagaId = Guid.NewGuid().ToString("N");
// Each completed step pushes the action that undoes it, with a name so a
// failed undo can say which one it was.
var compensations = new Stack<(string Name, Func<Task> Undo)>();
try
{
var orderId = await _orders.CreatePendingAsync(request, ct);
compensations.Push(("cancel-order", () => _orders.MarkCancelledAsync(orderId)));
var paymentId = await _payments.ChargeAsync(request.CustomerId, request.Total, ct);
compensations.Push(("refund-payment", () => _payments.RefundAsync(paymentId)));
await _inventory.ReserveAsync(orderId, request.Items, ct);
compensations.Push(("release-stock", () => _inventory.ReleaseAsync(orderId)));
await _orders.MarkConfirmedAsync(orderId, ct);
return SagaResult.Completed(orderId);
}
catch (Exception ex)
{
var outstanding = await CompensateAsync(compensations, sagaId);
// A saga whose compensations all succeeded is back to a consistent
// state. One whose compensations DIDN'T is not, and must not claim to
// be — "compensated" would hide a real charge that was never refunded.
return outstanding.Count == 0
? SagaResult.Compensated(ex.Message)
: SagaResult.CompensationFailed(ex.Message, outstanding);
}
}
/// Returns the names of the compensations that could not be completed.
private async Task<IReadOnlyList<string>> CompensateAsync(
Stack<(string Name, Func<Task> Undo)> compensations, string sagaId)
{
var outstanding = new List<string>();
while (compensations.Count > 0)
{
var (name, undo) = compensations.Pop();
try
{
await undo();
}
catch (Exception ex)
{
// The worst case: the forward step failed AND taking it back
// failed. Automation is out of options, so escalate to a human —
// and keep unwinding, because the remaining undos can still work.
outstanding.Add(name);
await _deadLetters.SendAsync(
$"compensation:{sagaId}:{name}",
$"Compensation '{name}' failed: {ex.Message}");
}
}
return outstanding;
}
}
public interface IDeadLetterQueue
{
Task SendAsync(string messageId, string reason);
}
public enum SagaStatus { Completed, Compensated, CompensationFailed }
public sealed record SagaResult(
SagaStatus Status, string? OrderId, string? Reason,
IReadOnlyList<string> OutstandingCompensations)
{
public static SagaResult Completed(string orderId) =>
new(SagaStatus.Completed, orderId, null, Array.Empty<string>());
public static SagaResult Compensated(string reason) =>
new(SagaStatus.Compensated, null, reason, Array.Empty<string>());
// The third state is the one people forget. Without it, "we tried to undo it"
// and "we undid it" are indistinguishable to every caller downstream.
public static SagaResult CompensationFailed(string reason, IReadOnlyList<string> outstanding) =>
new(SagaStatus.CompensationFailed, null, reason, outstanding);
}
// The participants, as seen by the orchestrator. Note that none of them
// knows the others exist — only the orchestrator knows the workflow.
public sealed record OrderRequest(Guid CustomerId, decimal Total, IReadOnlyList<string> Items);
public interface IOrderService
{
Task<string> CreatePendingAsync(OrderRequest request, CancellationToken ct);
Task MarkConfirmedAsync(string orderId, CancellationToken ct);
Task MarkCancelledAsync(string orderId); // compensation
}
public interface IPaymentService
{
Task<string> ChargeAsync(Guid customerId, decimal amount, CancellationToken ct);
Task RefundAsync(string paymentId); // compensation
}
public interface IInventoryService
{
Task ReserveAsync(string orderId, IReadOnlyList<string> items, CancellationToken ct);
Task ReleaseAsync(string orderId); // compensation
}
Compare this to the slide's version, which inlines the calls and the one refund
directly in the request handler. That works for two steps; the stack scales to ten without turning into nested
ifs.
Note the third outcome. Most implementations of this have two states — completed, or compensated. But if an undo failed, the saga is in neither: the customer's money is still gone and the order is still stuck. Reporting that as "compensated" is a quieter version of swallowing the exception, because every caller downstream — and every replay of that idempotency key — will now be told the rollback was clean. Give it its own state and carry the names of what is still outstanding.
And note what this code still does not do. It keeps the saga's progress in a local variable, exactly like the slide version — fine for reading, not fine for production; see the callout below on persisting saga state. It also escalates a failed undo immediately, where a real orchestrator would first retry it, since a refund usually fails for a transient reason. The M01 lab implements that properly: transient failures get a bounded retry ladder, permanent ones skip it, and the dead letter records how many attempts were actually made.
One caveat on the ordering. Reverse order is the right default, because a later step may depend on an earlier one and undoing the dependency first can strand it. It is not a law: independent compensations can safely run in parallel, and occasionally the business dictates its own order — you might insist on refunding the customer before releasing the stock, whatever sequence the forward path used. Use the stack because it makes the safe default automatic, not because reversal is universally required.
Choreography vs orchestration
Neither is "correct" — this is a genuine trade-off, and being able to argue it both ways is the point.
| Feature | Choreography (event-based) | Orchestration (command-based) |
|---|---|---|
| Complexity | Low — simple to start; no new component to build. | High — you must build, deploy and operate an orchestrator. |
| Coupling | Services are loosely coupled; each knows only its own events. | The orchestrator knows about all services (though the services still know nothing about each other). |
| Maintenance | Hard to track the flow as it grows — the workflow exists only as an emergent property of who happens to subscribe to what. | Easier to monitor: the saga's state lives in one place you can query. |
| Best for | Simple workflows (2–3 services). | Complex workflows with many steps or services. |
| Debugging a stuck order | Trace events across N services' logs and hope you attached a correlation id. | Read one row: "saga 7781, step 3 of 5, awaiting inventory reply". |
| Failure mode | Cyclic event storms and unclear ownership of the compensation. | The orchestrator is a single point of failure, and a bottleneck for change. |
Same three-service order flow, run under both coordination styles. Inject a failure and step through to watch the compensations run — and keep an eye on the message counter and the coupling readout.
For the exam: a saga is a chain of local transactions with compensating transactions on
failure, coordinated either by choreography or orchestration. The thing the slide code hides:
its orchestrator lives entirely inside one HTTP request, with the saga's progress held in local variables. If
that process is restarted between "payment charged" and "stock reserved" — a deploy, a crash, a pod eviction —
the saga simply evaporates, having taken the customer's money. A real orchestrator
persists saga state after every step, so a restarted instance can pick the saga back up and
finish it or compensate it. That is why frameworks exist for this (MassTransit's state machine sagas, Dapr
Workflow, Temporal, Camunda) rather than everyone hand-writing it. Also: sagas provide
eventual consistency and no isolation — another transaction can read the half-finished
state partway through, which is why saga-based systems lean on statuses like PENDING to make the
intermediate state explicit rather than pretending it isn't visible.
4. CQRS — Command Query Responsibility Segregation
In a high-scale system, the way you write data is very different from the way you read it. Writes want normalisation, validation and strict consistency. Reads want denormalisation, pre-joined shapes and raw speed — and there are usually a hundred times more of them. CQRS stops those two jobs fighting over one model by splitting them:
| Commands | Queries | |
|---|---|---|
| Job | Change state — create, update, delete | Read state |
| Returns | Success/failure (and maybe an id) — not data | Data — and never changes anything |
| Model shape | Normalised, validated, enforces invariants | Denormalised, pre-joined, shaped for one screen |
| Storage it likes | A relational database with real transactions | Whatever is fastest to read — a document store, a cache, a search index |
| Naming | PlaceOrderCommand, CancelOrderCommand |
OrderSummaryQuery, CustomerOrdersQuery |
One detail in the code below is easy to skip past and is the difference between a read model that eventually catches up and one that is permanently wrong. The command handler does not save the order and then publish an event as a second step — that is the dual-write problem from Chapter 10, and a crash in between would leave an order that the read model never hears about. Instead it writes the order row and an outbox row in one local transaction, and a separate relay publishes from the outbox.
// ---------- COMMAND SIDE: normalised, validated, returns an id — not a view ----------
public sealed record PlaceOrderCommand(Guid CustomerId, IReadOnlyList<OrderLine> Lines);
public sealed record OrderLine(string Sku, int Quantity, decimal UnitPrice);
public sealed class PlaceOrderHandler
{
private readonly IOrderWriteStore _write;
public PlaceOrderHandler(IOrderWriteStore write) => _write = write;
public async Task<Guid> HandleAsync(PlaceOrderCommand command)
{
if (command.Lines.Count == 0)
throw new ArgumentException("An order must have at least one line.");
var orderId = Guid.NewGuid();
var total = command.Lines.Sum(l => l.Quantity * l.UnitPrice);
// ONE local ACID transaction writes the order AND the outbox row, so there
// is no second, independently-failable "publish" step to get out of sync.
await _write.SaveWithOutboxAsync(
new OrderRow(orderId, command.CustomerId, command.Lines, total),
new OrderPlaced(orderId, command.CustomerId, total, LineCount: command.Lines.Count));
return orderId; // an id, not a view — the caller must QUERY for the view
}
}
// ---------- QUERY SIDE: denormalised, shaped for one screen ----------
public sealed record OrderSummary(
Guid OrderId, string CustomerName, string CustomerCity,
int LineCount, decimal Total, string Status);
public sealed class OrderSummaryQueryHandler
{
private readonly IOrderReadStore _read;
public OrderSummaryQueryHandler(IOrderReadStore read) => _read = read;
// One lookup against a pre-joined row. No joins, no aggregation at read time.
public Task<OrderSummary?> HandleAsync(Guid orderId) => _read.GetSummaryAsync(orderId);
}
// ---------- THE PROJECTOR: the only thing connecting the two ----------
public sealed class OrderSummaryProjector
{
private readonly IOrderReadStore _read;
private readonly ICustomerReadStore _customers;
public OrderSummaryProjector(IOrderReadStore read, ICustomerReadStore customers)
{
_read = read; _customers = customers;
}
// Upsert, not insert: at-least-once delivery means this can run twice for the
// same event, and a projector that is not idempotent corrupts the read model.
public async Task OnAsync(OrderPlaced e)
{
var customer = await _customers.GetAsync(e.CustomerId);
await _read.UpsertSummaryAsync(new OrderSummary(
e.OrderId, customer.Name, customer.City, e.LineCount, e.Total, "PENDING"));
}
}
// The event carries everything the read model needs. If the projector had to call
// back into the write side for the line count, the two would be coupled again.
public sealed record OrderPlaced(Guid OrderId, Guid CustomerId, decimal Total, int LineCount);
// The two stores are deliberately different shapes — that is the whole point.
public sealed record OrderRow(
Guid OrderId, Guid CustomerId, IReadOnlyList<OrderLine> Lines, decimal Total);
public interface IOrderWriteStore
{
/// Persists the order row and the outbox row in ONE local transaction.
/// A separate message relay polls the outbox, publishes, and marks rows sent.
Task SaveWithOutboxAsync(OrderRow order, OrderPlaced @event);
}
public interface IOrderReadStore
{
Task<OrderSummary?> GetSummaryAsync(Guid orderId);
Task UpsertSummaryAsync(OrderSummary summary);
}
public interface ICustomerReadStore
{
Task<CustomerView> GetAsync(Guid customerId);
}
public sealed record CustomerView(Guid Id, string Name, string City);
Send commands to the write model and query the read model. Drag the projection lag up and try querying immediately after a write — that stale window is the price of the pattern, and it is the thing users notice.
CQRS is about models: separate the type you write through from the type you read through. That is all it requires — you can do CQRS against a single database table. Event sourcing (Chapter 10) is about storage: keep the event log as the source of truth instead of current state. A materialised view is the artefact a projector maintains — the pre-joined read table itself. They are frequently used together (event sourcing almost forces CQRS, because an event log is unqueryable for reads), which is why people assume they are the same thing. They are not, and mixing them up is a classic exam trap.
For the exam: commands change state, queries read state, and in a FastAPI-style app that starts as separate request/response models and can go as far as a separate read database. What that undersells is the cost. Full CQRS with a separate read store buys you scale and shape, and charges you: two data stores to keep in sync, a projector to build and monitor, projection lag your UI has to cope with, and the read-your-own-writes problem (a user saves and immediately re-reads — standard workarounds are to return the new state directly from the command, or to read from the write model just for that user's own recent changes). Most systems should start with plain separate DTOs and one database, and only split the store when read load actually demands it. "Use CQRS everywhere" is how a codebase acquires three layers of indirection to fetch a customer's name.
5. Sidecar
The sidecar attaches a helper component to your main service — deployed alongside it, sharing its lifecycle and its local network — and hands that helper all the peripheral work: logging, metrics, tracing, TLS, retries, and service-to-service routing. Your service goes back to doing only its actual job.
The motorcycle sidecar the slides open with. The bike is complete on its own; the sidecar bolts on, goes everywhere the bike goes, and carries what the bike shouldn't have to. Nothing about the engine changes.
http://payment-service/… as normal — the
platform quietly redirects that traffic through the sidecar sitting next to it, which handles the network and
takes its policy from the control plane. Interception is transparent, which is exactly why adopting a mesh
needs no application changes.Toggle the sidecar and watch the responsibilities — and the code — move out of your service.
Because the sidecar handles the network, your service's only obligation is to be sidecar-aware — which in practice means exposing health endpoints so the platform can tell whether to route traffic to you and whether to restart you:
builder.Services.AddHealthChecks()
.AddCheck("self", () => HealthCheckResult.Healthy()) // is the process alive?
.AddNpgSql(connectionString, name: "orders-db", tags: ["ready"]); // can it actually serve?
// Liveness: "am I broken beyond recovery?" — failing this gets you RESTARTED.
app.MapHealthChecks("/health/live", new HealthCheckOptions
{
Predicate = _ => false // no dependency checks: a dead DB is not a reason to restart me
});
// Readiness: "should traffic come to me right now?" — failing this gets you
// REMOVED FROM THE LOAD BALANCER, then put back when you recover.
app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
Predicate = check => check.Tags.Contains("ready")
});
For the exam: a sidecar is a helper component attached to the main service handling peripheral concerns (logging, monitoring, service-to-service communication), usually provided at the infrastructure level by Envoy or Istio rather than written in your application — and your code must be sidecar-aware by exposing health checks. All correct. Three additions. (1) The liveness/readiness distinction above is the one people get wrong, and getting it backwards is genuinely dangerous: put a database check in your liveness probe and a brief database blip will cause the platform to restart every one of your pods at once, turning a small outage into a total one. (2) A sidecar is not free — it is an extra process per pod (memory and CPU multiplied by every instance) and an extra network hop each way, which is why sidecar-less mesh modes exist now as an alternative. (3) The pattern is broader than service meshes: log shippers, config reloaders, secret-refreshers and database proxies are all sidecars, and the same "attach a helper to the main container" idea is what an Adapter or Decorator does at the object level — the sidecar is that idea moved up to the deployment level.
Idempotency — the safety net that makes sagas work
In a distributed system, "exactly once" delivery is nearly impossible; "at least once" is the reality. A message broker that isn't sure whether you processed a message will send it again. A client whose request timed out will retry — even though the server may have processed the first attempt perfectly and just lost the response on the way back.
So every endpoint that changes something must be idempotent: calling it more than once with the same input has no additional effect beyond the first call. Without that, one dropped response packet charges your customer twice.
| Operation | Idempotent? | Why |
|---|---|---|
GET /orders/7781 | ✅ Naturally | Reads change nothing. |
PUT /orders/7781/address with a full address | ✅ Naturally | Setting a field to X twice leaves it at X. |
DELETE /orders/7781 | ✅ Naturally | Already deleted → still deleted. Note
that idempotency is about the effect on the server, not the reply: answering 204 the
first time and 404 the second is still idempotent. |
POST /orders | ❌ Not by default | Two calls create two orders. Needs a key. |
POST /accounts/42/charge {amount: 120} | ❌ Definitely not | Two calls charge $240. This is the one that ends up in the newspaper. |
UPDATE balance SET balance = balance - 120 | ❌ Not | Relative updates are never idempotent. Absolute ones (SET balance = 380) are. |
The idempotency key workflow
For operations that aren't naturally idempotent, the caller (a client, or your saga orchestrator) sends a
unique key with the request — conventionally an X-Idempotency-Key header — and the server remembers
what it did with that key.
Send a charge, then hit "network retry" to simulate the client resending a request whose response was lost. Run it once with the key off and once with it on.
Deep dive: idempotency done properly in C#
Two things make or break this. The claim must be atomic — "check, then insert" is a race two duplicates can both win. And the claim, the side effect and the stored response must commit together; three separate steps means a crash after the charge strands the key half-finished, with the money already gone and every retry refused. Both fall out of keeping the idempotency record in the same database as the business data, so one transaction covers all three:
using System.Data;
using System.Text.Json;
// Store the real status code AND the real body, so a replay is indistinguishable
// from the original response — including when the original was a failure.
public sealed record IdempotentResponse(int StatusCode, string Body);
// Three genuinely different answers, so make them three cases. A nullable
// response cannot express them: "null" would mean both "you claimed it" and
// "somebody else owns it but hasn't finished".
public abstract record ClaimResult
{
public sealed record Claimed : ClaimResult;
public sealed record Replay(IdempotentResponse Response) : ClaimResult;
/// Same key, DIFFERENT request — the caller reused a key by mistake.
/// Replaying an unrelated response would be worse than refusing.
public sealed record FingerprintMismatch : ClaimResult;
}
// Deterministic outcomes only. Anything transient — a timeout, a deadlock, the
// gateway being unreachable — must THROW rather than return Declined.
public abstract record ChargeOutcome
{
public sealed record Charged(string PaymentId) : ChargeOutcome;
public sealed record Declined(string Reason) : ChargeOutcome;
}
public interface IChargeStore
{
Task<IDbTransaction> BeginTransactionAsync();
/// ① CHECK + ② CLAIM, atomically — SQL in "Production details" below.
Task<ClaimResult> TryClaimAsync(
IDbTransaction tx, string key, string fingerprint, TimeSpan ttl);
Task<ChargeOutcome> TryChargeAsync(IDbTransaction tx, string customerId, decimal amount);
Task SaveResponseAsync(IDbTransaction tx, string key, IdempotentResponse response);
}
public sealed class ChargeHandler
{
private static readonly TimeSpan Ttl = TimeSpan.FromHours(24);
private readonly IChargeStore _store;
public ChargeHandler(IChargeStore store) => _store = store;
public async Task<IResult> ChargeAsync(string? rawKey, string customerId, decimal amount)
{
if (string.IsNullOrWhiteSpace(rawKey))
return Results.BadRequest("X-Idempotency-Key header is required.");
// Refuse at the edge what the currency cannot represent, so that nothing
// downstream is ever tempted to round it into place.
if (!Money.IsSupportedPrecision(amount))
return Results.BadRequest("Amount has more precision than the currency supports.");
var key = $"{customerId}:{rawKey}"; // scope per caller
var fingerprint = Fingerprint.Of(customerId, amount);
using var tx = await _store.BeginTransactionAsync();
switch (await _store.TryClaimAsync(tx, key, fingerprint, Ttl))
{
case ClaimResult.Replay(var stored): // ④ REPLAY
tx.Rollback(); // we did nothing
return Results.Content(stored.Body, "application/json",
statusCode: stored.StatusCode);
case ClaimResult.FingerprintMismatch:
tx.Rollback();
return Results.Conflict(
"This idempotency key was already used with different parameters.");
case ClaimResult.Claimed:
break; // we own it — carry on
}
// The claim, the side effect and ③ the stored response commit TOGETHER.
// If TryChargeAsync throws, the `using` rolls all three back, so the
// client's retry gets a genuine second attempt rather than finding a
// half-finished key. That is the whole reason they share a transaction.
var response = await _store.TryChargeAsync(tx, customerId, amount) switch
{
ChargeOutcome.Charged(var paymentId) => new IdempotentResponse(
201, JsonSerializer.Serialize(new { paymentId, amount })),
// A decline is a real answer, so it is stored and committed like any
// other. Retry this request and you must get the same 402 back.
ChargeOutcome.Declined(var reason) => new IdempotentResponse(
402, JsonSerializer.Serialize(new { error = reason })),
_ => throw new InvalidOperationException("Unhandled charge outcome.")
};
await _store.SaveResponseAsync(tx, key, response);
tx.Commit();
return Results.Content(response.Body, "application/json",
statusCode: response.StatusCode);
}
}
Production details: fingerprinting, the SQL claim, and what to do when the side effect isn't yours
The fingerprint. "Same key, different request" is a real case an idempotency store has to catch, and the only hard part is producing a canonical form without losing information:
using System.Globalization;
using System.Security.Cryptography;
using System.Text;
public static class Money
{
// Two decimal places is what this currency supports; anything finer is a
// bad request, not something to quietly round.
public static bool IsSupportedPrecision(decimal amount) =>
decimal.Round(amount, 2) == amount;
// Exact for any amount that passed the check above: decimal arithmetic is
// not binary floating point, so 120.00m * 100m is exactly 12000m.
public static long ToMinorUnits(decimal amount) => (long)(amount * 100m);
}
public static class Fingerprint
{
/// Canonicalise, keep it lossless, THEN hash.
/// · InvariantCulture, so a replica running under a comma-decimal locale
/// doesn't hash "120,00" while its neighbour hashes "120.00";
/// · integer minor units, so 120, 120.0 and 120.00 — the same money —
/// agree, without rounding anything away.
///
/// The trap: formatting with "F2" also makes those three agree, and looks
/// right. It is lossy. 120.001 and 120.004 both become "120.00", so two
/// genuinely different amounts hash identically and the second request
/// replays the first one's response. NEVER round only for the fingerprint —
/// validate the precision first (above), then hash a lossless form.
public static string Of(string customerId, decimal amount)
{
var canonical = string.Create(CultureInfo.InvariantCulture,
$"{customerId}|{Money.ToMinorUnits(amount)}");
return Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(canonical)));
}
}
The atomic claim, in SQL. Two statements, one transaction:
-- ① CHECK + ② CLAIM
INSERT INTO idempotency_keys (key, fingerprint, expires_at)
VALUES (@key, @fingerprint, @expires)
ON CONFLICT (key) DO NOTHING;
-- 1 row inserted => Claimed.
-- 0 rows inserted => somebody else owns it, so go and read it:
SELECT fingerprint, status_code, body
FROM idempotency_keys
WHERE key = @key
FOR UPDATE;
What you cannot do is add RETURNING to the insert and call it a day:
ON CONFLICT DO NOTHING returns no row for the conflicting case, so an empty result would mean
both "I claimed it" and "someone else has it". The FOR UPDATE on the follow-up select is what
makes a concurrent duplicate block until the original commits and then see its response, instead of
racing it.
Keys need a scope and an expiry. Scope to the caller (as above) so two tenants can never collide, and give the row a TTL — Stripe uses 24 hours — or the table grows forever. The TTL is also the honest limit of the guarantee: a key replays its stored response for as long as the record is retained, and no longer.
When the side effect isn't in your database. Everything above works because the charge is a row you control. Call an external provider you cannot enrol in your transaction and the crash window reopens: you need their idempotency key for the outbound call, plus a reconciliation sweep that finds records stuck mid-flight, asks the provider what actually happened, and finishes or fails them. Where you can't block on a row lock at all — as in the M01 lab, whose store is in memory — you need a fourth answer alongside completed/compensated/failed: "still in progress, I don't know yet." Guessing "it failed" there is the worst kind of wrong, because it is confidently wrong.
Why not a generic filter? You will see this written as an IEndpointFilter
wrapping any endpoint. That can only be an approximation: a filter sits outside your handler's transaction, so
its claim, the handler's write and the saved response commit separately, and it has to reconstruct the
response from whatever the handler returned rather than capturing the real status and body. If you write one,
buffer HttpContext.Response to capture it, and pair it with the reconciliation sweep above.
For the exam: check whether the transaction id / key already exists before processing; store it with a status; replay the stored response for duplicates. That is the right four-step story, and it is what the diagram above shows. Three things separate a working implementation from a broken one. (1) The claim must be atomic — check-then-act is a race two duplicates can both win. (2) Replay must return the same response — status code and body, failures included. (3) The line to draw is deterministic vs transient: a declined card is an answer, so commit it and replay it; a timeout is not, so roll back and let the retry try again. The mechanics behind all three — fingerprinting, the SQL claim, TTLs, reconciliation — are in Production details above.
Dead letter queues
Some messages simply cannot be processed. The payload is malformed, it refers to a customer who no longer exists, or a bug throws on every single attempt. Retrying such a poison message forever is worse than useless: it blocks the queue behind it and burns resources on a call that will never succeed.
A dead letter queue is where those messages go after N failed attempts — a holding area, off the main path, where a human can inspect them, fix the cause, and replay them. It is the human-in-the-loop safety valve: the system stops trying to be clever and asks for help instead of silently dropping data.
Deliver a message and watch the retry ladder. A transient failure recovers on its own; a poison message never will, and eventually gets set aside instead of blocking everything behind it.
Deep dive: a consumer with a retry ladder and DLQ, in C#
private const int MaxAttempts = 3;
public async Task ConsumeAsync(QueueMessage message, CancellationToken ct)
{
for (var attempt = 1; attempt <= MaxAttempts; attempt++)
{
try
{
await HandleAsync(message, ct); // must be idempotent — this can run twice
await _queue.AckAsync(message);
return;
}
catch (Exception ex) when (IsTransient(ex) && attempt < MaxAttempts)
{
// Exponential backoff with jitter: 1s, 2s, 4s (± a random slice) so that
// a thousand consumers don't all retry on the same tick.
var delay = TimeSpan.FromSeconds(Math.Pow(2, attempt - 1))
+ TimeSpan.FromMilliseconds(Random.Shared.Next(0, 500));
_logger.LogWarning(ex, "Attempt {Attempt} failed; retrying in {Delay}", attempt, delay);
await Task.Delay(delay, ct);
}
catch (Exception ex)
{
// Either non-transient (a poison message — retrying cannot help),
// or we've exhausted the ladder. Set it aside for a human.
await _queue.DeadLetterAsync(message, reason: ex.Message);
_logger.LogError(ex, "Message {Id} dead-lettered after {Attempt} attempts",
message.Id, attempt);
return;
}
}
}
Note the when (IsTransient(ex) && ...) filter: a malformed
payload should go to the DLQ on attempt one. Retrying a JsonException three times just
wastes nine seconds to reach the same conclusion.
For the exam: if a step fails repeatedly and even the compensation fails, the message goes to a dead letter queue for manual intervention. Correct. What makes a DLQ actually work: a DLQ nobody watches is a data-loss bin with extra steps, so the pattern is only complete with an alert on DLQ depth > 0 and a redrive tool to replay messages once the bug is fixed. Two more practicalities: dead-lettering a message from an ordered stream means every message after it has now overtaken it, so ordering guarantees are gone — your handlers need to cope. And distinguish the two failure classes early (as the code above does): retrying a poison message is pure waste, while not retrying a transient blip turns a two-second outage into lost work.
The three pillars of distributed safety
These three patterns are usually taught separately, and each is perfectly useful alone — an idempotency key protects an ordinary HTTP endpoint with no queue or saga anywhere in sight. But a workflow like this one needs all three, because each covers a failure the others leave open.
Design patterns inside a service
Everything above is architectural — patterns about how services talk to each other. But a microservice is still just a program, and inside it the ordinary GoF patterns from Chapters 3–5 do the work. The slides make the two levels explicit:
| Level | Question it answers | Examples |
|---|---|---|
| Architectural / systems patterns | How do services talk? | API Gateway, Circuit Breaker, Saga, CQRS, Sidecar |
| Code-level patterns | How is the code inside one service structured? | Factory Method, Adapter, Strategy — and the rest of the GoF catalogue |
The running example is a Notification Service that a checkout flow calls. Three patterns, three different jobs.
Creational Factory Method — choosing a driver from configuration
A microservice constantly needs to create a different driver based on configuration or on the request: send this notification by email, or SMS, or push. The calling code should say "give me a notifier for this channel" and know nothing about which class it gets.
Structural Adapter — surviving third-party APIs
Microservices constantly talk to external providers (Stripe, Twilio, MessageBird, AWS) whose APIs are strange, verbose and prone to change. The Adapter wraps each provider so that switching from Twilio to MessageBird means writing one new adapter — your business logic never learns that anything changed.
Twilio and MessageBird (now Bird) are Communications Platform as a Service vendors: they let you embed SMS, voice, video and chat into your application through an API. They do the same job with completely different method names and payload shapes — which is precisely why you want an adapter between them and you.
Behavioral Strategy — changing behaviour at runtime
A pricing service applies a different discount calculation on Black Friday than on a regular Tuesday. The
Strategy pattern makes each rule an interchangeable object
chosen at runtime, instead of an if chain that grows a new branch every promotion.
All three, in one checkout endpoint
Here is the slides' final example, ported to C#. Watch which pattern does what: Strategy decides the price, Factory picks the delivery tool, and an Adapter hides the fact that the SMS tool is a hostile third-party SDK.
// ===== BEHAVIORAL: Strategy — the pricing rule is chosen at runtime =====
public interface IPricingStrategy
{
string Name { get; }
decimal Apply(decimal price);
}
public sealed class RegularPricing : IPricingStrategy
{
public string Name => nameof(RegularPricing);
public decimal Apply(decimal price) => price;
}
public sealed class BlackFridayPricing : IPricingStrategy
{
public string Name => nameof(BlackFridayPricing);
public decimal Apply(decimal price) => price * 0.5m; // 50% off
}
// ===== STRUCTURAL: Adapter — a third-party SDK with an awkward shape =====
// Pretend this comes from a NuGet package you do not control.
public sealed class LegacySmsSdk
{
public string TransmitTextIso8859(string num, string payload)
=> $"SDK sent: {payload} to {num}";
}
// Our own interface — the shape our application actually wants.
public interface INotificationChannel
{
Task<string> NotifyAsync(string recipient, string message);
}
public sealed class SmsAdapter : INotificationChannel
{
private readonly LegacySmsSdk _sdk = new();
// Adapting our simple NotifyAsync call to the SDK's awkward method.
public Task<string> NotifyAsync(string recipient, string message)
=> Task.FromResult(_sdk.TransmitTextIso8859(recipient, message));
}
public sealed class EmailChannel : INotificationChannel
{
public Task<string> NotifyAsync(string recipient, string message)
=> Task.FromResult($"Email delivered to {recipient}: {message}");
}
// ===== CREATIONAL: a factory — pick the driver from configuration =====
public static class NotificationFactory
{
public static INotificationChannel Create(string channel) => channel switch
{
"sms" => new SmsAdapter(),
"email" => new EmailChannel(),
_ => throw new ArgumentException($"Invalid channel '{channel}'.")
};
}
// ===== The endpoint, bringing all three together =====
public static class Checkout
{
public static async Task<string> RunAsync(
decimal basePrice, string customerContact, string channel, bool isSale)
{
// 1. BEHAVIORAL Strategy decides the price.
IPricingStrategy pricing = isSale ? new BlackFridayPricing() : new RegularPricing();
var finalPrice = pricing.Apply(basePrice);
// 2. CREATIONAL factory gets the right tool.
var notifier = NotificationFactory.Create(channel);
// 3. Use the tool — which may well be a STRUCTURAL Adapter.
var message = $"Order confirmed! You paid ${finalPrice:F2}";
var deliveryReport = await notifier.NotifyAsync(customerContact, message);
return $"price_applied={finalPrice:F2}, strategy_used={pricing.Name}, delivery={deliveryReport}";
}
}
public static class Program
{
public static async Task Main()
{
Console.WriteLine(await Checkout.RunAsync(100m, "alice@example.com", "email", isSale: false));
Console.WriteLine(await Checkout.RunAsync(100m, "+94771234567", "sms", isSale: true));
}
}
Set the request parameters and send it. The trace shows which pattern made which decision — and notice that changing the SMS provider changes only the adapter line.
channel = "push". The factory is the
single place that fails — the pricing logic and the endpoint code have no idea a channel was even
added or removed. That is the whole payoff of putting creation behind a factory.For the exam: Factory Method for creating drivers from configuration, Adapter for wrapping
third-party APIs, Strategy for behaviour that changes at runtime — and the slide labels the notification
factory "Factory Method". Strictly, that example is a Simple Factory, not GoF Factory
Method. GoF Factory Method defines a creation method that subclasses override to decide the
concrete type; a static method with a switch over a string is the Simple Factory idiom (see
Chapter 3 and the
Confusable Patterns page for the distinction). Use the lecturer's
framing in the exam — but know the difference, because "which pattern is this really?" is exactly the
kind of thing a viva question probes. One more practical note: in a real ASP.NET service you would usually let
dependency injection do this — register the channels by key and resolve one by name — which is
the same pattern, just delegated to the container.
Chapter 11 quiz
Five architectural patterns, each replacing something the network took away: API Gateway (one front door, fewer round trips), Circuit Breaker (fail fast so failure doesn't cascade), Saga (local transactions plus compensations instead of one ACID transaction), CQRS (separate the write model from the read model), and Sidecar (cross-cutting concerns out of your code and into the platform). Bolt on idempotency and dead letter queues and the whole thing survives contact with an unreliable network. Then, inside each service, it is ordinary object-oriented design again: Factory, Adapter, Strategy and the rest of the catalogue. Practise choosing and justifying these in the Critique & Architecture Studio.