← All patterns
Messaging · Architecture pattern

Pipeline · Pipes and filters

Compose transformations as stages with explicit input, output and recovery boundaries.

The problem

A media processor bundles validation, transformation and publication into one operation, so any failure repeats all the work.

When it fits

Processing can be decomposed into well-defined stages that can be tested, retried or scaled separately.

The whole architecture

Architecture diagram

Independent stages transform an input into a published output, with retained intermediate work and a quarantine path.

Blue arrows are stage handoffs; purple arrows carry rejected or failed work.

100%

Scroll to pan. Select a component for its role.

pipeline architectureIndependent stages transform an input into a published output, with retained intermediate work and a quarantine path. Blue arrows are stage handoffs; purple arrows carry rejected or failed work.Processing stagesRecovery pathsinput → validate: inputvalidate → transform: valid worktransform → publish: complete outputpublish → output: publishtransform → checkpoint: retain progressvalidate → quarantine: invalid inputtransform → quarantine: failed workinputvalid workcomplete outputpublishretain progressinvalid inputfailed workA stable input identity makes stage retries easier to reconcile.Input sourceStable input identityReject unsupported inputs before expensive processing.Validate stageCheck input contractA retry should reuse or recognize already-completed work.Transform stageVersioned outputPublish only after required outputs are available.Publish stageExpose complete resultThe published representation references verified outputs.Output storeCompleted resultRetained stage results avoid repeating successful earlier processing.Stage checkpointsCompleted intermediatesA poison input needs a recovery policy instead of unlimited retries.QuarantineBounded retry / reviewFaster upstream stages can increase a downstream backlog.Flow controlBounded buffers, backlog age and slowest-stage capacity

Select a component, or focus it and press Enter, to read its role.

Walk through the design

Follow one operation

Select a step to highlight its participants. This conceptual walkthrough is separate from challenge scoring.

Arrows show data moving between stages. This simple sequence is a subset of the DAG used in video practice.

InputActive in this stepValidateActive in this stepTransformWaitingPublishWaiting
Step 1 of 3

Validate an input

The first stage accepts a known input format or rejects it with a clear outcome.

Trade-offs to make explicit

  • Stage boundaries improve reuse and recovery but add serialization, coordination and observability work.
  • Throughput is constrained by the slowest stage. Buffers need bounds; extra upstream capacity can increase backlog.

Failure modes to plan for

  • A stage can commit an output before a crash prevents acknowledgment, so retry must recognize completed output.
  • A poison input needs a bounded retry or quarantine policy. Publishing partial output can mislead downstream stages.
Before you build

Explain your decision

Which outputs can survive a retry, and how will an overloaded stage signal backpressure?

Use Plan & estimate inside practice to record your assumptions and trade-offs.

Connect concept to practice

Related challenges

Browse challenges →

These links open the original challenges. Existing round unlocks, completions and points apply.

Upload once, publish when ready

Practice stage dependencies, checkpointed retries and playable publication.

Open challenge →

Finish the nightly audience report

Practice bounded processing cost, memory pressure and skew.

Open challenge →

Further reading

Microsoft · Pipes and filters ↗

Original PractiseGrid explanations and conceptual diagrams, with primary references for further study. Dedicated scored exercises can be added as coverage grows.