Architecture 4

A feature flag can split traffic, but that does not automatically create an experiment.

Rollout asks whether the change is safe

new feature → 10%
observe health
expand if acceptable

Experiment asks which variation performs better

eligible population
      │
   randomize
   /         A         B
  │         │
measure outcomes
      │
statistical analysis

The experiment needs a defined population, stable assignment, trustworthy exposure data, metrics and an analysis method.

Exposure is not the same as assignment

A user may be assigned variation B but never reach the screen where B appears. Counting assignment as exposure can dilute the measured effect. Experiment platforms therefore pay careful attention to when the treatment was actually experienced.

Define the primary metric in advance

An experiment with twenty metrics and no primary outcome makes it easy to find something that moved by chance. Decide what success means before looking at results, then use guardrails to ensure a positive primary outcome is not purchased with unacceptable harm elsewhere.

Stable identity matters

If the same person moves between A and B because their targeting key changes, the comparison becomes harder to interpret. Choose the experimental unit deliberately: user, account, device, session, request, or another entity.

Not every change should be an experiment

Security fixes, legal requirements, obvious bug fixes and infrastructure changes with no meaningful product alternative may need controlled rollout but not A/B experimentation.

Feature management and experimentation can be separate platforms

Some organizations use one system to allocate variations and another warehouse or analytics platform to analyze outcomes. Others choose integrated products such as LaunchDarkly, Harness FME, Datadog or experimentation focused tools.

The architecture should make exposure data consistent no matter where the analysis occurs.

Feature Flags series

← Feature Flags, Progressive Delivery, and Observability   Feature Flags in the AI Era: Models, Prompts, and Agents →