Skip to main content
Version: Next

Setting up A/B Tests

This guide outlines how to run an application-level A/B test by running two distinct versions of your application simultaneously. Codesphere’s load balancer handles the traffic distribution natively, ensuring users are assigned to a version and stay there.

Environment Isolation​

To perform a test, you must isolate your "Control" (current production) from your "Variant" (the experiment).

  1. Deploy the Control (v1):
    • Deploy your current stable, production-ready code to a Landscape in Codesphere.
    • Name this Landscape production-control.
  2. Deploy the Variant (v2):
    • Create a duplicate Landscape (or clone the first one).
    • Name this Landscape production-variant.
    • Push your experimental code (new features, architectural changes, etc.) to this specific Landscape.
Why this matters

Because these are separate containers, you can test backend changes (like a new API structure or database query).

Analytics Instrumentation (PostHog Example)​

Since both Landscapes will eventually live on the same public URL, your analytics tool will not naturally know which version a user is seeing. You must explicitly tag the user session using "Super Properties" (or "User Traits").

In the Control Landscape Code​

Locate your analytics initialization script (usually in the header or a config file) and add the following immediately after initialization:

// App v1 (Control)
posthog.register({
    experiment_group: "Control"
});

In the Variant Landscape Code​

In the experimental codebase, update the snippet to reflect the new version:

// App v2 (Variant)
posthog.register({
    experiment_group: "Variant-A"
});
How it works

The posthog.register function attaches the experiment_group property to every subsequent event (pageview, click, purchase) sent by that user. You do not need to manually tag individual events across your codebase.

Traffic Routing​

Now that you have two running apps and analytics set up, you need to route real traffic to them. Codesphere handles this natively without requiring external tools like Nginx or HAProxy.

Configure the Domain​

  1. Navigate to the Domains tab in Codesphere.
  2. Select your public custom domain (e.g., app.example.com).
  3. Attach both landscapes:
    • production-control
    • production-variant
Result

Traffic Split
Codesphere automatically distributes incoming requests 50/50 between the two landscapes.

Session Stickiness
The load balancer sets a Sticky Session Cookie on the user's first visit.
If a user is routed to the Variant, they remain locked to that landscape for their entire session, ensuring they do not switch versions on reload.

Verification​

Before announcing the update, verify that the split and tagging are working correctly.

1. Open an Incognito Window​

Navigate to your public domain.

2. Inspect the Console​

Open your browser’s developer tools (F12) and run:

posthog.get_property('experiment_group')
Expected Result

The command should return either "Control" or "Variant-A".

3. Check Live Events​

Go to:

PostHog → Live Events

Ensure that the incoming events from your session include the experiment_group property.

4. Test Routing​

  • Close the Incognito window.
  • Open a new one.
  • Repeat multiple times.
Success Criteria

Eventually, the load balancer should route you to the opposing version, confirming that both environments are accessible.

Analysis & Decision​

Once traffic is flowing, analyze the data by segmentation rather than complex mathematics.

The Analysis​

  1. Open your dashboard (e.g., PostHog Insights).
  2. Create a trend graph or funnel (e.g., "User Signups").
  3. Apply a Breakdown by the property:
experiment_group
  1. Compare the results.

You will see two distinct lines:

  • Control
  • Variant-A

The Rollout​

Based on the results, execute one of the following scenarios:

Scenario A: The Variant Wins​

(Higher conversion or better performance)

  1. Go to the Codesphere Domains tab.
  2. Remove production-control from the domain.
    → 100% of traffic now flows to production-variant.
  3. Decommission the old Control landscape.

Scenario B: The Variant Fails​

(Bugs or lower conversion)

  1. Go to the Codesphere Domains tab.
  2. Remove production-variant from the domain.
    → 100% of traffic reverts to the stable production-control.
  3. Decommission the Variant landscape and investigate the code offline.