A/B Tests Overview

A/B tests let you compare variations of paywalls and App remote config against a Control and measure which performs best. Every A/B test runs on top of an existing Targeting — the test inherits that targeting's audience filters and platforms. You only choose which of its assets to vary.

Find A/B tests in the left sidebar under Growth tools → A/B tests.

How a test works

1. Pick an existing Targeting to define the audience and platforms.
2. Pick assets to vary — placements (with their paywalls) and/or App remote config.
3. Define variations — each variation gets its own paywall per a tested placement and/or its own app remote config JSON. The Control variation always serves what the Targeting already serves.
4. Allocate audience size (share of matching users entered into the test) and variation weights (how those users split across variations).
5. Launch.

Paywalls used as variations are frozen at launch — see Paywall snapshots.

A/B tests list

The list page has five tabs and a search box.

Columns: A/B test name (link to detail) · Status · Targeting · Audience filters · Audience size · Platform · Users · Launched on (sortable).

Click a row to open the test detail page.

Test states

Create a test

Two entry points:

• Growth tools → A/B tests → + New A/B test — opens the targeting picker.
• From a Targeting — opens the picker with that targeting preselected.

Step 1 — Pick a Targeting

The Start from selecting a targeting view lists every Targeting configured for your app.

Each row shows:

• Targeting name (chevron to expand and see the targeting's placements and paywalls).
• Audience filters — the conditions that define the targeting's audience.
• App remote config — green ✓ if the targeting has app remote config configured; otherwise -.

Targetings that already have a Running A/B test are highlighted in purple. Pick one and click Select & Continue. A confirmation modal asks you to confirm — once you start creating the test, the targeting can't be changed.

General information

The below needs to be configured first

• A/B test name (required) — name your test for the list view. Will be used as a filter in analytics.
• Audience size — slider, 0-100%. Default 100%. Use Edit audience size to change it.
• Manage test users — optional. Pin specific user IDs to specific variations for QA. See Manage test users.
• Right-side meta — the Targeting name, audience filters, and platforms (set by the Targeting, not editable here, e.g. "Subscription expired" and "Status: Expired").

Step 2 — Pick assets to vary

Click on Select testing assets button on the banner asking to select a testing object(s).

The Select what you want to test modal asks which of the Targeting's assets your test will vary:

• App remote config — checkbox at the top. Opt in to test different JSON configs delivered to your SDK. Can be tested alone, but without placements you'll only get Users and Overall Proceeds — add placements too if you want per-variation view/purchase metrics.
• Placements & paywalls — pick one or more of the Targeting's placements (each shows its currently-assigned paywall and screen).

Put checkmarks in all objects you'd like to test. Click Select assets. You land on the Draft test detail page with Control variation showing the selected objects.

Step 3 — Configure the Draft

The unsaved Draft detail page has two top-of-page actions:

• Cancel — deletes the traft Draft.
• Preview & create A/B test — proceed to the preview step.

Test variations & assets

• Every A/B test must have the Control variation (locked — it serves whatever the Targeting already serves) and at least one test variation. The minimum is Control plus Variation A — neither can be removed.
• To add more test variations, click + (the add button) at the end of the variation tabs — up to 5 test variations total (Variation A / B / C / D / E).
• To remove an added variation, open its tab and click Delete variation X (under Specify variation) — available from Variation B onward.
• Click a test-variation tab to configure it:
  • Variation name — defaults to "Variation A" / "B" / etc.; rename for clarity.
  • App remote config (if you opted in at Step 2) — JSON editor. Write the config this variation will deliver to the SDK.
  • Per-placement card — one card per placement you chose. The card shows the current paywall and a Configure button to override it for this variation. Configure opens the same paywall editor as in Paywalls, with two tabs — Modify existing (start from a copy of an existing paywall) or Create new (start from scratch). Either way, this never edits the live paywall — it always creates a new paywall scoped to this variation (see Paywall snapshots).

Variation weights

Distribute traffic across Control + test variations using whole-number ratios (e.g., 1:1 = 50/50, 1:2 = ~33/66). The UI shows both the ratio inputs and the resulting percentage bar.

Editable any time before launch — and also after launch (Running tests).

Step 4 — Review & launch

Once all variations are configured, click Review & launch test. The Preview A/B test testing assets page shows the final state:

• Per-variation rows with weight, paywall, products, and JSON ✓.
• A Placements not used in A/B test callout (if any) — placements that belong to the Targeting but weren't included in any test variation.

Click Create A/B test to save the Draft.

Launch A/B test

The saved Draft test page has three top-of-page options:

• Actions — allows to duplicate or delete the draft.
• Save & return - allows to adjust and save the draft
• Review & launch test — moves from Draft to Running and starts allocating users.

Audience size

The Audience size slider sets the share of matching users that enter the test.

• 100% — every user matching the Targeting is allocated to one of the variations.
• 0% — no new users are allocated. Users already in the test stay in their variation.
• Intermediate (e.g., 50%) — proportional allocation among new users matching the Targeting.

Users not allocated to this test (because audience size < 100% or because they don't match the Targeting) follow the normal Targeting rank and fall into the next matching Targeting.

Audience size is editable on a Running test.

Variation weights

Variation weights split the entered users across Control + test variations.

• Weights are entered as ratios (whole numbers). Control 1 · Variation A 1 = 50/50. Control 1 · Variation A 2 = ~33/66.
• The UI shows both the ratio inputs and the resulting percentages.

Weights are editable on a Running test.

Testing mode — verify on sandbox users

Before launching, run the test on specific sandbox users to check it works — we recommend verifying both the paywall visuals and the purchase flow. In Testing mode, only the sandbox IDs you list enter the test; production users are unaffected.

Switch a Draft into Testing mode

1. Open the Draft test. In General information, click Run test on selected user IDs.
2. In the User-specific test variations dialog, add each sandbox User ID and pick the Variation it should be shown — a test variation (Variation A / B / …) or Control variation. Use Add user ID for more rows, Clear all to reset.
3. Click Switch to Testing.

The test moves to Testing status — shown that way both in the A/B tests list and on its Targeting in Mission Control. A banner confirms the test is in Testing mode; you can still edit the test and see which variation each test user gets in the app.

• Only the listed sandbox IDs enter the test, each in the variation you assigned.
• Production users and unlisted sandboxes don't enter it. Listed users bypass audience-size and weight allocation.

Update the list while testing

While in Testing mode you can add more sandboxes — click View testing users, add IDs via Add user ID, and Save the list (new IDs take effect after saving).

Exit Testing mode

When done, open the dialog and click Cancel testing. A confirmation appears — "All testing user IDs will be removed, the A/B test will be switched to Draft." Confirm, and the test returns to Draft. From Draft, click Review & launch test to go live: the test becomes Running and moves to the top of the Targeting priority in Mission Control (see Test priority).

Paywall snapshots

Paywalls used as test variations are frozen copies — made when you create the test and rebuilt each time you edit the draft, but not refreshed at launch. The test keeps its own snapshot of each variation's paywall.

• Editing the original paywall in Product Hub doesn't change what the test serves.
• Deleting the original paywall doesn't affect the test — variations keep serving the snapshot.
• Product set, product order, and Custom JSON are frozen.

The Control variation always uses the live paywall that the Targeting currently serves — it's not a snapshot. Editing the Targeting's default paywall mid-test will update what Control variation users see.

Paywall Screens render the frozen product set but pull the live UI from Figma. You're responsible for keeping the Figma file unchanged during the test.

App remote config

App remote config is a JSON parameter object delivered to the SDK alongside paywalls. You can A/B test it as a separate asset.

In a test variation, write the JSON the SDK will receive for users assigned to that variation. Your app code reads the values and varies behavior — onboarding length, feature flags, copy strings, anything you map.

• The Control variation uses the Targeting's current app remote config (or no config if none is set).
• Each test variation uses its own JSON defined in the variation tab.

Several tests on one Targeting

You can run several A/B tests on one Targeting at once — for example, one test on the onboarding paywall and another on the freemium paywall — and adjust or complete each test on its own. There is no limit on the number of tests running on one Targeting at the same time, as long as they don't overlap. Parallel tests work with any SDK version.

• Tests can't overlap. A placement can be tested by only one Running or Testing test at a time, and only one of them can test App remote config. Otherwise launching is blocked — see Overlapping tests are blocked.
• Users join every test they qualify for. Each test has its own audience size, so a user can end up in some tests and not in others.
• The app receives one merged set of placements. Each test the user joined supplies only the placements it tests, taken from the user's variation. All other placements come from the Targeting as it is, and no placement appears twice. A user who isn't in any test gets the Targeting as it is.
• Views and purchases count for the right test. Each paywall view and each purchase is credited only to the test that owns the placement where the paywall was shown.
• Mission Control shows every test. On the Audience targeting tab, each placement of a Targeting shows the A/B test that tests it. Hover the test name to see when it was launched, its audience size and how many users it has affected; click it to open the test in a new browser tab.
• Rolling out a winner doesn't stop the other tests. It is blocked if another running test tests one of the placements you roll out, or the App remote config that the winner changes.
• A placement that is tested by a Running or Testing test can't be deleted.

Test priority and user reallocation

You can run multiple A/B tests at the same time — on different Targetings, or on one Targeting if they test different placements. When tests run on different Targetings, the SDK picks the variation for a given user based on Targeting rank.

• When a new A/B test is launched on a Targeting, that Targeting moves to the top of the rank. The most recently launched test always wins for overlapping audiences.
• When the test is completed, the Targeting returns to its original rank position.

If a user is currently in test A1 on a broader Targeting and a higher-priority test A2 is launched on a narrower Targeting they also match, the user is reallocated to A2 immediately. Revenue and events that already happened in A1 stay attributed to A1; everything new is attributed to A2.

If A2 is later completed and A1 is still Running, the user returns to A1.

Complete and evaluate

When you're confident in a variation, click Complete A/B test on the Running test detail page.

The Complete modal shows preliminary indicators per variation: View to Purchase, Trial Conversion, ARPU, ARPPU. Each variation also shows Effect (% change vs Control) and P-value (statistical significance). For metric definitions and significance rules, see Analyzing Experiments.

You must check I'm sure I want to complete the A/B test to confirm. The button stays disabled otherwise.

After completion:

• Users who were in the test stay assigned to their variation — they keep seeing the variation's paywall and their revenue keeps flowing to the test.
• Users newly matching the Targeting follow the normal Targeting rank — they go to the next matching Targeting.
• The Targeting returns to its original rank position (see Test priority).

Roll out the winning variation

Once a Completed test has a clear winner, you can apply that variation's assets to the test's Targeting permanently. From then on, every user matching the Targeting gets the winning paywall(s) and App remote config as the new default.

Available on Completed tests only. To roll out, you must first Complete the test.

How to roll out

1. Open the Completed test's detail page.

2. Scroll to Experiment analytics. The default view shows the Control variation tab. While on Control, you'll see a hint:

   Select your winner variation to roll out it's results to the targeting.

3. Click the test variation you want to roll out (Variation A / B / etc.). A blue banner appears:

   Roll out the "Variation X" configuration to the targeting. Once rolled out, these placements and app config will always be shown to this targeting.

   On the right of the banner: Roll out variation X button.

4. Click the button. The Roll out Variation X modal opens, listing every tested placement in a table: Placement, Current paywall, Winner paywall, How to roll out, and Identifier if need to create.

   For each placement, pick how its winning paywall is applied:

   • Update paywall content — overwrites the placement's current paywall content in place, keeping the same paywall id and identifier.

     🚧

     This updates the paywall everywhere it's used

     If the placement's current paywall is also used by other placements or schemes, they get the new content too.

   • Replace entire paywall (shown as Create new paywall when the variation's paywall was built from scratch with Create new) — puts a whole paywall on the placement instead of editing the current one. What happens depends on whether you changed the variation:

     • Unchanged copy — you picked an existing paywall under Modify existing and changed nothing. The placement is switched to that existing paywall. No new paywall is created, and Identifier if need to create is ignored. For example, if the placement currently uses paywall X and the variation is an unchanged copy of paywall Y, after roll-out the placement uses Y.
     • Changed copy or built from scratch — Apphud creates a new paywall with the variation's content. Identifier if need to create is pre-filled with a suggested ID built from the placement and the experiment's ID. Keep it or type your own: it must be unique among the app's paywalls, up to 120 characters, letters, numbers, dash and underscore only.

     "Unchanged" means an exact match with the original paywall as it is at roll-out time, not just a screen that looks the same: the screen, Custom JSON, other paywall settings, and the products with their order and per-product settings must all be identical. Changing any of them, even only the product order, makes it a changed copy.

   🚧

   Paywall-only apps: don't change the default selection

   If your app doesn't use placements — it's running the paywall-only experiment mode described in Step 2 above — Apphud already pre-selects Update paywall content for every eligible placement here, since Replace entire paywall would create a new identifier your app never reads. This lowers the risk of a mistake, but doesn't rule it out — don't switch a row to Replace entire paywall unless you know what you're doing.

   ❗

   Conflicting replacements

   If two placements in the variation share the same current paywall and both are set to Update paywall content with different winning content, roll-out is blocked with a "conflicts with another replacement for the same paywall" error. Set one of them to Create new paywall instead, or roll out separately.

   Click Roll out in the modal to confirm your choices.

5. A final confirmation dialog opens:

   Roll out the "Variation X" configuration to the targeting? Placements and app remote config from "Variation X" will be applied to "{targeting name}" targeting.

   This action cannot be reverted.

6. Click Roll out to confirm. The variation's paywall(s) — applied per the mode chosen for each placement — and App remote config replace the Targeting's current configuration for the corresponding assets.

What roll-out does

• Every placement in the variation updates the Targeting's paywall for that placement: its content is overwritten in place, the placement is switched to an existing paywall, or a new paywall is created — see the modes in How to roll out above.

• The variation's App remote config (if any) becomes the Targeting's App remote config.

• The change is irreversible — undo means manually re-configuring the Targeting back to the old assets via Mission Control → Targetings.

• After roll-out, the variation tab shows a confirmation badge:

  Variation X assets have been applied to the targeting.

  The Audience size area also shows: "You've applied assets from Variation X to this audience."

Duplicate a test

Available on Running and Completed tests via Duplicate A/B test.

Duplicate opens the targeting picker again — you must pick a Targeting (you can re-pick the same one).

• Control variation uses the new Targeting's current setup for the tested assets (paywall per placement, app remote config).
• Test variations carry over from the source test — paywalls, products, Custom JSON, app remote config.
• Paywalls in test variations are snapshotted again at the moment you Run the duplicate.

Using A/B tests in Observer Mode

You can use A/B tests on paywalls and App remote config even in Observer Mode. If you don't use Apphud SDK to make purchases, you need to tell Apphud which paywall (and optionally which placement) was used when the purchase happened — otherwise the SDK can't attribute the purchase to the right test / variation.

iOS — call willPurchaseProductFrom before the purchase:

Apphud.willPurchaseProductFrom(paywallIdentifier: "paywallID", placementIdentifier: "placementID")
YourClass.purchase(product) { result in
    ...
}

Android — call trackPurchase after a successful purchase, passing both identifiers:

Apphud.trackPurchase(
    purchase: Purchase,
    productDetails: ProductDetails,
    offerIdToken: String?,
    paywallIdentifier: String? = null,
    placementIdentifier: String? = null
)

FAQ

Apphud.setUserProperty(key: .init("first_paywall_closed"), value: true)
Apphud.forceFlushUserProperties { _ in
    Apphud.refreshUserData { _ in
        Apphud.fetchPlacements { placements, _ in
            // placements are refreshed
        }
    }
}