Article

We showed the offer to one player in twenty and sold the same number of packs

A scoring model picked who saw the starter pack. A randomised control measured what that was worth.

Causal A/B · 64,302 decisions · 18 days

A live mobile fighting game was showing its starter pack to everyone who crossed a threshold. There was no control group, so nobody could say whether the offer was doing the work, or whether the players who bought would have bought anyway.

We scored each player on their first thirty minutes of behaviour, showed the pack only to the players the model flagged, and held back a randomised control.

What came back

8.7×

conversion per offer shown, against a randomised control

Conversion went from 1.0% to 8.7%. If the model made no difference, a gap that size would turn up by chance less than once in a million tests. Both arms of the test produced it independently, at 8.0× and 9.4×.

The second number matters more to whoever runs the game day to day.

95%

fewer interruptions, and the same number of packs sold

Nineteen players in twenty stopped seeing an offer they were never going to take, and the studio sold what it sold before. Half the buyers came from the model’s top 10%.

What did not move

Retention, sessions, playtime and average revenue per user (ARPU) were all flat. Cutting 95% of your offer impressions can easily cost you something somewhere else, so those four flat lines are part of the result rather than a footnote to it.

How the impressions were counted

The game’s telemetry had no popup-impression event. An impression here means the player was eligible and flagged, not a confirmed rendered popup. The conversion result is measured against a randomised control and does not depend on that stand-in. The 95% figure does. If you take one thing from this to your analyst, take that.

Throughput

64,302 individual decisions across 18 days, one every 24 seconds
about 30 minutes from install to a scored decision
2,215 players re-scored mid-session as their behaviour changed

Nobody read a weekly report and decided who should see the pack. The model ranked, the offer fired, the control stayed clean, and the answer was readable in eighteen days.

← All articles