You spent three weekends on the update. It's live. RevenueCat's chart wiggles the way it always wiggles, and you cannot tell whether the release did anything. The problem is the question, not the tooling: "is revenue up since Tuesday?" isn't answerable, but "do users on 2.4 pay more than users on 2.3?" is.
Why the calendar comparison lies
The obvious move (revenue in the 7 days after release vs the 7 days before) fails for four separate reasons, each big enough to swamp a real effect at indie scale:
- Phased release means "after" isn't the new version. If you left phased release on (you should), the rollout reaches automatic updaters over seven days on Apple's fixed ramp — roughly 1, 2, 5, 10, 20, 50, then 100 percent. For the first half-week, most of your "after" users are still running the old build.
- Adoption lags the rollout. Even at 100%, updates install when devices are plugged in, on Wi-Fi, and asleep. Real version mix shifts over one to two weeks; day 3 always looks like nothing happened.
- Trials convert on a delay. A 7-day trial started by a new-version user becomes revenue a week later. Money impact shows up one trial-length after behavior impact, minimum.
- The window absorbs everything else from that week. Weekday vs weekend mix, a featuring placement, a keyword-rank move, a promo you forgot — calendar windows include all of it and hand you a confident wrong answer.
Anchor on version, not date
The app version your analytics SDK stamps on every event is the honest anchor. With behavior and purchases joined on one user record (the one-ID pattern again; everything downstream assumes it), "did the update work?" becomes two concrete comparisons:
Cut 1: new users, per version
Users whose first session ran on 2.4 vs users whose first session ran on 2.3, measured over windows of equal length. Compare trial-start rate, trial-to-paid rate, and revenue per user. This is the cleanest read because each user only ever experienced one version — no contamination from people who saw both. It's the release-over-release equivalent of the completed-vs-skipped cut from measuring onboarding against paid conversion, and it's the number that should decide whether the change stays.
Cut 2: existing users, before and after updating
For changes aimed at your installed base (a redesigned paywall, a new feature behind the subscription), watch users who updated: their session length, feature usage, and conversion after adopting the new version vs their own baseline before. Noisier than cut 1 (the same person, different weeks), but it's the only cut that can see re-engagement effects.
The timing rule
Stack the lags and you get a practical schedule for a subscription app with a 7-day trial:
| Days since release | What you can safely read |
|---|---|
| 0–7 | Crashes, ratings, obvious breakage. Not revenue. |
| 7–14 | Behavior shifts: trial starts, onboarding completion, feature usage per version. Directional only. |
| 14–21+ | First honest money read: trial-to-paid per version cohort, revenue per new user. This is when to decide. |
Painful but true: if you ship something new every week, no individual release ever gets a clean money read at indie volume. Either batch the changes you believe move revenue into fewer deliberate releases, or accept reading direction across several releases at once.
Match the metric to the change
"Did it make money" resolves differently depending on what you shipped, and picking the wrong primary metric is how good releases get rolled back:
- Paywall or onboarding changes: trial-start rate per new-user cohort, then trial-to-paid one trial-length later. Fastest feedback of the bunch.
- New features behind the subscription: conversion among existing free users after updating, and cancellation rate among subscribers. Slow: the effect compounds over weeks.
- Quality-of-life and bug fixes: retention per version cohort, not revenue. The money effect is real but arrives through renewals, months out — judging a stability release on week-two revenue guarantees disappointment.
One escape hatch worth knowing while you watch the early numbers: a phased release can be paused from App Store Connect if something looks broken. Users who already got the build keep it, but the ramp stops while you decide — you get days to diagnose instead of a same-day forced rollback.
Small-sample honesty, again
- ~50 users per bucket before acting. Twenty new users per version means one large family on a shared promo code decides your "result."
- Equal windows, matching weekdays. Compare 2.4's first two full weeks against 2.3's first two full weeks, not against whatever remained of 2.3's lifetime.
- Throw out polluted weeks. Price change, featuring, promo, a viral moment — any of these voids the comparison. Annotate them somewhere permanent; you will not remember in October.
- Gross revenue for the decision, proceeds for the books. Judge releases on what customers paid (RevenueCat), not on Apple's payout, which arrives smaller and a month later — the three money numbers are different tools.
- Exclude yourself. Your own devices run the new build for a week before anyone else and fire far more events per session than a typical user. Internal-user filtering isn't optional at indie scale.
What a real answer looks like
A hypothetical with honest shapes: 2.4 reworks the paywall. Three weeks post-release you look at first-session cohorts — 2.3's window had 240 new users, 19 trials, 8 paid; 2.4's had 235 new users, 31 trials, 13 paid. Trial rate moved from 7.9% to 13.2% on samples big enough to mean something; trial-to-paid held steady, so the gain is real reach, not junk trials. That's a keeper. The same percentages on 25 users per side would be statistically indistinguishable from no change at all.
FAQ
How long after a release can I judge revenue impact?
Phased rollout (7 days) plus adoption ramp plus one trial length: for a weekly-trial subscription app, two to three weeks is the earliest read worth acting on. Behavior signals arrive about a week sooner than money signals.
Should I compare before/after the release date?
No. Both sides of the line contain a mix of versions, and the window inherits every store-side event of that week. Version-anchored cohorts are barely more work and answer the actual question.
What if I ship weekly?
Then per-release verdicts mostly won't clear the noise floor. Batch the revenue-relevant changes, or read trends across releases and reserve per-release judgment for the big swings.
Downloads went up, revenue didn't. Why?
Downloads respond to store-side causes (featuring, keywords, icon), while revenue responds to conversion and retention. Check per-user conversion by version: if it's flat, the update changed acquisition or nothing, and the download spike has a different explanation than your release notes.