E-commerce A/B Test Experiments
Disciplined conversion experiments across seven D2C brands, from wins to instructive failures.
- Brands tested: KettlebellKings, Flygrubs, Silvon, RDN, Baby Merlin, TribeWod, HappyPo
- Testing tools: Convert, Shogun, Google Optimize
- Period: 2022–2025
Testing philosophy
Not every test wins. In fact, roughly 40% of our tests failed. But that's the point.
We ran disciplined experiments with clear hypotheses, measured rigorously, and learned from both wins and losses.
Process
- Identify opportunity via analytics (high traffic, low conversion pages)
- Form hypothesis based on competitive research or user behavior data
- Design variant with specific change isolated
- Run until statistical significance (minimum 2 weeks, 1000+ actions)
- Document results regardless of outcome
- Implement winners, learn from losers
Aggregate impact
- Tests run: 50+
- Win rate: ~60%
- Avg conversion lift on winners: 15–25%
- Avg conversion improvement across the portfolio: 25%
Selected experiments and results
Test 1: Flygrubs Collection Page — Subscription Highlighting
Brand: Flygrubs (Chicken feed, US market). Page: Collection page.
Hypothesis: Highlighting subscription benefits and bestseller products on the collection page will increase conversion and subscription sign-ups.
- Added subscription benefit callouts above product grid
- Redesigned UI to highlight bestseller products (recommended size)
- Clearer subscription pricing comparison
| Metric | Control | Variant | Change |
|---|---|---|---|
| Conversion rate | 6.53% | 7.68% | +17.5% |
| Add to cart rate | 13.80% | 15.03% | +9% |
| Avg session duration | 4:32 | 4:41 | +3% |
| Odds to win | – | 95% | – |
| Metric | Control | Variant | Change |
|---|---|---|---|
| Conversion rate | 10.75% | 12.38% | +15.2% |
| AOV | $43.94 | $46.58 | +6% |
| Odds to win | – | 97% | – |
Outcome: Winner. Implemented site-wide.
Learning: Customers needed clearer subscription value props at the browsing stage, not just on the PDP.
Test 2: KettlebellKings Theme & PDP Upgrade
Brand: KettlebellKings (Fitness equipment, US market). Page: Site-wide theme + product detail pages.
Hypothesis: Upgrading to the latest theme version and adding A+ content on PDPs will improve engagement and conversion.
- Upgraded entire theme codebase to latest Shopify version
- Added A+ content sections on all PDPs
- Improved image galleries
- Better mobile responsiveness
| Metric | Before | After | Change |
|---|---|---|---|
| Overall CVR | 1.97% | 2.48% | +25.8% |
| Sessions with PDP views | 42.4% | 53.8% | +26.8% |
Outcome: Winner. Major conversion lift from combined improvements.
Learning: Sometimes you need a bigger swing (a full theme upgrade) rather than incremental tweaks.
Test 3: RDN Hero Section Redesign
Brand: RDN (Home decor, US market). Page: Homepage hero section.
Hypothesis: A custom hero section based on competitive research will increase product discovery and engagement.
- Redesigned hero with category-focused navigation
- Added lifestyle imagery
- Clearer value proposition
- Improved visual hierarchy
| Metric | Change |
|---|---|
| Product views | +30.88% |
| Engagement | +20.36% |
| Bounce rate | +8.13% (worse) |
| Metric | Change |
|---|---|
| Product views | +0.67% |
| Engagement | +1.49% |
| Bounce rate | −0.80% (better) |
Outcome: Partial win. Mobile showed a significant product view lift; desktop was flat. We implemented for mobile and iterated on desktop.
Learning: Mobile and desktop users behave differently. Sometimes you need device-specific solutions.
Test 4: Silvon Homepage Improvements
Brand: Silvon (Bedding, US market). Page: Homepage.
Hypothesis: A new homepage template with better information hierarchy and competitive positioning will improve revenue.
- Redesigned homepage layout based on GA scroll data
- Added social proof sections
- Improved product discovery flow
- Better trust signals right in the first scroll
| Metric | Change |
|---|---|
| Conversion rate | +34.40% |
| Scroll depth | +6.78% |
| Bounce rate | +1.31% (slightly worse) |
| Metric | Change |
|---|---|
| Scroll depth | +23.92% |
| Avg session duration | +4.84% |
| Conversion rate | −13.64% |
Outcome: Split decision. Desktop revenue significantly up; mobile engagement up but revenue down. Implemented the desktop version and continued iterating on mobile.
Learning: Longer sessions don't always mean more purchases. Mobile users may have been overwhelmed by additional content.
Test 5: KettlebellKings Social Proof — Video on Homepage & PDP
Brand: KettlebellKings (Fitness equipment, US market). Page: Homepage and product pages.
Hypothesis: Adding customer videos will increase social proof, engagement, and conversion.
- Added customer video testimonials on homepage
- Added product-in-use videos on PDPs
- Embedded video player widgets
| Metric | Change |
|---|---|
| Engagement | +7.19% |
| Avg session duration | +7.19% |
| Bounce rate | −0.08% (slightly better) |
| Revenue goal | −1.81% |
| Metric | Change |
|---|---|
| Engagement | −5.07% |
| Avg session duration | −0.53% |
| Revenue goal | −1.92% |
Outcome: Failed. Engagement improved on mobile but didn't translate to revenue. Desktop was worse across the board.
Learning: Video content can increase time on site without improving conversion. Users may have been distracted rather than persuaded. We reverted and focused on static social proof instead.
Test 6: Baby Merlin PDP A+ Content
Brand: Baby Merlin (Baby products, US market). Page: Product detail page.
Hypothesis: Adding UGC images, testimonials, and detailed benefits below the fold will improve engagement and conversion.
- Added A+ content section with UGC
- Included detailed product benefits
- Added customer testimonials
- More lifestyle imagery
| Metric | Desktop | Mobile |
|---|---|---|
| CVR | −10.5% | −2.47% |
| ATC rate | −14.7% | −14.4% |
| Session duration | −10.3% | −13.4% |
Outcome: Failed. All metrics declined.
Learning: More content isn't always better. For Baby Merlin's audience of new parents, the original concise PDP was more effective: they wanted quick answers, not more information to process.
Key learnings
- Mobile and desktop need separate strategies: Results often diverged significantly between devices.
- More content can hurt conversion: Especially for audiences seeking quick decisions.
- Engagement ≠ revenue: Longer sessions and more scroll depth don't automatically mean more purchases.
- A ~60% win rate is healthy: If you're winning 90%+ of tests, you're not being ambitious enough.
- Compound small wins: 3% here, 5% there, 15% there — it adds up to 25%+ over time.