Top A/B Testing Tips from Acumen’s Lean Data Experiments

Posted by in Data & Impact

Lean data

One of the ways Acumen’s Impact Team is working to make Lean Data stronger is through implementing more A/B testing. They are increasingly experimenting with different combinations of tools, questions and techniques to increase response rates, improve data quality, and make the experience more delightful for customers. Below are a few of their recent findings as well as their top A/B testing tips for Lean Data mobile surveys.


The Essentials
Acumen’s Impact Team ran 12 separate quick experiments leading to these results:

  • Tool: IVR (Interactive Voice Response)
  • Provider: EngageSpark
  • Response rate: 59%
  • Country: South Africa
  • Sample size: 959 responses

What made this IVR survey so powerful?

  • Fun, engaging quiz format—including true or false questions
  • Upbeat “radio voice” recording by one of the company’s local brand ambassadors
  • Lottery incentive—”chance to win a Samsung smartphone”
  • In these 12 experiments, they found that:
    • An SMS priming message =  14% higher response rate
    • Local vernacular = 9% higher response rate
    • Time of day = 8% higher response rate

By running multiple, quick A/B tests, we increased the response rates by 32%!

lean data


1. Isolate 2-3 variables that may affect response rates and customer experience. Examples include:

  • Time of day
  • SMS priming
    • IVR example: “Hello [first name]. You will receive a call at 5 pm today with a fun, 1-minute quiz. Pick up & have a chance to win a Samsung smartphone! The call is free!”
    • Phone interview example: “[Company] is collecting feedback about the [product/program]. Please expect a call from us next week. Thank you! SMS STOP to opt out”
  • Incentive: airtime vs. lottery vs. no incentive at all
  • Language
  • Voice: female, male, different accents
  • And of course the questions themselves! Take note if a particular question causes drop-off.

2. Split your sample into the necessary amount of segments to ensure you are only changing one variable at a time.

3. Analyze the results to isolate the best combination of variables.

4. Continue testing!

Want to measure your impact with Lean Data? Sign up for our free course: Lean Data Approaches to Measure Social Impact

Interested in getting the latest Lean Data insights and tips from Acumen’s impact team right in your inbox? Sign up for the Lean Data Lowdown Newsletter here


Subscribe to our Blog
* = required field
No thanks! Don't show me this window again.