Experiments

EID 01 – Change colour of CTA

Hypothesis

If this experiment is successful, I predict the conversion rate from views to signups on our home page will increase by 10% for the following reasons:

  • The current color (yellow) doesn’t resonate with our brand colors (although it provides a good contrast with the page’s background)
  • I received anecdotal data from a friend who ran this experiment with a similar product where this increased their conversion rates.

Experiment Design

  1. Create two variations of the home page.  
    1. Variation a – Blue CTA
    2. Variation b – Green CTA
  2. Test against the Control (current home page) by evenly splitting traffic between the three.  
  3. Test for minimum of 50 conversions

Resource Estimation and Probability

Marketing Design Engineering Probability
0.5 Day 0 Day 0.2 Days High


Results

Variation Views Signups Conversion Rate
a 1034 204 19.7%
b 1058 312 29.4%
Control 1022 189 18.5%

 

Learnings

  • Users respond to Green CTA button much better than other colors. Possible reasons are that green resonates with our branding and it is a good contrast with the page’s background.
  • Although the color blue gels well together with our branding it doesn’t provide a good contrast and the button can be missed by users.

 

Action Items

  • Productize the winning variation (Green CTA)
Read more...

EID 02 – Ask users for reviews frequently

Hypothesis

If this experiment is successful, I predict:

  • Our app’s ranking will go up a few notches – at least 10 places 30 days after implementation
  • and that the conversion rate from views to installs on our app store page will increase by 15

Experiment Design

  1. We are currently asking user for reviews only once after they complete their first transaction
  2. We should ask users for reviews on all occasions they could feel happy while using our product
    • Send user first request for review after:
      • 10 app opens
      • at least one event where the user is rewarded or has purchased an item
      • During weekdays only (between 5 pm and 9 pm)
    • Send the second review after:
      • at least 25 app opens
      • at least three events where the user buys an item or a combination of events where the user is rewarded
      • and at least 2 weeks after the first review
      • and during weekends(time 8 am to 11 am or 9 pm to 12 am)
    • Send user the third review after:
      • at least 150 app opens
      • at least ten purchases or a combination of events where the user is rewarded a total of more than $25
      • and at least 1 month after the second review
      • during the weekday (if time is between 5 pm and 9 pm) or on weekends (time 8 am to 11 am)

 

Resource Estimation and Probability

Marketing Design Engineering Probability
0.25 Day 0 Day 0.1 Days High


Results

When Ranking Views Installs Reviews Install/View Review/Install
Before 44 16300 1170 50 7%  4%
After 26 18330 1560 90 9% 6%

Learnings

  • Higher frequency of asking reviews increased the:
    • Total  no of reviews
    • Total no of installs/day
    • No of views of the app on the app store
  • And most importantly reduced the app ranking, thereby magnifying organic user acquisition

Action Items

  • Do a user survey asking those who don’t complete the survey the reason why they don’t complete it
Read more...

EID 03 – Add new keywords to your adwords account

Hypothesis

If this experiment is successful, I predict:

  • Our google ad conversions will increase by 15% because we have found some really good cost effective exact and phrase match keywords from search terms developed by broad or modified keywords

Experiment Design

  1. Filter broad keywords and go to their search terms
  2. Filter further by the ones which are giving high CTR and conversions
  3. Start adding them to the new ad group (either phrase or exact)
  4. Change the ad text on this new ad group to enhance the quality score
  5. See if bids are high enough to ensure a strong >80% Search Impression Share

Resource Estimation and Probability

Marketing Design Engineering Probability
0.25 Day 0 Day 0 Days High


Results

When Impressions Clicks Conversions
Before 44123 16300 1170
After 51615 18330 1560

Learnings

  • Finding new keywords to target in a cost effective way by utilizing this experiment is a great way to continually increase the scale of the account

Action Items

  • Ensure you get higher Search share by playing around with bids or increasing budget
  • Make this a workout that is repeated every week or every 15 days
Read more...

EID 04 – Increase clicks by pushing bids on low ad ranked keywords

Hypothesis

If this experiment is successful, I predict:

  • Clicks to the website from adwords will increase by 25%

Experiment Design

  1. Filter keywords which are not garnering impressions due to low ad rank and where CPC < First Page Bid
  2. Add columns (Est Addition Clicks/Week [+50% Bid]) and (Est Additional Cost/Week [+50% Bid]) and sort descending based on Est Addition Clicks/Week [+50% Bid]
  3. Do the maths as indicated above, and see if it makes sense to bid higher and if yes, then by how much

 

Resource Estimation and Probability

Marketing Design Engineering Probability
0.25 Day 0 Day 0 Days High


Results

Metrics Before After
Clicks/Week 50 250
Est. Addition Clicks/Week (+50% Bid) 200 0
Est. Additional Cost/Week (+50% Bid) $125 0
Click Conversion Rate 6% 8%
Conversions/Week 3 20
CPC (50% increase in bid doesn’t translate to an equal increase in CPC) $0.5 $0.6
Cost/Week $25 $150
Cost/Conversion $8.3 $7.5

Learnings

  • We spent $125 extra and got 17 extra conversions
    • Increasing conversions by 600%
    • Reducing Cost/Conversion by 11%

Action Items

  • Try to repeat this exercise once in a while, maybe once in two months. It’s not a guaranteed way of getting better results however worth a try..
Read more...
Bitnami