21 {{ upvoteCount | shortNum }}
21 {{ upvoteCount | shortNum }}

How would you interpret these A/B marketing tests?

by kerrya
19 replies
I do a Lingerie Website for a client:
(link removed by mod per forum rules)

I wanted to do a small A/B Facebook ad test and here's what happened. I've having trouble drawing a conclusion from these results.

We created a loss leader. A pair of sexy woman's panties for only .99 (USD) with free shipping anywhere in the US or Canada.

Week #1 $5/day
Audience: Female age 18-39
Interests: Lingerie.

Results:
4 orders from new clients. = COA $8.75/new client

Week #2 $5/day
Audience: Female age 18-39
No interests specified.

Results:
1 order from new client = COA $35/new client

Conclusion was to go back to specifying the same target group with Interests in LIngerie, but double the ad spend.

Week #3 $10/day
Audience: Female age 18-39
Interest: Lingerie

Results
0 orders for the promotional products!
But one order received for other products but nothing from the promotion. = COA $70/client

So now I can't draw any conclusion at all given these results.

Other than the target interest group and the daily ad spend the ad remained unchanged during the test.

I'm thinking the failure of this test may be for one of the following reasons:

1) Sample size too small - results fall within a margin of error and are therefore useless.

2) Facebook ads are next to useless

3) (your thoughts please)

Any advice appreciated.

Thanks again

Kerry
#a or b #interpret #marketing #tests
Avatar of Unregistered
  • Profile picture of the author savidge4
    Originally Posted by kerrya View Post

    I'm thinking the failure of this test may be for one of the following reasons:

    1) Sample size too small - results fall within a margin of error and are therefore useless.

    2) Facebook ads are next to useless

    3) (your thoughts please)
    In order to get a sale there needs to be a series of conversions that take place.
    1) They click on the ad
    2) They click on an item
    3) They place the item in the cart
    4) They click to check out.
    5) They actually pay for the purchase.

    So we need to start with how many people saw the ad ( impressions ) and how many clicked? This ratio gives you what is called CTR ( Click through Rate ) if for example you have 1000 impressions and only 3 clicks there is something wrong with the ad.

    I am going to bet if you look specifically across the 3 weeks, week 1 and 3 should be about the same and week 2 should be higher it terms of impressions and lower in terms of clicks.

    IF you are no where near 3% CTR I would start here in terms of looking at what is failing. without seeing this I am going to assume the offer falls under "To Good To Be True" and that will lower your click rate.

    Throw a story behind the .99 offer " The boss is out of town for the week and told us to sell these... So we said "Sure" with a grin. for this week only get select items for $.99 AND free shipping " You will have to shrink it up a bit ( in terms of word count ) but you get the idea.

    Post your impressions and Click numbers and I can for sure give you better advice moving forward.
    Signature
    Success is an ACT not an idea
    {{ DiscussionBoard.errors[11559533].message }}
    • Profile picture of the author DABK
      You're assuming efficiency. The OP's system might have more steps. So, I'd suggest OP lists all the steps someone has to go through, or confirm the ones you listed are the only ones that exist.


      Originally Posted by savidge4 View Post

      In order to get a sale there needs to be a series of conversions that take place.
      1) They click on the ad
      2) They click on an item
      3) They place the item in the cart
      4) They click to check out.
      5) They actually pay for the purchase.

      So we need to start with how many people saw the ad ( impressions ) and how many clicked? This ratio gives you what is called CTR ( Click through Rate ) if for example you have 1000 impressions and only 3 clicks there is something wrong with the ad.

      I am going to bet if you look specifically across the 3 weeks, week 1 and 3 should be about the same and week 2 should be higher it terms of impressions and lower in terms of clicks.

      IF you are no where near 3% CTR I would start here in terms of looking at what is failing. without seeing this I am going to assume the offer falls under "To Good To Be True" and that will lower your click rate.

      Throw a story behind the .99 offer " The boss is out of town for the week and told us to sell these... So we said "Sure" with a grin. for this week only get select items for $.99 AND free shipping " You will have to shrink it up a bit ( in terms of word count ) but you get the idea.

      Post your impressions and Click numbers and I can for sure give you better advice moving forward.
      {{ DiscussionBoard.errors[11559537].message }}
    • Profile picture of the author kerrya
      Absolutely right about several things, but let's start with CTR. Reviewing the FB stats it shows only about a 1.1% CTR.

      I'm going to start with the "too good to be true" point and recreate a promotion with the same item, but packaged differently to show a different price and markup falling more in line with market norms.
      {{ DiscussionBoard.errors[11559547].message }}
      • Profile picture of the author DABK
        You might want to test all conversion points.


        Originally Posted by kerrya View Post

        Absolutely right about several things, but let's start with CTR. Reviewing the FB stats it shows only about a 1.1% CTR.

        I'm going to start with the "too good to be true" point and recreate a promotion with the same item, but packaged differently to show a different price and markup falling more in line with market norms.
        {{ DiscussionBoard.errors[11559550].message }}
        • Profile picture of the author savidge4
          Originally Posted by DABK View Post

          You might want to test all conversion points.
          As I practice this I do it in series... without proper traffic how can you know step 2 is working or not.. so to change without traffic to determine a block would / could be working in the wrong direction.

          So again as I do.. I actually stream line the process even further than above.

          1) They click on the ad
          2) They add item to cart
          3) They click to check out.
          4) They actually pay for the purchase.

          I land to a specific page that sells 1 and only 1 item. I do not give the opportunity to look at a store. I sell specifically what I am advertising... it reduces loss in the funnel.
          Signature
          Success is an ACT not an idea
          {{ DiscussionBoard.errors[11559565].message }}
          • Profile picture of the author DABK
            The OP is working on the ad. I would have worked on targeting too. I was once after 18-34 year-old women. Dropping the 18-24 increased results significantly. In my case, those with incomes over 50k a year but less than 100k responded better. College-educated better yet. Then, there was geography...



            I think OP should tinker with their targeting too to get better CTR.


            Originally Posted by savidge4 View Post

            As I practice this I do it in series... without proper traffic how can you know step 2 is working or not.. so to change without traffic to determine a block would / could be working in the wrong direction.

            So again as I do.. I actually stream line the process even further than above.

            1) They click on the ad
            2) They add item to cart
            3) They click to check out.
            4) They actually pay for the purchase.

            I land to a specific page that sells 1 and only 1 item. I do not give the opportunity to look at a store. I sell specifically what I am advertising... it reduces loss in the funnel.
            {{ DiscussionBoard.errors[11559751].message }}
            • Profile picture of the author savidge4
              Originally Posted by DABK View Post

              The OP is working on the ad. I would have worked on targeting too. I was once after 18-34 year-old women. Dropping the 18-24 increased results significantly. In my case, those with incomes over 50k a year but less than 100k responded better. College-educated better yet. Then, there was geography...



              I think OP should tinker with their targeting too to get better CTR.
              Not going to disagree wiih this.. playing with Married Singled Divorced and Separated in the case of "Lingerie" might be a wise move as well.
              Signature
              Success is an ACT not an idea
              {{ DiscussionBoard.errors[11559754].message }}
              • Profile picture of the author DABK
                I wish there was a way to target "married bu I've decided to leave their sorry ass but haven't made the announcement yet" as those are starting to think of improving themselves in preparation for dating again.


                Originally Posted by savidge4 View Post

                Not going to disagree wiih this.. playing with Married Singled Divorced and Separated in the case of "Lingerie" might be a wise move as well.
                {{ DiscussionBoard.errors[11559763].message }}
    • Profile picture of the author Jamell
      Thanks for sharing your knowledge. I am impressed by how you went into detail .
      {{ DiscussionBoard.errors[11560026].message }}
  • Profile picture of the author manonpoint
    I would say you probably don't have enough data to draw many conclusions.
    Signature
    ManOnPoint
    {{ DiscussionBoard.errors[11559543].message }}
  • Profile picture of the author IGotMine
    I would offer a coupon for the $0.99 item if they opt-in to your newsletter.
    Signature
    {{ DiscussionBoard.errors[11559551].message }}
    • Profile picture of the author savidge4
      Originally Posted by IGotMine View Post

      I would offer a coupon for the $0.99 item if they opt-in to your newsletter.
      Think about it this way.. its Just as easy to sell an item ( and then you have the e-mail address of a BUYER ) vs collecting a list with a "Freebie" and then trying to sell. In terms of e-commerce I practice that you want to get while the getting is good. Offer a product SELL the product.
      Signature
      Success is an ACT not an idea
      {{ DiscussionBoard.errors[11559566].message }}
      • Profile picture of the author IGotMine
        Originally Posted by savidge4 View Post

        Think about it this way.. its Just as easy to sell an item ( and then you have the e-mail address of a BUYER ) vs collecting a list with a "Freebie" and then trying to sell. In terms of e-commerce I practice that you want to get while the getting is good. Offer a product SELL the product.
        You don't understand what I mean.

        You still get the sale, it's just offering a "new customer coupon" vs a "too good to be true" offer.
        Signature
        {{ DiscussionBoard.errors[11559567].message }}
        • Profile picture of the author savidge4
          Originally Posted by IGotMine View Post

          You don't understand what I mean.

          You still get the sale, it's just offering a "new customer coupon" vs a "too good to be true" offer.
          I get what you are saying.... opt in to the newsletter and get a coupon for a .99 cent item with free shipping... But I am not sure you are getting past the "to good to be true" friction.

          More importantly is you are creating added steps in the process. I personally like to decrease steps, because it is proven to maximize sales. With Facebook ads, Google ads, Bing Ads, and Microsoft platform ads I run a 4 step conversion funnel consisting of 1 page and a payment pop up

          1) Click from Ad
          2) Selecting quantity and pressing buy now
          < payment pop up >
          3) Inserting payment info
          4) Clicking to create the conversion

          This ultimately is a numbers game. and @IGotMine this is more for others reading this but lets break down some numbers.

          If you have 3% clicking on the ad and going to the offer you have 3 people out of every 100 that are presented the ad that click. you then have to understand that the OFFER itself will convert on average at 3% That means you have to have the Ad presented 1000 times to get 30 people to your offer, to create 1 sale. In many circles this is considered a decent converting funnel.

          Any time you add steps you are adding what I call "bleed". As a customer has to make step after step to buy something you lose buyer after buyer by percentage that actually get to the end result of actually buying. With an average "Abandoned Cart Percentage" floating at 70% or so, I feel it is imperative to reduce steps and "Bleed" / loss of sales.

          Selling anything for .99 with free shipping is simply a losing proposition. ( and that's not even taking into account the expense of using paid advertising ) And then the idea that you are going to create sales based on life of customer - simply not a bet I would make. I look at e-comm sales as a 1 off effort. Sure I have product and sites that create multiple life time of customer sales, but by enlarge its not something that I would base a campaign for.

          I believe at all stages of e-comm you have to be making money, or all you will be doing for the sake of "sales" is losing, and that's simply not why you sell things.

          To the OP @kerrya if you actually have the product in hand.. I would suggest you are better off selling this on eBay. The idea of getting your offer in front of traffic is simply built in to the Millions that use eBay daily. In the end the 10% eBay fee for selling is probably far less than the cost of traffic you are experiencing now, minus the stress of exactly what you are going through now.
          Signature
          Success is an ACT not an idea
          {{ DiscussionBoard.errors[11559578].message }}
  • Profile picture of the author Skywriting
    Originally Posted by kerrya View Post

    I do a Lingerie Website for a client:
    (link removed by mod per forum rules)

    I wanted to do a small A/B Facebook ad test and here's what happened. I've having trouble drawing a conclusion from these results.
    100 times the data in needed. But I'm not sure on that number.

    There Is a formula but you are certainly in the random noise level of data.
    {{ DiscussionBoard.errors[11559556].message }}
  • Profile picture of the author Kam Ami
    for AB testing you can use free Google analytic tool. its very easy and totally free of cost.
    {{ DiscussionBoard.errors[11560050].message }}
  • Profile picture of the author kuchenchef
    1. it's not a real a/b test because those ads are not running at the same time. running test variants sequentially opens your test to all kinds of external influences, feature changes, the weather, holidays, political drama, ...

    2. your post compares daily budget to sales and cost of aquisition. better is to start with impressions, then list all important conversion steps, ad clicks, shopping cart, checkout, ..., check the numbers for each step and calculate the statistic significance of those microconversion numbers. this way you have a better understanding whether it's your ad or your target group or your website.

    3. for an ad campaign it's always advisable to have a specific landing page that mirrors/extends the look&feel and copy of your ad. if your ad clicks look ok but the conversion on your website sucks, that would be something to look into by testing different combinations of ads and landing pages, or at least by making sure that ads and landing page work together. e.g. when advertising a specific item, the landing page above the fold content should feature the item, show a clear call to action for purchasing the item and underline/expand on the value proposition/value theme/motif outlined in your ad. ad and landing page are a unit.

    but overall, from these numbers you presented i get the strong impression that there isn't the required sample size to draw meaningful, specific, actionable conclusions. the purpose of a test is to let the numbers speak. without the numbers there is not much talking possible. it could be the ad, it could be the target group, it could be the website, it could be that facebook ads suck in general for that type of product, ...
    {{ DiscussionBoard.errors[11560078].message }}
  • Profile picture of the author ChrisBa
    Originally Posted by kerrya View Post

    I do a Lingerie Website for a client:
    (link removed by mod per forum rules)

    I wanted to do a small A/B Facebook ad test and here's what happened. I've having trouble drawing a conclusion from these results.

    We created a loss leader. A pair of sexy woman's panties for only .99 (USD) with free shipping anywhere in the US or Canada.

    Week #1 $5/day
    Audience: Female age 18-39
    Interests: Lingerie.

    Results:
    4 orders from new clients. = COA $8.75/new client

    Week #2 $5/day
    Audience: Female age 18-39
    No interests specified.

    Results:
    1 order from new client = COA $35/new client

    Conclusion was to go back to specifying the same target group with Interests in LIngerie, but double the ad spend.

    Week #3 $10/day
    Audience: Female age 18-39
    Interest: Lingerie

    Results
    0 orders for the promotional products!
    But one order received for other products but nothing from the promotion. = COA $70/client

    So now I can't draw any conclusion at all given these results.

    Other than the target interest group and the daily ad spend the ad remained unchanged during the test.

    I'm thinking the failure of this test may be for one of the following reasons:

    1) Sample size too small - results fall within a margin of error and are therefore useless.

    2) Facebook ads are next to useless

    3) (your thoughts please)

    Any advice appreciated.

    Thanks again

    Kerry

    Can you share the traffic numbers and more details? My first thought is not enough data but need to verify.
    Signature
    {{ DiscussionBoard.errors[11560108].message }}
    • Profile picture of the author kerrya
      Originally Posted by ChrisBa View Post

      Can you share the traffic numbers and more details? My first thought is not enough data but need to verify.
      Here are some stats:

      Total reach of Campaign = 7316

      Women = 100%
      age 18-24 = 29%
      age 25-34 = 48%
      age 35-44 = 22%

      Link Clicks = 86
      Photo Clicks = 13
      Comments = 2
      Shares = 10
      {{ DiscussionBoard.errors[11560364].message }}
Avatar of Unregistered

Trending Topics