Split Testing Scripts are not reliable

by 17 replies
20
Do you think split testing scripts are reliable?

I doubt more and more...


I'm a test addict. I love testing. And I know how important it is.

I had my own custom made script developed 5 or 6 years ago,
and I use it all the time. I even have planed to released it for
free on the french speaking market, for which I registered the
domain name SplitTest.fr.

Problem is...

I'm confident of the quality of my script, and have relied on
it only until around 2 years ago.

Then I started to compare the number of sales (or subscribers)
recorded by my script and the number recorded by my
e-commerce application (or autoresponder).

Two years ago, 5-10% of sales were not recorded by my
split test script.

Last year, it was around 15-20%.

This year, it's around 25-30%.

I tried other scripts. And the results were the same: depending
the script and webpage, 20-30% of sales are not recorded by
the split test application!

These are numbers.

This evening, I stumbled upon the following screenshot on
Chris Rempel's site:


(taken from http://www.thelazymarketer.com/vip-multiply/)

As you can see, 13 + 25 + 16 = 54 total sales, 16 were not
recorded by the split testing script. That's 29.6%!

So I'm not alone!


1- If this is "common" problem (Chris doesn't hide this fact on
the screenshot above, so I start to think I'm not alone and this
might be "normal" situation), and if 20-30% of actions are not
recorded by split test scripts, we can't be sure the results of
our tests are accurate.

2- Anyone else has noticed that?

3- Anybody knows why such enormous number of actions are
not recorded (cookies didn't work, but why)?

Sure, I'll hire a programmer... or 2... or 3 to try to fix this issue
on my SplitTest.fr script!

All the best.

Didier
#main internet marketing discussion forum #accuracy #reliable #scripts #split #split test #split testing #testing
  • Hi Didier,

    Actually there was no problem on my end.

    The reason why those sales weren't logged is because they didn't come from my target landing page.

    In this case, my home page AND the split-tested landing URL were directly selling the product, and the home page produced 16 untracked sales in that case.

    Interesting observation though.

    Cookie-blocking would also affect this.

    -Chris
    • [1] reply
    • Oh yes! This also explain why figures increase with time.

      But 20-30%, that's a lot!

      Maybe we can consider that these 20-30% untracked actions come from
      *random* visitors, so tests are not biased...


      Hi Chris!

      How could you be sure the 16 untracked sales come from your homepage?

      (I didn't notice your report is about split testing until now...
      Obviously, I'm not trying to say your report is not good!)

      Didier
  • Anything to do with javascript perhaps??? or is your script all in php? Just an idea because I know many people disable js entirely.
    • [1] reply
    • It's coded in PHP, and it doesn't use any JavaScript code.

      "My" programmer doesn't have an explanation for now.

      Didier
  • Sorry to bump this thread. This is not only about my own script.

    Has anybody noticed 20-30% of untracked actions when using
    a split testing script?

    Thank you.

    Didier
    • [1] reply
    • It might be depend on the method your script tracks the sale:

      * Cookie method: As said above most people disable cookies / delete cookies often. So with out cookie, your script might not be able to track.

      * IP number: If your script tracks the sale thru ip number from sales page to download (tracking page), the ip can be changed in between because the person bought your product different time from visiting your sales page OR using proxies.

      * If every other thing is working, they bought visited the sales page at one time or using one computer and bought the product and goes to tracking page using different time and computer.

      There are more variables that difficult to track with a sale.

      .
      • [1] reply
  • This isn't necessarily a big problem.

    As long as the script ignores the untrackable sales and handles the other sales properly, you should still get valid results, right?

    A big problem would be if the script did NOT ignore those untrackables and mistakenly assigned them to one of your sales letter versions.

    In other words, if only 70% of sales can be properly tracked, the stats on which sales letter performed better should still be useful... certainly better than not testing at all.
  • Hi Didier

    There's no FOOLPROOF way of tracking visitors using cookies and IP. That's because the cookie can be deleted or blocked, and IP addresses frequently change.

    It's simply a FACT OF LIFE that not all sales will be tracked, even if you are using a combination of both cookie and IP address.

    As Chris pointed out, as long as the split testing script doesn't just "guess" where the sale came from, you should be alright, as long as you're getting a big enough sample.

    My program Power Split Tester can also track via a special code which can be incorporated into a mailing list, which is ideal if you have a sales process such as SQUEEZE PAGE > LIST > SALES LETTER.

    You can incorporate the code into the list message, so the visitor will see the same version of the sales letter.

    I created this feature because I kind of anticipated that cookies and IP tracking would gradually become more unreliable over time, as more people block them.

    However, I don't think we'll get to the stage when the majority of users will have cookies blocked, because they rely on them for too many other things.

    Hope this helps!
    • [1] reply
    • @ all: Thank you for your time and help!


      @ Chris Rempel:

      Ok. That's clear now.

      Thank you.



      Not sure about that. And it's my real concern here.

      It seems you say: just ignore the unrecorded sales.
      I don't agree with that.


      Simple example. 2 versions of a sales copy, A and B.

      Total impressions is, say 1,234 (approximately 50% from A, 50% from B).

      Sales from A: 48
      Sales from B: 31

      => Winner is A (statistical confidence: 0.95)

      Unrecorded sales (30% of the total): 33

      That's not possible to know how much of these 33 sales were
      presented A, and how much B.

      Maybe 21 buyers saw A, and 12 B.
      Or 10 saw A, and 23 saw B.
      Or 1 saw A, and 32 B.

      Meaning maybe the real result is B is the winner, or there's no
      clear winner. One then runs the risk of coming to a wrong
      conclusion.


      So, in your opinion one has to make the hypothesis that in the
      33 unrecorded sales, 50% were presented A, and 50% B (which,
      in fact, consist of ignoring these 33 sales), otherwise that would
      not be possible to draw a conclusion.

      Is this hypothesis valid? I can't see a reason why it's valid, because
      these 33 sales did happen. They're part of the sample.

      Tester got 30% of sales from unknown origin.
      It's a big bias that is introduced here if he doesn't take them into
      account. IMO, we can't just ignore these sales.

      Ignoring 10-15% "would be ok", but not 30%.


      Also, from a pure mathematical point of view, if you remove 30%
      of the sample, there's no way to be sure the statistical distribution
      (in fact the "probability density function") is still gaussian ("bell curve"),
      unless this part is taken at random.

      So no conclusion can be drawn using the chi square test, for example
      (PDF has to be gaussian for that).


      That would be ok to remove 30% of the sample, if this "sub-sample"
      was taken at random.

      Problem is, these unrecorded sales are not taken at random...


      You put 100 sales into a bag. You close your eyes and remove 30
      from the total. That would be ok. But that's not what happens here.



      Yes, good! But how do you track returning visitors?



      Didier

      P.S.: Damn! I wish I could speak english better!
      • [2] replies
  • Didier, I see the same thing I've seen anywhere from 25-50% inaccuracies depending on the script being used.

    One I've found to be very innacurate is Google's Adwords conversion tracking it is routinely off by about 30% no matter how I try to improve this I can't seem to get better results.

    I still use it because I like to have all my stats right there in Adwords but it's frustrating to see it so far off.
Join the discussion

Next Topics on Trending Feed