ROI TrackingWeb Analytics

Adaptive Multivariate Testing – Does It Work?

By February 6, 2009 December 3rd, 2012 121 Comments

If you like tinkering with your website, then you have probably heard of A/B or multivariate testing. This is where you can quickly test new things on your website, such as copy, images, call-to-action buttons, placement, etc., and see which combination effectively leads to more conversions. A/B testing is essentially testing two versions against each other that could be completely different, where as multivariate testing assesses multiple areas of the page and tests all possible combinations. So, if you had 3 versions of a headline, 3 versions of an image, and 3 versions of a button, you would have 27 possible combinations in a multivariate test.

Traditionally, multivariate testing has been where each possible combination gets equal play; meaning each combination is displayed equally to your traffic. Once a visitor is exposed to one of the test combinations, they are given a cookie so that each time they return while the experiment is still in progress, they will see the same combination. Google Website Optimizer is considered a traditional multivariate testing tool, where their algorithm will determine the winner based on how many combinations you have, and how many visitors is required to statistically determine a winner.  Their model shoots for a 12% minimum improvement in conversion rate at an 80% confidence level.

Now we have what’s called adaptive multivariate testing, which is offered by a company called Hiconversion. What they do is the same multivariate set up, but instead of giving equal play to each possible combination, they only test page combinations that are consistently performing in producing conversions. They claim that their methodology dramatically reduces the amount of traffic required to reach a statistically correct “winner”. This real-time adaptation also reduces the amount of loss leads or sale conversions that a typical test can produce.

This is sort of how Google AdWords works when you opt for the “auto-optimize” feature, which instead of displaying your ads evenly, they display the best performing ads more often.

Well, I personally was never a believer in AdWord’s optimization feature. I thought they determined a “winner” way too early, and so I’ve always turned off the optimization feature, and did my own split testing within the ad groups themselves.

I also had the same disbelief for the Hiconversion tool initially. First of all, how could you really determine a winner if all combinations were not played evenly? If one was played more than the other, then of course that combination would win!

So I began to ponder about the mathematics involved with this (I know…that’s what I do). First let me start by saying that I do not know how the Hiconversion algorithm works (I did ask them, but they said they would have to shoot me), so this is just my rationalization.

Let’s imagine that our multivariate test had 100 possible combinations. As the testing starts, each combination gets equal play, or 1%. The algorithm can quickly see which combination is starting to get higher conversion rates until it reaches a point to where it begins to test itself. (Bear with me). So if a few combinations are “starting” to look like good performers, the algorithm might say “OK, you got 5 conversions on 100 plays in X time…let’s see if you can get the same 5 conversions or better for the next 100 plays in the same time period.” If the combination meets or exceeds the mini-test, it moves up to the next “level” where it is now played 3% of the time, where if it doesn’t, and lets say it falls to 4, or 3, then perhaps it stays among all of the other combinations that are tested at 1% play. So the combinations in the experiment keep getting tested in this manner until there are only a few left, where a “winner” prevails. All of this happens very quickly, and in real-time. It’s kind of this survival of the fittest scenario because the combinations that can’t leap into the next levels, actually get played less and less over time, since the ones that are performing are eating up their play time.

Anyway, that’s my take on how it might be working, but again, I didn’t design the tool. My ultimate experiment would be to do identical experiments in Google Website Optimizer and Hiconversion and see if they arrive at the same result. Of course, I can’t be experimenting with all of our clients leads and sales, so maybe I’ll try this on my site one day.

121 Comments

  • Richard Hearne says:

    Hi

    Just wondered if you used their product, and if so how you found it?

    You dont seem to have subscribe to comments, so perhaps you could reply to my mail? If so many thanks in advance.

    Best rgds
    Richard

  • Derek says:

    Interesting idea, but this model is severely flawed as it is susceptible to time bias, such as time-of-day, day-of-week, holiday and news events, etc.

    For example, if the algorithm automatically adjusts what percent of each version is shown after only looking at Monday’s traffic, then the decision it makes is based solely on user preferences for Mondays, not for other days of the week.

    Similarly, suppose you are a flower or chocolates website … if the algorithm looks at traffic from Feb 1 to 13th, adjusting several times in that timeframe, it probably zeros in on what it thinks is the “best” version. But, come Feb 14th (Valentine’s day), everything changes because the big “event” has come and gone, and now the shopper’s dynamics are totally different. Where are you then? You basically need to start over.

    These are some of the fundamental issue with self-optimizing “auto pilot” systems. Even when we fly airplanes, we don’t let the auto pilot take us the whole way … there are many times that the human needs to take control of the stick.

    DM

  • Zee Aganovic says:

    DM, thank you for sharing your opinion about adaptive testing methodology.

    This is not just an ‘interesting idea’. This is a battle tested solution used by broad spectrum of companies over period of more than 2 years.

    During that time the method has consistently delivered speedy results, it has eliminated testing risk, and it was very easy to use.

    The algorithm is designed to take into account a time varying nature of website visitors. It collects combinations statistics over different day parts or days in the week. It’s decisions are driven by relational statistical model a not how an individual combination performs. As result it can effectively adopt to changes in visitor behavior.

    Now, lets talk about your hypothetical ‘February 14th chocolate or flower sales’ example that in your mind kills the adaptive methodology.

    Assuming that you are right about visitors behaving completely differently on February 14 then any other day, your options are:

    * Option 1 – Adaptive optimization: A system will adopt itself to a new situation, but because of a short burst nature of the change the adaptive response will be sub-optimal. Useful but not a perfect one.
    * Option 2 -Traditional multivariate testing solution: These methods will not react to change at all.
    * Option 3 – Let the human take control of the ‘stick’: This is a guessing game that can not guarantee consistent results.

    Today, to better manage their website, the most companies are applying the best practices and ‘gut feel’ measures. The end result is that we have 98% of general web traffic wasted.

    That is why we recommend use of a solution like http://www.hiconversion.com to merge the optimization science with a good marketing insight. There is not denying, an optimization test is only as good as input provided by a marketing person. Our tool gives you an ability to easily test many promotional ideas while helping you to consistently produce measurable increases in the online marketing ROI.

    Cheers,

    Zee
    President & CEO
    Hiconversion, Inc.

  • I do want to leave a comment here because the post here helps me a lot in dealing with the WordPress program! I could not delete anything in the program in the past and luckily, with this post, I know how to solve the problem now!

  • i emailed this to my mom….thanx

  • Thanks, bookmarked this blog.

  • Jake Muney says:

    Thank you for the intelligent critique. Me and my neighbour were just setting up to do some research about this. I am very glad to see such great information being shared freely out there.

  • Adam says:

    Hi, interesting article, but without fully testing the “losing” treatments aren’t you in danger of producing false positives? How can you statistically prove a treatment to be false, it you haven’t sampled it enough times. Doesn’t this work in reserve of GWOs principle of flood testing the losers to get them out of the way more quickly?

  • How you find ideas for articles, I am always lack of new ideas for articles. Some tips would be great.

  • https://waterfallmagazine.com
    Great blog here! Also your web site loads up very fast!
    What web host are you using? Can I get your affiliate link to your host?
    I wish my site loaded up as quickly as yours lol

  • I just want to mention I am very new to blogs and certainly liked you’re web page. Likely I’m likely to bookmark your website . You definitely have impressive articles. Regards for sharing your web page.

  • Evan Hladik says:

    I take delight in reading what you had to express, You have an remarkable knowledge on the subject informationa nd I look forward to examing more of what you have to say. I will pay attention and bookmark your post and come back to your internet site when an update is posted.

  • Wonderful blog! I found it while browsing on Yahoo News.Do you have any tips on how to get listed in Yahoo News? I’ve been trying for a while but I never seem to get there!

  • If you want to increase your knowledge only keep visiting this website and be updated with the hottest information posted here.

  • I am also writing to let you be aware of what a cool discovery my wife’s daughter went through studying your web page. She realized several things, which include what it’s like to possess an amazing giving spirit to get the others without problems grasp chosen complex subject areas. You undoubtedly exceeded visitors’ expected results. Thank you for coming up with those effective, trustworthy, informative and as well as cool thoughts on that topic to Emily.

  • Fantastic post but I was wanting to know if you could write a litte more on this topic? I’d be very thankful if you could elaborate a little bit further.Thanks!

  • Thank you for the auspicious writeup. It in fact was a amusement account it.Look advanced to far added agreeable from you! By the way,how can we communicate?

  • My brother recommended I might like this website.He was entirely right. This post actually made my day.You can not imagine simply how much time I had spent for this info!

  • Your style is unique in comparison to other people I’ve read stuff from. Many thanks for posting when you have the opportunity, Guess I’ll just bookmark this blog.

  • posters says:

    Hi, I do believe this is a great blog. I stumbledupon it 😉 I’m going to return once again since I book marked it. Money and freedom is the best way to change, may you be rich and continue to help other people.

  • Spot on with this write-up, I truly feel this site needs far more attention. I’ll probably be returning to read more, thanks for the information.

  • I truly love your blog.. Very nice colors & theme. Did you create this site yourself? Please reply back as I’m trying to create my own personal site and would like to know where you got this from or just what the theme is named. Appreciate it!

  • Do you believe past life hypnosis? Do you think past lives regression is real?

  • When I originally left a comment I appear to have clicked on the -Notify me when new comments are added- checkbox and from now on whenever a comment is added I receive four emails with the exact same comment. There has to be a means you are able to remove me from that service? Thanks a lot.

  • I was able to find good info from your articles.

  • One more issue is really that video gaming has become one of the all-time main forms of fun for people spanning various ages. Kids play video games, and also adults do, too. The XBox 360 is amongst the favorite video games systems for those who love to have a lot of activities available to them, along with who like to play live with people all over the world. Thanks for sharing your notions.

  • This blog was… how do I say it? Relevant!! Finally I’ve found something that helped me. Thanks a lot!

  • Good article! We are linking to this great content on our website. Keep up the good writing.

  • tiktokbeats says:

    Your style is unique in comparison to other folks I’ve read stuff from. Thank you for posting when you have the opportunity, Guess I will just bookmark this blog.

  • Good information. Lucky me I came across your site by accident (stumbleupon). I have saved it for later!

  • An impressive share! I’ve just forwarded this onto a coworker who has been doing a little research on this. And he actually ordered me breakfast because I discovered it for him… lol. So allow me to reword this…. Thank YOU for the meal!! But yeah, thanks for spending time to discuss this matter here on your internet site.

  • MetroClick specializes in building completely interactive products like Photo Booth for rental or sale, Touch Screen Kiosks, Large Touch Screen Displays , Monitors, Digital Signages and experiences. With our own hardware production facility and in-house software development teams, we are able to achieve the highest level of customization and versatility for Photo Booths, Touch Screen Kiosks, Touch Screen Monitors and Digital Signage. Visit MetroClick in NYC at http://www.metroclick.com/ or , 121 Varick St, New York, NY 10013, +1 646-843-0888.

  • Sapid Agency is a Search Engine Optimization company in New York City that provides SEO Services. Their proprietary SEO strategies help struggling websites and aspiring business owners to rank their websites higher in multiple search engines like Google , Yahoo and Bing. They provide local and gmb map ranking for businesses in NYC and many other local areas. Find more at https://www.sapidagency.com/ @ 145 E 57TH NEW YORK, NY 10022, USA, +1 971 341 5608 USA

  • This is a topic that is near to my heart… Cheers! Where can I find the contact details for questions?

  • g first says:

    Hey! This is my 1st comment here so I just wanted to give a quick shout out and tell
    you I truly enjoy reading your articles. Can you suggest any
    other blogs/websites/forums that cover the same subjects?
    Thanks for your time!

  • FreeSpinner will help you to attract the audience to your website.

  • you g says:

    Fantastic site. A lot of helpful info here. I am sending it to a few friends ans additionally sharing in delicious.

    And of course, thanks on your effort!

  • T-shirts says:

    Hi there, I do think your blog may be having internet browser compatibility problems. Whenever I look at your website in Safari, it looks fine however, if opening in IE, it’s got some overlapping issues. I simply wanted to give you a quick heads up! Besides that, great blog!

  • g in says:

    Keep on working, great job!

  • Great article! We will be linking to this particularly great article on our website. Keep up the great writing.

  • Wedding says:

    Very good post! We are linking to this great content on our website. Keep up the good writing.

  • Wow that was strange. I just wrote an extremely long comment but after I clicked submit
    my comment didn’t appear. Grrrr… well I’m not writing all that over again. Anyways, just wanted
    to say great blog!

  • sellefied says:

    Hi there, I think your blog could possibly be having web browser compatibility issues. Whenever I look at your website in Safari, it looks fine however when opening in I.E., it has some overlapping issues. I simply wanted to provide you with a quick heads up! Apart from that, wonderful site.

  • I do trust all of the concepts you’ve offered for your post. They are really convincing and can certainly work. Still, the posts are very quick for novices. Could you please extend them a little from next time? Thanks for the post.

  • ufa888a says:

    bookmarked!!, I love your blog.

  • cars says:

    Excellent write-up. I certainly love this site. Keep writing!

  • I truly appreciate this post. I’ve been looking all over for this! Thank goodness I found it on Bing. You have made my day! Thx again!

  • I’ve recently started a web site, the information you offer on this website has helped me tremendously. Thank you for all of your time & work. “Character is much easier kept than recovered.” by Thomas Paine.

  • I uncovered your blog website on google and also examine a few of your early blog posts. Remain to maintain the great run. I simply extra up your RSS feed to my MSN Information Reader. Seeking onward to learning more from you later!?

  • Excellent article. I certainly appreciate this site. Keep it up!

  • You made some decent points there. I checked on the internet to find out more about the issue and found most people will go along with your views on this site.

  • wooyart says:

    I like reading through a post that can make men and women think. Also, thank you for allowing me to comment.

  • trim repairs says:

    Pretty! This was an incredibly wonderful post. Thank you for providing these details.

  • Fantastic web site. A lot of helpful information here. I’m sending it to several pals ans additionally sharing in delicious. And of course, thanks on your sweat!

  • You have made some decent points there. I checked on the web to find out more about the issue and found most people will go along with your views on this site.

  • tinyurl.com says:

    Hello colleagues, how is everything, and what you
    want to say concerning this post, in my view its genuinely awesome in favor of me.

  • Hi there very nice blog!! Guy .. Excellent .. Wonderful .. I’ll bookmark your website and take the feeds additionally…I’m happy to search out so many useful information right here within the publish, we need work out extra techniques on this regard, thank you for sharing.

  • Have you ever considered creating an ebook or guest authoring on other sites?
    I have a blog centered on the same subjects you discuss
    and would really like to have you share some stories/information. I
    know my visitors would appreciate your work. If you’re even remotely interested,
    feel free to shoot me an e-mail.

  • Hello friends, how is all, and what you want to say on the topic of this piece of writing, in my view its actually awesome designed for me.

  • low quality says:

    This is a topic which is near to my heart… Best wishes! Where can I find the contact details for questions?

  • Hiya there, just turned aware about your blog site through Yahoo and bing, and found that it’s genuinely educational. I will appreciate in the event you continue this approach.

  • wonderful issues altogether, you simply won a new reader. What might you suggest about
    your post that you just made some days in the past?
    Any certain?

  • obviously like your web-site but you have to check the spelling on several
    of your posts. Several of them are rife with spelling
    problems and I to find it very troublesome to inform the
    truth then again I will definitely come back again.

  • Good post. I learn something new and challenging on websites I stumbleupon everyday. It’s always helpful to read articles from other writers and practice a little something from other websites.

  • pmbok 6th says:

    Hi, I do believe this is a great website. I stumbledupon it 😉 I will revisit once again since I book marked it. Money and freedom is the best way to change, may you be rich and continue to help others.

  • It’s practically not possible to find well-educated viewers on this content, still, you come across as like you be aware of those things you’re indicating! Many Thanks

  • Like!! Really appreciate you sharing this blog post.Really thank you! Keep writing.

  • I learn something new and challenging on blogs I stumbleupon everyday.

  • Hi there, after reading this amazing paragraph i am as well delighted to share my knowledge here with friends.

  • SMS says:

    These are actually great ideas in concerning blogging.

  • Oh my goodness! Awesome article dude! Thank you, However I am having troubles with your RSS. I don’t know the reason why I am unable to join it. Is there anybody else having identical RSS issues? Anybody who knows the answer will you kindly respond? Thanx!

  • Very nice article. I absolutely appreciate this site. Stick with it!

  • I am curious to learn what blog system you’re using?
    I’m experiencing some small security issues with my latest site and I would want to find some thing safe.

    Have you got any recommendations?

    my web-site – Chic Sparrow Folio

  • Faytech North America is a touch screen Manufacturer of both monitors and pcs. They specialize in the design, development, manufacturing and marketing of Capacitive touch screen, Resistive touch screen, Industrial touch screen, IP65 touch screen, touchscreen monitors and integrated touchscreen PCs. Contact them at http://www.faytech.us, 121 Varick Street, New York, NY 10013, +1 646 205 3214.

  • Best CBD Oil says:

    Oh my goodness! Impressive article dude! Thank you so much, However I am experiencing difficulties with your RSS. I don’t know the reason why I am unable to join it. Is there anybody else having the same RSS problems? Anyone that knows the answer will you kindly respond? Thanks!

  • It’s impressive that you are getting thoughts from this article as well as from our dialogue made here.

  • maxcape says:

    Good post. I learn something new and challenging on sites I stumbleupon every day. It’s always useful to read through content from other authors and use something from other websites.

  • Sapid Agency is a Search Engine Optimization company in New York City that provides SEO Services. Their proprietary SEO strategies help struggling websites and aspiring business owners to rank their websites higher in multiple search engines like Google , Yahoo and Bing. They provide local and gmb map ranking for businesses in NYC and many other local areas. Find more at https://www.sapidagency.com/ @ 145 E 57TH NEW YORK, NY 10022, USA, +1 971 341 5608 USA

  • I have realized some points through your blog post post. One other point I would like to convey is that there are many games in the marketplace designed mainly for preschool age young children. They contain pattern acknowledgement, colors, animals, and designs. These normally focus on familiarization in lieu of memorization. This makes little ones occupied without sensing like they are studying. Thanks

  • home says:

    Hi here, just turned receptive to your website through The Big G, and discovered that it’s pretty beneficial. I will appreciate should you decide maintain this approach.

  • VR Headsets says:

    I’m pretty pleased to find this great site. I want to to thank you for ones time just for this wonderful read!! I definitely really liked every part of it and i also have you book-marked to look at new stuff in your blog.

  • There’s certainly a lot to know about this issue. I really like all the points you’ve made.

  • Charms says:

    An outstanding share! I’ve just forwarded this onto a colleague who was doing a little research on this. And he actually bought me dinner because I stumbled upon it for him… lol. So allow me to reword this…. Thank YOU for the meal!! But yeah, thanks for spending time to talk about this issue here on your internet site.

  • hip hop says:

    Way cool! Some extremely valid points! I appreciate you writing this article and the rest of the site is really good.

  • Aw, this was a really nice post. Spending some time and actual effort to generate a very good article… but what can I say… I hesitate a lot and never manage to get nearly anything done.

  • That is a really good tip especially to those new to the blogosphere. Simple but very precise info… Thank you for sharing this one. A must read post.

  • mcdvoice com says:

    I could not resist commenting. Well written!

  • rr com says:

    Hi, I do believe this is a great blog. I stumbledupon it 😉 I may return yet again since i have book marked it. Money and freedom is the best way to change, may you be rich and continue to guide others.

  • recommended says:

    I could not refrain from commenting. Well written!

  • You need to take part in a contest for one of the highest quality sites on the net. I will recommend this web site!

  • Saved as a favorite, I really like your web site!

  • Millennials says:

    There is certainly a lot to learn about this topic. I really like all of the points you’ve made.

  • Click here says:

    Very nice post. I just stumbled upon your weblog and wanted to say that I have truly enjoyed surfing around your blog posts. After all I’ll be subscribing to your feed and I hope you write again very soon!|

  • MetroClick specializes in building completely interactive products like Photo Booth for rental or sale, Touch Screen Kiosks, Large Touch Screen Displays , Monitors, Digital Signages and experiences. With our own hardware production facility and in-house software development teams, we are able to achieve the highest level of customization and versatility for Photo Booths, Touch Screen Kiosks, Touch Screen Monitors and Digital Signage. Visit MetroClick in NYC at http://www.metroclick.com/ or , 121 Varick St, New York, NY 10013, +1 646-843-0888.

  • This site was… how do you say it? Relevant!! Finally I’ve found something which helped me. Kudos.

  • Hi! I could have sworn I’ve visited your blog before but after browsing through many of the articles I realized it’s new to me. Regardless, I’m definitely happy I found it and I’ll be bookmarking it and checking back frequently.

  • The next time I read a blog, I hope that it doesn’t fail me as much as this one. I mean, I know it was my choice to read, nonetheless I really believed you would probably have something useful to say. All I hear is a bunch of whining about something you could fix if you were not too busy looking for attention.

  • Way cool! Some very valid points! I appreciate you writing this write-up plus the rest of the website is very good.

  • Porn says:

    Very descriptive post, I liked that a lot. Will there be a part 2?|

  • Porn says:

    Hello, I believe your web site could possibly be having internet browser compatibility issues. When I look at your site in Safari, it looks fine but when opening in I.E., it has some overlapping issues. I merely wanted to give you a quick heads up! Other than that, wonderful site!

  • Aaron Lal says:

    It is perfect time to make some plans for the future and it’s time to be happy. I’ve read this post and if I could I want to suggest you some interesting things or suggestions. Maybe you could write next articles referring to this article. I want to read more things about it!|

  • An intriguing discussion is worth comment. I do believe that you need to publish more on this subject matter, it may not be a taboo matter but usually people do not talk about these subjects. To the next! Cheers.

  • An impressive share! I’ve just forwarded this onto a co-worker who was conducting a little research on this. And he in fact bought me dinner simply because I discovered it for him… lol. So let me reword this…. Thanks for the meal!! But yeah, thanks for spending time to discuss this subject here on your web site.

  • This site was… how do you say it? Relevant!! Finally I have found something that helped me. Many thanks.

  • I love looking through an article that will make people think. Also, thank you for allowing for me to comment.

  • This is the perfect site for anybody who wants to understand this topic. You know so much its almost hard to argue with you (not that I really would want to…HaHa). You certainly put a fresh spin on a subject that’s been written about for many years. Excellent stuff, just wonderful.

  • Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point. You clearly know what youre talking about, why waste your intelligence on just posting videos to your weblog when you could be giving us something enlightening to read?|

  • An intriguing discussion is definitely worth comment. I do believe that you ought to publish more on this subject matter, it may not be a taboo subject but usually people do not discuss these topics. To the next! Many thanks!

  • Greetings! Very useful advice in this particular article! It is the little changes that make the biggest changes. Many thanks for sharing!

  • I’m using the same blog platform asyours and I’m ha

  • This website was… how do I say it? Relevant!! Finally I have found something that helped me. Kudos!

  • This website really has all of the information and facts I wanted concerning this subject and didn’t know who to ask.

  • I simply want to say I’m newbie to blogs and truly enjoyed you’re web site. More than likely I’m going to bookmark your blog post . You certainly come with impressive articles and reviews. Many thanks for revealing your website page.

  • Hi there, i read your blog occasionally and i own a similar one and i was just wondering if you get a lot of spam comments? If so how do you stop it, any plugin or anything you can advise? I get so much lately it’s driving me mad so any assistance is very much appreciated.|

Leave a Reply