What is A/B testing?
A/B testing is a modern name for the classic scientific ‘controlled experiment’. If you’ve heard claims such as:
“We took two ordinary white shirts and washed one using detergent A and the other in detergent B.”
The reasoning is simple, there are two situations which are identical except for factor X. Since factor X is the only difference between the two situations, the new outcome was caused by factor X.
A/B testing is a reliable way to confirm that an outcome was caused by a particular factor provided that experimental controls to remove systematic bias are made and statistical methods are used to interpret the results.
What is MultiVariate testing?
A-B tests show a single factor has an effect. You test a version at price A of and count the purchases, Then test another with price B and compare.
Multivariate testing is A-B testing on steroids – it lets more than one factor be tested and this is important.
This is more than a mere convenience. Testing factors in parallel means that each factor can be compared with each other. Some will be more important than others and with a large enough set Multivariate testing can explain over 70%-95% of the reasons behind an outcome.By contrast, a weakness of A-B testing is that it is focused on a single factor. While the factor tested might be interesting and significant, it may only explain 1% of the all reasons behind an outcome, with 99% being unexplained.
What is Choice Modelling?
Choice Modelling is uses to predict the outcome in any situation where human choice is involved.
Generally, if a company has a large volume of interactions with customers and wants to greatly increase its return on those interactions, Choice Modelling is the answer.
A constant challenge facing companies is to understand the level and nature of demand for new and existing products or services. Traditional surveys and market research methods only capture a rough picture of consumer intentions using weak methods such as ratings. More troublesome is the reliance on respondents’ own explanations of their intentions.
Choice Modelling is different. Using a controlled experiment, the influences leading to a decision, even those outside the conscious awareness of the respondent, can be identified and measured.
Choice Modelling allows organisations to accurately estimate demand and know exactly how and why customers are making decisions.
The applications of Choice Modelling are many. It has been used to estimate demand of existing products and to forecast demand for new ones. It has been used to observe the effects of subtle layout and wording changes to marketing materials and even to measure the direct dollar valuation of brands.
Choice Modelling allows the exploration of vast numbers of possible configurations – many trillions – something infeasible using traditional research methods.
What is the difference between Predictive Analytics and Multi Variant testing?
Predictive Analytics is a broad term that includes Multivariate testing as one of its tools.
MultiVariate testing is an accepted mathematical method used by scientists to find effects of multiple variable. Predictive Analytics is a general term and may include ad hoc non-scientific techniques such as ‘credit-scoring’ or ‘technical analysis’ where there is no solid scientific theory to underpin the technique.
What’s the Process, how does it work?
A Choice Modelling project starts with laboratory style controlled experiments. Subjects are given hypothetical choice situations in the area we are interested in, for example buying a new car. These scenarios are strictly constructed to isolate independent effects from each factor such as Price, Colour, Make, Age, Condition etc. This is known as an ‘Experiment Design’.
The experiment data is collected, usually within a web browser, and modelled using Multinomial Regression.
The magnitude and significance of each effect tested can be compared with each other so see what is influencing choice, in this case decision to buy a new car.
The model can then be manipulated in a number of ways to generate more meaningful information such as:
- Probability of purchase for every and any combination of factors.
- Discovery of the optimal combination of factors to maximise sales
- When combined with cost information, discovery of the most profitable combination of factors
- Calculation of Implied Willingness-to-Pay in dollar (or Euros) for factors such as Car Colour or Condition.
- When combined with actual sales data, models can be scaled to accurately predict market-share for any course of action usually within less than 1%.
Isn’t real world data better than experiments?
Not always…
Using experiments allows freedom to explore new product concepts, features and prices – something not always possible with real world data.
In fact, live testing with other experiment platforms, such as Optimizely or Google are heavily constrained by what can be actually tested. As an example, price cannot be tested in a live setting without a cascade of enquires creating a customer management nightmare.
Using experiments, allows radical new concepts to be tested. Vast numbers of product combinations several trillion or more can be tested with minimal user input.
How does it save money and time? Why switch?
Because a model can accurately estimate what price a customer will pay for a feature, this powerful piece of knowledge eliminates the guesswork of deciding which features are profitable and which are not.
An exercise was done by one of the four major Australian banks on their credit cards prices and features. A choice model experiment demonstrated how to reshuffle interest rate, annual fee and interest free days so that customers were as happy with the cards as before. In practice however, customers were paying an additional $80 per year in fees and rates translating roughly to A$50M additional annual revenue for the bank.
Its possible to use the online platform without using expert resources or having to perform data manipulation. Strategically, a choice model cleanly separates what is important from what is irrelevant allowing strategic decisions to focus entirely on what matters.
How can you say it’s so accurate and dependable?
Choice Modelling boasts the highest academic credentials with two Nobel prizes being awarded to the founders of the theory.
Commercially it is proven technology and is routinely used by the largest companies in the world on bespoke projects.
SurveyEngine has fourteen years experience in Choice Modelling. In every case, Choice Models produced concrete and specific predictions that could be independently verified against observation. Typical variation is less than 1% from the actual. SurveyEngine has developed the only commercial Choice Modelling software and was awarded a US patent in 2014.
Is Choice Modelling new?
The science of Choice Modelling has been known for more than thirty years and in was in development for at least twenty years before that. Fortune 500 companies have been using it for many years. The field of Choice Modelling is one of the most vibrant in the social sciences, with two recent Nobel Prizes for economics awarded to contributors: Daniel McFadden (2000) and Daniel Kahneman (2002) with McFadden’s early and best-known application of discrete choice analysis was his work with the California Bay Area Rapid Transit Authority (BART).
Who uses Choice Modelling?
- Financial institutions including the Commonwealth Bank of Australia and Westpac to model customer valuation of price variables such as interest rates, fees and rewards Agencies including Saatchi and DDB for concept testing, new product development and brand equity tracking.
- Healthcare including BUPA and RTA to understand how people value aspects of personal health and healthcare options for insurance packages and policy decisions.
- Construction for CSR and Hebel to develop pricing strategies for commodity building products. Experiments covered pricing, product benefits, after sales service and warranties allowing the user to price products competitively against alternatives.
- Fast food companies including McDonalds, Subway and Coca-Cola to explore the myriad combinations of fast food ingredients, drinks, special offers, price, promotional messages and delivery in order to develop optimal bundled orders.
- Fast Moving Consumer Goods like Burgen to explore the billions of possible pricing and packaging configurations of new FMCG product variants.
What business problems can Choice Modelling solve?
Choice Modeling can answer questions such as:
- What aspects of a product do customers value when choosing?
- How can I make my company’s offerings more attractive?
- Is there a real demand for my new service?
- How can I make our marketing messages more effective?
Choice Modelling is an invaluable tool for marketers and business strategists for planning purposes. It is particularly powerful when linked to a company’s business planning or budgeting process. This allows the evaluation of countless combinations of offer design alternatives and potential competitive responses in real time.
By running a series of ‘what if’ scenarios on the models, managers can assess not only the demand and profit potential of strategic moves but also the risk exposure if competitors react defensively.