This is a Q&A blog post with pricing expert, Mark Stiving. If you have a pricing question for Mark, ask on his LinkedIn page.
Question: Should I A/B test our pricing?
Answer: The best answer is, “it depends”. But a more generic answer would be probably not. I’d say a very small percentage of companies could get away with this. However, there are probably better methods you could use besides A/B testing.
A/B testing probably doesn’t make sense for the following reasons:
1. You need a large sample size to make good conclusions.
In an election cycle you often hear something like, “Candidate A is polling 4% higher than candidate B. The margin of error is plus or minus 3%.” What they don’t tell you is they are only 95% confident that difference is between 1% and 7%. Which means they will be wrong on average 1 out of 20 times. Wow. And, in order to get to that 95% confidence of +/-3%, they had to survey over 1,000 people. There are sample size calculators you can use to get the exact numbers, but if you have a big market your numbers will be similar.
When you do A/B testing, you are doing something very similar. Instead of two candidates, you are looking at two prices and instead of votes, you are looking at conversion rates. How easy is it for you to get 1,000 buyers (non-customers) to visit your pricing page? This may be reasonable for a B2C company with a great brand name and lots of visitors. Most companies don’t have the luxury of waiting this out.
2. You may upset customers.
Almost 20 years ago now, Amazon got major egg on its face because they were caught charging different customers different prices. It almost seems like a sport for bloggers to look for and write about companies that charge different prices to different buyers. You don’t want to be one of those companies. Sometimes buyers see a price on one visit to your website, and then, if they revisit the same site using a different browser or device, they may see a higher price. This causes angst and even anger. Similarly, in B2B a worker finds a great product and price, tells his boss, the boss goes to the website and sees a higher price.
3. There are ethical considerations.
Some people think there are ethical problems with charging different buyers different prices. I am not one of them. Companies commonly use price segmentation, which is exactly that. The only difference I see is that price segmentation is focused on buyers with different willingness to pay while price testing randomly assigns pricing.
If it’s appropriate for the company, I like the idea of price testing. Consumer goods companies like Proctor and Gamble often do it, but they do it in different cities at retail stores. It is very unlikely one buyer will see multiple prices. Price testing can be valuable, but we have to consider the experience and emotions of our buyers.
If you want to set up eCommerce A/B price testing, here are two thoughts on how to execute it:
- You should have a high volume of traffic so it’s quick and easy to get meaningful data.
- Consider doing what P&G does. Instead of normal random A/B testing, intentionally choose areas of the country or cities as samples. This makes it less likely any single person will see different prices. However, you still may see one price in Columbus while your parents in Wichita see another.
Of course, there are other techniques you can use to get data to help figure out what price to charge. The simplest is Van Westendorp’s Price Sensitivity Meter. I’ll write more on that in a later post.
To wrap this one up, if you’re going to use A/B price testing, go into it carefully so you don’t anger your market.
