Gleb Romanyuk is a Principal Economist at Wayfair. He leverages his Ph.D. in Economics and his extensive experience in the tech industry to develop cutting-edge solutions for competitive pricing and economic analysis.
In this episode, Gleb emphasizes the underutilization of data by most companies. He recommends harnessing its power to gain insights into business performance and develop optimized pricing strategies.
Podcast: Play in new window | Download
Why you have to check out today’s podcast:
- Understand an economist’s role in analyzing data and optimizing pricing
- Learn to note the distinction between forecasting and prediction versus the causal inference
- Test price elasticity with causality
“I would encourage people to take advantage of their sales data when looking into setting prices. If you don’t have it, then start recording it.”
– Gleb Romanyuk
Topics Covered:
01:56 – How he got introduced into pricing
03:46 – What is an economist’s role in a company
06:04 – Distinguishing forecasting versus prediction
09:23 – Proving causality even with just observational data
15:06 – Case in point: does faster shipping improves revenue and profitability [how to go about the test and the variables used]
18:31 – What he thinks of the fact that most companies don’t know how to use the data they gather
19:41 – Talking about more job opportunities for economists
20:49 – Highlighting a fascinating finding on using value pricing
23:16 – Sharing his insights on Mark’s comment of not favoring taking into account price elasticity for pricing
24:46 – Gleb’s impactful pricing advice
Key Takeaways:
“You can learn a lot about the performance of your business [by taking advantage of your sales data] and do better than just pricing by the market or pricing by the cost.” – Gleb Romanyuk
“Can we measure the long-term elasticity that takes into account the competitor’s response? Usually we can’t really get at it because it involves strategic interactions. I think what we do about it is we take market price into account. But also it’s important to understand to which extent your products are differentiated from the competitors and how fierce the competition is.” – Gleb Romanyuk
People /Resources Mentioned:
- Wayfair: https://www.wayfair.com/
- Amazon: https://www.amazon.com/
Connect with Gleb Romanyuk:
- LinkedIn: https://www.linkedin.com/in/glebromanyuk/
- Email: [email protected]
Connect with Mark Stiving:
- LinkedIn: https://www.linkedin.com/in/stiving/
- Email: [email protected]
Full Interview Transcript
(Note: This transcript was created with an AI transcription service. Please forgive any transcription or grammatical errors. We probably sounded better in real life.)
Gleb Romanyuk
I would encourage people to take advantage of their sales data when looking into setting prices. If you don’t have it then start recording it.
[Intro]
Mark Stiving
Today’s podcast is sponsored by Jennings Executive Search. I had a great conversation with John Jennings about the skills needed in different pricing roles. He and I think a lot alike. If you’re looking for a new pricing role, or if you’re trying to hire just the right pricing person, I strongly suggest you reach out to Jennings Executive Search. They specialize in placing pricing people. Say that three times fast.
Mark Stiving
Welcome to Impact Pricing, the podcast where we discuss pricing, value, and the complex relationship between them. I’m Mark Stiving and our guest today is Gleb Romanyuk. And here are three things you want to know about Gleb before we start. He is the principal economist at Wayfair. That sounds impressive, doesn’t it? He was an economist at Amazon Web Services for five years, and he started as a data scientist and then got his PhD in economics from Harvard. Oh. And he likes Latin dancing. Welcome, Gleb.
Gleb Romanyuk
Thank you, Mark. Glad to be here.
Mark Stiving
Yeah. I hope you don’t mind the way I say Harvard, but it’s just a habit of mine. I can’t help myself.
Gleb Romanyuk
That’s totally fine. I’m not a native. I don’t have this way of pronouncing, but this works. Yeah.
Mark Stiving
How did you get into pricing, usually the first question I ask. So take a shot at it. Even though I wouldn’t say you’re in pricing per se.
Gleb Romanyuk
I am not strictly in pricing. I am mainly an economist and the current title of my job reflects that. So let me step back. Indeed, I got a PhD in economics from Harvard. and well, economists cover a lot of different topics. I focused on, in my research in microeconomics, microeconomic theory, industrial organization. After the PhD I decided not to go to academia, but decided to go to industry. And the tech industry was closer to my heart due to my previous background. And I joined Amazon. And then in the industry, there are few areas where economists kind of find themselves to be the most valuable. And pricing is one of them. I think economists are trained to think through markets, how prices come around, what influences them, how to measure markets, how to measure price elasticity and things like that. And this naturally lent for me to develop an interest and experience in pricing. And I did various things at my position at AWS and in Wayfair I had other projects, but I think the central theme was pricing.
Mark Stiving
Nice. so in my PhD I’ve got a doctorate in marketing. And I spent a lot of time doing microeconomics as well, which I find fascinating. Right? I mean, it’s a fascinating topic and it’s a way of thinking, but I wonder what the heck does an economist do when you get hired for a company? Because those of us who work for companies, we’re thinking, okay, how do we make this decision and tweak this one little thing to make more money? What’s an economist doing?
Gleb Romanyuk
Yeah. It may be actually quite similar to these economists in tech, and, okay, there are, and I’m going to be speaking about tech companies and largely obviously drawing on my personal experience at Wayfair and the Amazon. So economists are kind of the most similar to data scientists at these companies. It’s our econometric toolkit, which I think the employers seek. So answering questions such as, hey, we want to introduce fast shipping on Wayfair. Can you please measure, tell us what would be the lift in revenue in profitability, retention as a result of launch of this program? Or so now closer to pricing we want to drive profitability. I want you to measure price elasticity of demand as opposed to cost-plus pricing. And I believe we might get to this at some point. And tell us what you recommend. so economists develop and build models and work with data. So this is collecting data, using the corporate data, maybe also using public data sets to build models that help customers make decisions around business metrics. Like friendly profitability, but also evaluation of the programs, as I mentioned as in my first example. So that’s kind of a summary of it.
Mark Stiving
This is actually making sense to me in the following. If I were most pricing people, most product managers, people inside companies don’t have the ability to think in terms of big data and forecasting models and testing decision processes. Because it’s just not the way they’ve been trained or they think even though we have all this data and these data scientists inside our company. And so what an economist is really doing is helping these other departments try to make really smart decisions using, let’s call them complex forecasting tools or complex modeling tools. Would that make sense?
Gleb Romanyuk
Yeah. I think that’s right. Here I would like to emphasize an important distinction between forecasting and prediction on one side and the causal inference on the other side. and economists can do both. Another example is predicting sales data or demand for like, for the next quarter, for the next year is like a pure prediction task. Economists also do that, but I think what usually sets apart economists, at least in the places where I worked, is the causal entrance is the thinking through the causal relationships between like forces and the metrics. Like everyone knows that correlation is not causation but for you actually acting on this.
Mark Stiving
I was going to say, I wish everybody actually knew that, but go ahead.
Gleb Romanyuk
Okay. I think many at least heard the saying, maybe I’m wrong. Maybe, you might know better.
Mark Stiving
I think a lot of people have heard it, but they don’t understand what it actually means.
Gleb Romanyuk
Yeah. And for prediction tasks, the distinction may not matter much. Like given, if you want to predict the, let’s say, sales in the next quarter you have the series of the, in the past, you have other variables, you have your trends, and it’s essentially what you’re doing is estimating correlations for this kind of problem. Now a different kind of problem is what happens if we lower our prices by 15% for like, large chunk of our catalog, given that we have never done it before. It’s kind of like the new strategy, maybe something you want to try. So we can’t just look at the existing data. There is no existing data like exactly answering this question. And here the trap that often happens is, we do look at some data that we have in the past and we point to the like fourth quarter when our prices were lower by 15% because it was like seasonal discounts and our sales were like 2X, and we’re like, look, 15% discount, 2X sales. Well, that’s definitely fallacy because there are admitted variables here. In this case, it’s the holiday season. So there are other factors happening at the same time. Everyone shopping at this time, yes, looking for lower prices, but still there is just a lot of demand throwing in. So this is an example of where correlation acquisition kind of generalizing this observation to the rest of the year will be a mistake.
Mark Stiving
So that’s absolutely true. How do you prove causation? And I say that in a really weird sense because I’m almost, let’s say anal about it where I’ll tell you that smoking cigarettes doesn’t cause cancer because you can’t do a random study and say, you guys smoke and you guys don’t smoke, and let’s see who gets cancer. Now, there’s a ton of evidence that says this is a true statement. You can’t prove it because of the way we think of causality. So now, if all you have is historical data, how are you going to prove causality?
Gleb Romanyuk
Okay, that’s a very good question. That’s kind of the question that defines the field or a few in economics. So let me tell you how economists define causality and then how we can get it, I’ll tell you how we can get at it using the observational data only. So the definition of causality for economists is, there are two alternative universes. You either smoke, okay, let’s say, let’s take this like you are now. You either have smoked for 15 years or have not. And then we can compare the results, okay? That’s kind of theoretical. In practice, we say, okay, there is a gold standard of randomized control trial or A/B test. If we took a group of people randomly, drew half of them and made them smoke and not to others, then the difference in results is our causal change.
Mark Stiving
I’d buy that one completely.
Gleb Romanyuk
Yeah. That’s kind of the gold standard. That kind of serves like a definition. If you are able to perfectly conduct this experiment, this is your true causal effect. Now, of course it’s never impossible to do such an ideal test. but you can get closer and closer to it. Okay? But now you even asked me a harder question. Okay, let’s say we’re not even testing anyone, right? Like, we can’t, like, aren’t randomizing, we’re not asking people to smoke or not to smoke. What do we do if we only have observations of the different people smoking and not in the past? then you again, try to approximate this design.
So for example, one way is, okay, let’s look, let’s take a group of people who are smokers, right? And then you create a synthetic control group of people who are not smokers, but by all other observable characteristics they’re the same or not like on average. Okay? I can’t find who your twin is actually, I don’t know if you’re a smoker or not. but we can’t create your synthetic twin but let’s say on average and let’s say the same distribution of age, the same distribution of gender, the same distribution of other medical problems, the same distribution of lifestyles. So we created the group which is exactly the same, but who doesn’t smoke. This creates kind of a plausible design where we can say, okay, the difference in the outcomes for these two groups will be a causal effect. This is how we often approach the problem when we only have the observational data.
Mark Stiving
And so I think that’s all you can do, right? I mean, that’s the best you can do. And of course the counter argument is that there happens to be a spurious variable in there, and that’s a genetic gene that makes you want to smoke, that also causes cancer. And you can’t rule that out. And that’s the problem.
Gleb Romanyuk
Yeah. That’s right. It’s still not perfect. It’s a good example. I actually haven’t heard this story that there is this gene.
Mark Stiving
But by the way, I’m not saying that that’s a true thing at all.
Gleb Romanyuk
Totally . Yeah, it’s true. You can try to poke holes in the arguments and this is kind of why these projects can take some time, because let’s say I…
Mark Stiving
I want to bring this back to pricing or to economics for just a second, right?
Gleb Romanyuk
Yeah, sure.
Mark Stiving
Because I think the point is, unless you can do a control study, you can’t prove causality. But there comes a point where you say, I’m going to infer causality.
Gleb Romanyuk
That’s right.
Mark Stiving
How do you make that point? Right? Is it like a, it is almost like in statistics we say we’ve got a P value of 95 or something.
Gleb Romanyuk
Yeah. That’s right. and I agree for serious questions. Like for drugs the FDA has strict procedures that require testing. When companies often… you do not have the luxury of being able to do tests, even though sometimes you do how often you do not. yeah. So there is statistical power to this test. Yes. kind of you measure the statistical significance of the difference between the outcomes for treatment and control. if you have a large enough sample, then the distinction is very significant. We are more certain that it’s a true effect as opposed to the statistical fluke noise. Another part is less quantitative. It’s like towards what you pointed out. What if there is a gene that kind of controls both? And so here the conversation becomes, okay, how many stories can we refute like this?
So let’s say someone like we, we get into the room for a meeting, someone says, oh, what about this gene? Then we try to include the data on this gene, maybe collect the data on the gene, like bring it into the study. Let’s measure this gene into the people and bring into the study and check if it actually matters. And so then someone else can say, okay, what about this story? This might be an emitted variable. Then we try to include that variable. And when we kind of like shut down like a bunch of plausible stories, then we are more and more certain that the effect is real.
Mark Stiving
What was the example you gave at the very beginning? You were talking about at Wayfair we could see if faster shipping, improved revenue and profitability. How would you go about doing that? Right. What’s the model look like or what are the variables you’re going to use?
Gleb Romanyuk
Yeah. Let me kind of make a more specific case for this. So let’s say we are in the very beginning of the projects, we haven’t tested anything. We only have an idea, let’s add a badge on the products that can ship fast, but we don’t have this badge yet. What we have is just the history of orders. And some orders had, like in the past, shorter shipping times than others had longer shipping times. So this is the variation that we can try to exploit. Okay. So kind of like economists often, like usually hunting for variation in the data. They love variation. We want to see differences in some variables because if there is a variation, we can compare things. so we want to find the variation in the lead times in the history and use it.
And from here, kind of the story goes similar to what I just said, let’s pretend that the products that happen to ship faster were kind of, it was like intentional even though in the fact it wasn’t, like in the first stage of such an analysis, you usually almost always find that the products that happen to ship faster observably different from those that didn’t ship faster. Because it just may be smaller products. It’s just the small boxes as opposed to the large couches or the seller is just better by other characteristics, reviews, maybe incident rates. So we notice that they are different and this provides us with control for these differences. And from here, this is what I was saying, we try to actually create two groups of the products that shipped faster and longer that are similar on all the other characteristics.
Mark Stiving
This is the virtual twin that you were talking about earlier?
Gleb Romanyuk
Exactly. And usually kind of if you can imagine if there is like a cloud of different orders, usually those and you can classify some of them as that shipped fast or that shipped slower. And those kind of around the middle will be similar in their characteristics. So we can exploit those groups of products for those for whom it is possible to find synthetic twins. And we compare those to make statements about how much fast shipping helps in terms of the conversion revenue, just like order volume. I can go into and talk about specific models like mathematical that we use and, but…
Mark Stiving
Although I would find that fascinating, I find everything we’re talking about fascinating. I doubt most of our listeners would care too much about the specific models. One of the things I find so interesting though is the idea that says, here’s a question we’d like the answer to. Now, can we go figure out how to get the data to tell us? And I find that interesting because companies have so much data, but they don’t really have people who think through what does all that mean? Or what can we get out of it?
Gleb Romanyuk
Yeah, I agree. And I think, yeah, there is often, there’s like a trove of data sitting somewhere. and I think even for tech forward companies like Wayfair, we are not kind of effectively using a hundred percent of the data we’re gathering. But on the other hand, I think I kind of, I may have a bias of working like in tech companies. So people like digging into the data.
Mark Stiving
I think that’s a true statement. The tech companies probably are a little better than non-tech companies, but if you just think about it, data analysts, right? If you’re trained as a statistician or a big data analyst, that’s great, but you haven’t been trained to think about business and what are the correlations we’re actually looking for? And so you really need someone who’s somewhere between the, hey, we’re trying to maximize profit and the, hey, let me go crank through some stats for you.
Gleb Romanyuk
Yeah, I think that’s right. I’d love to believe that economists kind of fit into this niche and this bridge in between, and that’s why companies value us.
Mark Stiving
Well, the good news is there’s now a job for economists.
Gleb Romanyuk
Yeah, that’s true. It hasn’t always been, I think when I just started my PhD, few companies were hiring economists as economists. Many were hiring them and coalescing them into data scientists or analysts and it’s not exactly the same, even though we can do these jobs too.
Mark Stiving
Yeah. I have to say, prior to this conversation, I had not thought of the role of economists inside companies. And so this is truly fascinating. I have thought of the gap between data analysts and people who are trying to make business decisions, but I hadn’t filled it with an economist, and I think that’s a really brilliant thought.
Gleb Romanyuk
Yeah. I’d say, it’s still a small profession but I think we’re making our rounds into the companies.
Mark Stiving
Nice. So, since we have a few minutes left, can you share a fascinating finding that you found at any of your previous companies doing this? I’m just curious. I love this stuff.
Gleb Romanyuk
Whew. Okay. I will tell you about a finding, okay, not mine, but that happened at the Wayfair is at some point Wayfair was just doing a cost-plus pricing for their catalog. And the Wayfair catalog is broad. and then people decided to, okay, can we do better than that? Because the catalog is also diverse and it’s impossible for a group of people to manually look into each group of products. And they hired economists to build a price elasticity model to measure demand and try to price not based on the cost, but based on the demand. I guess we can call it value pricing. And at first it was just an experiment because it’s kind of, okay, I didn’t know what are you going to do? Like moving the margin by one percentage point, it doesn’t even matter.
It’s complex. But it turned out that after testing this model, the profitability exploded. Like even they were able to increase profitability, holding the revenue fixed. So it wasn’t just kind of finding the right price point for the entire catalog, even that was part of that too, but also distributing the margin throughout the catalog in a more efficient way so that we are able to lower the products that have higher elasticity and higher rate of lower elasticity. And that was a fascinating finding that this actually works.
Mark Stiving
Nice. That actually is fascinating. I want to share one of my thoughts on price elasticity with you and tell me that you’re welcome to agree, disagree, doesn’t matter. But I personally do not like price elasticity. And the reason I don’t like it is because it doesn’t take into account my competitor’s actions. So if I drop my price and I get a big boost in sales and I say, wow, that was great, I should do that again. And then my competitor says, ouch, this hurts. I think I’m going to have to bring my price down as well. Now we just hurt industry profits instead of actually gaining from our reduced price. So what are your thoughts since you guys are using price elasticity and making these decisions?
Gleb Romanyuk
Yeah, that’s great actually. That’s indeed the constraint of this approach. it’s kind of a long term, you need to take into account the response of your competitor. And again here the ideal for the measurement team is, okay, can we measure the long-term elasticity that takes into account the competitor’s response? Usually we can’t really get at it because it involves strategic interactions. I think what we do about it is we like to take market price into account, into our measurements. but also it’s important to understand to which extent your products are differentiated from the competitors. and how fierce is the competition? Do your users come to your website, compare the prices with the others, like do most of them compare or in a small fraction, if you have a loyal base, then as a company you have more opportunity to rely on price elasticity. Then if you do have only those who cross shop.
Mark Stiving
You just caused a whole bunch of thoughts to run through my head. Gleb, I could do this all day. This is fascinating. I love these conversations.
Gleb Romanyuk
Likewise.
Mark Stiving
But we’re going to have to start wrapping this up. Let me ask you the final question. What is one piece of pricing advice you would give our listeners that you think could have a big impact on their business?
Gleb Romanyuk
I don’t know your audience well, but I would encourage people to take advantage of their sales data when looking into setting prices. If you don’t have it, then start recording it. I think you can learn a lot about the performance of your business and do better than just pricing by the market or pricing by the cost.
Mark Stiving
Fabulous answer. Gleb, thank you so much for your time today. If anybody wants to contact you, how can they do that?
Gleb Romanyuk
You can reach me by my personal email, [email protected]. So that’s my first name, G LE B dot R O M A N Y U K, last name @gmail.com. That would be the best way to reach me.
Mark Stiving
Excellent. And to our listeners, thank you so much for your time today. If you enjoyed this, would you please leave us a rating and a review? And finally, if you have any questions or comments about the podcast or pricing, feel free to email me, [email protected]. Now, go make an impact!
Mark Stiving
Thanks again to Jennings Executive Search for sponsoring our podcast. If you’re looking to hire someone in pricing, I suggest you contact someone who knows pricing people contact Jennings Executive Search.
