S1E27: Interview with Kyle Kretschman, head of economics at Spotify

In this week’s episode of The Mixtape with Scott, I had the pleasure of interviewing Kyle Kretschman, Head of Economics at Spotify. It was a great opportunity for me because Kyle is one of the first economists I have spoken to who didn’t enter tech as a senior economist (e.g., John List, Susan Athey, Michael Schwarz, Steve Tadelis). Kyle entered tech straight out of graduate school. He spent much of his career at Amazon, a firm that has more PhD economists than can be easily counted. Under Pat Bajari’s leadership there, Kyle grew and his success was noticed such that he was then hired away by Spotify to lead up their economics team. At the end of the interview, I asked Kyle an economics article that has haunted his memories and he said “BLP”, which is affectionate shorthand that “Automobile Prices in Market Equilibrium” by Berry, Levinsohn and Pakes 1995 Econometrica goes by. I really enjoyed this interview, and despite the less than ideal sound quality at times, I hope you will too.

But before I conclude, I wanted to share some more of my thoughts. This series I’ve been doing on “economists in tech”, which has included interviews with John List, Susan Athey, Michael Schwarz and Steve Tadelis, comes from a complex place inside me. First there is the sheer curiosity I have about it as a part of the labor market for PhD economists. As I have said before on here, the tech sector has exploded in the last decade and the demand for PhD economists has grown steadily year over year. Tech demand selects on PhD economists with promising academic style research inclinations. There is substantial positive selection in this market as firms seek out strong candidates can be produce value for them. This is reflected in both junior market salaries, but also senior. Job market candidates are economists with technical skills in econometrics and economic theory, not to mention possess competent computer programming skills in at least one but often several popular coding languages. They are also candidates who were often entertaining careers within academia at the time they entered tech, and in those academic careers, they envisioned themselves writing academic articles about research they found personally and scientifically important and meaningful. Going into tech, therefore, would at least seem to involve choice that may go far beyond merely that of taking one job over another. It may involve a choice between a career in academia and a career outside it, which for many of us can feel permanent, as though we are leaving academia. And for many economists, it may be the first time they have ever contemplated such a thing. If they do internalize the story that way, if they do see taking a job in tech as “leaving academia”, then I can imagine that for at least some economists, that may be complicated, at least.

But there’s another reason I have been wanting to talk to economists in tech and that is I am very concerned about the welfare of our PhD students. In a recent article published in the Journal of Economic Literature, economists interviewed graduate students in top economics programs. They found there incredibly high rates of depression, anxiety, loneliness and even suicidality. This is a common feature of graduate studies, but it is interesting that PhD economists have incredibly good employment opportunities and yet the depression and anxiety plague there too. One of the things that struck me in that study was the disconnect between what graduate students felt about their work and what their advisors felt about their own work. Many students, for instance, do not feel they are properly supported by advisers, do not believe their advisers care about their research success and do not even care about them as a person. Whereas most Americans (and faculty) feel that their work has a positive impact on society, only 20% of PhD students in economics feel that way. (I discussed the article as well as my own research on the mental health of PhD students here.)

I suppose part of me feels a great sigh of relief to see the labor market for PhD economists expanding in light of those troubling statistics. If students know that life is full of infinite possibilities, then perhaps they can begin to process earlier what they want to do in the short years they have on this small spinning ball of rock we call Earth. If students do not in the end want to become professors, if they do not have the opportunities to become one, they should know that there is no “failure” involved there. Careers are just that — careers. They do not tell us who we are. The sooner a student can detach from the unhelpful story that our value is linked to a vita listing our accomplishments, the sooner they can begin their own life work of choosing their meaning. Can having more labor market opportunities with more employers competing for them help do that? Well no, not really. At least, not exactly. It can disrupt certain equilibrium, but then the new equilibrium can just as easily cover that up too. Still, I do like the idea that to keep students in academia, universities and departments must fight harder for them, pay attention to them, and invest in them as people. I like the idea that students have more options and that the options are diverse. Will it help their depression? Well, that’s another matter, as that’s complex. And presumably the economists in the survey I mentioned were themselves well aware of the career options they had since they were coming from the nation’s top 10 PhD programs in economics.

I suppose my point is that ultimately, the burden of life really cannot be resolved with money or career. We are trained to look there because we have boundless appetites. But ultimately the hard work of navigating life can only be helped so much by a job. We must still decide for ourselves what meaning we will choose for ourselves. But one thing I know, and one thing which I think our profession is profoundly bad at saying out loud, is that if we make our identity connected to vitas, we will not just be miserable, we will be hopeless, and probably poisoned. Such a mindset leads to endless laps on a brutalizing treadmill of meaningless performance in which a person chases for first place in a race they don’t remember signing up for and which they cannot win. They compare themselves with others running, not knowing that they too are brutalized by their own treadmill, not realizing that it is impossible to catch up with someone else as there is always someone else ahead of us. The sooner we learn that the joy we long for will not come when we get a top 5, the sooner we can look elsewhere. It has taken me many years to relearn a lesson I learned decades ago — I am whole now. I am complete now. I still run, and I still chase, but I am not chasing completeness. I am not chasing my own wholeness. Being whole and complete has nothing to do with a career. Careers are ultimately orthogonal to hope, which does not mean they do not matter — they absolutely matter. But if asked to deliver meaning, we will find that our jobs are as weak as wet spaghetti at such a task as that.

So, I suppose in some ways I simply want to announce — there are incredible opportunities for economists inside government, commerce and academia. But the weight of this life is not likely to be lighter in any one of them, for the weight we feel in life is largely self imposed, inside us, in the stories we tell about who we are and for many of us who we are not. Those stories are real, because we feel them and because we believe them, but they are not true. All stories are wrong, but some are useful, and the story that our lives can only matter if we have certain types of jobs or certain types of success, while it may be useful to getting a paper out or accomplishing something important, in a much bigger sense it is hollow at best and pure poison at worst.

TRANSCRIPT

This transcript will be updated once the more complete transcript is finished; for now it was transcribed using voice-to-text machine learning.

Kyle Kretschman:Might not have prepared myself well enough to be attractive for some of the most pop most top tier schools. Scott Cunningham:In this week's episode of the mix tape with Scott, I had the pleasure of interviewing Kyle kretchma the head of economics at the streaming platform. Spotify. Before I dive into the interview, though, I wanted to give you a bit of a heads up about the sound quality. Unfortunately, the sound quality in the interview on Kaza side is a bit muffled. We discussed refilming. It tried to find a way to tweak it, but there were certain constraints on the actual sound itself that kept us from being able to do it. And we didn't feel that refilming, it would be good because we thought that the interview had a lot of serendipitous kind of spontaneous tangents and things spoken about that. We thought students and people in academia would want to know, would need maybe even need to know. And I doubted that I could recreate it, cuz I don't even know why it happened. Scott Cunningham:So I'm gonna post a video version of this at my subs, for those who feel that a video version would help them kind of follow it in so far as the audio might be at times challenging. So check out the subst for those of you that wanna watch, watch it instead of just listen to it, hopefully that'll help. I won't say much here by way of introduction, except to say a few things about Kyle, because I wanted to let Kyle tell you his story in his own words, cuz it's his story to tell. And it's an interesting story. Kyle's a PhD economist though from the university of Texas Austin, which is down the road from where I live and work at Baylor, where he wrote on topics in graduate school and applied econometrics, empirical industrial organization or empirical IO and public choice after graduating, Kyle went to Amazon, not academia. Scott Cunningham:In fact, given we might start the boom of tech hiring PhD economists in the early to mid 20 2010s. You could say Kyle maybe was sort of one of the earlier hires among that second wave of PhD economists that went there. He worked for several years at Amazon before being hired away by Spotify to head up and lead a new economics team there, perhaps this is part of a broader trend of tech firms building up more internal teams, not just of data scientists, but like Amazon departments of economists who knows recall though from an earlier interview with Susan athe where, when I asked Susan why she said pat Maja had done something amazing at Amazon, she said he made economists productive. And in time he made many of them productive and very in productive from what I've been able to follow. And Kyle is from what I can gather someone whose skills matured and deepened under the leadership of Papa jar at Amazon and other leaders at and other economists at Amazon. Scott Cunningham:And he was ultimately hunted down by a major tech term to create an economics team there I'm by no means an expert on the labor market for PhD economists. I just have been very intrigued and curious by the, the, the Mar the labor market for PhD economists in tech, because well, partly because of realizing first that cause of inference was really valued in tech, but then to sort of realize that there was just this very large community of economists there, but I don't think it's controversial to say over the last 10 to 15 years, the tech industry really has been disruptive in the labor market for PhD economists. They continue to hire at the junior and senior market in larger and larger volume selecting more and more on people who likely would've gone into academia into tenure track or tenured positions. They pay very high wages, some of the very, some of the highest wages in the country, both at the junior level and especially at the, at the higher end at the, at the more advanced levels, people can earn compensation packages by the, in the, by the time they're in their thirties, that many of us didn't know were possible. Scott Cunningham:It's in my mind, historically novel, and I might be wrong about this, but it, it seems historically novel that the PhD economists who likely would've produced academic research papers in tenured and tenure track jobs have begun to branch out of academia, but maintain those skills and maintain that research output. It's partly driven best. I can tell, buy Amazon, I might be wrong, but by Amazon and paja, as well as Jeff Bezos own view, that economists are what I guess we would just say value added for many firms. Therefore I'm continuing to wanna speak with economists in tech to help better trace out the story. This interview with Kyle follows on the back of earlier interviews with people in tech like John list, you know, a, a distinguished professor of economics at the university of Chicago, but also the former chief economist that Lyft and Uber now Walmart Michael Schwartz, former professor of economics at Harvard. Now, chief economist at Microsoft and Susan athe former chief economist at Microsoft professor at Stanford and now chief economist at the DOJ. I hope you find this to be an interesting dive into the industry. Learn a little bit more about economists there, but by, by learning the about one particular important economist, there a, a young man named Kyle crutch, head of economics at Spotify, my name's Scott Cunningham. And this is the mix tape with Scott. Scott Cunningham:Well, it's my pleasure today to have, as my guest on the mix tape with Scott, Kyle crutch, Kyle, thanks so much for being on the call. Kyle Kretschman:Hey Scott, thanks for having me really appreciate the time to talk Scott Cunningham:Well before we get started with your career and, and everything. I was wondering if you could just tell us your name and your title and where you work. Kyle Kretschman:Sure. Yeah. As you said, I'm Kyle kretchma, I'm the head of economics at Spotify, Scott Cunningham:Head of economics at Spotify. Awesome. Okay. I can't wait to talk. So let me, let me, let's get started. I was wondering if you could just tell me where you grew up. Kyle Kretschman:Sure. So most of the time I grew up in outside of Pittsburgh, Pennsylvania, about an hour north of the city, real real small town probably had one stop light. And maybe the, the funny story that I can share is what I took my wife there. She asked where's the Starbucks. And I said, no Starbucks here. There's no Scott Cunningham:Starbucks. Kyle Kretschman:Yeah. So pretty small town called Chippewa township in Pennsylvania. Scott Cunningham:Oh, okay. Is that near like Amish stuff or anything like that? Kyle Kretschman:No, that's the other side of the state. So this would be Western Pennsylvania about near the end of the turnpike, about five minutes from the Ohio border. Scott Cunningham:Oh, okay. Okay. You said, but you, did you mention, you kind of grew up in different places? Kyle Kretschman:Yeah. So before that, my father worked in civil engineering and so would do build roads and bridges basically across every, across the nation. So I was actually born in Louisiana, lived there with, I think for a whole two, three weeks. I don't quite remember. Cause I was pretty young obviously, but then Michigan and then spent some time in Philadelphia before moving out to Pittsburgh around second grade. Scott Cunningham:Oh, that's kinda like, that's like when people described their parents being in the military, just kind of moving around a lot. Kyle Kretschman:Yeah. A little bit. So, but Scott Cunningham:Then you settled in the second grade Kyle Kretschman:That's right. Yeah. So outside of Pittsburgh and then stayed in Pittsburgh through high school and even through undergrad. Scott Cunningham:Oh, okay. Oh, you went to undergrad in Pennsylvania. Kyle Kretschman:Yeah, I did. So I went to undergrad at the university of Pittsburgh. Oh, okay. It was, yeah. If, I guess maybe continuing the story growing up in a town with no Starbucks. I was, I was pretty intrigued by going to a city. Yeah. And find out that lifestyle and yeah, we might have lived pretty close, like an hour away, but we didn't go down to the city very much. So Pittsburgh was just really, really enticing for a city to, for, to go to undergrad in. And so I basically looked at all schools that were in cities and so the proximity plus then the, the ability to just spread my wings and explore what it's like to be in a city was really, really enticing. Scott Cunningham:Did any of your friends go to pit with you? Kyle Kretschman:Yeah, so there's probably, I grew, I graduated from a class of about a little over 200 people in high school and I think there was like five or six people from high school that went to pit for my class. So definitely had some really good friends who went and kept in touch with, through undergrad. Scott Cunningham:Mm. Yeah. So it wasn't, were you sort of an early generation or you weren't, were you a first generation college student in your family or did your parents go to college Kyle Kretschman:Combination? So my dad went to Penn state civil engineer, as I mentioned, me and my mom actually graduated from undergrad the same week. So my mom went back to school later in life after me, after we went to school. And so yeah, we, we were able to celebrate graduation cuz she went to a small private school right outside of the city also. Scott Cunningham:Oh, okay. Okay. Yeah. Well, so what did you like to do in high school? Kyle Kretschman:So I played a lot of sports before high school and then I kind of switched into, and this was a traditional sports of football, basketball, baseball, but then I switched into tennis in high school. And so that kept me busy, but along with a lot of academics and really, really liked computer science. So played a lot of video games growing up, really enjoyed like that aspect in combination. Scott Cunningham:What games were your, were you, did you play on a, on a video game, plat platform? Like an Nintendo or did you play? Kyle Kretschman:Yeah, no, we played a lot of plays very much into like role playing games. Some of the arcade games like Marvel versus Capcom. So yeah. Yeah. Very, very interested in gaming. Yeah. Maybe I was a little too early for that. Cause you know, every, everybody in the 1990s was like, oh, I could make pu money playing video games, which wasn't true back, which wasn't true back then, but that's right. You know, nowadays Scott Cunningham:You can that's right. Yeah. You know, that's right. You can do it. There's all kinds of ways you can make money doing things today that nobody knew was possible 10, 10 or 15 years ago. Even Kyle Kretschman:My Scott Cunningham:That's cool. Yeah. I, I, it's funny, you know, computer games can keep a, keep a kid in high school going, you know, like especially I think they're kind of misunderstood. I, I had a lot of friends that, well, I mean, I, I, I had, when I didn't have a lot of, we moved from a small town in Mississippi to Memphis and I, those, those that first year when I didn't have a friends, I did bulletin boards and played Sierra online games like Kings quest. And it's like, it's like, you know, not intertemporal smoothing, but like inner temporal socializing, smoothing, you know, so that you just kind of get through some periods that would otherwise be a little lonelier. Kyle Kretschman:Yeah, for sure. And I mean, I mean for this audience, like most video games are some sort of form of constrained optimization. So there was, there was the inkling that I, I liked understanding how economies worked in high school through this and yeah. Going back to my mom, my mom always said like she encouraged it and she encouraged education. And there was actually kind of like that nexus, whenever I took economics in high school, it was like, oh, you know, some of these games really are full economies that are constrained and constrained in a way that you can understand and complete in, you know, under a hundred hours. Right. But there was that combination that was kind of showing itself of computer science, computer gains and economics of putting itself together. Scott Cunningham:So you were kind of thinking even in high school about economics in that kind of like, you know, optimizing something and like, like almost that modern theory that we get in graduate school. Kyle Kretschman:I think more, I had the intuition when I didn't have know how to say what it was in high school because my high school was pretty forward and that it offered both advanced computer science courses that could get you through definitely through first year of undergrad, maybe even through second year with advanced placement. And then they also offered advanced placement economics. And so I, I ended up taking advanced place in economics my junior year when most people took senior year. And so whenever I was going small Scott Cunningham:Town, even in that small town, they had, you had good your high school. Good econ. Kyle Kretschman:Yeah. It was a real, it was a really good high school that would put together good curriculum that did a lot of college preparatory work though. They, wow. They really leaned into the advanced placement, the AP courses to get students ready to go to school. Scott Cunningham:Wow. Wow. So even at, as a junior, you're taking AP econ, you know, you don't have to take AP econ. That kind of is say that, that sounds like somebody that was kind of interested in it. Kyle Kretschman:Yeah, very much. Yeah. And again, as soon as I, I definitely didn't get to the graduate level of understanding, like, you know, LaGrange multipliers, but the, the micro and macro sequence just made intuitive sense to me. It was like, it was kind of where I was like, yeah, this fit. And this is how I think. And some people might criticize me now that I think too much like an economist. Right. Like, but at the same time, it just like, it started to put together that language and even more so some of the frameworks that really kind of drew me into it. Scott Cunningham:Well, did you, did you, did you notice that you had this interest in computer science and this interest in economics and that they might be one, did you get a feeling that they could be in conversation with each other? Kyle Kretschman:Not Scott Cunningham:At first, our ancestors a hundred years ago. Didn't, you know, those economists didn't think that way, but now it's just so natural for this generation of economists to be almost one half, you know, one third mathematician, one third economist, one third computer scientist. Kyle Kretschman:Yeah. So not at first, but I, I feel like I made have like lucked into it, honestly, because whenever I chose to go to Pitt, I chose to start as computer science because I knew what that pass was. I was inspired by my older brother, the great teacher in high school. And like, I was definitely like, okay, a software software development engineer career is great. It's cutting edge. It's there. But after probably like the first year, it just didn't feel that end state didn't feel right. And so I made kind of the hard decision to choose, honestly, to switch into economics as a major, because I wasn't sure what the end state would be, where I was going with it. Cuz it was definitely felt more amorphous, you know, it's a social science, so yeah. It didn't feel like it was gonna be as clear cut and as, and have as much certainty. But pretty quickly, like after a year was like, oh, well we're doing, we're using E views at the time. All right, this is coding. I know how to do this. This is great. Right. And starting, starting to see some of that in undergrad was like the, kind of the aha moment that like, yeah, this is, this is a place where I can apply this love of coding and problem solving, but problems and solutions that I find really, really hard and interesting. Scott Cunningham:It was because of econometrics though. It was in that. Kyle Kretschman:Exactly. Yeah, yeah. Scott Cunningham:Yeah. Wow. That's, that's really interesting because you know, I think it's still the case that, you know, you can easily end up with an econometrics class that remains purely theoretical and doesn't end up, you know, exposing the student with a lot of actual coding, but it sounds like your professors were, were getting you into working with data. Kyle Kretschman:That's correct. Yeah. Both. Both within the class. So like I said, we used E views at the time. Yeah. And again, kind of like learning as a go, I, I don't think I really knew what I was doing whenever we were typing commands and E views, but the computer scientist in me was like, okay, well this is a function. I know functions. Didn't put outputs, but definitely didn't understand necessarily things that were going under the hoods or you know, all of the theory that goes with it. Oh, right, right, right. So it was, you Scott Cunningham:Knew the coding part, you knew you were coding, but you did, but like the, the actual statistical modeling was kind of the new part, but that was a way for you to kind of engage it a little bit. Kyle Kretschman:Yep, exactly. Scott Cunningham:Oh, that's interesting. That's interesting. Well, so what were you gonna have to choose between a computer science and an econ major did or did you end up doing both? Kyle Kretschman:So I chose an econ major, but then I had what I would call basically minors or concentrations in computer science, but then also in statistics and also in math, because once, once I had an internship at a bank and was doing data entry and I was like, eh, I don't think this is what I wanna use my economics degree for. Yeah. I had a couple professors at pit named Steve Houston and Frank Giani who brought me on as a research assistant, an undergrad to start being part of some of like their survey projects and data collection. And even, even one of 'em I don't, Steve was crazy, but he even let me TA classes on undergrad, so oh, wow. But he kinda, I mean, I, I say that jokingly because it was formative for me, it was like, okay, this is great. How do I do more of this? And he was like, well, you go get your econ PhD. And I was like, so I can be a teacher with computer science and doing economics altogether. He goes, yeah, let's do that. And so it was with the help and support of some of these really good professors and education to kind push me on this path consider to get Ancon PhD. Scott Cunningham:Mm. And that's when you were like, so how, how, what, what year would you have been in your program? Kyle Kretschman:Probably. I think I was in my junior year where I was starting to explore this. And then in my senior year is where I was like, okay, I'm actually gonna be doing more more of this and applying to grad school because going back, as I said, I entered with some credits. So my senior year was very, I didn't need a full course load. So I was looking for other things to keep me busy, which maybe, maybe that's one of the themes of this conversation is I kinda kind of like the variety and really have variety seeking behavior too. Yeah, Scott Cunningham:Yeah, yeah. Yeah. So you graduate, was there like a field that you were mostly interested in? Kyle Kretschman:I thought I would be going into macro economics. Macro. Yep. Yeah, because Steve worked on the council of economic advisors and I was really inspired by that and the application of economics within, within policy and just again, always applied economics, not necessarily theoretical. So yeah. Then again was, that would be sort of like labor and macro was like the initial idea, but finally Scott, I didn't do all my homework and like, think about like what grad school looked like or all it looked like. I kind of went a little bit more naive than I think other people with, again, ideas of how I could become like a teacher, an educator with some of these tools versus like how disciplined and single thread you need to be on research to be within an econ PhD program and to see that. Scott Cunningham:So you, so you kind of were like, so when you were thinking about graduate schools, what, how, what, what did you sort of, can you walk me through like what you were thinking and how you went about trying to apply to graduate school and where you ultimately chose? Kyle Kretschman:Yeah, sure. So applied probably the, the top 10 and the top 10 probably said no thanks. But also then was targeting specific schools that we had relationships with that I knew would provide computer science and macros. So university at the Iowa at the time, this was 2000 and had a really strong macro program. And then also at the university of Texas with Dean Corbe there, they also had one in Russ Cooper. And so those were like the two that I was like targeting at outside of what the top schools were. But yeah, as I, I kind of mentioned, I, I might not have prepared myself well enough to be attractive for some of the most pop with top tier schools because kind of, you know, as I said, bounced around and would be yeah, a little bit working on it a little bit different things and have computer science versus being solely focused on like economics and math and things that might be more of what the top tier schools were looking for. Scott Cunningham:Yeah. Yeah. You know, you know, it's like the, I mean, I'm the same way. I didn't ha have any econ classes in college. I was a English major, but the, the, the diff there's so many students that sort of seem to almost for whatever reason, know a lot sooner what they want to do and then like make those choices. And then there's just many of us that are, you know, in a process of search yeah. That when you're in a process of search, well, you, you know, by definition, that's like you're using that time to search. Kyle Kretschman:That's exactly right. As Scott Cunningham:Opposed to saying, I've gotta take, I've gotta become a triple major computer science, math, econ, and have to do like, you know, these set of these set of steps that, you know, there's no way I could even have known to do it unless somebody had told me it's weird. I mean, it's just funny how the little things can have such big repercussions for your whole life, but it's, but it, it worked out great. So you end up, where do you end up going? Kyle Kretschman:I went to the university of Texas at Austin. Scott Cunningham:Yeah. Yeah. What year was that? And Kyle Kretschman:So, so this would've been 2002. Scott Cunningham:Oh, okay. So you go to oh 6 0 7. Kyle Kretschman:Okay. And so ended up working. So I ended up working a lot with Jason, Ava. Yeah. And who came in and became the, the head of the department. Yeah. Applied econometrician who just did an amazing job going back to whenever I said, I didn't know how things worked under the hood, in those formulas. He didn't even let us use those formulas. So anytime we were doing applied econometric econometrics with them, not only we learning to teach, we're learning the theory, but he said, you have to code it yourself. You have to do the matrix algebra, you have to calculate standard errors. You can't really call those functions. So that was probably again, that wasn't until the third year, but yeah, in the first year to go back a little bit, Scott Cunningham:I, that played to your strengths though. I bet that played to your strengths. Yeah. Just at the end of the day, wanting to be someone that, that wrote down the raw code. Kyle Kretschman:That's exactly right. And, but the first year I didn't play my strength. Yeah. Yeah. So the first year I felt, I felt a little bit outta water and I was like, this is, I remember when we were proving what local non association. And I was like, this is, this is one hard, but also like, again, going back to like, that is this actually how I wanna be spending my time and right. I, I was like, yes, I do. But I was like, I, I knew that I needed to get to those applied applications. Yeah. And so that's, again, why I was thankful to be able to work with Jason and Steve Trayo and a few other, they applied econometricians at Texas that really encouraged me to explore starting in the second year. They didn't us like pin it down. And so I, I thought I, at the second year I worked like wrote the first, a paper on school choice and trying to see if I could find some sort of instrument on school selection on public versus private. And again, so that led to like that idea of like applied econometrics was really, really the thing that like, I was like, okay, now this fits again. Once we got into second and third year Scott Cunningham:Was, was picking up that intuition, that kind of like labor style identification, causal inference kind of approach. Was that something you picked up from Jason or was that just like from your labor people? Oh, okay. Kyle Kretschman:Yeah. That's yeah. From Jason and Steve a lot. They did a great job of doing that. And yeah. So then, yeah. Then I, then I threw in, I knew threw a little bit of a switch in there also, and my co-author Nick master and Arti and closest friend and classmate in Texas was very theoretical and very interested in applied empirical IO. And so we started working in that field also together. And so then I got to work with the Han me vet and Ken Hendrix on using empirical IO. So, oh, wow. Yeah. And so again, Scott Cunningham:This is the more structural, more structural econometric. So you've got this like reduced, you've kind of got this like traditional labor reduced form type of, part of your brain. And then you've got this empirical IO structural part of your brain kind of emerging at the same time. Kyle Kretschman:That's right. That's exactly right. Yeah. And then we threw, we threw everybody for a loop. I also saying we wanted to study study politics and how money turns into vote using both using all these tools. So yeah, I can see here kind of saying in hindsight, like it all makes sense in this story that I'm telling you, but at the time it was more of what you were talking about. It was searching. It was, I wanna be working on really interesting applied problems. I love the toolkit that economics provides in framing. And yeah. I have to be coding to be able to utilize these tools that I've had built up in the past. Scott Cunningham:Yeah, yeah. Yeah. So, so matching with Nick was really important Kyle Kretschman:Very much. Scott Cunningham:And why, if you hadn't to match with Nick, I mean, just kind of outta curiosity, if you could articulate the value added of that whole partnership, what was it? Kyle Kretschman:Yes. Sure. So, so we matched basically from math camp going into, going into the first year because Nick came both from the pure math and physics background and also had some experience in the air force. So the air force was sending him to Texas and he, we were, we were definitely, we definitely didn't have a lot of vend overlap on the fact. He's like, well, I would have the intuition and some of the computer skills, Nick would have the theoretical math skills, Scott Cunningham:The theoretical math skills. Yep. Kyle Kretschman:And then we just had, we had the common factor that we wanted to work hard together and learn together and we're willing to, we're willing to intellectually hash out really tough things together. Yeah. So yeah, he huge credit to him through being able to put up with me. And he says, he says the same thing once in a while. But again, matching with somebody that had the, the more real analysis proof based understanding of math was so valuable for me. And especially, Scott Cunningham:I think some empirical IO, especially empirical IO, just being able to, you know, think like an economist in the area of IO is thinking real deep about, you know, a rich set of models and modeling approaches. Kyle Kretschman:That's Scott Cunningham:Exactly right. That's definitely not what you're learning in your econometrics classes, even though they might go together. Kyle Kretschman:Yep. So, so yeah, it was just a, it was a really good match from the beginning. And so we complimented each other and we're, we're able to build a strong enough relationship to be able to be able to hash out, have really long nights yelling at each other, we say in the office, but it never, it was always for educational purposes and lifting each other up. Scott Cunningham:Was that different than what you thought grad school was gonna be like? Kyle Kretschman:Yeah. So I knew the research component a little bit. I just didn't under understand the unstructured research on how that was gonna go and like the cadence and where it was gonna and how that was gonna be so required to develop your own viewpoint. Yeah. I thought it would be more directed cuz as a 22 year old, that was the experience I had generally. So that was the big one was the undirected and I liked it, but it was also very difficult. Scott Cunningham:How would you describe what you're talking about to your college self? Who kind of like, you know, he, he doesn't really, he doesn't even have the vocabulary for what you're describing. What would you say? It was like, Kyle Kretschman:I think you use a good term. You have to be not only wanting to search, you have to be willing to search, but you also, then you have to put in the guardrails yourself to keep it focused because you're not necessarily gonna have those external guardrails that you will have from an alternative path of going to either like a master's program that's gonna be more structured or going in an industry or going to get a job. Right. Like I mentioned at a bank for like a 22 year old where entry level jobs are gonna be more structured. Yeah. So yeah, I just, I, I probably knew it, but I didn't know what it meant to be and what, what it meant to experience it. Scott Cunningham:So how did Jason and, and Steve kind of, and any other faculty, how, how did they, how did they, I, so I did this interview with Susan athe and she was saying that, you know, the amazing thing that pat Maja did at Amazon was he managed to make economists productive, which kind it was kind of a weird, weird way of saying it. And so in a way it could, in a way you could imagine a department that sort of has like a, you know, this idea of like research has got to come. There's like a, there's like a, a journey that a graduate student has to come on to just to basically make a decision to be a researcher. Yeah. You know, and you could imagine that creating the conditions for that is, is involves faculty member, doing stuff that's not necessarily obvious. What, how did they, how do you think they contributed to that for you personally? Kyle Kretschman:For me personally, at the time, again, it goes back to encourage the exploration versus mandating or saying that I need to be on one path. So like even Nick and I at the time explore the idea of a private company and how, what, what that would be into like pinching, pitching a venture capitalist on, on that. So all those things, again, in grad school, they, they were encouraged, but they weren't structured at the time. Yeah. So yeah, I can, I can, I understand Susan's comment because I was, I was one of those economists who started pretty early with pat and we, we have a lot of good mechanisms that we've learned and built at Amazon when I was there at the time through pat, through lay other people who were willing to make the jump into this entrepreneurial space that hit the election and the, of coalesce of economists doing open book, empirical research, along with data science. Right. Just becoming more and more valuable and applicable, but is kind of what Susan piloting that we can, we can talk more about if you Scott Cunningham:Want. Yeah. I do wanna talk about that. I wanna talk about the, the decision though, you know, to, to be, because you, you sort of started off in college, you know, you said things like, oh, you can become an educator and then you've gone in this non-academic direction and you know, it, it, and that's like a, that's a more common story now, you know, right. Of, of top talent, very talented PhDs that you could have easily seen 20 years ago, would've been an academia. Their counterfactuals are, are following you. And so, you know, it's, it's a, it's a big part of our, you know, collective story as economists that this, this new labor market that didn't, that didn't exist historically now exists and draws in so much talent. And I was just curious in a way you're kind of like a, a first generation person like that, you know, when you think about it, right. Cause text's not very old, right. Facebook, Facebook, what it's like 2007. And so, you know, so you've got this, you, you, you've got this, this chance to kind of say like, it must have been, so I don't wanna put words in your mouth, but I guess I was just wondering, what were the feelings like as you considered not taking an academic track and when did it start to be something in your mind that you thought that's gonna be something I'm explore Kyle Kretschman:Probably pretty early, because if you wanna really trace the roots of like tech economists back, it starts obviously with Hal varying at Google and me and Nick, actually, we, we sent an email to Hal, probably 2008 saying, do you have any, have any use for some summer interns who can do some empirical IO? And he said, no, not, not at this time, but so, but he Scott Cunningham:Answered the email. Kyle Kretschman:He did answer the email. Yeah. It was nice, nice of him to answer. Cause we knew he was probably pretty busy, but so it, honestly, when Amazon started hiring economists, I was probably searching for about a year to move into tech. If you wanna move back to the decision point coming outta grad school, honestly it was a challenging labor or a challenging job market for me, somebody who is a lover variety, who is working on empirical IO problems with campaign, policy, campaign, finance reform, policy recognition. That's, that's not fitting a lot of the standard application process. Yeah. Once again, that's so that's probably a theme for me. And again, at the time it was hard. I was, I was in the running for jobs at VA wakes force that I thought would be really good fit because they're the EDU the emphasis would be on education with the research ability to do research and work on problems that were more widely probably policy oriented. Yeah. But neither neither of them came through. So I just always knew that I industry was gonna be an option. And so Scott Cunningham:What year is this? What, Kyle Kretschman:What, what this would've been in this would've been in Scott Cunningham:20 11, 20 11. Okay. Oh, so you moved through the, you moved through the program or kind of relatively quickly. Oh 7, 4, 4, 5 years. Okay. Kyle Kretschman:Five years. Yeah. Five years. Yeah. Oh six to 11. Okay. But so for about a year, about six. Yeah. Yeah. And so starting in 2013 is whenever I started applying to the first tech job as a data scientist and got it went great until I talked to the VP who was a business part, like pure business person. When I was talking to the hiring manager at the time, it was a company who was providing college counseling as a software service. And so they would do this at their, their clients were both for profit and not for profit companies. And we were talking like, we'd get into details about treatment effects models and how we could measure the impact of their intervention. It went great. But then I had the flyout scheduled, but then the interview with the VP, he said, well, how am I gonna monetize your algorithm? Right. And I was like, I'm not sure I know what algorithm means, but right. I, I wasn't prepared for that language and that application and how you turn econometric modeling and measurement into, into business impact at the time. Yes. Right. So spent another year looking around with different opportunities like that and honestly learning again. So, so whenever Amazon, so this would've been in 2014 and then Amazon was hiring its first big cohort with pat. So this was a cohort that was about, I think there was about 13 of us. It was a no brainer. Kyle Kretschman:Whenever, whenever we did the interview, it just was like, all right, this is exactly right for me. I was hop. I was hoping it was right on the other side. And I could probably tell you some funny stories about the interview process, but I was like, this is, this is what's meant to be. Yeah. So it, it, it was like a 10 year journey from 2004 when I switched outta computer science into 2014 being like this, just this fit. Scott Cunningham:Right. Right. Right. So outta curiosity, you know, is, is there, is there something that you think is supposed to be learned by the fact that when you were on the job market and you had that interview with that, that gig and the, and you get to the VP and he articulates questions that are not traditional econ questions, or even econometrics questions like business profitability to act, it's kind of ironic, isn't it like to everybody? That's not an economist. That's actually what we, they think we do, you know, is like, they think we do all that stuff. And then they don't know that we're like, like you said, you know, trying to set up a Lara and solve, solve it, like what's a Lara, but do you think your competition at that time did know how to answer questions like that? Like non-economists in those positions Kyle Kretschman:Probably at an inflection point. Yeah. Because this is the same time. Wherever machine learning is becoming more common toolkit with an industry. So there would be like machine learning algorithms that are designed for, you know, prediction, problem sequencing, anything like that that are specifically designed to be used in a business setting to monitor. Scott Cunningham:So they, they not only know machine learning, it's like, they also can kind of immediately articulate why this would be profitable. Kyle Kretschman:I think so. Yeah, because again, the computer, so it's like in learning the language and this is the language that would probably be more understood within a machine learning computer science version is okay, well, I'm gonna use this to change the recommendation engine right. Is very common one. Yeah. That's obviously gonna be, so how are you gonna monetize it? I'm gonna improve the match and the recommendation engine it's gonna have this. So I think at the time there was a little bit of it, but, you know, hopefully I think, I think I learned pretty quick that you can, you can use econometrics in a similar vein. As I said, it's a flavor of data science, Scott Cunningham:Have you had to become a blue collar machine learner? Kyle Kretschman:I've had to understand it, but not, I think you mean by blue collar, you mean like implementing it Scott Cunningham:And yeah, I just, when I, I usually say blue collar in the sense of like, you know, you, don't like, you know, you basically are picking up these skills, but you weren't like, you know, you didn't get a PhD in computer science. You know, Kyle Kretschman:The answer was then that answer is definitely yes. So like as we, as our cohort and as we grew, the economics discipline at Amazon, that was a big part of it is how one could we bring in some machine learning scientist help educate and teach us. Mm. And yeah. So, and even in, sometimes in lecture style, we would do that because it was so important, but then even more so learning to so that you can interact with different stakeholders specifically, like machine learning scientists. Mm. Then understanding when you can actually implement it and marry it within the econometric models was definitely a huge part of the education process. Scott Cunningham:So you go to Amazon, is that right? That's like your first entry into tech Kyle Kretschman:That's Scott Cunningham:Right. Is Amazon, what's your title? Kyle Kretschman:So Scott Scott Cunningham:A scientist or economist. Kyle Kretschman:I, it was something like business intelligence engineer. There wasn't an economist job family. There was, as you said, it was kinda the forefront. I think it was this. Yeah. I think that's what it was, but Scott Cunningham:Cause it is now right. Baja has a that's Kyle Kretschman:Right. Scott Cunningham:He created a job title called economist. Kyle Kretschman:That's right. Yeah. And that got set up about a year in, so like, and I was part of the group. So we would set these, we would set up like these people and process mechanisms that allow economists to be so influential and productive within Amazon. Scott Cunningham:Mm, okay. So how is he doing it? Why, why is Susan saying he performed a miracle by making economist productive? Can you kind of describe, like, if you had to just guess at like the counterfactual, if it hadn't been, you know, pat, it hadn't even been an economist that was hired into Pat's position. Like, what is it that he, what, what is