Experiencing Data w/ Brian T. O’Neill

Today I sit down with Vijay Yadav, head of the data science team at Merck Manufacturing Division. Vijay begins by relating his own path to adopting a data product and UX-driven approach to applied data science, andour chat quickly turns to the ever-present challenge of user adoption. Vijay discusses his process of designing data products with customers, as well as the impact that building user trust has on delivering business value. We go on to talk about what metrics can be used to quantify adoption and downstream value, and then Vijay discusses the financial impact he has seen at Merck using this user-oriented perspective. While we didn’t see eye to eye on everything, Vijay was able to show how focusing on the last mile UX has had a multi-million dollar impact on Merck. The conversation concludes with Vijay’s words of advice for other data science directors looking to get started with a design and user-centered approach to building data products that achieve adoption and have measurable impact. In our chat, we covered Vijay’s design process, metrics, business value, and more: Vijay shares how he came to approach data science with a data product management approach and how UX fits in (1:52) We discuss overcoming the challenge of user adoption by understanding user thinking and behavior (6:00) We talk about the potential problems and solutions when users self-diagnose their technology needs (10:23) Vijay delves into what his process of designing with a customer looks like (17:36) We discuss the impact “solving on the human level” has on delivering real world benefits and building user trust (21:57) Vijay talks about measuring user adoption and quantifying downstream value—and Brian discusses his concerns about tool usage metrics as means of doing this (25:35) Brian and Vijay discuss the multi-million dollar financial and business impact Vijay has seen at Merck using a more UX driven approach to data product development (31:45) Vijay shares insight on what steps a head of data science might wish to take to get started implementing a data product and UX approach to creating ML and analytics applications that actually get used (36:46) Quotes from Today’s Episode “They will adopt your solution if you are giving them everything they need so they don’t have to go look for a workaround.” - Vijay (4:22) “It’s really important that you not only capture the requirements, you capture the thinking of the user, how the user will behave if they see a certain way, how they will navigate, things of that nature.” - Vijay (7:48) “When you’re developing a data product, you want to be making sure that you’re taking the holistic view of the problem that can be solved, and the different group of people that we need to address. And, you engage them, right?” - Vijay (8:52) “When you’re designing in low fidelity, it allows you to design with users because you don’t spend all this time building the wrong thing upfront, at which point it’s really expensive in time and money to go and change it.” - Brian (17:11) "People are the ones who make things happen, right? You have all the technology, everything else looks good, you have the data, but the people are the ones who are going to make things happen.” - Vijay (38:47) “You want to make sure that you [have] a strong team and motivated team to deliver. And the human spirit is something, you cannot believe how stretchable it is. If the people are motivated, [and even if] you have less resources and less technology, they will still achieve [your goals].” - Vijay (42:41) “You’re trying to minimize any type of imposition on [the user], and make it obvious why your data product is better—without disruption. That’s really the key to the adoption piece: showing how it is going to be better for them in a way they can feel and perceive. Because if they don’t feel it, then it’s just another hoop to jump through, right?” - Brian (43:56) Resources and Links: LinkedIn: https://www.linkedin.com/in/vijyadav/

Episode Description In one of my past memos to my list subscribers, I addressed some questions about agile and data products. Today, I expound on each of these and share some observations from my consulting work. In some enterprise orgs, mostly outside of the software industry, agile is still new and perceived as a panacea. In reality, it can just become a factory for shipping features and outputs faster–with positive outcomes and business value being mostly absent. To increase the adoption of enterprise data products that have humans in the loop, it’s great to have agility in mind, but poor technology shipped faster isn’t going to serve your customers any better than what you’re doing now. Here are the 10 reflections I’ll dive into on this episode: You can't project manage your way out of a [data] product problem. The more you try to deploy agile at scale, take the trainings, and hire special "agilists", the more you're going to tend to measure success by how well you followed the Agile process. Agile is great for software engineering, but nobody really wants "software engineering" given to them. They do care about the perceived reality of your data product. Run from anyone that tells you that you shouldn't ever do any design, user research, or UX work "up front" because "that is waterfall." Everybody else is also doing modified scrum (or modified _______). Marty Cagan talks about this a lot, but in short: while the PM (product managers) may own the backlog and priorities, what’s more important is that these PMs “own the problem” space as opposed to owning features or being solution-centered. Before Agile can thrive, you will need strong senior leadership buy-in if you're going to do outcome-driven data product work. There's a huge promise in the word "agile." You've been warned. If you don't have a plan for how you'll do discovery work, defining clear problem sets and success metrics, and understanding customers feelings, pains, needs, and wants, and the like, Agile won't deliver much improvement for data products (probably). Getting comfortable with shipping half-right, half-quality, half-done is hard. Quotes from Today’s Episode “You can get lost in following the process and thinking that as long as we do that, we’re going to end up with a great data product at the end.” - Brian (3:16) “The other way to define clear success criteria for data products and hold yourself accountable to those on the user and business side is to really understand what does a positive outcome look like? How would we measure it?” - Brian (5:26) “The most important thing is to know that the user experience is the perceived reality of the technology that you built. Their experience is the only reality that matters.” - Brian (9:22) “Do the right amount of planning work upfront, have a strategy in place, make sure the team understands it collectively, and then you can do the engineering using agile.” - Brian (18:15) “If you don’t have a plan for how you’ll do discovery work, defining clear problem sets and success metrics, and understanding customers’ feelings, pains, needs, wants, and all of that, then agile will not deliver increased adoption of your data products. - Brian (36:07) Links: designingforanalytics.com: https://designingforanalytics.com designingforanalytics.com/list: https://designingforanalytics.com/list

Today I’m talking about how to measure data product value from a user experience and business lens, and where leaders sometimes get it wrong. Today’s first question was asked at my recent talk at the Data Summit conference where an attendee asked how UX design fits into agile data product development. Additionally, I recently had a subscriber to my Insights mailing list ask about how to measure adoption, utilization, and satisfaction of data products. So, we’ll jump into that juicy topic as well. Answering these inquiries also got me on a related tangent about the UX challenges associated with abstracting your platform to support multiple, but often theoretical, user needs—and the importance of collaboration to ensure your whole team is operating from the same set of assumptions or definitions about success. I conclude the episode with the concept of “game framing” as a way to conceptualize these ideas at a high level. Key topics and cues in this episode include: An overview of the questions I received (:45) Measuring change once you’ve established a benchmark (7:45) The challenges of working in abstractions (abstracting your platform to facilitate theoretical future user needs) (10:48) The value of having shared definitions and understanding the needs of different stakeholders/users/customers (14:36) The importance of starting from the “last mile” (19:59) The difference between success metrics and progress metrics (24:31) How measuring feelings can be critical to measuring success (29:27) “Game framing” as a way to understand tracking progress and success (31:22) Quotes from Today’s Episode “Once you’ve got your benchmark in place for a data product, it’s going to be much easier to measure what the change is because you’ll know where you’re starting from.” - Brian (7:45) “When you’re deploying technology that’s supposed to improve people’s lives so that you can get some promise of business value downstream, this is not a generic exercise. You have to go out and do the work to understand the status quo and what the pain is right now from the user's perspective.” - Brian (8:46) “That user perspective—perception even—is all that matters if you want to get to business value. The user experience is the perceived quality, usability, and utility of the data product.” - Brian (13:07) “A data product leader’s job should be to own the problem and not just the delivery of data product features, applications or technology outputs. ” - Brian (26:13) “What are we keeping score of? Different stakeholders are playing different games so it’s really important for the data product team not to impose their scoring system (definition of success) onto the customers, or the users, or the stakeholders.” - Brian (32:05) “We always want to abstract once we have a really good understanding of what people do, as it’s easier to create more user-centered abstractions that will actually answer real data questions later on. ” - Brian (33:34) Links https://designingforanalytics.com/community