If any of the people listening to this are intrigued by this idea of kind of fumbling forward toward a learning organization, I'd recommend you check out the work of a guy named Steve Speer who has devoted his work in the past several years to studying cultures like Toyota or Navy submarines or places where they absolutely have to get things right and have to get things better on an ongoing basis and in all of what that means from a cultural transformation point of view. There are the good news is there are a lot of people who ablaze this trail. I mean there's a lot of really excellent organizations and institutions that have taken one step after another on this road to quality improvement and they're in a place now where we can learn from them and start adopting some of these practices. As we learned in the last episode it's not always easy or intuitive to think upstream of a problem to be preventive by nature to design organizations and incentives around trying to solve problems before they ever happen but it can be incredibly rewarding to our efforts. In today's episode we continue this discussion with Dan Heath. Dan is the author of so many incredible books including being the co-author of Two New York Times bestsellers made to stick and switch. Dan is also co-author of the book decisive this changed the way that I think about decision making but in today's episode we're discussing a new book that Dan has written called upstream. I encourage you to go and check out the book of course but also listen to what Dan has to say in this episode about what it means to solve problems before they happen. If you missed the first part of the episode I encourage you to go back and listen to that because there's so many good pieces of wisdom that Dan has to offer to us as engineers. Thank you so much for listening to today's episode let's jump straight into the second part of my interview with Dan Heath. In the book you get into these details a little bit more kind of mechanically specifically talking about you know uniting people and what are the changes actually that you need to make to a system? How do you determine some of those things? Finding leverage one thing I'd like to talk about specifically is how do you know when this is succeeding? I mean in the point of you know when we're talking about the children that are that are drowning it might make sense that if you had a rate of children drowning one every five minutes and that drops to one every 24 hours then you know that that might make a a good measurement but that's it's not always that easy right? No it's not and I think the reality is we live in a world where in the fictitious parable world I mean my guess is that in normal corporate America what people would be measured on is is the speed of rescue you know and in fact there's there's an example in the book that I think illustrates as well it's about Expedia the online travel site and back in 2012 this kind of Ryan O'Neill is studying some data about the call center at Expedia so if you book a flight or a hotel or something and something goes wrong with your reservation you can call the 1-800 number what he found made his jaw drop he found that for every hundred customers who booked a transaction 58 of them ended up calling the call center for help which which would pretty much seem to nullify the whole point of having an online self-service travel site and so he starts digging in to figure out why are so many people calling us and the number one reason people are calling I mean to the tune of 20 million calls in 2012 was to get a copy of their itinerary that was it 20 million calls can I get a copy of my tinerary and and so he and his boss just they're like this is madness we've got to do something about this and they make the case to the CEO to create a special team to work on this and they do and the technical solutions as you might expect are pretty simple they they change the way they send it's not like they forgot to send the itineraries they were always sending them as just they would end up in spam or customers would delete them thinking they were ads or that sort of thing so they changed their strategy and emailing they added self-service tools on the IVR and online and so forth and they basically took 20 million calls and whittled them down to zero so from a from a technical perspective this is a trivial problem but I think what's interesting about this story is is why this problem got to this level like you would think that there would be an alarm bell that would go off somewhere once you reached like your your three millionth call for an itinerary like people would start to take this seriously but but the deal is that Expedia like like virtually every other company has to organize itself or chooses to organize itself in in silos and so you've got a marketing team whose job it is to to attract customers to Expedia versus kayak or someone else and then you've got a product team whose job it is to make the site so smooth and intuitive that the customers are funneled toward a transaction then then you've got the tech team that makes the plumbing run and keeps uptime high and you've got the call center that's trying to minimize you know the the response time to field a customer issue and and to keep them happy via net promote rescores something like that and from a silo perspective like all of those goals make sense but but the problem is when you ask a very basic question like whose job is it to keep customers from needing to call us the answer was nobody yeah yeah and it was even worse than that like none of those silos even stood to benefit if the number of calls went down and so that's something I think that's that's really interesting about upstream problems is that it's often very easy to find owners for downstream problems like your house catches on fire it's the fire departments problem at that point like it may not be an easy problem but it's an easy problem to define an owner for versus if you flip things around and you say whose job is it to keep customers from calling or whose job is it to keep your house from catching on fire well that's a very different thing right and it might require the effort of lots of people across different silos that often these these challenging upstream issues fall in the gaps between silos so if you're facing an ongoing problem in your work that you think is preventable my guess is it it might require some new kind of integration that the organization of your company makes counterintuitive or makes a bit more effortful than that it probably should be integration and incentive are the two words that keep on coming to my mind here integration is you know this idea that you've got these two jobs and if you were to combine them in some way or find you know you can imagine that it might be somebody's job to make material that is flame resistant right and this that seems like a reasonable downstream effort we're going to make material that doesn't catch on fire very easily and then it's easy to see that firefighters are going to fight fires but is there a connection between the material making the fighting the firefighting where is that kind of if you were to travel up the tree you know where is the the shared branch right that that cares about both of those things yeah this is this is this is a really important question and it's something that I talk about in the book under the framing of of accountability that that one of the barriers to us thinking upstream is is a lack of ownership uh so to your point I mean at Expedia the reason they got to 20 million calls in in a year is because no one owned this that it wasn't obvious who should step up and and be the owner and in fact it took a CEO level intervention to crack it but it's a winner owner in some ways right yeah I mean you just keep having to work your way up the chain until you get to someone who's not stuck in a silo that defies collaboration but it's it's not always that bad I mean that's a pretty extreme story to have to go all the way up to the top of the chain to get a resolution and like one of the examples I give in the book this is kind of a long story I'll try to tell up a really reduced version of it but there was a single pediatrician in Tennessee guy named dr bob sanders who is probably responsible more than any other human being for they're being mandatory car seat laws I've got a couple of little girls and it's absolutely unthinkable to me that I would go on some long road trip without them being car seats that's just my world you know but but it wasn't that long ago you know i.e. in the 70s when it would have been perfectly normal for a couple of toddlers to be flopping around in the back seat not only not in car seats but maybe not even in seat belts and as a result of that there were just huge numbers of kids being maimed or killed in car crashes and so the chain of events here was a couple of of advocates wrote a no tour it not notorious it's a wrong spin it was like a a famous article in the journal pediatrics that called on pediatricians to own this issue the issue of car safety because they were saying if you really are in the business of protecting your patient's health your patients being the children that you serve they presented the damning statistics that included this one that more children in that era were dying inside cars than outside them from all other sources put together so they said if you really are are concerned with the health of your patients no you're not in the car with them when they're driving no you're not an auto manufacturer no you're not an auto safety advocate but you are probably the number one person in their life that's concerned with their health and you need to pick this up and doctor Bob Sanders that kind of hit him in the heart he was like by god you're right I do need to do something about this and and I'll spare you the whole chain of events but he got interested in politics he starts crusading within the state he eventually manages to get passed in the Tennessee state legislature the first state act to mandate the use of car seats in cars and then within a matter of years the other 49 states followed but if you notice something about that story that's kind of weird this is a big deal it's a big problem they're high stakes and yet the leadership of this Bob Sanders and his colleagues were basically kind of volunteering right then this is something that's that's so strange about the distinction between downstream and upstream that when something downstream happens action is almost obligated you know the kid drowning in the river it obligates our work you know the website goes down we're obligated to respond but but things on the upstream side of the equation even though they may be of equal or larger significance a lot of times it requires somebody to kind of raise their hands and say hey we didn't create this problem but we're going to be the ones to fix it and and so maybe some of the people listening to this that there's some problem that that you're facing in your company or maybe even in your community that lacks an owner maybe you're the person to be that owner to be that doctor Bob Sanders character it seems like in emergencies we have emergent ownership right that's kind of the the whole idea is it it imposes itself on us and but it doesn't even necessarily have to be you know a dire situation to be considered an emergency in a given scenario an emergency can be something quite small but it's like you're saying it it creates the sense of obligation to anyone who's around or anyone who has that particular skill set to solve that problem and in some way is that seems directly tied to that heroics conversation we're having earlier the idea that hey you know I'm in I'm in this spot and I suddenly became an owner and you hear stories like my instincts kicked in right and and it seems something again very core whereas like you're saying there's there's some level of volunteerism to solve upstream yeah now here's here's the question that I may be a difficult question to ask here how do organizations create a more systematic approach you know I hear very often in many of the companies that I've been involved with and people that I've talked to who work as Developer The kind of organizational leaders are asking people to take ownership it's a this constant struggle between people who don't know you know what level of ownership they're quote allowed to take right and then leaders who wish that their people would take more ownership how do we design organizations where ownership becomes maybe more accessible or maybe the incentives are better aligned so that the average person is more likely to volunteer I think a lot of these issues especially to do with organizational culture are ultimately emotional that that people are very sensitive whether we know it or not to what what earns you respect and what earns you acclaim and so so part of the answer has to be we have to do a better job a heroicizing the people who prevent the need for heroics I actually heard from a reader last week who was sharing an anecdote from somewhere they had worked where early we were talking about this cycle of heroics and they were saying you know that they they dealt with a lot of that to the point where it seemed to be feeding on itself and so they made a conscious effort to change that and they created something they called the Smokey the Bear Award for people who prevented crises and I thought that was just genius you know this idea that we have to be careful what we reward it not in terms of literally financially although that would be nice too if there's some way to manage that but even emotionally like who who is praised who is held up and next time we're about to praise the hero who saved the day could could we at least pay some attention to hey you know just to kind of tune into current events I would love to know which organizations were actually pretty well prepared for this pandemic and and obviously it's going to differ by sector the tech companies are going to be better prepared than others but but if you could control for sector wouldn't it be interesting to see like what staples are office depot better prepared or was lift or timber better prepared because I think if you if you could analyze the differences you'd really be getting a good index of leadership you know which set of leaders were better armed to take their company remote because there was nothing unforeseeable about this crisis and by the way there were a hundred other different flavors of crisis that would have required the same solution i.e. remote work and so this was something that a good competent leader should have anticipated and been building some muscles for and so it's like the old Warren Buffett quote like it until the tide goes out you don't know who swimming naked like tide just went out and and what can we learn about the quality of our leadership from how much of an emergency was sparked in the situation today's episode is sponsored by Linode Linode has 11 data centers worldwide including the newest data center in Sydney Australia with enterprise grade hardware s3 compatible storage options and the next generation network linode delivers the performance you expect at a price that you don't you get started on linode with as little as five dollars a month and you can get 20 dollars with a credit for being a listener of Developer Tea now some other things you should know about linode linode is a company of Developer That means that they have developer friendly tooling for example they have an API that is already at version four that actually means something with linode they even have a Python CLI and an open source a whole collection of open source tools including their single page app it's a cloud manager go and check it out at cloud.linode.com of course you can find them on github and they're actively updating this stuff go and check them out on github as well so linode is a company of developers making tools for developers once again you can get started with a 20 dollar credit by heading over to linode.com slash Developer Tea it's linode.com slash Developer Tea all one word and use the promo code Developer Tea 2020 that's the the word Developer Tea and then two zero two zero the numbers at checkout to get that 20 dollars worth of credit thanks again to linode for sponsoring today's episode of Developer Tea. Yeah I saw a wonderful graph that made this very real it was a it was a projection graph and the projection of of deaths total deaths from from this virus and of course it's a solemn and very real thing to to imagine that you know just 10 or 15 days ago the projections were half a million people and this is a little animation the graph stepped forward one day at a time and the projections each day lowered and so the all of the stories that we read the you know the New York Times article about flattening the curve I think it was New York Times that was visually made real in that moment and went from 500,000 over 500,000 down to 75,000 approximately 75,000 in about 12 days and the the person who posted this mentioned they said this is why we're staying home and it was this powerful moment but what made it even more powerful to me was someone who commented on it and they said one of those 500,000 people could have been my mother could have been your father could have been anyone that you know and now it won't be and that was such a you know bringing those numbers into a real picture for me was was such a powerful moment and it immediately made me think believe it or not of this upstream thinking that to me was the same kind of heroic reward that I would have felt had somebody saved someone's life in that moment I had that same feeling that someone's life was saved yeah I completely relate to that I mean it's like the the thought experiment of the police chief that the the policeman goes to that intersection save someone from getting in a wreck but they don't know and the police officer doesn't know but but we see it in the data and I think I mean maybe this is too meta since we're still in the middle of this crisis and people are still dying but I feel like this is this is a very powerful and unsolicited training exercise for the human species that we're going through right now that we didn't ask for this but then in a way we need this if we're going to survive that we need to understand the the power of preparation and the significance of preparing for things that may or may not happen on a smaller scale I talked to a guy named Jon Cuskinan who who was the Y2K's are you know back at the turn of the millennium and you know there weren't many stories where Y2K was going to kill a lot of people but people worried a ton about the collapse of the economy and power grids and so forth and so on and I think it's got some similar dynamics to pandemic preparation because Cuskinan told me he knew when Bill Clinton asked him to take this role that it was a total no-one position because if everything went to hell he was going to be the most obvious guy to blame and if everything went really smoothly then people would just say well what was all that you about there wasn't any problem here this is just a big scam you know and I think the same thing could well happen with this I mean it wouldn't surprise me if if three months from now a lot of people have died but far fewer than than a lot of the predictions or forecast suggested and that people conclude from that well we just made a lot of fuss over something that wasn't as bad as we thought which is exactly the wrong conclusion right that the right conclusion was this was a massive threat that we responded to dramatically and at great cost and as a result we saved a lot of lives but but this is not a kind of logic that's taught in school you know and there's nothing particularly intuitive or natural about it there's nothing in in evolution that prepares us for these kinds of thought experiments you know there's no monkey sitting around analyzing the data of you know lion attacks avoided it's it's just something that I feel like we have to grow into yeah yeah I think it's it is something that if you've had a chance to work with you know simulations or simulation theories or the ability to run you know algorithmic predictions that kind of thing you have a little bit more of a tangible grasp on statistics and how they work and why they matter but I don't think that the average person thinks in this particular way and we see that all the time we see that people use anecdote rather than data to make their point for example right that's one of many many things that kind of distorts the picture for a lot of people but my impression by the way is that the developers are are much more natural upstream thinkers than other disciplines you think that's accurate it's a good question and I think that you know I think it depends on which layer you're talking about because there are certain layers where developers feel that they're not at home that that that that it's not their pro as you mentioned in your book this isn't my problem right this is this is not in my realm you know I'm not the owner of this someone else should be responsible for this yeah and so the the upstream there's always more stream and there's some source and no one's responsible for the ocean you know or or for the rain right at some point you know the the responsibility we draw a line and I think developers very often they draw that line further downstream than perhaps they're capable of solving and I think that's that's an organizational issue but it's also a cultural issue amongst developers how so I think so that's a good question I think developers often feel that their solution should be hard solutions when I say hard I mean tangible direct measurable metal solutions and that anything that that starts to dip into intangibility whether that's you know marketing for example is a really good it's a perfect example marketing or or something that it has to do with less measurable outcomes is where they start to draw the line and say I can't care about that because I don't have a way of measuring my success mm-hmm mm-hmm and so if I can't measure it then what am I shooting for right what am I optimizing for here I'm just kind of optimizing to a black hole I can't do that and so instead I'm going to work on things I know for a fact I can succeed at I think that's probably a healthy natural instinct is to be looking for data or other kind of measurements to use for navigational purposes it's just a lot of devil in the details about which data you're using for that purpose but but virtually all of the upstream successes I talk about in the book profited from collecting the right information and using it to to inform you know the next set of steps that they would take so it's a good instinct I think yeah agreed and and that means that you know developers have the opportunity and I think it is an organizational responsibility to push that you know and perhaps this is how we decide where we are on that stream map you know what points on the stream are we at well we're at the point that we can measure anything that we can't or anything that is intangible inaccessible to us it's difficult to go up there right it's costly to go up there because we don't know how far to go mm-hmm yeah and just with with the caution that that we also have to be very diligent about kicking the tires on on whatever scheme of measurement we're using like there's an old story about AT&T that it wanted to boost the productivity of its developers and so they decided they got really clever and started paying per line of code yes and of course how to developers respond by writing writing videos and yeah and so that's the kind of thing that we have to be thoughtful about is is it's it's it's almost inconceivable that we could succeed at preventing a problem before it happens without the use of smart measurement but but it has to be smart yeah absolutely Dan I know we're coming up on time here and I just want to ask you a very short a couple of questions kind of quick hit questions before we wrap if that's okay sure the first question and I like to ask this of all my guess what is something that you wish more people would ask you about oh good question um I think within the realm of of this book it it might be what we were just talking about the kind of devils in the details aspect of of incentives and and measurement schemes there's a whole chapter in the book on just this problem and and I think that we've all heard this this idea or this phrase of people gaming measures in various ways and we often tell these stories in in kind of a a smirky way like isn't it isn't it kind of funny that people get clever and play these tricks to try to earn better numbers or get their bonus or whatever but I think it's much much more serious than that that I think in in fact if we get the wrong measurements scheme in different places and there's some pretty sobering tales in the book including one from the New York Police Department of how measurement schemes can go wrong that if we get these things wrong it can actually doom the mission it's not just a small thing it's it's possibly the main thing and then the flip side of that is equally true that if we get the measures right especially if we can align short term tangible measures that we can collect every week or every every month with long term goals in a productive way that can be the secret that unlocks you know problem solving at a level not thought possible yeah it's wonderful advice and I think that developers who understand this they know that this applies not only to the work that we do but also the processes that we run our teams and certainly managerial work and the engineering managers who are listening to this right now they see immediately how this applies and how important it is to think really clearly about the measures that you're taking we do this this kind of work in in in tech with OKRs this is such a common way to do this it's a kind of operationalize these measuring pieces and to try to balance the scales between two measures for example it's certainly as worth taking time to think about it I'll ask you one last question and it's a short one if you had 30 seconds to give developers of all backgrounds and experience levels what would advice what would you tell them oh dear god um 30 seconds to glory um I tell you what I'm going to mix it up and pull a concept out of a past book that I wrote with my brother chip called switch and it's the notion of finding bright spots which is to say in times of change our attention is is usually drawn by what's not working and we tend to analyze what's not working we we study projects that have gone wrong or we do after action reviews but often the best clue to a way forward is to study what is working an unusually successful project or an unusually successful employee or an unusually profitable client and that often by studying bright spots we can find the secrets to our next moves man that is that is good advice and certainly relevant for engineers Dan thank you so much for taking time to talk with me what is the best way for people to find you and also where do you prefer to send people to pick up your your new book upstream I am agnostic about book retailers so whatever tickles your fancy the important thing is to buy that book of course of course no I'm kidding uh you if you want to learn more about the book you can come to upstreambook.com and there actually are a bunch of of free resources at that site a a book club guide and a podcast and a one-page framework summary and a bunch of goodies that uh that you can get there for free. Excellent Dan thank you so much for your time it has been an honor to talk with you. Hey thanks this has been a fun conversation. Another huge thank you to Dan Heath for joining me on today's episode Dan has been one of the people that I've wanted to interview for a very long time on this episode he didn't disappoint I could imagine doing three or four of these interviews and still learning something new from Dan in every part of the conversation so I greatly appreciate Dan's time and joining me on this show. Thank you again to today's sponsor Linode head over to Linode.com slash Developer Teato get $20 worth of credit and he had started with his little it's $5 a month on Linode's Nanode plan and you'll get root access to Linux server of course you can go all the way up to having a dedicated CPU physical core for yourself head over to Linode.com slash Developer Teato learn more. Today's episode was produced by Sarah Jackson and the show is a part of the spec network spec dot FM as a bunch of other podcasts that are relevant to you as a developer especially if you are growing in your career if you're interested in becoming a better developer a better human go and check it out head over to spec dot FM thanks so much for listening to today's episode my name is Jonathan Cutrell and until next time enjoy your tea