Hey everyone, today's episode is about maintainability. It is an interview with Robby Russell. My name is Jonathan Cutrell, you're listening to Developer Tea, my goal on this show is to help driven developers like you find clarity, perspective, and purpose in their careers. Robby is going to help you do some of those things in today's episode and we talk a lot about maintainability. This is something that Robby and his team really invest a lot in. He is also the original creator of Omizze Shell, which is open source. There are many creators, but Robby is the one that started that project. Let's get straight into my interview with Robby Russell. Robby, welcome to Developer Tea. Thank you so much for having me. Thank you for having me on your show previously. You want to tell folks about that show really briefly before we dive into some of the topics that we've got today? Sure. I'm the host of Maintainable, which is a software development podcast that we talk with people like Jonathan about how to overcome the problems, often associated with a technical debt and legacy code in older code bases. It's an interview style format. You can find that on maintainable.fm. Maintainable.fm. I'm excited to talk with you today because you've got a couple of topics that you mentioned in advance to me that you are interested in talking about. It just so happens that some of these topics are things that I'm personally experiencing or have experienced multiple times in my own career. I'm certain that other people who are listening to this show are either currently or will one day or have in the past experience it and I think you have a unique insight on this. But for people who don't know about your work, can you kind of back up and provide just a little bit of insight of what your day to day looks like? Yeah. I am the CEO of a company called Planet Organ. We've been around for a little over 18 years and we primarily are working these days for the past 15, 16 years or so with Ruby on Rails. We're a consultancy. We come in and help organizations that have already built Ruby on Rails application or have it as part of their infrastructure and we come in and consultative supporting role where we're either helping them with overdue upgrades or helping them work on performance updates, security issues, adding and building and extending new features in their applications or more often we're also a company that will take over those applications for companies where they don't necessarily have their own internal team or maybe they had one or two developers for a really long time that have since left to go join a larger team and now they're like, well, we don't know that we're the best fit to manage software developers full time in our company but we can work with a company like Planet Organ where they have more than a few people that can provide a more well-rounded team from design development and back-end type of approach to that and so then we become the team that kind of manages that project for a number of years going forward. So primarily we're consultants. Yeah, that makes sense and is such a kind of a unique position to be in. There's so much that people like me who are not in consultancy role can learn from you and I want to talk about some of those things today. Also just a quick mention or shout out to probably a project that many people who are listening to this podcast are using OmizhL, OmizhL, whatever you call it. What do you call it officially? But I actually just used Twitter's new little feature where there's like you can upload an audio file. Oh, nice. And so last night I posted on OmizhL. OmizhL. That's how I read it. That's the official word as spoken on Twitter these days. Yeah, so obviously an incredibly popular open source project that provides, well, can you give the elevator pitch for that? And then we'll get into the topic. Sure, sure. So what you're doing, Robby? The OmizhL is an open source framework for managing your configuration for your terminal with your using ZShout as your primary shell. These days the Mac OS comes with ZShout as a default shell. Prior to that, it was bash. And so if you're familiar with bash as you shellfish, it's a framework for, you know, it's got a couple hundred different plugins and themes that come bundled in it so that helps Developer That are working on specific programming languages or frameworks kind of make some nice little features when it comes to auto completing and just some things that helps kind of add some more delight to your terminal experience and a lot of fun and little emojis and colorizing of your terminal prompt. There's a lot of customization there. And so it was a project that I started a little over 11 years ago, primarily for the people on my team so that I can kind of indoctrinate them to all use ZShout as well. And so, and now a lot of people on the internet use it. Yeah, if you've ever watched a screencast and you thought, wow, how did they get their terminal to look like that? There's a pretty good chance that it's OmizhL behind the scenes there. Cool. And, Robby, I'm really excited to talk to you about a couple of things that I think you have a unique perspective on, a unique vantage point. And I want to start with this idea of being a guest or coming into an established code base because, as you already said, you know, that is essentially what you do all the time. You're coming into an established code base most of the time and taking it over as a team, but also as individuals, I'm curious, you know, what are some of the things that you've experienced or that you've seen go wrong when you come into a code base as an outsider? Yeah. And I think just for that context of, you know, I think there's the scenario where we're coming at as a consultant and there's kind of, there's, let's make an assumption that you're going to be there for a short period of time. You're not going to be there indefinitely. It's not going to be the same as if you're becoming a full-time employee. And I think there's a lot of other things that go along with that. I think we can probably dig into. There's probably definitely some overlap. But going, when things go wrong, usually as a consultant, we'll get pulled in. You're usually a problem, whether that be a resource-ing constraint problem on the client side, you know, on, on, or if you're, let's assume, get Jonathan, you have a company, you're, you're, you're part of a team that has existing application and you're calling a company like mine and to come and provide a couple of Developer To help us. So we might be coming in to provide some staff augmentation or to help you fix some maybe very particular pain points within your, your, your application or your tech stack or what have you. And so, usually if we don't have a very clear goal that is shared with the stakeholders and the developers of the existing internal team, that can be a huge challenge and red flag to work through because sometimes we'll get hired by the stakeholders because they're might be a perception of they've lost confidence in their development team. Whether that is true or not, you know, is, it can be hard to determine, but the developers might think that that might be the case. Internally, they might feel like, well, I don't know who these people are coming in, you know, there's kind of this, if you think about, and like think of the movie like Office Space, where consultants come in and people don't really know what they're doing over there and they're coming in and like, oh, they're going to get rid of all of us. They're going to throw everything away or just point out all the problems that were, you know, that we're kind of, that we have here. And so, first I think you need to kind of figure out how to navigate that whole messy world of like, who's bringing me in? Why are they bringing me in? What's their goal? And then make sure you get some time to speak with the other people and the other team members to make sure that you can kind of connect with their goals are as well to the stakeholders goal or vice versa. So, things that have gone wrong is where, you know, I think it's sometimes that consultants will come in and be like, okay, great. I'm here to prove my value as quickly as possible. And so, one challenge I think to acclimate yourself not just to the technical and the code stack, but as to the team is to be too quiet. So I think if you're, if you're the type of person, you're just like, I can put my headphones on, give me access to the repository, and I'll let you know what I need some help. Just to start assigned me a few tickets in JIRA or whatever tool you're using. And I'll be in touch if I have a problem. And so, I think that sometimes can be a problem where, because it's as always, too, that if you're coming in as a consultant, you have maybe a little bit more expertise than say a junior or maybe even a mid developer, a level developer. You know, usually consultants are coming because they have a, you know, perceived to have an air quoting perceived, some level of expertise there. You should come in to provide some guidance or help solve a problem or you have some experience with some tool sets that maybe the internal team is struggling with. So if you come in as a consultant and are quiet for too long and you, you don't really start to foster a collaborative working relationship with the other teammates, there's going to be this mystery about what you're doing. And so the longer that happens, the more people will need to fill in their own assumptions of what you're focusing and working on and knowing that you're probably being perceived as an expensive person or individual or just a cost to the project now. There's going to be those sorts of things will start chipping away over time and erode their trust in you. So I think you need to figure out, so I think there's some some strategies to help navigate that, but being quiet for too long is not an effective, I think, way to be, join a project as a consultant. And definitely not if you're say a new hire to a team as well. Yeah, that's such an interesting problem to solve because I think, you know, coming in you have, if you are like most people, you don't want to over assert your newness, right? And because it's very easy to quickly kind of ruin the relationships with the people around you, if you come in and you act like you've got all the solutions to every problem that you're finding. And the people who you're talking to, you know, just kind of bolster that position, you know the people that you're talking to have spent more time with this code than you have. So it's totally reasonable to think that, you know, the best thing you can do is sit back and wait. But it's also totally reasonable, I think, and you can speak to this a little bit more. To take advantage of the fresh perspective that you have, right? Your eyes are not kind of used to what you're seeing in the code. And so you have an opportunity to say, Hey, has this been thought through a second time? Has this been revisited? And I think that's a huge advantage that gets lost over time. What do you think about that? No, I think that's a really good point. And so, you know, I think there's a couple of things that I've always been trying to encourage people on the team, my team here when they're entering these situations is to think about the, you know, trying to think of a good way to explain this. But we have to realize that, you know, people on the client side, so, you know, for in our perspective, the client being those with their own product or application that they're bringing us in, you know, those developers. And if you think about your own projects that you've been part of, Jonathan, I'm sure there's areas of the code base and your documentation that you're potentially embarrassed about. Like, I haven't had a lot of time to really spend focusing on those specific areas of the code base. And you know, I need to improve the documentation or there's not enough test coverage there. And you have probably your reasons for why, you know, from the day to day things that pop up the reasons, you have a number of reasons why you haven't been able to get to that yet. And so, so there's this kind of like this interesting thing where if like any, you know, think about any project you have and like someone's getting hired to come in and help you work on that project and they might start asking, like, what's going on over here? And you're like, it's like a little, a weird little store. So, you know, it's a store spot for you. And you're like, ah, and so you have to kind of internally work through this little hurdle of being like, okay, I need, this person's going to help. I need to be helpful. I need to be available. I need to make time for this person, not to mention you might have felt like you haven't had enough time to work on this project as it is. Now, there's another resource for multiple people potentially joining the project you need to find time to answer questions that you've already had to answer in the past or other people on your team or what have you. And you're like, okay, how am I going to get my stuff done? And this new, help this new person be successful as well. Knowing that it's part of your responsibility is a bit of a challenge. So I think as a consultant, you need to know that going in like that person is going to be nervous and about because there's a lot of unknowns. You don't, they don't know how you're going to perform. They don't know the questions that you're asked. You don't, they don't know whether or not you're going to write up a report and just outline all of the problems in the code and send it to their boss and say, what is going on with this person, this person's following short, can hit like those of those very sore subjects for you. You might be feeling imposter syndrome already and this person's going to come in and potentially just highlight all those things and you'll be caught. You know, put this spotlight on you and oh no, like, now is your job at Ingepathy. So there's a lot of mixed emotions going around, nervousness. So as a consultant, you need to be aware of that and just come into that space and acknowledge it and try to think how can you start establishing a some level of healthy trust between you and the other people that you're working with. You might have to do it with multiple people. Maybe there's one or two primary people in the project and hopefully it's more of that than the former, at least in this first several days. So I think some of the ways that I've kind of helped kind of across that bridge at times is to think about, let's say you're going to come in and you need to get the application up and running on your own development work environment. So say you have a laptop, you come in or they give you one and you're going to go through some documentation and get things set up. So as you're doing that, you might hit a couple of stags on the way and obviously anyone will tell you, like, please help us update the readme or the documentation if you, if there's some gaps, that's one of the benefits of something new come into any project, whether you're a new employee or a contractor is that you can help make those kind of early contributions. And that could be your first pour request or two is just improving the documentation to clear up some things that maybe you hit some bumps. So, but and then if someone helped you solve that problem, obviously share some appreciation of that person. But another thing that I think often gets overlooked is to, if the documentation that you're relying on was really helpful, look at the revision history and look and see who contributed to that and reach out to those people and say, Hey, this is some of the best documentation I've seen in a long time. I work on a lot of projects. I just wanted to say thank you and give them some appreciation because then it helps reinforce and not only is the read the documentation valuable, but they feel like there's new now like established, you know, and do that with like if it actually is helpful, don't just do it because you feel like you just feel like you need to give them some feedback. But if it was helpful, let them know that and it will just help reinforce why it's important to update the documentation on a regular basis. So that's like a really good like first little like those are small little things you can start doing to do that. But then that way you can start fostering to get into some of the messier sections later on. Like we're going to ask like, Hey, I noticed over in this area, it seems to be lacking in some documentation. Who would be the best person to talk to you about that? You know, is there some any documentation on where this was last when we made this, when this decisions were made to approach it this way, thinking about just being empathetic not only to that person, whether I right now, but to whoever worked on that project in the past because it's easy to go in and just be like, This is wrong. This is wrong. This is wrong. And I think we all know as developers, there's not always this booleanness of, is that a word booleanness? Anyways, but a sense of like something right or wrong, does it work or not? I think often is the more appropriate way to think about that, but like, is it working anymore? And if not, what do we need? How do we need to adapt and like move this thing forward to better adapt to the realities of the business today? Then, you know, three years ago when this was first implemented. So I always think about just that early relationship building to just think about that person's nervous, they might feel threatened about their reputation and their livelihood might be feeling at stake because maybe there's just fear, maybe they had bad experiences with previous consultants that came in and just made a mess of things or didn't produce a lot or just came in and just pointed out a bunch of problems and said, You know, let me know when you want to hire me to fix it. So can you need to go in there with like a different sort of approach? And I think you can apply that as a new hire as well in a lot of those ways. I agree with this. And especially, you know, you mentioned something that I think is really critical to understand as a new hire. And it also is critical to another role that we're going to talk about in a minute, the intern. But we'll get to that in just a minute. The idea that documentation is almost never a bad idea. Yeah, almost. And encouraging, especially when you find that good documentation, encouraging it even further. This is something that, of course, like you said, you know, the very few times, very few jobs that I've had, very few teams that I've joined even for, you know, a momentary time for a limited amount of time. Many few teams would say we don't want you to write, to fill in the gaps in our documentation, which tells me two things. Number one, it's something that everybody wants. And number two, it's a problem that everybody has, right? This is a pretty consistent issue that it's much easier to write code and move on to the next problem than it is to reflect and take the time once you've kind of completed the task, right? To go back and say, okay. And perhaps it's part of the reason why we have, you know, certain kind of paradigms of programming like documentation-driven design or whatever you want to call that, where you have to kind of write out what you want initially, you write out the design and document it first before you implement it and then post implementation, go back and correct wherever you were wrong when you were kind of predicting that future. So it is a problem that everybody has. And I do think what you're saying about, you know, coming into a new team and being, I don't even know if assertive is the correct word, but understanding that everyone else is, you know, it's a very content, it can be a very contentious moment especially for the situation where you have a contractor coming in. It may be a little different for someone, for a new team member, but kind of the general principle here is change is scary. So whatever team you are joining, you are changing that team. You're bringing something scary, regardless of how good of a developer you are, regardless of how much promise you have, no matter what you do, change can be a scary thing for the people that you are kind of, I guess, bringing that change to, right? So even if you are considered one of the best developers, you know, you're bringing in all of this freed up time for them, it could still be something that they're concerned about. And I love the way that you articulated, you know, to be aware of that that perhaps even their livelihood is on the line partially because you're there, right? Would you agree with that kind of perspective that, you know, the teams that you're coming onto or the teams that you're joining, they may feel threatened by you, even if you are going to give them a better quality of life, better quality of software, whatever it is. Definitely. There is a, there's this interesting balance there of where, you know, you, you mentioned like you want to come in and be assertive, but you can also be over assertive. Very, there's like a fine line between this person is just cocky or what I mean, trying to think of like some better language there, but it just in terms of you can be a really disruptive person really quickly and because in a lot of ways as a consultant, you're given a little bit more leeway to speak your truth, I think in some ways, because that's what the, you know, the business owners probably want to hear the hard stuff and you're, they're hiring you to come in and tell them what they might suspect might be true or, or to, hopefully, alleviate their concerns. So I think back to, there was a, there was a project in the last couple of years that we, we came in and it was interesting because there was a, it was a software project where they had someone that had worked at the company for many decades, like, like, right there I think there's like three plus decades. And so they have been there for a really long time and lots of different technical stacks and they were, they're nearing the end of their, their, you know, their career and you're going to be retiring soon and they have new people that have been really, really struggling to come in as employees and work amongst that environment where there's the kind of the old guard or the senior old guard. So like, you know, I think we often don't get that scenario where that, there's that big of a difference and just how the tenure of people within an organization, but that, you know, that happens. And so that person's kind of looking at a certain way of like, print, knowing that they need to pass the baton to the other people on the team, but they don't necessarily agree with some of the new styles and new approaches and maybe don't even see the value in some of the, the development methodologies that are being more popular in the last decade or two. And so there's this interesting culture clash that we were witnessing and so we got pulled in because we were perceived to have more experience than their internal newer recruits. So maybe we can come in and help kind of create a bridge like between the person that's leaving because they know they need to make sure that this thing can keep operating beyond after the retire and the newer people can start to start incorporating it and take some ownership of everything because there was a lack of trust that like, well, I'll take care of that because I don't think they're ready to take, take over that feature or move that architecture. But maybe, but they felt confident enough to bring in us because they knew who I was back in the day in the ReBron Realist community or what it have you. And so it was a really interesting challenge where we came in coming in and we're like, okay, process was like a big thing. We were trying to help them with and so trying to help kind of bridge that gap, but it was still, it was interesting that we had a scenario where we came in and so we did like, there's documentation updates you can do. But we started making some updates to just like cleaning up because we're using some winters and just cleaning up some really basic syntax things, cleaning up some comments in the code. And so we said to Porequus, like within the first few days, just a few little things that someone was like, just some tidying up and that person had been there for a long time said, why are you doing that? It's not important. And we're like, like, you need to be focused on these other things. And we're like, but we were here just tidying up while we're going. Why this took like 15 minutes? Why are we having a 15 minute conversation about such a small thing? And we were realizing we're still having to, like, we were, we, we immediately fell into the same sort of scenario that the other people that had been hired full time there were extraordinary experience. So now we're like, okay, we're now we're part of this weird situation. I'm trying to figure out how we're going to even just change the culture of this team. And when that, so that type of scenario could be really interesting too, but it's I'm trying to get back to what like my, my original thinking for telling that specific story it was. I don't think it was so much that they thought that they were worried about their, their livelihood. It was more about their legacy. And and how much they had helped that organization for multiple decades get to where they were. And then to feel like in some ways like the changing of the guard was going to kind of remove them in some way from and their, their impact or people see. Yeah. And that was a, that was a tricky one and admittedly we didn't, we weren't super successful. We, we worked with them for a number of months, but it wasn't, it wasn't something that we were really able to push through and get to the other side with. And so, um, and then COVID happened. And so I think there's some challenges related to that as well. But the, um, but I think about those types of things and we're like, oh, that how do we, if anything, I felt a lot of it, more empathy towards that individual and thinking, okay, how do I get a bond and establish a trusted relationship? So we can do these sorts of things without them being kind of doing the thing that they would do with their junior or mid-level developers, being like, don't focus on those things, but we're like, but those are the things that keep contributing to a lot of the problems you're having, you know, six or 12 months down the road. And so you have a mess. We need to help clean it up, but you don't want us to focus on the mess. You want us to build a new feature because that's what the stakeholders want. But we, there's kind of like a tricky balancing act there to kind of navigate. Today's episode is sponsored by Linode. With Linode, you can simplify your infrastructure and cut your cloud bills in half. With their Linux virtual machines, develop, deploy, and scale your modern applications faster and easier. Whether you're developing a personal project or managing larger workloads, you deserve simple, affordable, and accessible cloud computing solutions. You can get started on Linode today with $100 worth of free credit. That's $100 in free credit. You can find all the details at linode.com slash Developer Tea. Linode has data centers around the world with the same simple and consistent pricing regardless of your location. You can choose the data center that's closest to you. And of course, you receive 24-7 top tier support. Here's the thing. Linode doesn't have any tiers of support, no handoffs, regardless of your plan size. You can choose shared and dedicated computants, or you can use your $100 of credit on S3 compatible object storage, managed Kubernetes, and more. If it runs on Linux, it can run on Linode. Head over to linode.com slash Developer Tea and click on the Create Free Account button to get started today. Thanks again to Linode for sponsoring today's episode of Developer Tea. This actually speaks to another question that I have about the consulting vantage point that you have, because you've had a chance to see a lot of these different junctures for businesses that are critical enough that they've decided to make this kind of, we can call it not necessarily a drastic decision, but certainly a change, a juncture in that business's lifecycle that they've decided to make a change. And I'm curious about a couple of things about that juncture. The first one is, what is the most common reason that you see people needing to bring in outside help, whether that's augmentation, like you mentioned before, or something is going wrong and they can't put their finger on it. What are those types of things that you start seeing consultants like your company being brought in? What is the most common couple of things that you see? Yeah, I think staff augmentation is the most common that companies come to us for. It is the thing that we do least often, usually because we're more of the, that usually involves a longer term, like 100% someone on their project for six, 12 months at a time and that's not quite the word. I don't mean interrupt you here. I just want to kind of clarify for people who are listening who may not be familiar with this, the idea of staff augmentation and kind of what the, what does that provide to a business versus just hiring somebody. Can you kind of just give the, you know, the very acne version of what that is? Well, in a really high level, it's kind of almost like you're acting as a very specialized staffing company where a company, a large company or even small company might be looking at their roadmap and be like, we want to build up a lot of features over the next six months. We don't know if we're going to need people beyond that six months at this point and we might want to hire people of a certain skill level to help us get there maybe quicker. And so they'll look to consultant companies like us to say, hey, can we get a two senior Developer That have at least X number of years with, and have worked with these specific technologies to work on this project for the next, you know, full time for the next three, six months or what have you. And so then they kind of just get embedded in the team and work on that project and then once that contract is up, you know, then they get re, you know, we put them on other projects and so they kind of benefit to the company there is to, it's not a permanent decision. The other part is if it's not working out within a certain period of time, you could probably pull the plug and, you know, not be on the hook for having to worry about everything else when it comes to, you know, all this stuff that the company needs is, you know, they would normally need to think about with a full time hire when it comes to like benefit programs and everything else in terms of, you know, like what they're, you know, even just like having to provide, you know, like one on ones, maybe might not even be need to have or I'm just thinking about all the ways you would need to get rid of a developer if they weren't like a good fit for your company and needed to terminate that relationship. It's different. So it's kind of like we kind of take that risk on as our company and so we were the ones that kind of do that, but to have to navigate all that. So it's, I think on a really high level that that's the basic thing is to help move yourself faster and not be making a permanent decision. So you can kind of bring in that expert talent that you might not be able to get by just putting it up a job ad and in finding that right. Like who are, so I think it's a way to speed up the, you know, maybe overcome some of the higher, the how long it might take to hire and bring people on. So like this happens every year. Like they're all usually, there's a couple organizations that we know that their busiest time of the year is like, because they're an e-commerce. So like between in the US, between Thanksgiving and, you know, Christmas is like their big season. And so they'll usually contact us September, maybe October saying, hey, can we get a few people for that period of time? We are the from now until the end of Christmas so that we have some extra people that have a lot of experience that we don't have to spend a lot of time, you know, ramping up. And then we don't, we don't know if we're going to need them, you know, in January. And so it's, that's, it's a way to kind of like add some velocity to your project and then kind of quickly pull that off and not still be responsible for keeping figuring what those people are going to work on after that. Yeah, I've heard it described as kind of a faucet where you can turn on the resources or turn them back off without, you know, having to have the overhead of finding individual contractors and, you know, especially if you have a pool of resources. So let's say you, you know, maybe you start with one developer and you evaluate, you know, the quality of the work and to make sure that things are going well and then you can scale up to two or three all within the same company, you know, that is kind of the idea that at least the way that I understand it so that you can instead of having to, you know, negotiate each individual contractor's rates, for example, or evaluate each individual contractor's quality of work, you have a little bit of a buffer by working through that staffing, yeah, or I guess a staff augmentation layer. And those people, but you mentioned that that's not really what you guys are doing most often. What are you doing more often than that? Well, the thing that we end up typically, you know, outside of the taking over projects and being in the primary development team of those projects, which is like kind of the core of what we do. And those are usually for like projects that are like, there might be a large company that they definitely have their own internal development teams, but this project is maybe too small of a thing for them to have a team for. So rather than having like one person working on it, they'll kind of outsource that to a team like us to be responsible for it. And a lot of the reasons other companies will contact us as well is because they're behind on the versions, their application. So they need help with either with helping perform an upgrade on say the the foundation life, their application. So we primarily work with our band around. So there's a lot of like upgrade related projects and something that we've been doing more of and focusing on as not only maybe just doing the projects because actually we've started to take this opinion that we don't think we should be the team that handles like shouldn't outsource entirely an upgrade. I think you might probably would probably understand why, but there are people out there like, well, we don't have time to do it right now. So why don't we hire a team that's really competent at that, let them do it, and then we'll deal with the big merge and then we'll be done with it until it needs to happen again. And so one of the things we've been trying to focus on the last year is working with teams on helping them build up those skill sets themselves so that they're regularly thinking about how they're going to keep their dependencies and underlying frameworks up to date so that they're not following for five years behind and then needing to call a company like us when they're when they realize that their underlying version of rebound around or Ruby itself are no longer getting security patches and that raises some alarms on their end. So we just took quite often it's usually like, oh no, security problems because we're not getting security patches and we need to call companies like plan our own to come help us and help get us out of that mess. But what we want to do really is like help them fix that problem so it's a they can keep it going themselves long term. Yeah, yeah. And so it seems like it's these to the name maintainable, it's about maintainability. And that brings me to perhaps the closing question on this, you know, in this section, I guess, of the interview. The idea that the software is un-maintainable in what ways do you see most commonly software becoming poorly maintained? And I guess kind of what I'm hoping to hear from this is not necessarily a specific thing but a kind of category of things. What are people doing? What are those, I guess, the behaviors, the habits that engineers are ultimately practicing in that leave their software un-maintained? What are you seeing most commonly with that? So I'm glad you asked that. I think they're, you know, I talked to a lot of people on the podcast and get their interpretation of what they believe is maintainable software. They're thinking about their contributing factors to it and in my own experience, there's a tendency for Developer To, I think, to not really stay consistent to some degree with their habits. And I think when it comes to, like, okay, I'm going to document this or I'm going to like, you know, I think especially on projects early on in their life cycle where you might be building an MVP for a project, you don't even know if it's going to be around any year or two. So there's always that balance of like, you're trying to build something up and you're going to see if it's got a product market fit and then maybe the team will grow in the future and you know, it gets an investment or you actually make money with the product and you can build out your team and expand it. And that's a great situation. And then so then you have to come back and then usually start backfilling and figuring out how you're going to clean up things or help things scale. And so, but sometimes there's a lot of projects out there that they don't really end up scaling a lot in terms of Developer Tea size. And so there might be a really small team. And so whoever those original people on it, they kind of set the tone on projects for how much they're going to invest in things like testing, documentation, just consistency in the code or even just like how, you know, if you think about even like the simple thing, something that I always look for, like even just like when you look at someone's like someone applies for a job and you look at their portfolio projects and you go look at their own, like say get up profile and they'll see the projects that they've worked on in their portfolio. Just go look at their get commit messages to get a sense of how are how do they operate and deal with how do they document why they're making changes in their get commit messages themselves when they're kind of working on their own to get a sense of like, you know, I think I think that can tell a story doesn't mean that there's anything that's inherently bad with that. But I think those types of things happen on real world projects a lot too. And you know, there's a, I don't know how many times you've, you've might have seen a project where you look at the first, you know, 50, 100 comments or get commit messages or whatever versioning tool you're using. And there's just a lot of like really one, two word type thing, fixed type of, you know, updated whatever and there's not any explanation of what, why those things are changing. And so I think you can set a tone with as you bring other people into a project where the, the patterns that you've established is what people kind of learn, keep building off of unless you get hired, unless someone comes in that has more experience and says, okay, this is, we need to improve upon this and they make a conscious decision to change the nature and the, let's say the little microcosm of the culture of that specific application. And you know, if they start making some intentional choices, like we're going to start doing this now, we're all agreeing to do this. Great. And you start moving in that direction, but you will look at code where, you know, there's always, you know, there's conversations about finding the, let's say the God objects in your application where there's just too much complexity like that there's all high churn in those, in those areas. There's a lot of really big objects in your application. And so when you bring in other people in the project, they're like, well, this seemed to be where people added things last time. I'm going to add something else here. Then it kind of just keeps reinforcing that that becomes a bigger mess. And so, you know, a lot of coding is done to some degree can be copy-pasting or at least like copy and pasting an idea in a very similar area. And it might not be the best long term approach, but we take those kind of easy outs a lot in that all those little micro decisions start to add up where five years down the road, you're like, wow, there's so much, this is everything is brittle because there's maybe a lack of test coverage or there's like just a lot of interconnectedness in the code that, you know, if I, people don't really have a good holistic view of it anymore, you know, we talk about documentation, can also get really outdated. You know, there's like schema diagrams that people might produce and nobody keeps updated. They might not even be valuable anymore. I don't think people sometimes consider deleting documentation enough because it feels like that might be not helpful or they might be needed one day, but sometimes all that extra noise can also be distracting and make things more difficult so people become less reliant on it because there's not like a, like, if people start to dismiss documentation as, well, that's outdated. Like, why is it existing anymore? This is going to get rid of it then. In the same way that we should feel comfortable deleting code at times if it's not being used. Let's get rid of documentation and other resources that are just noise because I think the no easier things can be in an application environment. The harder it is for new people coming into the project and really go, okay, this is how this team works and operates. But if you, there's an undercurrent of, well, the culture in this code base is to just leave stuff. Okay, then I guess I'll keep doing that. And that's an easy decision to make and you don't have to, people don't have to, you know, think too critically about it. So I think those are some things that I see that are minor, but they do kind of stack up over a period of time. Yeah, yeah. I think it's really interesting that you mentioned going and looking at GitHub commits. There's this difficulty of trying to understand what is going on on a team. And this is kind of the underlying theme of what we're talking about in this whole, in this whole episode so far. If you're joining a team, whether you're joining as a contractor, if you're joining the team long term, if you're coming in as a manager even, or let's say that you're getting moved from one team in an organization to another team in an organization, people who have experienced this, they understand for sure that trying to get an idea of what is actually happening on that team, it seems like it would be the easiest task in your onboarding checklist, right? But this is the one that takes the most investment, trying to get a handle on, you know, what is the history of this team? What are the people on this team care about? What do they not care about? What are the kind of consistent problems? What are the, you know, the dead angles, the missing information? What do they not know about themselves, even? What are they, you know, kind of misunderstanding about the software that they're working on? As a new person coming in, it's easy to have misconceptions, very easy to have misconceptions about the people on the team already. For example, I've joined teams where I thought, wow, you know, the engineers on this team know this code front to back. Every line of code that's here was put here intentionally and, you know, me walking in today is, you know, I need to, like we said earlier, I need to take a step back and just listen, that's almost never true, right? It's almost never true. And so it's, and again, especially if you're a junior engineer coming onto a team, it's very difficult to know what's actually happening on that team. It seems like it would be very simple, right? You know, and your point of looking at, you know, get commits. This is a really good example of go and try to find data about what's going on on this team at the individual atomic level, right? At the smallest level possible, what kinds of changes are being made to the code? Is there improvement type changes or are there quick fixes? Is there a lot of kind of responding to errors? Is it, you know, what is the common pattern? You can kind of think about this as like the direction of the wind in a way, right? Like that's, which way is the wind blowing? Oh, it's blowing in the error direction, right? Like, we can kind of think about, you know, the code is going out as responsive to errors right now. Or you can think about it as, oh, the wind is actually going down the maintainable track. We're adding to the test base where, you know, improving, we're doing refactoring. We're all in that improvement space. Where you can think about it going in the direction of new features where that's being driven. And it is interesting because even in the most hyper optimized teams that have everything dialed in, they've got, you know, the team is jailed. There's the, you know, everybody's happy with the team. They're happy with their production. There still is a balance to strike between, you know, refactoring, maintainability, etc., new stuff, right? New, new features, new code. So I'm wondering, how do you, especially with your kind of bias towards maintainability, how do you see that playing out on a healthy team, the balance between those two things? Newness, new features, all of that stuff. And the necessary and critical maintainability that acts as a basis for that new work. It's interesting in that I think I have a skewed world view on that. And because I don't work on a product team. A lot of the projects we work on, where consultants will be, they might be a product type of organization or have like a SaaS, but a number of our projects we work on, it's not a lot of really, there's some element of the drive from the, our clients is not necessarily always new client, or new features as much as it is, maybe improving performance or just keeping things updated and maintained. And so there's periodically new features and such that are so quite often, and depending on the projects we're working and we're responsible for, there could be like, oh, we're doing a really good job because we took over a project that was kind of buggy. It's less buggy, but a lot of the work we're doing right now is bug fixes or data tracking down data problems and helping improve the stability of their data and their system and helping make more sense of that. So it's an interesting one. I don't feel like I have a good grasp on how to like come up with a healthy ratio there given that I don't work in, I think where a lot of companies might be, or a lot of people listening might be in more of a product driven type of environment. Or sometimes we're working on applications where it's internal. So it's about just being, keeping things stable and iteratively improving upon things. And so when we get to work on features, that can be great, but we don't always get to focus on that necessarily. So because stability and taking care of a lot of like the support stuff. So back in there, we're providing maintenance and, you know, occasionally for those projects we might have a handful of features or buildouts that we're doing for those projects to extend what they're capable of, but that's not always the primary focus of like our clients are hiring us. Yeah, that makes total sense. And, you know, to, you know, having been on many product teams, I know that this balance is really hard to strike. And part of the reason for this is, apparently because of the way the decisions are being made about products, right? Most of the time, somebody like an engineering manager and somebody like a product manager or product owner will meet and determine priorities together. This is a very common pattern for it. While the engineers are working on, you know, the current sprints or whatever your iteration is called, you're kind of grouping of work. The engineering leader and the product leader are meeting together to talk about what's coming next. And the engineering leader, their job is to try to understand and cut through all the hay's about which of these maintainability issues is going to cripple us in the long run or in the short run. And how do we, you know, prioritize that versus these new features, right? And this is notoriously a huge problem in startup world, but it also pervades in non-star companies because the perception is very easy to say, I don't see anything wrong, right? It's very hard to see those invisible issues that engineers are very aware of, but products, perhaps products owners and the product organization or suborganization department, whatever, they're not necessarily seeing the pain from those. So finding ways of elevating what those maintainability issues, what are they going to cause in the long run, right? In a very simple way, you could say, okay, well, we have security problems that are leaving us vulnerable today, right? That's a very simple one. What are some other ones that you've encountered? Yeah, so there's the security is always like one of the best ways to kind of at least help, I think in some ways like help pitch or sell, dealing with like upgrade type related projects in general, but there's also, you know, this could be, we can improve the performance of the application so that, you know, the application will run a bit faster and that's going to be, you know, that's proven to be more effective on mobile devices when people aren't on a Wi-Fi connection or, you know, the, I think another part of it, if you can kind of, there's always a business value associated to the work that you're doing. So if you're going to make some improvements or to refactor something, it could just speed up the development life's like on the long run or like, or if you can calculate, we're having to work around these problems and every time we have to do with this area of the code base, you know, it's costing us a couple of extra hours or a couple of days to to navigate that or because it's, we lack enough testing in this area of the code base or it's just inefficient with how we're trying to navigate things and we have to do some like data migrations to make to anytime we make a change in this area and that needs to be tested and that has some associated cost. If we can refactor this area, we think we can remove some of that, I mean, a decent percentage of that extra work that we feel like we're doing right now. So that can be another way of like helping, like we're removing obstacles, we're going to save time in the long run if we invest time now to tackle those types of challenges. And another thing that often gets overlooked, I think can be really effective with the right stakeholders would be to talk about employee retention, that if you're dealing with an older code base or a very painful to develop an environment where you're feeling like your team has high turn in terms of needing to bring on new people. So it could be the ramp up time for onboarding is super slow because of the way things are like we need to improve the some areas of application because it'll help speed up how quickly we can introduce new people into the project and or we're losing people because they're excited about the opportunity to work with their new versions of some of these technologies or other technologies potentially entirely is often a reason why people go move elsewhere. And so like if you stay to like rest in too much like, well, we don't want to invest in improving things, you might lose your people and all the knowledge that they currently have of the your infrastructure and application will walk out the door with them and you know, you might have a good process for like document everything before you leave, but we all know that not everything gets documented because we don't know what everything that's in each other's head that we need to gather from them. So that I think that's oftentimes a thing to think about is like we if we fall too far behind on some of these problems or if we don't improve these processes, we're going to lose people and that's going to cost us in a lot of other ways too. I totally agree. I totally agree. And there's this like I said, it is a continuous problem in in product organizations of all sizes and it's necessary for engineering leaders especially to learn what are the most important kind of shared concerns that I can focus on to make sure that our software doesn't fall into a very difficult state. One of those is certainly hiring, right? Or we're not necessarily hiring, but although hiring is another one, right? But you know, if you were out recruiting and you said, hey, we have a policy for example, and we focus a certain amount of our time on maintainability issues and upkeep and sweeping out from under the rug, so to speak, that is a very attractive prospect to most engineers. I think it is. And there's also this idea that there's another expectation that I think sometimes where companies could also in some ways, I've seen this where I think developers can also know that the audience is primary developers, but they can allow their own kind of personal interest of learning new technologies, help make technical decisions that might actually be really harmful to the organization that they're working with too. Like, oh, we're going to, we're really interested in using microservices, but you're a team of three. And you spend a much of time building up these things and start going down this path and then they leave and then you or maybe they don't leave yet, but we'll see what we get called in to help out companies where it might be a small team, but they have a lot of things that they're really interested in exploring, but some of those people aren't there anymore. It's only like a year after they made that decision and they're like, we don't really know what's going on. The developers said they wanted to use this new framework. And so they built out that I'm like, well, this feels really complicated for what you're really doing. And it starts to feel a little bit like resume driven development kind of took over for a bit because the stakeholders would trust the Developer To make good decisions, but I don't always think that developers are always acting in the best interest of the company as much as they're also looking to scratch their own itch. And that could be a really challenging thing to navigate. And so the people that have to come in and help navigate taking over those projects are in kind of for a fun ride of having to kind of make sense of all that too. And so which might be a way to recruit new people for new technology, but then if you're also learning a new technology on the job and you don't have anyone else helping establish good foundations for what your workflow is going to look like, your development patterns are going to be or making that elusive. Another back to the whole theme of maintainables. One of my goals is to fight the big rewrite because oftentimes, rewrite fail because the people drastic under estimate what it really takes to go through to not only build out a new application, but to keep the existing one going in parallel and then dealing with that migration at some point. And so a lot of failed upgrade projects out there. A huge thank you to Robby for joining me on today's episode. If you don't want to miss out on the second part of our discussion, go ahead and subscribe and whatever podcasting app you use to listen to Developer Tea. The next episode of the show comes out on Wednesday, so if you don't want to miss out, make sure you subscribe. This episode and every other episode of Developer Tea can only happen because you listen and you share this podcast with other people. You sharing this with someone that you believe would appreciate it. This episode, specifically this episode, that's going to help this podcast grow because you may find someone who decides to also continue being a listener of the show and our listeners are the reason we do the show. If we don't have you, then there's no reason to do the show. Seems pretty simple, but no pressure. The fate of the show is in your hands. We have no plans of stopping anytime soon. So thank you so much for listening. Thank you for leaving reviews as another way to help us out. Of course, this episode and every other episode of Developer Teacan be found at spec.fm. There's show notes there, there's show notes. Also in your podcast app, most likely if you use a normal podcast app, the links to our sponsors are there, including today's sponsor, LynneauheadovertoLynneau.com slash Developer Teato get $100 worth of free credit as a listener of this podcast. This episode is produced by Sarah Jackson, my name is Jonathan Cutrell, and until next time, enjoy your tea.