Hey everyone and welcome to Developer Tea. My name is Jonathan Cutrell and in today's episode I continue my interview with Sam Lambert. Sam is the director of systems at GitHub. In today's episode we start out by talking about Hubat. If you missed out on the first part of the interview, make sure you go back and listen to it by going to spec.ethm. Of course every episode from the past of Developer Teacan be found at spec.ethm. Make sure you go to spec.ethm. That's where you find the show notes for the episodes and there are other podcasts that you should check out there as well. So, spec.ethm, that's where you'll find the previous episode, the first part of the interview with Sam. So I've asked you guys to do this for a few interviews in a row now. I'm going to ask you to go on the Twitter and thank Sam Lambert for coming on the podcast. His Twitter name is I.Sam Lambert. That's I.S.A.M. L.A.M. B-E-R-T. Of course that will be in the show notes as well. Just go and thank Sam for being on the show while you are listening today. He will appreciate the thank you and I will as well. I'm very excited to welcome a brand new sponsor today for today's episode. The sponsor is Rollbar. With Rollbar you can put errors in their place. In your application you will detect, diagnose and defeat errors. You get the context, the insights and the control you need to find and fix bugs faster. We will talk more about Rollbar and what Rollbar provides to Developer Tealisteners later on in today's episode. But as always, first we're going to jump straight into the second part of the interview with Sam Lambert. HuBot. You've called HuBot the hardest work in GitHub in the past. I'd love for you to just give kind of the elevator pitch for why HuBot deserves the biggest raise from anybody in the company. HuBot's commit history. HuBot does a lot of work. For people who don't know HuBot is our chatbot. Also a central focal point of our culture. You can do anything in chat via HuBot. You haven't finished shipping something into it. There's that way of using controlling via HuBot. The concept of chat-ups is fascinating to me. It's so interesting because it brings the place you work, a chat room to life and brings everyone in there with you. It's like having all your teammates in your terminal window. You're controlling infrastructure, you're controlling services and processes and people are watching you and seeing it happen. That just removes so much communication overhead. If you call up a graph into chat and you start looking at something, I've seen it. It's probably happened today already. Someone has pulled a graph into chat and discussed it with someone else. You just see it happening in line. You can just go in and track that conversation happening. Not only you're involved, you're watching people do their work and that's where it happens. It's just fascinating. There's something truly amazing about it when you see it done right. It's wonderful. HuBot allows for people to essentially issue commands in a chat window, mention HuBot or whatever. You guys use an internal chat system or what do you use? We're using Slack for chat at GitHub. We previously had a half-done system that we built ourselves. We were running our own client, but now we've moved over into Slack with HuBot. The reality is you can use pretty much any chat system with HuBot. I guess it's worth mentioning that right now, developers, you can go and look at HuBot. It's obviously it's open source. You can go and clone HuBot into your company and use it for the same types of things that GitHub uses it for. Yeah, absolutely. There's loads of open source scripts and plugins. There's lots of companies that are contributing to the HuBot ecosystem. We do as well and we build a lot of stuff internally. It's really easy just to get going and start adding things. Companies like Haroku have added ways of being able to deploy from HuBot. It's fantastic. The range of things you can do is huge. We just have fun things to very serious things. You can command our entire fleet of servers from HuBot while everyone watches and collaborates with you doing things. You can deploy servers from HuBot. I was doing it recently. You essentially tell HuBot you want X number of hosts of this class and they will come back within 10, 15 minutes. They'll be ready and available all from your chat window. It's the one common interface I think that most of the teams around the company use is HuBot. You're right. I think chat ops is a big part of our future. It's like assisted intelligence. A side question that I have for you. Do you have any plans of implementing machine learning into HuBot into the future? Is that just too much? I don't know. I don't even say we had plans. It could be where someone in the community takes it. But I wouldn't say we had planning on necessarily adding machine learning to HuBot. I would be interesting if there's anyone listening that wants to try. He wants open source. He accepts all requests. It would be interesting to see what you could do that. But no. There's no specific problem to solve with machine learning. Chat can solve problems in how you can surface information but not necessarily specifically to HuBot. Chat brings up an interesting problem that I think a lot of people are facing right now in agencies. That is trying to maintain focus while also staying connected. For example, at Whiteboard where I work, we have chat windows that will fill up enough that for me to catch up, it takes more time than for me to just ask somebody in that moment, update me on what's going on. So I'm interested to know how do you see focus alongside working with a chat program? In other words, do you stay connected pretty much constantly or are there times where people are kind of disconnected from chat and working in code for a little while and then they come back or how does that work? Absolutely. Again, no dictated workflow if one works differently but sometimes I'll close the chat window and just ignore it. That's the great thing. It's very hard to close down your office when you want to zone out. You see all people around you, people might tap you on the shoulder and talk to you but when it's all happening in chat, you can close the chat window and just go and do whatever makes sense to you in that focused environment. You can turn notifications off and chat. It's really flexible because it's like this pipe of information that you can choose to subscribe to at the right time. Again, it's about social definition whether you'd codify these or not, I don't know but it becomes around your culture. Do you do everything in chat or do you take it to a point where the next conclusion is we open an issue and discuss it there. It's very good for quick fire discussions that's not quite a topic yet and then you lead that into an issue or well, the best answer is roughly talking about things in chat and then start hacking on it, build up a loose prototype and talk about that from there. Talking about work is working the abstract. You can apply it to code and send a poor request and then that's where it starts to begin and it's fantastic to see actually. I see this all the time. I see team members and engineers, they spike out a concept and that's the beginning. My tip for anyone who uses poor requests is open that poor request as early as possible. Quite often I'll make it a near empty commit. I'll lay out some file structure commit, send a poor request. Alex, I'll talk about my intent for this poor request. This is where by the time I get to the stage of closing or merging this poor request, I want to go through this cycle and it's about communicating and bringing those people with you in that journey. It's fantastic when you see someone who's incredibly experienced in their field, join the company. Be shown how to do a good poor request and you watch. This is my intent and I will iterate towards that goal while communicating that intent all the way through and it's a fantastic way to work again. It's rich with information and it tells the story of how you built something. If I was to give a tip to any of the listeners, it's absolutely open poor requests early. They shouldn't be, I don't think they should be the last thing you do to get a change make. Then you haven't collaborated. You've worked on your own, told the way and then you're presenting this final thing. The potential of what it could be is so much more when you're discussing it all the way through and the tools are there within a poor request to be able to do that. It's a very powerful process for sure. I think multiple issues can be closed in a single poor request. Let's say you have three or four issues that have cropped up in the past week or something and you want to be able to close all those. I would say, you could correct me how you guys do things here, but opening a poor request and then mentioning those issues is better than trying to go and work on each of the issues on their own. Because everything is networked together and so you could just mention an issue and it'll show up in both sides, both feeds basically. Yeah, absolutely. In the poor request acts, it's like the central trunk essentially. Imagine you were rolling out and you cast a host, for example, and you're doing poor requests and you hit an issue. A problem has come up. You can either CC the team that could help you with that into the poor request or you can open issue on the repositories they look at. We organize around repositories as well. But the rich cross-referencing is amazing. When I joined GitHub, GitHub had been a company for four years, just over four years. I was able to go and look at poor requests that are years old. Because the people that were working in the company at that time understood this rich tapestry of information you can weave and build by communicating clearly through issues and poor requests, I would say see a setting in a file, for example. I'd want the history of how that was set that way. First, EBA at the company, people before then were very skilled, but not necessarily experts. In Mask Girl, for example, I would be like, okay, this is a variable that's set for a reason. What is that reason? We look at the Git Blame. You go back to the poor request that introduced it. You look at the discussion. People have pasted in graphs from the incident that caused people to set that setting. You then look at the issues that were opened before then. You just get this huge piece of information all from just the history. So you should, if you treat poor requests and issues correctly, you build up this really amazing body of information. Why spend your time documenting playbooks and wikis and things when you can just the process of writing code is self-documented? It's a living discussion. In fact, one of our major projects that we're working out now started as a poor request that was never merged. It was two engineers hacking on a concept, throwing stuff around and then closing the poor request and extracting out the real learnings from that poor request and then really getting going on the project. The final comment isn't always merging the poor request. It's enabling the debate around the change that you're trying to make. Yeah. That actually brings up kind of an interesting point that I'd love to shift over to, which is features at GitHub. Before we talk about features at GitHub, today's episode is sponsored by Rollbar. With Rollbar, you can put errors in their place. Rollbar allows you to detect, diagnose and defeat errors. You will have the contacts, the insights and the control you need to find in fixed bugs faster. You know, dealing with errors sucks. There are so many different types of errors that you can deal with and a lot of times they are difficult to find, especially for trying to dig through the logs. And of course, that is not a fun situation to be in, especially when you're in the high intensity situation of trying to fix a server that is experiencing a bunch of errors under load. Rollbar works with all major languages and frameworks and you can start tracking production errors and deployments in eight minutes or less. So you can integrate your rollbar into existing development workflows. So you can send the alerts to Slack or HipChat or you can create issues automatically in GitHub, for example. How appropriate we're talking about GitHub today anyway. Or pretty much any other service that you already use. Now, customers that use Rollbar include Heroku, Twilio, Kayak, Instacart, Zendesk and Twitch. Now, Rollbar is offering Developer Teaa free 90 day bootstrap plan. That's 300,000 errors that you get to track totally for free, simply by going to rollbar.com slash Developer Tea. Of course, that link will be found in the show notes at spec.ethm. Thank you so much to rollbar for sponsoring Developer Tea. So from birth to shipping, what does that flow look like? Where do the features come from? Who suggests them? Who decides eventually what's going to happen? Or what's going to actually be put into development? I think it's different for nearly every feature. Some features become really obvious at a certain point. You hear customers talking about these features. Or we use the product ourselves. It's the best dog food experience you can imagine when you're using the thing to build the thing. The tagline on the GitHub repository is you're looking at it and it's true. You are. We're using our own product to build our product and that's this amazing experience. We've got these wonderful Developer That have access to change the thing that they care about. They use to do their day job. So features come out of just that being a thing. That being possible. Other features come from listening to people that maybe work in a different area of software or within a different type of company. And they share with us how they believe GitHub could be made better for them. And we think about that. We look at data. We look at research. We have research teams that we'll go into studies on how GitHub is being used and we'll learn. Because you can only empathize with your users to a certain degree. And then it becomes really you have to look for that. You have to mind the information and the usage patterns that tell you how things can be made better. And you have to quantify that somehow. So features come out of those processes. Or they just come out of a random late night discussion in chatware. Oh, you know, this would be cool. Oh, I can start hacking on. I can build that up. And then suddenly other people get involved in it. Roles and rolls and it becomes ready to go. And so it's different every time. Again, not prescribed too much process around that. We didn't hit bit different ways of doing things. We always explore new ways. And they work for certain areas that we're working in. And the next project may just it might not be applicable. That's really interesting. Because it sounds like a relatively organic process where things can come from pretty much anywhere. Yeah. And we have a wonderful product team that tend to the future of the product. They think about how the product should grow and be cared for. And there's something you can't underestimate that either. There's some things that just don't necessarily always happen organically. You have to you have to really proactively think about your product and care for it. And that's and our product team do that. Yeah. So what's what's amazing about this to me is that GitHub has over 100 engineers at this point. And having that many minds and that many different types of experience and different contexts. That's a powerful machine. Right. If you only have one gatekeeper that decides, well, this feature is good enough to put some some energy behind. Well, that's not really a useful at nearly as useful as taking advantage of that many mindsets, that many creative people. So I love the idea of allowing for that organic growth of features and you know, marrying the the process oriented stuff where you're going out and interviewing your customers really to determine, you know, what are some features that we never could have come up with ourselves that our customers are, you know, pushing us towards. But also using the product internally and saying these features make sense for us. They're probably going to make sense for other companies as well. Absolutely. With that said, what are what are some of the big problems that GitHub is facing today? Well, it's getting a workload that's relatively new. The use of Git is growing daily and especially on GitHub during this podcast and conversation, you know, are the number of bytes we're storing for Git has grown and it's always growing and that brings new problems. You have to revisit your approaches and you have to think about the workload that you're scaling and it's a fairly new one. You know, there's more there's better understood workloads across the industry like a web apps database workload. It's fairly understood, you know. But it changes as we as Git as the use of Git grows, it becomes a larger and larger scaling problem for us. So it's keeping ahead of that and building systems that are intelligent and resilient to the different workloads that will come up and fail in predictable ways. So there's a continual growing challenge and we have the typical challenges that a growing company has and we also, I guess we have this problem that you could essentially you could imagine the future of any feature on GitHub. It's so open ended almost and it's about keeping the core essence of why GitHub is important and relevant to our users and not diluting that too heavily. You know, you can get lost in just a mile of features and it becomes confusing and how to use part of the reason I love using Git Hub is it's simple and it's fairly opinionated about a workflow and I believe it's the right workflow and it's in workflow that I've seen work very effectively. And so that's a challenge as well is how does it evolve to include everyone's workflow without just being a clutch of features that represent necessarily every flow. Yeah. Well, that's it is a powerful and validated many times over by large and small projects I like and so I think that I think you're on to something about the workflow there. So yeah, I think it's a good workflow. Again, no workflow should be prescribed to anybody in different industries and different types of people who use a different workflow and that's the great thing about we have an API. We have various different ways that you can just plumb in how you want to do things and you see really novel and interesting ways that GitHub is being used. It's fascinating. It's open ended enough to be flexible but still simple and elegant and the way that things are done. Sure. Yeah. So Sam, this is been a great discussion. We could go on for hours I think and really unpack all these topics but this show is short. It's intentionally short. We don't have T here today. I feel like that was definitely a shame. I can pull some T from him. I'll have a hand stash. But I do want to ask you two more quick questions, questions that I always ask every guess that comes on the show. The first question is if you could talk to every developer and give them 30 seconds of advice, what would that advice be? I'd say don't believe in magic. There's things that will be rented you throughout your industry, throughout the industry throughout your career. There will appear like magic and the truth is none of it's magic. You really need to understand every trade-off that you're making. You're afterwards being an experienced engineer is really you get quicker understanding trade-offs that you're making. The thing I love about working with really experienced engineers and this is the thing that's unique to truly experienced engineers is they understand the trade-off that they're making. They're constant. You'll always be making a trade-off. You know, there's just, it gets to a stage where it's physics at the end of the day and not acknowledging the trade-off just diminishes your ability to truly understand the systems that you're working on. Go deeper than what's presented to you. Understand the trade-offs that software is making for you. In the day-based, well, the day-based is the promised thing. We'll do everything for you. They'll do automated scaling and failover and high availability and data consistency and all those things all at once. It's not possible. There's something that gives and the point is you understand the trade-off you're making and you understand the failure case and you build around it. You build systems that fail predictably, not that don't fail. That's a, you're never going to achieve the systems that don't fail. Power supplies go out. It just happens. Don't hold onto that as something you can reasonably achieve, but instead try and understand the way it's definitely going to fail and make sure you understand that failure case. It's great advice. Don't believe in magic. The second question I like to ask every developer and this is really a question that I like to ask pretty much anybody I come in contact with, but what do you wish more people would talk to you about? What topic do you wish you could talk with people more about? I think we could have more production. Productive conversation doesn't industry if we talked about the type of innovation that we shouldn't shouldn't be doing. I feel like people feel tasked with innovating every area of their stack. I don't actually believe that's the right thing to do. We have important things to build that are unique to what we're doing. If you choose to spend your time innovating on inventing your own database, you're going to waste your time because it's done. A lot of these problems have been solved. Unless you're at a huge scale, what's good for most companies around your scales is good for you. Don't be unusual. Build off boring technologies. There's technologies that are in users' huge scale that are fine. If there's fine that that scale is fine for you, just understand that you don't need to be completely unique and different in every case. Make your product unique. Make the things you offer your users unique. Don't necessarily have to have a weird and interesting or stack. I think we have more productive discussions if we just said this is a solve problem. This is a problem that maybe we just don't even need to care about and then count and continue and do what's actually interesting and what we can do with technology. We can sit in right around databases day in day out if we want. Is that really the best use of the time of our engineers when we could actually use technologies to do interesting and new things in the world? So yeah, people should ask me about the things I don't care about. You know, boring stuff. Yeah, that's great. Well, perfect. Thank you so much for joining me on the show. And thank you to GitHub for creating a product that all of us use so pretty much. Yeah, thanks for coming. Thanks for everyone that uses GitHub and to the open source community and the communities that have surrounded us as a company is wonderful to see the way technology is changing and growing. I'm just excited to be at the company that's in the middle of all of that. Awesome. Thank you. Thank you. Thanks for listening to today's episode of Developer Tea. Of course, if you missed out on the first part of my interview with Sam Lambert, you can go and check it out at spec.fm. Thank you again to GitHub for inviting me to come and visit the GitHub office. And thank you for listening to today's episode. This podcast would not exist without you, the listeners. So thank you so much for being here and listening to me through your headphones or through your car speakers or however you're listening to this podcast. By the way, if you're enjoying Developer T, make sure you leave a review for Developer T in iTunes. There is a link directly to Developer T page and iTunes in the show notes. So you can find those in your podcasting app. Of course, you can find them at spec.fm. And if you leave a review, this is the best way to help other Developers find Developer Tea. And it's the best way that you can help the show continue doing what we do. So thank you so much for listening and for leaving a review. And thank you to today's sponsor, Rollbar. With Rollbar, you can detect, diagnose and defeat errors in your application. Go and check out rollbar.com slash Developer Tea. Thank you so much for listening to today's episode. Until next time, enjoy your tea.