Claim Ownership

Author:

Subscribed: 0Played: 0
Share

Description

 Episodes
Reverse
About SterenSteren is a Group Product Manager at Google Cloud. He is part of the serverless team, leading Cloud Run. He is also working on sustainability, leading the Google Cloud Carbon Footprint product.Steren is an engineer from École Centrale (France). Before joining Google, he was CTO of a startup building connected objects and multi device solutions.Links Referenced: previous episode: https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/google-cloud-run-satisfaction-and-scalability-with-steren-giannini/ Google Cloud Carbon Footprint: https://cloud.google.com/carbon-footprint Google Cloud Region Picker: https://cloud.withgoogle.com/region-picker/  Google Cloud regions: https://cloud.google.com/sustainability/region-carbon  TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don’t leave managing your database to your cloud vendor because they’re too busy launching another half-dozen managed databases to focus on any one of them that they didn’t build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. My guest today was recently on the show. Steren Giannini is the product lead for Google Cloud Run, and we talked about that in a previous episode. If you haven’t listened to it, you might wish to go back and listen to it, but it’s not a prerequisite for what we’re about to talk about today. Because apparently Google still does it’s 20% time, and one of the things that Steren decided to do—because, you know, everyone needs a hobby—you decided to go ahead and start the Google Cloud Carbon Footprint, which is—well, Steren, thanks for coming back. What the hell is that?Steren: Thanks for having me back on the show. So yes, we started with Cloud Carbon Footprint, and this is a product that now has launched publicly, available to every Google Cloud customer right out of the box of the Google Cloud Console.Corey: I should also point out, because people always wonder and it’s the first thing I always check, yes, this one is free. I’m trying to imagine a scenario which you charge for this and I wasn’t incensed by it, and I can’t. So, good work, you aren’t charging anything for it. Good job. Please continue.Steren: So, Google Cloud Carbon Footprint helps a Google Cloud customer understand and reduce their gross carbon emissions linked to their Google cloud usage. So yeah, what do we mean by carbon emission? Just so that we are all on the same page, these are the greenhouse gases that are emitted due to the activity of using Google Cloud that are notably responsible for climate change. And we report them in equivalent of carbon dioxide—CO2—and you know, the shortcut is just to say ‘carbon.’ Corey: Now, I’m going to start with something relatively controversial. It’s an opinion I have around this sort of thing. And I should also disclaim, I am not in any way, shape, or form, disputing the climate change as caused by humans is real. It is. If you don’t believe that, please go listen to something else, maybe Infowars. I don’t know and I don’t care. I just don’t want you around.Now, the problem that I have with this is, on some level, it feels like a cloud provider talking to its customers about their carbon footprint is, on some level, shifting the onus of responsibilities in some way away from the cloud provider and onto the customer. Now, I freely admit that this is a nuanced topic, but how do you view this?Steren: What I mentioned is that we are exposing to customer their gross carbon emissions, but what about their net carbon emissions? Well, Google Cloud customers, net operational carbon emissions are simply zero. Why? Because if you open Google’s environmental report, you will see that Google is purchasing as much renewable energy globally for the year as it is using. So, that means that on a yearly basis worldwide, every kilowatt hour of electricity has been matched with renewable energy.And you know, this Google has been doing since 2017. Since 2007, Google was already matching its carbon footprint with carbon offsets. But 2017, Google went beyond and is matching the purchase of the electricity with renewable energy. So, in a sense, your net operational emissions are zero.Now, that’s not sufficient for our customers. They have some reporting obligations; they need to know before this renewable matching, what were their gross emissions? And they also need to know what are their emissions coming from, not only the electricity usage, but maybe the data center or manufacturing. And this is all of what we expose in Google Cloud Carbon Footprint.  They are before offset, before renewable energy matching.And you’re right also to say that this is not only the customer’s problem, and indeed, Google itself has set a goal to get to a hundred percent carbon-free electricity for every hour in every location. The big goal for 2030 is that at every hour, every location, the electricity comes from carbon-free sources. This is very ambitious and never done before, of course, at the scale of Google, but this is the next goal for Google.Corey: The challenge that I have—in the abstract—with cloud providers, more or less, shaming customers—not to say that’s what you’re doing here—about their carbon usage and their carbon footprint is, okay, I appreciate that this is everyone’s problem, and yes, it is something that we should be focusing on extensively. The counterargument is that I don’t recall ever getting a meeting invite to a Google or Amazon or Microsoft or Oracle negotiation with any of your power bills or power companies or power sourcing. I have no input whatsoever as a customer on those things. And, on some level, it’s “Ooh, you’re causing a particular amount of carbon to be used by your usage of these services.” Like, well, at some level, it feels like that is more of a you thing than a me thing.And I want to be clear, I’m speaking more in the abstract to the industry rather than the specifics of Google Cloud, not to unfairly put you in the position of having to speak for everyone.Steren: No, but you’re right. If you were to do nothing, Google is constantly working hard to sign more power purchase agreements with some renewable energy sources or optimizing its data centers. Google Cloud data centers are one of the most optimized data centers in the industry with a power usage effectiveness of 1.1, which is basically saying that the energy that is used to power the facility over the energy used to actually power the server is 1.1. So, not that much loss in between.So, all of that to say, Google Cloud and Google are working very hard anyway to reduce Google Cloud’s carbon footprints and the carbon footprint of Google Cloud customers. So, if you were to do nothing, the charts that you’re seeing on Google Cloud Carbon Footprint should trend to zero. But in the meantime, you know, that’s not the case, so that’s why we show the data. And, like, many customers want to know or have the obligation to report on this data.Corey: One of the challenges that I see—and I believe this might even be related to the carbon footprint tool you have built out on top of Google Cloud—is when I am looking at… at where to place something—first, let me just say the region experience is wildly different than I’m used to with AWS. In the AWS universe, every region is basically its own island; it is very difficult to get a holistic view between regions. Google Cloud does not have that approach. There are advantages and disadvantages to both. I’m not passing any particular value judgment—for once—on this topic in this context. But where do I want to spin something up? And I have a dropdown of the regions that I can put it in. And some of these now have a green leaf next to them and others do not. I’m going to go out on a limb and assume you had a hand in this somewhere.Steren: Exactly. That’s something I worked on with the team. So, you will see on the Google Cloud Console location selectors on the Google Cloud location page, on the Google Cloud documentation, you will see a small low CO2 indicator next to some regions. And this indicator is basically saying that this region meets some criteria of high carbon-free energy percentage or low grid carbon intensity. So, you don’t need to go into the details; you just need to know that if you see this small leaf, that means that for a given workload, the emissions in that particular region will be way lower than on another region which doesn’t have the leaf.Often at Google, when we do a change we A/B test it. We A/B tested those small low CO2 indicators because, you know, that’s a console-wide change so we want to make sure that it’s worth is. And well, it turns out that for people who were in the experiment—so people will be seeing the leaf—among new Google Cloud users, they were 50% more likely to pick a low-carbon region when the leaf was displayed. And among all users, it was 19%. So, you see how just by surfacing the information, we were able to significantly influence customers’ behavior towards reducing their carbon emissions.And, you know, if you ask me, I think picking the cleanest region is probably one of the simplest action you can take—if possible, of course—to reduce your gross carbon emissions because, you know, they don’t require to change your architecture or your infrastructure; it just requires you to make the right choice in the first place. And just by letting people know that some regions are emitting much less carbon than others we basically allow them to reduce their footprint.Corey: A question I have is that as you continue to move up the stack, one of the things that Google has done extraordinarily well is the global network. And we talked previously about how I run the snark.cloud URL shortener in Google Cloud. That is homed out of us-central1 as far as regions go. But given that thing is effectively stateless, it just talks to Google Sheets for its source of truth, but then just runs a Docker invocation on every request, cool, I can see a scenario in which that becomes much more of a global service.In other words, if you can run that in pops in every region around the world on some level, there is no downside, from my perspective, on doing that. What I’m wondering then, as a result of that, is as you start seeing the evolution of services becoming more and more global, instead of highly region-specific, does that change the way that we should be thinking potentially about carbon footprint and regional selection? Or is that too much of a niche edge case to really be top of radar right now?Steren: Oh, there are many things to talk about here. The first one is that you might be hinting at something that Google is already doing, which is location shifting of workloads in order to optimize power usage, and, you know of course, carbon emissions. So, Google itself is already doing that. For example, I guess, to process YouTube videos, that can be done, not necessarily right away and that can be done in the location in which, for example, the sun is shining. So, there are some very interesting things that can be done if you allow the workloads to be run in not necessarily a specific region.Now, that being said, I think there are many other things that people consider when they pick a region. First, well, maybe they have some data locality constraints, right? This is very much the case in European countries where the data must stay in a given region, by law. Second, well, maybe they care about the price. And as you probably know, [laugh] the price of cloud providers is not the same in every region.Corey: I’ve noticed that and in fact, I was going to get into that as our next transition, but you’ve just opened Pandora’s Box early. It’s great to have the carbon-friendly indicator next to the region, but I also want number of dollar signs next to it as well. Like in AWS-land, do you have the tier one regions where everything is the lowest price: us-east-1, us-west-2, and a few others escaped me from time to time, where Managed NAT Gateways are really expensive. And then you go under some others and they get even more expensive, somehow. Like, talk about pushing the bounds of cloud economics. It’s astonishing to me.Steren: Yes. And so—Corey: Because I want that display, on some level—Steren: Exactly.Corey: —as a customer, in many cases.Steren: So, there is price, there is carbon, but of course, you know, if you are serving web requests, there is probably also latency that you care about, right? Even if—for example, Finland is very low carbon. You might not host your workloads in Finland if you want to serve US customers. So, in a sense, there are many dimensions to optimize when you pick a region. And I just sent you a link to something that I built, which is called Google Cloud Region Picker.It’s basically a tool with three sliders. First one is carbon footprint; you tell us how much you care about that. Hopefully, you put it to the right. The second one is lower price. So, how much do you want the tool to optimize to lower your bill? And third one is latency, and then you tell us where your users are coming from and if you care about latency.Because some workloads are not subject to latency requirements. Like, if you do batch jobs, well, that doesn’t serve a user request, so that can be done asynchronously at a later time or in a different place. And what this tool does is that it takes your inputs and it basically tells you which Google Cloud region is the best fit for you. And if you use it, you will see it has very small symbols like three dollars for the most expensive regions, one dollar for the least expensive ones, three leaves for the greenest regions, and zero leaves for the non-green one.Corey: This is awesome. I’m a little bit disappointed that I hadn’t seen this before. This is a thing of beauty.Steren: Yeah. Again, done by me as a 20%. [laugh]. And, you know, the goal is to educate, like, of course, it’s way more complex. Like, you know that price optimization is way more complex than a slider, but the goal of this tool is to educate and to give a first result. Like, okay, if you are in France and care about carbon, then go here. If you are in Oregon, go here. And so, many parameters that this tool help you optimize in a very simple way.Corey: One of the challenges I think I get into when I look at this across the board, is that you have a couple of very different ends on a confusing spectrum, by which I mean that one of the things I would care about from a region picker, for example, is there sufficient capacity in that region for the things I want to run. At my scale of things where right now on Google Cloud I run a persistent VM that hangs out all the time, and I run some Google Cloud Run stuff. Great. If you have capacity problems with either one of those, are you really a cloud?But then we have other folks who are spinning up tens or hundreds of thousands of a very particular instance type in a very specific region. That’s the sort of thing that requires a bit more in the way of capacity planning and the rest. So, I have to imagine for those types of use cases, this tool is insufficient. The obvious reason, of course, if you’re spinning up that much of anything, for God’s sake, reach out and talk to your account manager before trying to do it willy-nilly but yes.Steren: That’s exactly right. So, as I said, this tool is simplified tool to give, like, the vast majority of users a sense of where to put their workloads. But of course, if you’re a very big enterprise customer who is going to sign a very big deal with Google Cloud, talk to your account manager because if you do need a lot of capacity, Google Cloud might need to plan for it. And not every regions have the same capacity and we are always working with our customers to make sure we direct them in the right place and have enough capacity. A real-life example of a very high profile Google Cloud customer was that they were selecting a region without knowing its carbon impact, and when we started to disclose the carbon characteristics of Google Cloud regions—which is another link we can send to the audience—this customer realized that the region they selected—you know, maybe because it was close to their user base—was really not the most carbon friendly.So, they decided to switch to another one. And if we take an example, if you take Las Vegas, it has a carbon-free energy percentage of 20%. So, that basically means that on average, 20% of the time, the electricity comes from carbon-free sources. If you are to move to Oregon, this same workload, Oregon has a carbon-free energy percentage of 90%. So, you can see how just by staying on the West Coast, moving from Las Vegas to Oregon, you have drastically reduced your carbon emissions. And your bill, by the way because it turns out Oregon is one of the cheapest Google Cloud Data Center. So, you see how just being aware of those numbers led some very important customers who care about sustainability to make some fundamental choices when it comes to the regions they select.Corey: I guess that leads to my big obvious question, where I wind up pulling up my own footprint in Google Cloud—again, I don’t run much there—and apparently over the last year, I’ve had something on the order of two kilograms of carbon. Great. It feels like for this scale, I almost certainly burn more carbon than that just searching Google for carbon-related questions about where to place things. So, for my use case, this is entirely academic. You can almost run my workloads based upon, I don’t know, burning baby seals or something, and the ecological footprint does not materially change.Then we go to the other extreme end of the spectrum with the hundreds of thousands of instances, where this stuff absolutely makes a significant and massive difference. My question is, when should people begin thinking about the carbon footprint of their cloud workload at what point of scale?Steren: So, as you said, a good order of magnitude is one transatlantic flight is a thousand kilogram of equivalent CO2. So, you see how just by flying once, you’re already, like, completely overshadowing your Google Cloud carbon footprint. But that’s because you are not using a lot of Google Cloud resources. Overall, you know, I think your question is basically the same as when should individuals try to optimize reducing their carbon footprint? And here I always recommend there are tons of things you can optimize.Start by the most impactful ones. And impactful means an action will have a lot of impact in reducing the footprint, but also the footprint reduction will be significant by itself. And two kilograms of CO2, yes indeed, it is very low, but if you start reaching out into the thousands of kilograms of CO2 that starts to represent, like, one flight, for example. So, you should definitely care about it. And as I said, some actions might be rather easy, like picking the right region might be something you can do pretty easily for your business and then you will see your carbon emissions being divided by, you know, sometimes five.This episode is sponsored in part by our friends at Lambda Cloud. They offer GPU instances with pricing that’s not only scads better than other cloud providers, but is also accessible and transparent. Also, check this out, they get a lot more granular in terms of what’s available. AWS offers NVIDIA A100 GPUs on instances that only come in one size and cost $32/hour. Lambda offers instances that offer those GPUs as single card instances for $1.10/hour. That’s 73% less per GPU. That doesn’t require any long term commitments or predicting what your usage is gonna look like years down the road. So if you need GPUs, check out Lambda. In beta, they’re offering 10TB of free storage and, this is key, data ingress and egress are both free. Check them out at lambdalabs.com/cloud. That's l-a-m-b-d-a-l-a-b-s.com/cloud.Corey: I want to challenge your assertion, incidentally. You say that I’m not using a whole lot of Google Cloud resources. I disagree. I use roughly a dozen different Google Cloud resources tied together for some of these things, but they’re built on serverless design patterns, which means that they scale to nothing. I’m not sitting there with an idle VM—except that one—that is existing on a persistent basis.For example, I look at the things that show up on the top five list. Compute Engine is number one, Cloud Run, Cloud Logging, Cloud Storage, and App Engine are the rest that are currently being used. I think there’s a significant untold story around the idea of building in a serverless way for climate purposes.Steren: Yes. So, maybe for those who are not aware of what you are seeing on the dashboard, so when you open this Google Cloud Carbon Footprint tool on the Cloud Console, you saw a breakdown of your yearly carbon footprint and monthly carbon footprint across a few dimensions. The first one is the regions because as we said, this matters a lot; like, the regions have a lot of impact. The second one are the month; of course, you can see over time, how you’re trending. The third one is a concept called Google Cloud Project, which is, for those who are not aware, it’s a way to group Google Cloud resources into buckets.And the third one is Google Cloud services. So, what you described here is, which of your services emits the most and therefore which ones should you optimize first? Like, again, to go back to impactful actions. And to your point, yes, it is very interesting that if you use products which auto-scale, basically, the carbon attributed to you, the customer, will really follow this auto-scaling behavior. Compare that to a virtual machine that is always on, burning some CPU for almost nothing because you have a server that doesn’t process requests. That is wasting, in a sense, resources.So, what you describe here is very interesting, which is basically the most optimized products you’re going to pick, the less waste you’re going to have. Now, I also want to be careful because comparing one CPU hour of Cloud Run and one CPU hour of Compute Engine is not comparing apples to apples. Why? Because when you use Cloud Run, I’m not sure if you know, but you are using a regional product. So, a product which has built-in redundancy, which is safe in case of one zone going down in a region.But that means the Cloud Run infrastructure has to provision a little bit more machines than if it was a zonal product. While Compute Engine, your virtual machine lives in one zone and there is only one machine for you. So, you see how we should also be careful comparing products with other products because fundamentally, they are not offering the same value and they are not running on the same infrastructure. But overall, I think you are correct to say that, you know, avoiding waste, using auto-scaling products, is a good way to reduce your footprint.Corey: I do want to ask—and this is always a delicate topic because you’re talking about cultural things—how much headwind did you have internally at Google when you had the idea to start exposing this? How difficult was it to bring this to fruition?Steren: I think we are lucky that our leadership cares about reducing carbon emissions and understood that our customers needed our help to understand their cloud emissions. Like, many customers before we had this tool, we’re trying to some kind of estimate their cloud emissions. And it was—you know, Google Cloud was a black box for them. They did not have access to what you said, to some data that only Google has access to.And you know, to build that tool, we are using energy measurement of every machine in every data center. We are using, you know, customer-wide resource usage. And that is something that we use to divide the footprint among customers. So, there is some data used to compute those numbers that only Google Cloud has access to. And indeed, you’re correct; it required some executive approval which we received because many of our leaders believe that, you know, this is the right thing to do, and this is helping customers towards the same goal as Google, which is being net-zero and carbon-free.Many of our customers have made some sustainability commitments, and they need our help to meet those goals. So yeah, we did receive approval, first to share the per-region characteristics. This was already, you know, a first in the industry where a cloud provider disclosed that not every region is equal and some are emitting more carbon than others. And second, another approval which was to disclose a per customer carbon footprint, which is broken down by service, project, region, using some, you know, if you touch a little bit on the methodology, you know, it uses energy consumption, resource usage, and carbon intensity coming from a partner of ours to compute, basically, a per customer footprint.Corey: My question for you is, on some level, given that Google is already committed to being net-zero across the board for all of its usage, why do customers care? Why should they care? Effectively, haven’t you made that entirely something that is outside of their purview? You’ve solved the problem, either way.Steren: This is where we should explore it a bit more the kinds of carbon emissions that exist. For a customer, their emissions linked to the cloud usage is all considered the indirect emissions. This, in the Greenhouse Gas Protocol Standard, this is called Scope 3. So, our Google Cloud emissions are the customers’ Scope 3 emissions; they are all indirect for them. But those indirect emissions, what I mentioned as being net-zero are the emissions coming from electricity usage.So, to power those data centers, those data centers are located in certain electricity grids. Those electricity grids might be using energy sources that emit more or less carbon, right? Simply put, if in a given place, the electricity comes from coal, it will be emitting a lot of carbon compared to when electricity comes from solar, for example. So, you see how the location itself determines the carbon intensity. And these are the emissions coming from electricity usage, right?So, these are neutralized by Google purchasing as much renewable energy. But there are also types of emissions. For example, when a data center loses connection to the grid, we startup diesel generators. Those diesel generators will directly emit carbon. This is called Scope 1 emissions.And finally, there is the carbon emissions that are coming from the manufacturing of those servers, of those data centers. And this is called Scope 3 emissions. And the goal of Google is for the emissions coming from electricity to be always coming from carbon-free sources. So, this is a change that we’ve recently released to Google Cloud Carbon Footprint, which is now we also break down your emissions by scope. So, they are all Scope 3 for you, the customer, they are all indirect emissions for you, the customer, but now those indirect emissions, you can see how much is coming from diesel generators, how much is coming from electricity consumption, and how much is coming from manufacturing of the data center, and other, like, upstream, downstream activities. And yeah, overall, this is something that customers do need to report on.Corey: I think that’s very fair. I do want to thank you for taking so much time to speak with me. And instead of the usual question I’d like to ask here of where can people go to find out more because we have a bunch of links for that, instead, I want to ask something a little bit different here, which is, what are the takeaways that customers or prospective customers should really have around their carbon footprint when it comes to cloud?Steren: So, I would recommend our audience to consider carbon emissions in your cloud infrastructure decisions. And my advice is, first, move to the cloud. Like, we’ve talked that Google Cloud has very well-optimized data centers. Like, your cloud gross carbon emissions are anyway going to be much lower than any on-premise carbon emissions. And by the way, if you use Google Cloud, your net operational emissions are zero.Second action is pick the region with the lowest carbon impact. Like we discussed that this is probably a low-effort action, if possible, that will have a lot of impact on your gross carbon emissions. And you know, if you want to go further, try to schedule those workloads when the electricity is the greenest, you know, when the sun is shining, the wind is blowing, for example, or try to schedule those workloads in regions which have the lowest impact. And yeah, Google Cloud gives you all the tools to do that, the tools to optimize your region selection, and the tools to report and reduce your gross carbon emissions. We haven’t talked about it, but Google Cloud Carbon Footprint will even send you some proactive recommendations of things to do to reduce your emissions.For example, if you have a project, a machine that you forgotten, Google Cloud Carbon Footprint, will recommend you to delete it and we’ll tell you how much carbon you would save by deleting it, as well as dollar, of course.Corey: It’s funny because I feel like there’s a definite alignment between my view of cloud economics and the carbon perspective on this, which is step one, everyone wins if you turn things off when you’re not using them. What a concept. I sometimes try and take it too far of, ‘turn off all of production because your company’s terrible.’ Yeah, it turns out, that doesn’t work super well. But the idea of step one, turn it off, especially when you’re not using it. And if you’re never using it, why would you want to pay for it? That becomes a very clear win for everyone involved. I think that in the fullness of time, economics are what are going to move the needle on driving further adoption about this. I have to guess that you see the same thing from where you are?Steren: Yes, very often working to reduce your carbon footprint is also working to reduce your bill. And we’ve also observed—not always—but some correlation between regions that have the lowest carbon impact and regions that are the cheapest. So, in a sense, this region selection, optimizing for price and carbon is often optimizing for the same thing. It’s not always true, but it is often true.Corey: I really want to thank you for spending so much time to talk with me about this. This has definitely giving me a lot of food for thought, and I have to imagine that this will not be our last conversation around the topic.Steren: Well, thanks for having me. And I’m very happy to talk to you in the podcast, of course.Corey: Steren Giannini, product lead for Google Cloud Carbon Footprint and Google Cloud Run. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry screed about how climate change isn’t real as you sit there wondering why it’s 120 degrees in March.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About AlexAlex Su is a lawyer who's currently the Head of Community Development at Ironclad, the #1 contract lifecycle management technology company that's backed by Accel, Sequoia, Y Combinator, and other leading investors. Prior to joining Ironclad, Alex sold cloud software to legal departments and law firms on behalf of early stage startups. Alex maintains an active presence on social media, with over 180,000 followers across Twitter, LinkedIn, Instagram, and TikTok. Links Referenced: Ironclad: https://ironcladapp.com/ LinkedIn: https://www.linkedin.com/in/alexander-su/ Twitter: https://twitter.com/heyitsalexsu Instagram: https://www.instagram.com/heyitsalexsu/ TikTok: https://www.tiktok.com/@legaltechbro TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it’s hard to know where problems originate. Is it your application code, users, or the underlying systems? I’ve got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it’s more than just hipster monitoring.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it’s still somebody’s problem. And a big part of that responsibility is app security from code to cloud. And that’s where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you’re actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That’s S-N-Y-K.co/screamCorey: Welcome to Screaming in the Cloud. I’m Corey Quinn. I’ve been off the beaten path from the traditional people building things in cloud by the sweat of their brow and the snark on their Twitters. I’m joined today by Alex Su, who’s the Head of Community Development at Ironclad, and also relatively well-renowned on the TikToks, as the kids say. Alex, thank you for joining me.Alex: Thank you so much for having me on the show.Corey: It’s always been an interesting experience because I joined TikTok about six months or so ago, due to an escalatingly poor series of life choices that continue to fail me, and I have never felt older in my life. But your videos consistently tend to show up there. You are @legaltechbro, which sounds like wow, I hate all of those things, and yet your content is on fire.How long have you been doing the public dance thing, for lack of a better term? I don’t even know what they call it. I know how to talk about Twitter. I know how to talk about LinkedIn—sad. LinkedIn is sad—but TikTok is still something I’m trying to wrap my ancient brain around.Alex: Yeah, I felt out of place when I first made my first TikTok. And by the way, I’m known for making funny skits. I have actually never danced. I’ve always wanted to, but I don’t think I have that… that talent. I started posting TikToks in, I will call it—let’s call it the fall of 2020. So, after the pandemic.Before that, I had been posting consistently on LinkedIn for, gosh, ever since 2016, when I got into legal tech. And during the pandemic, I tried a bunch of different things including making funny skits. I’d seen something somewhere online if somebody’s making fun of the doctor life. And so, I thought, hey, I could do that for legal too. And so, I made one with iMovie. You know, I recorded it on Zoom.And then people started telling me, “Hey, you should get on this thing called TikTok.” And so, I resisted it for a while because I was like, “This is not for me.” But at some point, I said, “I’ll try this out. The editing seems pretty easy.” So, I made a couple of videos poking fun at the life of a law firm lawyer or a lawyer working for a corporate legal department.And on my fourth video, I went massively viral. Like, unexpected went viral, like, millions of—I think two million or so views. And I found myself with a following. So, I thought, “Hey, I guess this is what I’m doing now.” And so, it’s been, I don’t know, a year-and-a-half since then, and I’ve been continuously posting these skits.Corey: It’s like they say the worst thing can happen when you go into a casino and play for the first time is you win.Alex: [laugh].Corey: You get that dopamine hit, and suddenly, well now, guess what you’re doing for the rest of your life? There you go. It sounds like it worked out for you in a lot of fun ways. Your skits about big law of life definitely track. My wife used to work in that space, and we didn’t meet till she was leaving that job because who has time to date in those environments?But I distinctly remember one of our early dates, we went out to meet a bunch of her soon-to-be-former coworkers at something like eight or nine o’clock in Los Angeles on a Friday night. And at the end of it, we went back to one of our places, and they went back to work. Because that is the lifestyle, apparently, of being in big law. I don’t have the baseline prerequisites to get into law school, to let alone get the JD and then go to work in big law, and looking at that lifestyle, it’s, “Yeah, you know, I don’t think that’s for me.” Of course, I say that, and then three days later, I was doing a middle of the night wake up because the pager went off.Like, “Oh, are you a doctor?” And the pager is like, “Holy shit. This SSL certificate expires in 30 days.” It’s, yeah. Again, life has been fun, but it’s always been one of those things that was sort of, I guess, held in awe. And you’re putting a very human face on it.Alex: Yeah. You know, I never expected to be in big law either, Corey. Like, I was never good at school, but as I got older, I found a way to talk my way into, like, a good school. I hustled my way into a job at a firm that I never imagined I could get a job at. But once I got in, that’s when I was like, “Okay, I don’t feel like I fit in.”And so, I struggled but I still you know grinded it out. I stayed at the job for a couple of years. And I left because I was like, “This is not right for me.” But I never imagined that all of those experiences in big law ended up being the source material for my content, like, eight years after I’d left. So, I’m very thankful that I had that experience even if it wasn’t a good fit for me. [laugh].Corey: And on some level, it feels like, “Where do you get your material from?” It’s, “Oh, the terrible things that happened to me. Why do you ask?”Alex: That’s basically it. And people ask me, they say, you know, “You haven’t worked in that environment for eight years. It’s probably different now, right?” Well, no. You know, the legal industry is not like the tech industry. Like, things move very slowly there.The jokes that made people laugh back then, you know, 10 years ago, even 20 years ago, people still laugh at today because it’s the same way things have always worked. So, again, I’m very thankful that that’s been the case. And, you know, I feel like, the reason why my content is popular is because a lot of people can resonate with it. Things that a lot of people don’t really talk about publicly, about the lifestyle, the culture, how things work in a large firm, but I make jokes about it, so people feel comfortable laughing about it, or commenting and sharing.Corey: I want to get into that a little bit because when you start seeing someone pop up again and again and again on TikTok, you’re one of those, “Okay, I should stalk this person and figure out what the hell their story is.” And I didn’t have to look very far in your case because you’re very transparent about it. You’re the head of community development at a company called Ironclad, and that one threw me for a little bit of a loop. So, let’s start with the easy question, I suppose. What is Ironclad?Alex: We’re a digital contracting technology that helps accelerate business contracts. Companies deal with contracts of all types; a lot of times it gets bogged down in legal review. We just help with that process to make that process move faster. And I never expected I’d be in this space. You know, I always thought I was going to be a trial lawyer.But I left that world, you know, maybe six years ago to go into the legal technology space, and I quickly saw that contracts was kind of a growing challenge, contracting, whether it’s for sales or for procurement. So, I found myself as a salesperson in legal tech selling, first e-discovery software, and then contracting software. And then I found my way to Ironclad as part of the community team, really to talk about how we can help, but also speaking up about the challenges of the legal profession, of working at a law firm or at a legal department. So, I feel like it’s all been the culmination of all my experiences, both in law and technology.Corey: In the world in which I’ve worked, half of my consulting work has been helping our clients negotiate their large-scale AWS contracts and the other half is architectural nonsense of, “Hey, if you make these small changes, that cuts your bill in half. Maybe consider doing them.” But something that I’ve learned that is almost an industry-wide and universal truism, is that you want to keep the salespeople and the lawyers relatively separate just due to the absolute polar opposites of incentives. Salespeople are incentivized to sell anything that holds still long enough or they can outrun, whereas lawyers are incentivized to protect the company from risk. No, is the easy answer and everything else is risk that has to be managed. You are one of those very rare folks who has operated successfully and well by blending the two. How the hell did that happen?Alex: I’m not sure to this day how it happened. But I think part of the reason why I left law in the first place was because I don’t think I fit in. I think there’s a lot of good about having a law degree and being part of the legal profession, but I just wanted to be around people, I wanted to work with people, I didn’t want to always worry about things. And so, that led me to technology sales, which took me to the other extreme. And so, you know, I carried a sales quota for five years and that was such an interesting experience to see where—to both sell technology, but also to see where legal fit into that process.And so, I think by having the legal training, but also having been part of a sales team, that’s given me appreciation for what both teams do. And I think they’re often at tension with one another, but they’re both there to serve the greater goals of the company, whether it’s to generate revenue or protect against risk.Corey: I think that there’s also a certain affinity that you may have—I’m just spitballing wildly—one of the things that sales folks and attorneys tend to have in common is that in the public imagination, as those roles are not, shall we call it, universally beloved. There tend to be a fair number of well, jokes, in which case, both sides of that tend to be on the receiving end. I mean, at some level, all you have to do is become an IRS auditor and you’ve got the holy trifecta working for you.Alex: [laugh]. I don’t know why I gravitated to these professions, but I do think that it’s partly because both of these roles hold a significant amount of power. And if you look at just contracting in general, a salesperson at a company, they’re really the driver of the sales process. Like, if there’s no sale to be made, there’s no contract. On the flip side, the law person, the lawyer, knows everything about what’s inside of the contract.They understand the legal terms, the jargon, and so they hold an immense amount of power over advising people on what’s going to happen. And so, I think sometimes, salespeople and legal people take it too far and either spend too much time reviewing a contract and lording it over the business folks, or maybe the salesperson is too blase about getting a deal done and maybe bypasses legal and doesn’t go through the right processes. By the way, Corey, these are jokes that I make in my TikToks all the time and they always go viral because it’s so relatable to people. But yeah, that’s probably why people always make jokes about lawyers and salespeople. There’s probably some element of ridiculing people with a significant amount of power within a company to determine these transactions.Corey: Do you find that you have a better affinity for the folks doing contract work on the seller side or the buyer side? Something they don’t tell you when you run companies is, yeah, you’re going to spend a lot of time working on contracts, not just when selling things, but also when buying things and going back and forth. Aspects of what you’re talking about so far in this conversation have resonated, I guess, with both sides of that for me. What do you have the affinity for?Alex: I think on the sales side, just because of my experience, you know, I think when you go through a transaction and you’re trying to convince someone to doing something, and this is probably why I wanted to go to law school in the first place. Like I watched those movies, right? I watched A Few Good Men and I thought I’d be standing up in court convincing a jury of something. Little did I know that that sort of interest [crosstalk 00:10:55]—Corey: Like, Perry Mason breakthrough moment.Alex: That moment where—the gotcha moment, right? I found that in sales. And so, it was really a thrill to be able to, like, talk to someone, listen to them, and then kind of convince them that, based on what challenges they’re facing, for them to buy some technology. I love that. And I think that was again, tied to why I went to law school in the first place.I didn’t even know sales was a possible profession because I grew up in an immigrant community that was like, you just go to school, and that’ll lead to your career. But there’s a lot of different careers that are super interesting that don’t require formal schooling, or at least the seven years of schooling you need for law. So, I always identify with the sales side. And maybe that’s just how I am, but obviously, the folks who deal with the buy side, it’s a pretty important job, too.Corey: There’s a lot of surprise when I start talking to folks in the engineering world. First, they’re in for a rough awakening at times when they learn exactly how much qualified enterprise salespeople can make. But also because being a lawyer without, you know, the appropriate credentials to tie into that, you’re going to have a bad time. There are regulatory requirements imposed on lawyers, whereas to be a salesperson, forget the law degree, forget the bachelor’s, forget the high school diploma, all you really need to be able to do from an academic credential standpoint is show up.The rest of it is, can you actually sell? Can you have the conversations that convince people to see the outcome that benefits everyone? And I don’t know what that it’s possible, or advised necessarily, to be able to find a way to teach that in some formalized way. It almost feels like folks either have that spark or they don’t. Do you think it’s one of those things that can be taught? Do you think it’s something that people have to have a pre-existing affinity for?Alex: It’s both, right, because part of it is some people will just—they don’t have the personality to really sell. It’s also like their interest; they don’t want to do that. But what I found that’s interesting is that what I thought would make a good salesperson didn’t end up being true when I looked at the most effective sellers. Like, in my head, I thought, “Oh, this is somebody who’s very boisterous, very extroverted,” but I found that in my experience in B2B SaaS that the most effective sellers are very, very much active listeners. They’re not the people showing up and talking at you. They are asking you about your day-to-day asking about processes, understanding the context of your situation, before making a small suggestion about what you might want to do.I was very impressed the first time I saw one of these enterprise sellers who was just so good at that. Like, I saw him, and he looked nothing like what I imagined an effective sales guy to look like. And he was really kind and he just, kind of, just talked to me, like, I was a human being, and listened to my answers. So, I do think that there is some element of nature, your talent when it comes to that, but it can also be trained because I think a lot of folks who have sales talent, they don’t realize that they could be good at it. They think that they’ve got to be this extroverted, happy hour, partying, storyteller, where —Corey: The Type A personality that interrupts people as they’re having the conversation.Alex: Yeah, yeah.Corey: Yeah.Alex: So anyways, I think that’s why it’s a mix of both.Corey: The conversations that I’ve learned the most from when I’m talking to prospects and clients have been when I asked the quote-unquote, dumb question that I already know the answer to, and then I shut up and I listen. And wow, I did not expect that answer. And when you dig a little further, you realize there’s nuance that—at least in my case—that I’ve completely missed to the entire problem space. I think that is really one of the key differentiators to my mind, that separate people who are good at this role from folks who just misunderstand what the role is based upon mass media, or in other cases—same problem with lawyers—the worst examples, in some cases, of the profession. The pushy used car salesperson or the lawyer they see advertising on the back of a bus for personal injury cases. The world is far more nuanced than that.Alex: Absolutely. And I think you hit the nail on the head when you said, you know, you ask those questions and let them talk. Because that’s an entire process within the sales process. It’s called discovery, and you’re really asking questions to understand the person’s situation. More broadly, though, I think pitching at people doesn’t seem to work as well as understanding the situation.And you know, I’ve kind of done that with my content, my TikToks because, you know, if you look at LinkedIn, a lot of people in our space, they’re always prescribing solutions, giving advice, posting content about teaching people things. I don’t do that. As a marketer, what I do is I talk about the problems and create discussions. So, I’ll create a funny video—Corey: I think you’re teaching a whole generation that maybe law school isn’t what they want to be doing, after all there is that.Alex: There is that. There is that. It’s a mix of things. But one of the things I think I focus on is talking about the challenges of working with a sales team if you’re an in-house lawyer. And I don’t prescribe technology, I don’t prescribe Ironclad, I don’t say this is what you need to do, but by having people talk about it, they realize, right—and I think this is why the videos are popular—as opposed to me coming out and saying, “I think you need technology because of XYZ.” I think, like, facilitating the conversation of the problem space, that leads people to naturally say, “Hey, I might need something. What do you guys do, by the way?”Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don’t leave managing your database to your cloud vendor because they’re too busy launching another half-dozen managed databases to focus on any one of them that they didn’t build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: It sounds ridiculous for me to say that, “Oh, here’s my entire business strategy: step one, I shitpost on the internet about cloud computing; step two, magic happens here; and step three people reach out to talk about their AWS bills.” But it’s also true. Is that the pattern that you go through: step one, shitpost on TikTok; step two, magic happens here; and step three people reach out asking to learn more about what your company does? Or is there more nuance to do it?Alex: I’m still figuring out this whole thing myself, but I will say shitposting is incredibly effective. Because I’m active on Twitter. Twitter is where I start my shitposts. TikTok, I also shitpost, but in video format, I think the number one thing to do is figure out what resonates with people, whether it’s the whole contracting thing or if it’s frustrations about law school. Once you create something that’s compelling, the conversation gets going and you start learning about what people are thinking.And I think that what I’m trying to figure out is how that can lead to a deeper conversation that can lead to a business transaction or lead to a sale. I haven’t figured it out, right, but I didn’t know that when I started creating content that spoke to people when I was a quota-carrying salesperson, people reached out to me for demo requests, for sales conversations. There is something that is happening in this quote-unquote, “Dark funnel,” that I’m sure you’re very familiar with. There’s something that’s happening that I’m trying to understand, and I’m starting to see.Corey: This is probably a good thing to the zero in on a bit because to most people’s understanding of the sales process, it would seem that you going out and making something of a sensation out of yourself on the internet, well what are you doing that for? That’s not sales work? How is that sales? That’s just basically getting distracted and going to do something fun. Shouldn’t you be picking up the phone and cold calling people or mass-emailing folks who don’t want to hear from you because you trick them into having a badge scanned somewhere? I don’t necessarily think that is accurate. How do you see the interplay of what you do and sales?Alex: When you’re selling something like makeup or clothing, it’s a pretty transactional process. You create a video; people will buy, right? That’s B2C. In B2B, it’s a much more complex processes. There’s so many touchpoints. The start of a sales conversation and when they actually buy may take six months, 12 months, years. And so, there’s got to be a lot of touch points in between.I remember when I was starting out in my content journey, I had this veteran enterprise sales leader, like, your classic, like, CRO. He said to me, “Hey, Alex, your content’s very funny, but shouldn’t you be making cold calls and emails? Like, why are you spending your time doing this?” And I said, “Hey, listen, do you notice that I’m actually sourcing more outbound sales calls than any other sales rep? Like, have you noticed that?”And he’s like, “Actually, yeah, I did notice that. You know, how are you doing it?” And I was like, “Do you not see that these two are tied? These are not people I just started calling. They are people who have seen my content over time. And this is how it works.”And so, I think that the B2B world is starting to wise up to this. I think, for example, Ironclad is leading the way on creating a community team to create those conversations, but plenty of B2B companies are doing the same thing. And so, I think by inserting themselves in a conversation—a two-way conversation—during that process, that’s become incredibly effective, far more so than, like, cold-calling a lawyer or a developer who doesn’t want to be bothered by some pushy salesperson.Corey: Busy, expensive professionals generally don’t want to spend all their time doing that. The cold outreach emails that drive me nuts are, “Hey, can we talk for half an hour?” Yeah, I don’t tend to think in terms of billable hours because that’s not how I do anything that I do, but there is an internal rate that I used to benchmark and it’s what you want me just reach into my pocket and give you how much money for a random opportunity to pitch me on something that you haven’t even qualified whether I need or not? It’s like, asking people for time is worse, in some ways, than asking for money because they can always make more money, but no one can make more time.Alex: Right, right. That’s absolutely right.Corey: It’s the lack of awareness of understanding the needs and motivations of your target market. One thing that I found that really aided me back when I was working for other folks was trying to find a company or a management structure that understood and appreciated this. Easy example, when I was setting out as an independent consultant after a few months I’d been doing this and people started to hear about me. But you know, it turns out that there are challenges to running a business that are not recommended for most people. And I debated, do I take a job somewhere else?So, I interviewed at a few places, and I was talking to one company that’s active in the cloud costing space at the time and they wanted me to come aboard. But discussions broke down because they thought I was, quote, “More interested in thought leadership than I was and actually fixing the bills themselves.” And looking at this now, four years later or so, yeah, they were right. And amazing how that whole thing played out, but that the lack of vision around, there’s an opportunity here, if we can chase it, at least in the places I was at, was relatively hard to come by. Did you luck out in finding a role that works for you in this way or did you basically have to forge it for yourself from the sweat of your brow and the strength of your TikTok account?Alex: It was uphill at first, but eventually, I got lucky. And you know, part of it was engineered luck. And I’ll explain what I mean. When I first started out doing this, I didn’t expect this to lead to any jobs. I just thought it would support my sales career.Over time, as the content got more popular, I never wanted to do anything else because I was like, I don’t want to be a marketer. I’m not a—I don’t know anything about demand gen. All I know is how to make funny videos that get people talking. The interesting that happened was that these videos created this awareness, this energy in our space, in the legal space. And it wasn’t long before Ironclad found me.And you know, Ironclad has always been big on community, has always done things like—like, our CEO, our founder, he said that he used to host these dinners, never talking about Ironclad, but just kind of talking about law school and law with potential clients. And it would lead to business. Like, it’s almost the same concept of, like, not pushing sales on people. And so, Ironclad has always had that in its DNA. And one of our investors, our board members, Jessica Lee from Sequoia, she is a huge believer in community.I mean, she was the CEO of another company that leveraged community, and so there’s this community element all throughout the DNA of Ironclad. Now, had I not put myself out there with this content, I may not have been discovered by Ironclad. But they saw me, they found me, and they said, “We don’t think about these things like many other companies. We really want to invest in this function.” And so, it’s almost like when you put yourself out there, yes, sometimes some people will say, “What are you doing? Like, this makes no sense. Like, stop doing that.” But there’s going to be some true believers who come out and seek you out and find you.And that’s been my experience here, like, at Ironclad. Like, people were like, “When you go there, are they going to censor you? Is your content going to be less edgy?” No. Like, they pulled me aside multiple times and said, “Keep being yourself. This is what we want.” And I think that is so special and unique. And part of it is very much lucky, but it’s also when you put yourself out there kind of in a big way, like-minded people will seek you out as well.Corey: I take the position that part of marketing, part of the core of marketing, is you’ve got to have an opinion. But as soon as you have an opinion, people are going to disagree with you. They’re going to, effectively, forget the human on the other side of it and start taking you for a drag on social media and whatnot. So, the default reaction a lot of people have is oh, I shouldn’t venture opinions forward.No. People are always going to dislike you for something and you may as well have it be for who you are and what you want to be doing rather than who you’re pretending to be. That’s always been my approach. For me, the failure mode was not someone on Twitter is going to get mad about what I wrote. No one’s going to read it. That’s the failure mode. And the way to avoid that is make it interesting.Alex: That is a hundred percent relatable to me because I think when I was younger, I was scared. I did worry that I would get in trouble for what I posted. But I realized these people I was worried about, they weren’t going to help me anyways. These are not people who are going to seek me out and help me but then say, “Oh, I saw your content, so now I can’t help you.” They were not going to help me anyways.But by being authentic to myself and putting things out there, I attracted my own tribe of people who have helped me, right? A lot of my early results from content came not because I reached my target customers; it was because somebody resonated with what I put out there and they carried my message and said, “Hey, you should talk to Alex.” Something special happens when you kind of put yourself out there and say an opinion or share a perspective that not everyone agrees with because that tribe you build ends up helping you a lot. And meanwhile, these other people that might not like it, they probably weren’t going to help you either.Corey: I maintain that one of the most valuable commodities in the universe is attention. And so, often there’s so much information overload that’s competing for our attention every minute of every day that trying to blend in with the rest of it feels like the exact wrong approach. I’m not a large company here. I don’t have a full marketing department to wind up doing ad buys, and complicated campaigns, and train a team of attacking interns to wind up tackling people to scan their badges at conferences. I’ve got to work with what I’ve got.So, the goal I’ve always had is trigger the Rolodex moment where someone hears about a problem in the AWS billing space—ideally—and, “Oh, my God, you need to talk to Corey about that.” And it worked, for better or worse. And a lot of it was getting lucky, let’s be very clear here, and people doing me favors that they had no reason to do and I’ll never be able to repay. But being able to be in that space really is what made the difference. Now, the downside, of course, when you start doing that is, how do you go back to what happened before?If you decide okay, well, it’s been a fun run for you and Ironclad. And yeah, TikTok. Turns out that is, in fact, for kids; time to go somewhere else. Like, I don’t know that you would fit into your old type of job.Alex: Yeah. No, I wouldn’t. But very early on, I realized, I said, “If I’m going to find meaningful work, it’s okay to be wrong.” And when I went to big law, I realized this is not right for me. That’s okay. I’m just not going to get another big law job.And so, when people ask me, “Hey, now that you’ve put yourself out there, you probably can’t get a job at a big firm anymore.” And that’s okay to me because I wasn’t going to go back anyways. But what I have found, Corey, is that there’s this other universe of people, whether it’s a entrepreneur, smaller businesses, technology companies, they would be interested in working with me. And so, by being myself, I may have blocked out a certain level of opportunities or a safety net, but now I’m kind of in this other world where I feel very confident that I won’t have trouble finding a job. So, I feel very lucky to have that, but that’s why I also don’t worry about the possibility of not going back.Corey: Yeah, I’ve never had to think about the idea of, well, what if I go have to get a job again? Because at that point, it means well, it’s time to let every one at the company who is depending on the go, and that’s the bigger obstacle because, let’s be honest, I’m a white guy in tech, and I look like it. My failure mode is basically a board seat and a book deal because of inherent bias in the system.Alex: [laugh]. Oh, my god.Corey: That’s the outcome that, for me personally, I will be just fine. It’s the other people took a chance on me. I’m terrified of letting them down. So far, knock on wood, I haven’t said anything too offensive in public is going to wind up there. That’s also not generally my style.But it is the… it is something that has weighed on me that has kept me from I guess, thinking about what would my next job be? I’m convinced this is the last job I’ll ever have, if for no other reason that I’ve made myself utterly unemployable.Alex: [laugh]. Well, I think many of us aspire to find that perfect intersection of what you love doing and what pays the bills. Sounds like you’ve found it, I really do feel like I found it, too. I never imagined I’d be doing what I do now. Which is also sometimes hard to describe.I’m not making TikToks for a living; I’m just on the community team, doing events—I’m getting to work with people. I’m basically doing the things that I wanted to do that led me to quit that job many years ago, that big law job many years ago. So, I feel very blessed and for anybody who’s, like, looking for that type of path, I do think that at some point, you do need to kind of shed the safety nets because if you always hang on to the safety nets, whether it’s a big tech job or a big law job, there’s going to be elements of that that don’t fit in with your personality, and you’re never going to be able to find that if you kind of stay there. But if you venture out—and, you know, I admire you for what you’ve done; it sounds like you’re very successful at what you do and get to do what you love every day—I think great things can happen.Corey: Yeah, I get to insult Amazon for a living. It’s what I love. It’s what I would do if I weren’t being paid. So, here we are. Yeah—Alex: [laugh].Corey: I have no sense of self-preservation. It’s kind of awesome.Alex: I love it.Corey: But you’re right. It’s… there’s something to be said for finding the thing that winds up resonating with you and what you want to be doing.Alex: It really does. And you know, I think when I first made the move to technology, to sales, there was no career path. I thought I would—maybe I thought I might be a VP of Sales. But the thing is, when you put yourself out there, the opportunities that show up might not be the ones that you had always seen from the beginning. Like if you ask a lawyer, like, “What can I do if I don’t practice law?” They’re going to give you these generic answers. “Work here. Work there. Work for that company. I’ve seen a lot of people do this.”But once you put yourself out there in the wilderness, these opportunities arise. And I’ve been very lucky. I mean, I never imagined I’d be a TikTokker. And by the way, I also make memes on Twitter. Couldn’t imagine I’d be doing that either. I learned, like, Mematic, these tools. Like, you know, like, I’m immersed in this internet culture now.Corey: It is bizarre to me and I never saw it coming either. For better or worse, though, here we are, stuck at it.Alex: [laugh].Corey: I really want to thank you for taking so much time to speak with me today. If people want to learn more about what you’re up to and follow along for the laughs, if nothing else, where’s the best place for them to find you?Alex: The best way to find me is on LinkedIn; just look up Alex Su. But I’m around and on lots of social media platforms. You can find me on Twitter, on Instagram, and on TikTok, although I might be a little bit embarrassed of what I put on TikTok. I put some crazy gnarly stuff out there. But yeah, LinkedIn is probably the best place to find me.Corey: And we will put links to all of it in the show notes, and let people wind up making their own decisions. Thanks so much for your time, Alex. I really appreciate it.Alex: Corey, thank you so much for having me. This was so much fun.Corey: Alex Su, Head of Community Development at Ironclad. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry insipid comment talking about how unprofessional everything we talked about is that you will not be able to post for the next six months because it’ll be hung up in legal review.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About JonA husband, father of 3 wonderful kids who turned Podcaster during the pandemic. If you told me in early 2020 I would be making content or doing a podcast, I probably would have said "Nah, I couldn't see myself making YouTube videos". In fact, I told my kids, no way am I going to make videos for YouTube. Well, a year later I'm over 100 uploads and my subscriber count is growing.Links Referenced: LinkedIn: https://www.linkedin.com/in/jon-myer/ Twitter: https://twitter.com/_JonMyer jonmyer.com: https://jonmyer.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it’s hard to know where problems originate. Is it your application code, users, or the underlying systems? I’ve got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it’s more than just hipster monitoring.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: Welcome to Screaming in the Cloud, I’m Corey Quinn. Every once in a while I get to talk to a guest who has the same problem that I do. Now, not that they’re a loud, obnoxious jerk, but rather that describing what they do succinctly is something of a challenge. It’s not really an elevator pitch anymore if you have to sabotage the elevator before you start giving it. I’m joined by Jon Myer. Jon, thank you for joining me. What the hell do you do?Jon: Corey, thanks for that awesome introduction. What do I do? I get to talk into a microphone. And sometimes I get to stare at myself on camera, whether it makes a recording or not. And either I talk to myself or I talk to awesome people like you. And I get to interview and tell other people’s stories on my show; I pull out the interesting parts and we have a lot of freaking fun doing it.Corey: I suddenly feel like I’ve tumbled down the rabbit hole and I’m in the wrong side of the conversation. Are we both trying to stand in the same part of the universe? My goodness.Jon: Is this your podcast or mine? Maybe I should do an introduction right now to introduce you onto it and we’ll see how this works.Corey: The dueling podcast banjo. I liked the approach quite a bit. So, you have done a lot of very interesting things. For example, once upon a time, you worked at AWS. But you have to go digging to figure that out because everything I’m seeing about you in your professional bio and the rest is forward-looking, as opposed to Former Company A, Former Company B, and this one time I was an early investor in Company C, which means, that’s right, one of the most interesting things about me is that I wrote a check once upon a time, which is never something I ever want to say about myself, ever. You’re very forward-looking, and I strive to do the same. How do you wind up coming at it from that position?Jon: When I first left AWS—it’s been a year ago, so I served my time—and I actually used to have ex-Amazonian on it and listed on it. But as I continuously look at it, I used to have a podcast called The AWS Blogger. And it was all about AWS and everything, and there’s nothing wrong with them. And what I would hear—Corey: Oh, there’s plenty wrong with them, but please continue.Jon: [laugh]. We won’t go there. But anyway, you know, kind of talking about it and thinking about it ex-Amazonian, yeah, that’s great, you put it on your resume, put it on your stuff, and it, you know, allows you that foot in the door. But I want to look at and separate myself from AWS, in that I am my own independent voice. Yes, I worked for them; great company, I’ve learned so much from them, worked with some awesome people there, but my voice in the community has become very engaging and trustworthy. I don’t want to say I’m no longer an Amazonian; I still have some of the guidelines, some of the stuff that’s instilled in me, but I’m independent. And I want that to speak for itself when I come into a room.Corey: It’s easy as hell, by the way, for me to sit here and cast stones at folks who, “Oh, you’re going to talk about this big company you worked for, even though you don’t work there anymore.” Yeah, I really haven’t worked anywhere that most people would recognize unless they’re, you know, professionally sad all the time. So, I don’t have that luxury; I had to wind up telling a story that was forward-looking just because I didn’t really have much of a better option. You have that option and decided to go in a direction where it presents, honestly as your viewpoint is that your best days are yet to come. And I want to be clear that for folks who are constantly challenged in our space to justify their existence there, usually because they don’t look like our wildly over-represented selves, Jon, they need that credibility.And when they say that it’s necessary for them, I am not besmirching that. I’m speaking from my own incredibly privileged position that you share. That is where I’m coming from on this, so I don’t want people to hear this as shaming folks who are not themselves wildly over-represented. I’m not talking about you fine folks, I assure you.Jon: You can have ex-Amazonian on your resume and be very proud of it. You can remove it and still be very proud of the company. There’s nothing wrong with either approach. There are some conversations that I’ll be in, and I’ll be on with AWS folks and I’ll say, “I completely understand where you’re coming from. I’m an ex-Amazonian.” And they’re like, “Oh, you get us. You get the process. You get the everything.”I just want to look forward that I will be that voice in the community and that I have an understanding of what AWS is and will continuously be. And I have so much that I’m working towards that I’m very proud of where I’ve come from, but I do want to look forward.Corey: One of these days, I really feel like I should hang out with some Amazonians or ex-Amazonians who don’t know who I am—which is easier to find than you think—and pretend that I used to work there and wonder how long I can keep the ruse going. Just because I’ve been told a few times that I am suspiciously Amazonian for someone who’s never worked there.Jon: You have a lot of insights on the AWS processes and understanding. I think you could probably keep it going for quite a while. You will have to get that orange lanyard though, when you go to, like—Corey: I got one once when I was at a New York Summit a couple years ago. My affiliation then, before I started The Duckbill Group, was Last Week in AWS, and apparently, someone saw that and thought that I was the director of Take-this-Job-and-Shove-it, but I’ll serve out my notice until Friday. So, cool; employee lanyard, it was. And I thought this is going to be awesome because I’ll be able to walk around and I’ll get the inside track if people think I work there. And they treated me like crap until I put the customer lanyard back on. It’s, “Oh, it’s better to be a customer at an AWS event than it is to be an employee.” I learned that when the fun way.Jon: There is one day that I hope to get the press or analyst lanyard. I think it would be an accomplishment for me. But you get to experience that firsthand, and I hate to switch the tables because I know it’s your podcast recording, not mine, but—Corey: Having the press analyst lanyard is interesting because a lot of people are not allowed to speak to you unless they’ve gone through training. Which, okay, great. I will say that it is a lot nicer walking the expo floor because most of the people working the booths know that means that person is press, generally—they’re not quite as familiar with analysts—but they know that regardless that they’re not going to sell you a damn thing, so they basically give you a little bit of breathing room, which is awesome, especially in these pandemic times. But the challenge I have with it is that very often I want to talk to folks who are AWS employees who may not have gone through press training. And I’ve never gotten anyone in trouble or taken advantage of things that I hear in those conversations and write about them.Everything I write about is what I’ve experienced in public or as a customer, not based upon privileged inside information. I have so many NDAs at this point, I can’t keep track, so I just make sure everything I talked about publicly cited I have that already.Jon: Corey, I got to flip the script real quick. I got to give you a shout-out because everybody sees you on Twitter and sees, like, “Oh, my God, he’s saying this negative, that negative towards AWS.” You and I had, I don’t know, it was a 30, 45 minute at the San Francisco Summit, and I think every Summit, we try to connect for a little bit. But that was really the premise I kicked off a lot of our conversations when you joined my podcast. No, this is not my podcast, this is Corey’s, but anyway—Corey: And just you remember that. Please continue.Jon: [laugh]. But you know, kind of going off it you have so much insight, so much value, and you kind of really understand the entire processes and all the behind the scenes and everything that’s going on that I was like, “Corey, I got to get your voice out there and show the other side of you, that you’re not there trying to get people in trouble, you never poke fun of an AWS employee. I heard there was some guy named Larry that you do, but we won’t jump into that.”Corey: One of the things that I think happened is, first and foremost, there is an algorithmic bias towards outrage. When I say nice things about AWS or other providers, which I do periodically, they get basically no engagement. When I say something ridiculous, inflammatory, and insulting about a company, oh, goes around the internet three times. One of the things that I’m slowly waking up to is that when I went into my Covid hibernation, my audience was a quarter of the size it is now. People don’t have the context of knowing what I’ve been up to for the last five or six years. All they see is a handful of tweets.And yeah, of course, you wind up taking some of my more aggravated moment tweets and put a few of those on a board, and yeah, I start to look a fair bit like a jerk if you’re not aware of what’s going on inside-track-wise. That’s not anyone else’s fault, except my own, and I guess understanding and managing that perception does become something of a challenge. I mean, it’s weird; Amazon is a company that famously prides itself on being misunderstood for long periods of time. I guess I never thought that would apply to me.Jon: Well, it does. Maybe that’s why most people think you’re an Amazonian.Corey: You know, honestly, I’ve got to say, there are a lot of worse things people can and do call me. Amazon has a lot to recommend it in different ways. What I find interesting now is that you’ve gone from large companies to sort of large companies. You were at Spot for a hot minute, then you were doing the nOps thing. But one thing that you’ve been focusing on a fair bit has been getting your own voice and brand out there—and we talked about this a bit at the Summit when we encountered each other which is part of what sparked this conversation—you’re approaching what you’re doing next in a way that I don’t ever do myself. I will not do it justice, but what are you working on?Jon: All right. So Corey, when we talked at the New York Summit, things are actually moving pretty good. And some of the things that I am doing, and I’ve actually had a couple of really nice engagements kind of kick off is, that I’m creating highly engageable, trustworthy content for the community. Now, folks, you’re asking, like, what is that? What is that really about? You do podcasts?Well, just think about some of the videos that you’re seeing on customer sites right now. How are they doing? How’s the views? How’s the engagement? Can you actually track those back to, like, even a sales engagement in utilizing those videos?Well, as Jon Myer—and yes, this is highly scalable because guess what I am in talks with other folks to join the crew and to create these from a brand awareness portion, right? So, think about it. You have customers that you want to get engaged with: you have products, you have demos, you have reviews that you want to do, but you can’t get them turned around in a quick amount of time. We take the time to actually dive into your product and pull out the value prop of the exact product, a demo, maybe a review, all right? We do sponsors as well; I have a number of them that I can talk about, so Veeam on AWS, Diabolical Coffee, there’s a couple of other I cannot release just yet, but don’t worry, they will be hitting out there on social pretty soon.But we take that and we make it an engaging kind of two to three-minute videos. And we say, “Listen, here’s the value of it. We’re going to turn this around, we’re going to make this pop.” And putting this stuff, right, so we’ll take the podcast and I’ll put it on to my YouTube channel, you will get all my syndication, you’ll get all my viewers, you’ll get all my views, you’ll get my outreach. Now, the kicker with that is I don’t just pick any brand; I pick a trusted brand to work with because obviously, I don’t want to tarnish mine or your brand. And we create these podcasts and we create these videos and we turn them around in days, not weeks, not months. And we focus on those who really need to actually present the value of their product in the environment.Corey: It sounds like you’re sort of the complement to the way that I tend to approach these things. I’ll periodically do analyst engagements where I’ll kick the tires on a product in the space—that’s usually tied to a sponsorship scenario, but not always—where, “Oh, great. You want me to explain your product to people. Great, could I actually kick the tires on it so I understand at first? Otherwise, I’m just parroting what may as well be nonsense. Maybe it’s true, maybe it’s not.”Very often small companies, especially early stage, do a relatively poor job of explaining the value of their product because everyone who works there knows the product intimately and they’re too close to the problem. If you’re going to explain what this does in a context where you have to work there and with that level of intensity on the problem space, you’re really only pitching to the already converted as opposed to folks who have the expensive problem that gets in the way of them doing their actual job. And having those endless style engagements is great; they periodically then ask me, “Hey, do you want to build a bunch of custom content for us?” And the answer is, “No, because I’m bad at deadlines in that context.”And finding intelligent and fun and creative ways to tell stories takes up a tremendous amount of time and is something that I find just gets repetitive in a bunch of ways. So, I like doing the typical sponsorships that most people who listen to this are used to: “This episode is sponsored by our friends at Chex Mix.” And that’s fine because I know how to handle that and I have that down to a set of study workflows. Every time I’ve done custom content, I find it’s way more work than I anticipated, and honestly, I get myself in trouble with it.Jon: Well, when you come across it, you send them our way because guess what, we are actually taking those and we’re diving deep with them. And yes, I used an Amazon term. But if you take their product—yeah [laugh]. I love the reaction I got from you. But we dive into the product. And you said it exactly: those people who are there at the facility, they understand it, they can say, “Yeah, it does this.”Well, that’s not going to have somebody engaged. That’s not going to get somebody excited. Let me give you an example. Yesterday, I had a call with an awesome company that I want to use their product. And I was like, “Listen, I want to know about your product a little bit more.”We demoed it for my current company, and I was like, “But how do you work for people like me: podcasters who do a lot of the work themselves? Or a social media expert?” You know, how do I get my content out there? How does that work? What’s your pricing?And they’re, like, “You know, we thought about getting it and see if there was a need in that space, and you’re validating that there’s a need.” I actually turned it around and I pitched them. I was like, “Listen, I’d love for you guys to be a sponsor on my show. I’d love for you to—let me do this. Let me do some demos. Let’s get together.”And I pitched them this idea that I can be a spokesperson for their product because I actually believed in it that much just from two calls, 30 minutes. And I said, “This is going to be great for people like me out there and getting the voice, getting the volume out there, how to use it.” I said, “I can show some quick integration setups. You don’t have to have the full-blown product that you sell out the businesses, us as individuals or small groupings, we’re only going to use certain features because, one, is going to be overwhelming, and two, it’s going to be costly. So, give us these features in a nice package and let's do this.” And they’re like, “Let’s set something up. I think we got to do this.”Corey: How do you avoid the problem where if you do a few pieces of content around a particular brand, you start to become indelibly linked to that brand? And I found that in my early days when I was doing a lot of advisory work and almost DevRel-for-hire as part of the sponsorship story thing that I was doing, and I found that that did not really benefit the larger thing I was trying to build, which is part of the reason that I got out of it. Because it makes sense for the first one; yeah, it’s a slam dunk. And the second one, sure, but sooner or later, it feels like wow, I have five different sponsors in various ways that want me to be building stories and talking about their stuff as I travel the world. And now I feel like I’m not able to do any of them a decent service, while also confusing the living hell out of the audience of, “Who is it you work for again anyway?” It was the brand confusion, for lack of a better term.Jon: Okay, so you have two questions there. One of them is, how do you do this without being associated with the brand? I don’t actually see a problem with that. Think of a race car; NASCAR drivers are walking around with all their stuff on their jackets, you know, sponsored by this person, this group, that group. Yeah, it’s kind of overwhelming at times, but what’s wrong with being tied to a couple of brands as long as the brands are trustworthy, like yourself? Or you believing those, right? So, there’s nothing wrong with that.Second is the scalability that you’re talking about where you’re traveling all over the world and doing this and that. And that’s where I’m looking for other leaders and trustworthy community members that are doing this type of thing to join a highly visible team, right? So, now you have a multitude and a diverse group of individuals who can get the same message out that’s ultimately tied to—and I’m actually going to call it out here, I have it already as Myer Media, right? So, it’s going to be under the Jon Myer Podcast; everything’s going to be grouped in together under Myer Media, and then we’re going to have a group of highly engaging individuals that enjoy doing this for a living, but also trust what they’re talking about.Corey: If you can find a realistic way to scale that, that sounds like it’s going to have some potential significant downstream consequences just as far as building almost a, I guess, a DevRel workshop, for lack of a better term. And I mean, that in the sense of an Andy Warhol workshop style approach, not just a training course. But you wind up with people in your orbit who become associated, affiliated with a variety of different brands. I mean, last time I did the numbers, I had something like 110 sponsors over the last five years. If I become deeply linked to those brands, no one knows what the hell I do because every company in the space, more or less, has at some level done a sponsorship with me at some point.Jon: I guess I’ll cross that when it happens, or keep that in the top-of-mind as it moves forward. I mean, it’s a good point of view, but I think if we keep our individualism, that’s what’s going to separate us as associated. So, think of advertising, you have a, you know, actor, actress that actually gets on there, and they’re associated with a certain brand. Did they do it forever? I am looking at long-term relationships because that will help me understand the product in-depth and I’ll be able to jump in there and provide them value in a expedited version.So, think about it. Like, they are launching a new version of their product or they’re talking about something different. And they’re, like, “Jon, we need to get this out ASAP.” I’ve had this long-term relationship with them that I’m able to actually turn it around rather quickly, but create highly engaging out of it. I guess, to really kind of signify that the question that you’re asking is, I’m not worried about it yet.Corey: What stage or scale of company do you find is, I guess, the sweet spot for what you’re trying to build out?Jon: I like the small to medium. And looking at it, the small to medium—Corey: Define your terms because to my mind, I’m still stuck in this ancient paradigm that I was in as an employee, where a big company is anything that has more than 200 people, which is basically everyone these days.Jon: So, think about startups. Startups, they are usually relatively 100 or less; medium, 200 or less. The reason I like that type of—is because we’re able to move fast. As you get bigger, you’re stuck in processes and you have to go through so many steps. If you want speed and you want scalability, you got to pay attention to some of the stuff that you’re doing and the processes that are slowing it down.Granted, I will evaluate, you know, the enterprise companies, but the individuals who know the value of doing this will ultimately seek me and say, “Hey, listen, we need this because we’re just kicking this off and we need highly visible content, and we want to engage with our current community, and we don’t know how.”Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don’t leave managing your database to your cloud vendor because they’re too busy launching another half-dozen managed databases to focus on any one of them that they didn’t build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: I think that there’s a fair bit of challenge somewhere in there. I’m not quite sure how to find it, that you’re going to, I think, find folks that are both too small and too big, that are going to think that they’re ready for this. I feel like this doesn’t, for example, have a whole lot of value until a company has found product-market fit unless what you’re proposing to do helps get them to that point. Conversely, at some point, you have some of the behemoth companies out there, it’s, “Yeah, we can’t hire DevRel people fast enough. We’ve hired 500 of them. Cool, can you come do some independent work for us?” At which point, it’s… great, good luck standing out from the crowd in any meaningful way at that point.Jon: Well, even a high enterprise as hired X number of DevRels, the way you stand out is your personality and everything that you built behind your personal brand, and your value brand, and what you’re trying to do, and the voice that you’re trying to achieve out there. So, think about it—and this is very difficult for me to, kind of, boost and say, “Hey, listen, if I were to go to a DevRel of, like, say, 50 people, I will stand out. I might be one of the top five, or I might be two at the top five.” It doesn’t matter. But for me why and what I do, the value that I am actually driving across is what will stand out, the engaging conversations.Every interview, every podcast that I do, at the end, everybody’s like, “Oh, my God, you’re, like, really good at it; you kind of keep us engaging, you know when to ask a question; you jump in there and you dive even deeper.” I literally have five bullet points on any conversation, and these are just, like, two or three sentences, maybe. And they’re not exact questions. They’re just topics that we need to talk about, just like we did going into this conversation. There is nothing that scripted. Everything that’s coming across the questions that you’re pulling out from me giving an answer to one of your questions and then you’re diving deep on it.Corey: I think that that’s probably a fair approach. And it’s certainly going to lead to a better narrative than the organic storytelling that tends to arise internally. I mean, there’s no better view to see a lot of these things than working on bills. One of my favorite aspects of what I do is I get to see the lies that clients tell to themselves, where it’s—like, they believe these things, but it no longer matches the reality. Like developer environments being far too expensive as a proportion of the rest of their environment. It’s miniscule just because production has scaled since you last really thought about it.Or the idea that a certain service is incredibly expensive. Well, sure. The way that it was originally configured and priced, it was and that has changed. Once people learn something, they tend to stop keeping current on that thing because now they know it. And that’s a bit of a tricky thing.Jon: That’s why we keep doing podcasts, you keep doing interviews, you keep talking with folks is because if you look at when you and I actually started doing these podcasts—and aka, like, webinars, and I hate to say webinars because it’s always negative and—you know because they’re not as highly engaging, but taking that story and that narrative and creating a conversation out of it and clicking record. There are so many times that when I go to a summit or an event, I will tell people, they’re like, “So, what am I supposed to do for your podcast?” And we were talking for, like, ten minutes, I said, “You know, I would have clicked record and we would have ten minutes of conversation.” And they’re like, “What?” I was like, “That’s exactly what it is.”My podcast is all about the person that I’m interviewing, what they’re doing, what they’re trying to achieve, what’s their message that they’re trying to get across? Same thing, Corey. When you kick this off, you asked me a bunch of questions and then that’s why we took it. And that’s where this conversation went because it’s—I mean, yeah, I’m spinning it around and making it about you, sometimes because obviously, it’s fun to do that, and that’s normally—I’m on the other side.Corey: No, it’s always fun to wind up talking to people who have their own shows just because it’s fun watching the narrative flow back and forth. It’s kind of a blast.Jon: It’s almost like commentators, though. You think about it at a sporting event. There’s two in the booth.Corey: Do a team-up at some point, yeah.Jon: Yeah.Corey: In fact, doing the—what is it like the two old gentlemen in the Sesame Street box up in the corner? I forget their names… someone’s going to yell at me for that one. But yeah, the idea of basically kibitzing back and forth. I feel like at some level, we should do a team up and start doing a play-by-play of the re:Invent keynotes.Jon: Oh… you know what, Corey, maybe we should talk about this offline. Having a huge event there, VIP receptions, a podcasting booth is set up at a villa that we have ready to go. We’re going to be hosting social media influencers, live-tweeting happening for keynotes. Now, you don’t have to go to the keynotes personally. You can come to this room, you can click record, we’ll record a live session right there, totally unscripted, like everything else we do, right? We’ll have a VIP reception, come in chat, do introductions. So, Corey, love to have you come into that and we can do a live one right there.Corey: Unfortunately, I’m going to be spending most of re:Invent this year dressed in my platypus costume, but you know how it works.Jon: [laugh]. Oh man, you definitely got to go for that because oh, I have a love to put that on the show. I’m actually doing something not similar, but in true style that I’ve been going to the last couple of re:Invents I will be doing something unique and standing out.Corey: I’m looking forward to it. It’s always fun seeing how people continue to successfully exceed what they were able to do previously. That’s the best part, on some level, is just watching it continually iterate until you’re at a point where it just becomes, well frankly, either ridiculous or you flame out or it hits critical mass and suddenly you launch an entire TV network or something.Jon: Stay tuned. Maybe I will.Corey: You know, it’s always interesting to see how that entire thing plays out. Last question before we call it a show. Talk to me about your process for building content, if you don’t mind. What is your process when you sit down and stare at—at least from my perspective—that most accursed of all enemies, a blank screen? “All right time to create some content, Jackwagon, better be funny. And by the way, you’re on a deadline.” That is the worst part of my job.Jon: All right, so the worst part of your job is the best part of my job. I have to tell you, I actually don’t—and I’m going to have to knock on wood because I don’t get content block. I don’t sit at a screen when I’m doing it. I actually will go for a walk or, you know, I’ll have my weirdest ideas at the weirdest time, like at the gym, I might have a quick idea of something like that and I’ll have a backlog of these ideas that I write down. The thing that I do is I come down, I open up a document and I’ll just drop this idea.And I’ll write it out as almost as it seems like a script. And I’ll never read it verbatim because I look at it and be like, “I know what I’m going to say right now.” An example, if you take a look at my intros that I do for my podcast, they are done after the recording because I recap what we do on a recording.So, let’s take this back. Corey will talk about the one you and I just did. And you and I we hopped on, we did a recording. Afterwards, I put together the intro. And what I’m going to say the intro, I have no freaking clue until I actually get to it, and then all of a sudden, I think of something—not at my desk, but away from my desk—what I’m going to say about you or the guest.An example, there was a gentleman I did his name’s called Mat Batterbee, and he’s from the UK. And he’s a Social Media Finalist. And he has this beard and he always wears, like, this hat or something. And I saw somebody on Twitter make a comment about, you know, following in his footsteps or looking like him. So, they spoofed him with a hat and everything—glasses.I actually bought a beard off of Amazon, put it on, glasses, hat, and I spoofed him for the intro. I had this idea, like, the day before. So, thank goodness for Prime delivery, that I was able to get this beard ASAP, put it on. One take; I only tried to do one take. I don’t think I’ve ever recorded any more.Corey: I have a couple of times sometimes because the audio didn’t capture—Jon: Yeah.Corey: —but that’s neither here nor there. But yeah, I agree with you, I find that the back-and-forth with someone else is way easier from a content perspective for me. Because when you and I started talking, on this episode, for example, I had, like, three or four bullet points I wanted to cover and that’s about it. The rest of it becomes this organic freewheeling conversation and that just tends to work when it’s just me free-associating in front of the camera, it doesn’t work super well. I need something that’s a bit more structured in that sense. So apparently, my answer is just never be alone, ever.Jon: [laugh]. The content that I create, like how-to tutorials, demos, reviews, I’ll take a lot more time on them and I’ll put them together in the flow. And I record those in certain sections. I’ll actually record the demo of walking through and clicking on everything and going through the process, and then I will actually put that in my recording software, and then I will record against it like a voiceover.But I don’t record a script. I actually follow the flow that I did and in order to do that, I understand the product, so I’ll dive deep on it, I’ll figure out some of the things using keywords along the way to highlight the value of utilizing it. And I like to create these in, like, two to three minutes. So, my entire process of creating content—podcast—you know what we hop on, I give everybody the spiel, I click record and I say, “Welcome.” And I do the introduction. I cut that out later. We talk. I’ll tell you what, I never edited anything throughout the entire length of it because whatever happens happens in his natural and comes across.And then I slap on an ending. And I try to make it as quick and as efficiently as possible because if I start doing cuts, people are going to be, like, “Oh, there’s a cut there. What did he cut out?” Oh, there’s this. It’s a full-on free flow. And so, if I mess up and flub or whatever it is, I poke fun of myself and we move on.Corey: Oh, I have my own favorite punching bag. And I honestly think about that for a second. If I didn’t mock myself the way that I do, I would be insufferable. The entire idea of being that kind of a blowhard just doesn’t work. From my perspective, I am always willing to ask the quote-unquote dumb question.It just happens to turn out but I’m never the only person wondering about that thing and by asking it out loud, suddenly I’m giving a whole bunch of other folks air cover to say, “Yeah, I don’t know the answer to that either.” I have no problem whatsoever doing that. I don’t have any technical credibility to worry about burning.Jon: When you start off asking and say, “Hey, dumb question or dumb question,” you start being unsure of yourself. Start off and just ask the question. Never say it’s a dumb question because I’ll tell you what, like you said, there’s probably 20 other people in that room that have the same question and they’re afraid to ask it. You can be the one that just jumps up there and says it and then you’re well-respected for it. I have no problem asking questions.Corey: Honestly, the problem I’ve got is I wish people would ask more questions. I think that it leads to such a better outcome. But people are always afraid to either admit ignorance. Or worse, when they do ask questions just for the joy they get from hearing themselves talk. We’ve all been conference talks where you there’s someone who’s just asking the question because they love the sound of their own voice. I say, they, but let’s be serious; it’s always a dude.Jon: That is very true.Corey: So, if people want to learn more about what you’re up to, where’s the best place to go?Jon: All right, so the best place to go is to follow me on LinkedIn. LinkedIn is my primary one, right? Jon Myer; can’t miss me. At all. Twitter, I am active on Twitter. Not as well as Corey; I would love to get there one day, but my audience right now is LinkedIn.Else you can go to jonmyer.com. Yes, that’s right, jonmyer.com. Because why not? I found I have to talk about this just a little bit. And the reason that I changed it—I actually do own the domain awsblogger, by the way and I still have it—is that when I was awsblogger, I had to chan—I didn’t have to change anything’ nobody required me to, but I changed it to, like, thedailytechshow. And that was pretty cool but then I just wanted to associated with me, and I felt that going with jonmyer, it allowed me not having to change the name ever again because, let’s face it, I’m not changing my name. And I want to stick with it so I don’t have to do a whole transition and when this thing takes off really huge, like it is doing right now, I don’t have to change the name.Corey: Yeah. I would have named it slightly differently had I known was coming. But again, this far in—400 some-odd episodes in last I checked recorded—though I don’t know what episode this will be when it airs—I really get the distinct impression that I am going to learn as I go and, you know, you can’t change that this far in anymore.Jon: I am actually rounding so I’m not as far as you are with the episodes, but I’m happy to say that I did cross number 76—actually 77; I recorded yesterday, so it’s pretty good. And 78 tomorrow, so I am very busy with all the episodes and I love it. I love everybody reaching out and enjoying the conversations that I have. And just the naturalness and the organicness of the podcast. It really puts people at ease and comfortable to start sharing more and more of their stories and what they want to talk about.Corey: I really want to thank you for being so generous with your time and speak with me today. Thanks. It’s always a pleasure to talk with you and I look forward to seeing what you wind up building next.Jon: Thanks, Corey. I really appreciate you having me on. This is very entertaining, informative. I had a lot of fun just having a conversation with you. Thanks for having me on, man.Corey: Always a pleasure. Jon Myer, podcaster extraordinaire and content producer slash creator. The best folks really have no idea what to refer to themselves and I am no exception, so I made up my own job title. I am Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment telling me that I’m completely wrong and that you are a very interesting person. And then tell me what company you wrote a check to once upon a time.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About TimTim’s tech career spans over 20 years through various sectors. Tim’s initial journey into tech started as a US Marine. Later, he left government contracting for the private sector, working both in large corporate environments and in small startups. While working in the private sector, he honed his skills in systems administration and operations for large Unix-based datastores.Today, Tim leverages his years in operations, DevOps, and Site Reliability Engineering to advise and consult with clients in his current role. Tim is also a father of five children, as well as a competitive Brazilian Jiu-Jitsu practitioner. Currently, he is the reigning American National and 3-time Pan American Brazilian Jiu-Jitsu champion in his division.Links Referenced:Twitter: https://twitter.com/elchefe TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it’s hard to know where problems originate. Is it your application code, users, or the underlying systems? I’ve got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it’s more than just hipster monitoring.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it’s still somebody’s problem. And a big part of that responsibility is app security from code to cloud. And that’s where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you’re actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That’s S-N-Y-K.co/screamCorey: Welcome to Screaming in the Cloud. I’m Corey Quinn. A bit of a sad episode Today. I am joined by Duckbill Group principal cloud economist, Tim Banks, but by the time this publishes, he will have left the Duckbill nest, as it were. Tim, thank you for joining me, and can I just start by saying, this is sad?Tim: It is. I have really enjoyed being with Duckbill and I will never forget that message you sent me. It’s like, “Hey, would you like to do this?” And I was like, “Boy would I.” It’s been a fantastic ride and I have enjoyed working with a friend. And I’m glad that we remain friends to this day and always will be, so far as I can tell.Corey: Yes, yes. What you can’t see while recording this, I’m actually sitting in the same room as Tim with a weapon pointed at him to make sure that he stays exactly on message. Yeah, I kid. There’s been a lot that’s happened over the last year. We only got to spend time together in person once at re:Invent. I think because re:Invent is such a blur for me, I don’t remember who the hell I talk to.Someone can walk up and say, “Oh yeah, we met at re:Invent,” and I’ll nod and say, “Oh yeah,” and I will have no recollection of that whatsoever. But you don’t argue with people. But I do distinctly remember hanging out with you there. But since then, it’s been a purely distributed company, purely distributed work.Tim: Yeah, that’s the only time I’ve seen you since I’ve worked here. It’s the only time I met Mike. But it’s weird because it’s like, someone you work with you see every day virtually and talk to, and then you actually get to, like, IRL them and like, “Oh, wow. I had all these, kind of, conceptions of, you know, what you are or who you are as a person, and then you get to, like, check yourself. Was I right? Was I wrong?” I was like, “Oh, you’re taller than I thought; you’re shorter than I thought,” you know, whatever it was.But I think the fun part about it was we all end up being so close by the nature of how we work that it was just like going back and seeing family after a while; you already know who they are and how they are and about them. So, it felt good, but it felt familiar. That’s a great feeling to have. To me, that’s a sign of a very successful distributed culture.Corey: Yeah, it’s weird the kinds of friendships we’ve built during the pandemic. When I was in New York for the summit, I got to meet Linda Haviv at AWS for the first time, despite spending the past year or so talking to her repeatedly. As I referred to her the entire time I was in New York, this is Linda, my new old friend because that is exactly how it felt. It’s the idea of meeting someone in person that you’ve had a long-term ongoing friendship with. It’s just a really—it’s a strange way Everything’s new but it’s not, all at the same time.It reminds me of the early days of the internet culture where I had more friends online than off, which in my case was not hard. And finally meeting them, some people were exactly like they were described and others were nothing at all like they presented. Now that we have Zoom and this constant level of Slack chatter and whatnot, it’s become a lot easier to get a read on what someone is like, I think.Tim: I think so too, you know, we’ve gotten away—and I think largely because of the pandemic—of just talking about work at work, right? The idea of embracing, you know, almost a cliche of the whole person. But it’s become a very necessary thing as people have dealt with pandemic, social upheaval, political climates, and whatever, while they’re working from home. You can’t compartmentalize that safely in perpetuity, right? So, you do end up getting to know people very well, especially in what their concerns are, what their anxieties are, what makes them happy, what makes them sad, things that go on in their lives.You bring all that to your distributed culture because it’s not like you leave it at the door, when you walk out. You’re not walking out anymore; you’re walking to another room, and it’s hard to walk away from those things in this day and age. And we shouldn’t have to, right? I feel like for a successful and nurturing culture—whatever it is, whether it’s tech culture, whether it’s whatever kind of work culture—you can’t say, “I only want your productivity and nothing else about you,” and expect people to sustain that. So, you see these companies are, like, you know, “We don’t have political discussions. We don’t have personal discussions. We’re just about the work.” I’m like, “All right, well, that’s not going to last.” A person cannot just be an automaton in perpetuity and expect them to grow and thrive.Corey: And this is why you’re leaving. And I want to give that a little context because without, sounds absolutely freaking horrifying. You’ve been a strong advocate for an awful lot of bringing the human to work, on your philosophy around leadership, around management. And you’ve often been acting in that capacity throughout, I would say, the majority of your career. But here at The Duckbill Group, we don’t have a scale of team where you being the director of the team or leader of the team is going to happen in anything approaching the near or mid-term.And so, much of your philosophy is great and all because it’s easy to sit here at a small company and start talking about, “Oh, this is how you should be doing it.” You have the opportunity to wind up making a much deeper impact on a lot more people from a management perspective, but you do in fact, need a team to manage as opposed to sitting around there, “Oh, yeah. Who do you manage?” “This one person and I’m doing all of these things to make their life and job awesome.” It’s like, “Yeah, how many hours a week are you spending in one-on-ones?” “20 to 25.”Okay, maybe you need a slightly larger team so you can diffuse that out a little bit. And we are definitely sad to be losing you; super excited to see where you wind up going next. This has been a long time coming where there are things that you have absolutely knocked out of the park here at The Duckbill Group, but you also have that growing—from what I picked up on anyway—need to set a good management example. And lord knows this industry needs more of those. So first, sad to lose you. Secondly, very excited for where you wind up next and what they’re in for, even though it has a strong likelihood that they don’t know the half of it yet.Tim: One of the things that I like about The Duckbill Group and how my time here has been is the first thing that I was asked in the interview was very sincere, like, “Well, what’s your next job?” And I was very clear. It’s like, “After this, I want to be a director or VP of engineering because I would like to be a force multiplier, right?” I would like to make engineering orgs better. I would like to make engineering practices better. I want to make the engineers better, right?And not by driving KPIs and not by management, right, not administrative functions. I want to do it via leadership. I want to do it by setting examples, making safe places for people, making people feel like they’re important and invested in, nurturing them, right? I’ve said this before I—this analogy was getting me somewhere else and I love, it’s like, if I plant a tree and I want it to grow apples, right, I’m not going to sit there and put a number down of apples it’s expected to produce, and then put it on a performance plan if it doesn’t get that number of apples, right? I need to nurture the tree, I need to fertilize it, I need to protect it, I need to keep it safe, I need to keep it safe from the elements, I need to make sure that it doesn’t have parasites, I need to take care of that tree.And if that tree grows and it’s healthy and it’s thriving, it will produce, right? But I’m not—I can’t just expect apples if I’m not taking care of the tree. Now, people are not trees, but you still have to take care of the people if you want them to do things. And if you can’t take care of the people, if you can’t manage the environment that they’re in to make it safe, if you can’t give them the things they need to be successful, then you’re just going to be holding numbers over someone and expecting to hit them.And that doesn’t work. That’s not something that’s sustainable. And it doesn’t really—it’s not even about how much you pay them. You must pay them well, right, but it has to be more than just that if you want people to succeed. And that doesn’t necessarily mean—like, one thing is at the Duckbill Group I love, succeeding doesn’t necessarily mean that I’m going to stay at—or your engineer is going to stay at one place in perpetuity. If you mentor and train and coach and give an engineer opportunity to grow and thrive and what they do is they go to another job for a title increase and a pay increase or something like that, you did your job.Corey: A lot of companies love to tell that lie and they almost convince themselves of it where I look at your resume, and great you have not generally crossed the two-year mark at companies for the last decade. I never did until I started at this place. But we magically always liked to pretend in job interviews that, “Oh yeah, this is my forever job—” like you’re a rescue dog getting adopted or something, “—and I’m going to work here for 25 years and get a gold watch and a pension at the end of it.” It’s lunacy. I have never seen the value in lying to ourselves like that, which is why we start our interviews with, “What’s the job after this and how do we help you get there?”It’s important that we ask those questions and acknowledge that reality. And the downside to it—if you can call it a downside—is you’ve got to live by it. It’s not just words, you can slop onto an interview questionnaire; you actually have to mean it. People can see through insincerity.Tim: And it’s one of the things, like, if you run an org and you grow your people and you don’t have a place for them to grow into, you should expect and encourage them to find those opportunities elsewhere. It is not reasonable, I feel like, as a leader for you expect people to stay in a place where they have grown past or grown out of. You need to either need to give them a new pot to grow into or you need to let them move elsewhere and thrive and grow. And moving elsewhere—like, if you have a retention problem where you can’t retain anybody, that’s a problem, but if you have your junior engineers who become senior engineers at other places, right, and everyone leaves on good terms, and they got the role and you gave them a great recommendation and they give glowing recommendations to you, there’s nothing wrong with that. That’s not a failure; that’s success.Corey: One bit of I would say pushback that I suspect you might get when talking to people about what’s next is that, “Well, you are just a consultant, on some level, for a year.” You always know that someone is really arguing in good faith when they describe what you did with the word ‘just,’ but we’ll skip past that part. And it’s, “You’re just a consultant. What would you possibly know about team management and team dynamics?” And there is a little bit of truth to that insofar as the worst place in the world to get management advice is very clearly on Twitter.It turns out that most interpersonal scenarios are, one, far too personal to wind up tweeting about, and two, do not lend themselves to easy solutions that succinctly fit within 280 characters. Imagine that. The counter-argument though, is that you have—correctly from where I sit—identified a number of recurring dynamics on teams that you have encountered and worked with deeply as a large number of engagements. And these are recurring things, I want to be clear. So, I’m not talking about one particular client. If you’re one of our clients and listen to this thinking that we’re somehow subtweeting you with our voices—I don’t know what that is; subwoofing, maybe?Tim: [crosstalk 00:12:05]—Corey: Is that what a subwoofer is? I’m not an audio person.Tim: Throwing shade, we’ll just say—call it throwing shade.Corey: Yeah, we’re not throwing shade at any one person, team, or group in particular; these are recurring things. Tim, what have you seen?Tim: And so, I think the biggest thing I see is folks that are on the precipice of a big technological change, right, and there is an extraordinary amount of anxiety, right? I’ve seen a number of customers through our engagements that, “We are moving away from this legacy platform,” or from this thing that we have been doing for X amount of time. And everyone has staked the other domains, staked out their areas of expertise and control and we’re going to change that. And the solution to that is not a technical solution. You don’t fix that by Helm charts, or Terraform, or CloudFormation. You fix that by conversations, and you fix that by listening. You fix that by finding ways to reassure folks and giving them confidence in their ability to adjust and thrive in a new environment.If you take somebody who’s been, you know, an Oracle admin for 20 years, and you going to say, “Great. Now, you’re going to learn, you’re going to do this an RDS,” that’s a whole new animal, and folks feel like, well, you know, I can’t learn something new like that? Well, yeah you can. If you can learn Oracle, you can learn anything. I firmly believe that.But that’s one of the conversations we have, it’s never, almost never a technical problem folks have. We need to reassure people, right? And so, folks who reach out to us, it’s typically folks who are trying to get their organizations in that direction. Another thing we see sometimes is that we find that there’s a disconnect between leadership and the engineers. They have either different priorities or different understandings of what’s going on. And we come in to solve a problem, which may be cost but that’s not the problem we actually solve. The problem we actually solve is fixing this communication bridges between management and leadership.And that’s almost an every time occurred. At some point or another, there’s some disconnect there. And that’s the best part of the job. Like, the reason I do this consulting gig is not because I want to bang away at code. If I’ve had to do that, that’s an anomaly for sure because I want to have these conversations.And people want to have these conversations; they want to get these problems solved and sometimes they don’t know how to. And that is the common thing, I think, through all of our customers. Like, we need some amount of expertise to help us find solutions to these things that aren’t necessarily technical problems. And I think that’s where we run into problems as an industry, right, where we think a lot of things are technical problems or have technical solutions, and they don’t. There are people problems. They’re—Corey: Here at The Duckbill Group, we’re basically marriage counseling for engineering and finance in many cases.Tim: We really are.Corey: This is why were people not software.Tim: Yeah. And I will say this very firmly and you can quote me on this: like, you cannot replace us. You cannot replace the kind of engagements we do with software. You can’t. Can’t be done, right? Software is not empathetic.Corey: There are a whole series of questions we ask our clients at the start of an engagement and the answers to those questions change what we ask them going forward. In fact, even the level-setting in the conversation that we have at the start of that changes the nature of those. We’re not reading from a list; we’re trying to build an understanding. There is a process around what we do, but it’s not process that can ever be scoped down to the point where it’s just a list of questions or a questionnaire that isn’t maddening for people to fill out because it’s so deeply and clearly misses the mark around context of what they’re actually doing.Tim: Mm-hm. Our engaged with their conversations. That’s all they are. They’re really in-depth conversations where we’re going to start asking questions and we’re going to ask questions about those answers. We’re start pulling out strings and kicking over rocks and seeing what we find.And that’s the kind of thing that, you know, you would expect anyone to do who’s coming in and saying, “Okay, we have a problem. Now, let’s figure it out.” Right? Well, you can’t just look at something on the surface, and say, “Oh, I know what this is.” Right? You know, for someone to say, “Oh, I know how to fix this,” when they walk in is the surest way to know that someone doesn’t know what they’re talking about, right?Corey: Oh, easiest thing in the world is to walk in and say, “This is broken and wrong.” That can translate directly to, “Hi, I am very junior. Please feed my own ass to me.” Because no one shows up at work thinking they’re going to do a crap job today on purpose. There’s a reason things are the way that they are.Tim: Mm-hm. And that’s the biggest piece of context we get from our customers is we can understand what the best practices are. You can go Google them right now and say, “This is the ten things you’re supposed to do all the time,” right? And we would be really, really crappy consultants if we just read off that list, right? We need to have context: does this thing make sense? Is this the best practice? Maybe, but we want to know why you did it this way.And after you tell us that way, I’m like, “You know what? I would do it the exact same way for this use case.” And that’s great. We can say like, “This is the best way to do that. Good job.” It’s atypical; it’s unusual, but it solves the problems that you need solving.And that’s where I think a lot of people miss. Like, you know, you can go—and not to throw shade at AWS’s Trusted Advisor, but we’re going to throw shade at AWS’s Trusted Advisor—and the fact that it will give you—Corey: It is Plausible Advisor at absolute best.Tim: [laugh]. It will give you suggestions that have no context. And a lot of the automated AI things that will recommend that you do this and this and this and this are pretty much all the same. And they have no context because they don’t understand what you’re trying to do. And that’s what makes the difference between people. There’s these people problems.And so, one of the things that I think is really interesting is that we have moved into doing a shorter engagement style that is very short. It’s very quick, it’s very kind of almost tactical, but we go in, we look at your bill, we ask you some questions, and we’re going to give you a list of suggestions that are going to save you a significant amount of money right away, right? So, a lot of times, folks when they need quick wins, or they don’t really need us to deep-dive into all their DynamoDB access patterns, right? They just want like, “Hey, what are the five things we can do to save us some money?” And we’re like, “Well, here they are. And here’s what we think they’re going to save you.” And folks who really enjoyed that type of engagement. And it’s one of my favorite ones to do.Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: I can also predict that people are going to have questions for you—probably inane—of, well, you were a consultant, how are your actual technical chops? And I love answering these questions with data. So, I have here pulled up the last six months of The Duckbill Group’s AWS bills. And for those who are unaware, every cloud economist has their own dedicated test account for testing out strange things that we come across. And again, can the correct answer in many consulting engagements is, “I don’t know, but I’ll find out.”Well, this is how we find out. We run tests and learn these things ourselves. I suppose we could extend this benefit, if you want to call it that, to people who aren’t cloud economists but I’m not entirely sure what, I don’t know, an audio engineer is going to do with an AWS account that isn’t, you know, kind of horrifying. To the audio engineer that is editing this podcast, my condolences if you take that as a slight, and if there is something you would use an AWS account for, please let me know. We’ll come talk about it here.But back to topic, looking at the last six months of your bill for your account—that’s right, a ritualistic shaming of the AWS bill—in January you spent $16.06. In February, you spent 44 cents. And you realized that was too high, so back in March, you then spent 19 cents. And then $3.01 back in April. May wound up $10.02, and now you’re $9.84 as of June. July has not yet finalized as of this recording.And what I want to highlight—and what that tells me when I look at these types of bills—and I assure you as the world’s leading self-described expert in AWS billing, I’m right; listening to me is a best practice on these things—that shows the exact opposite of a steady-state workload. There’s a lot of dynamism to those giant swings because we don’t have cloud economists who are going to just run these things steady-state for the rest of our lives. Those are experiments of building and testing out new and exciting things in a whole bunch of very weird, very strange ways. Whenever I wind up talking to someone in one of the overarching AWS services at AWS and I pull up my account, a common refrain is, “Wow, you use an awful lot of services.” Right. I’m not just sitting here run and EC2 instances forever. Imagine that. And your account is a perfect microcosm of that entire philosophy.Tim: Well, I don’t know all the answers, right? And I will never profess all the answers. And before I say, “You should do this—” or maybe I will say, “You might be able to do this. Let me go save as possible.” [laugh]. Right? And so, just let me just see, can you do this? Does this work? No, I guess it doesn’t. Or AWS docu—especially, “The AWS documentation says this. Let me see if that’s actually the case.”Corey: I don’t believe that they intend to lie, but—Tim: No.Corey: —they also certainly don’t get it correct all the time.Tim: And to be fair, they have, what, 728 services by this point, and that’s a lot of documentation you’re not going to get—Corey: Three more have launched since the start of this recording.Tim: I—yeah, actually—well, by the time this hits, they’re probably going to have 22. But we’ll [laugh] see. But yeah, no. And that’s fine. And they’re not going to have every use case, and every edge, kind of like, concern handled, and so that’s why we need to kick the tires a little bit.And what I think more than anything else is, you know, sometimes we just do things out of convenience. Like, “Well, I don’t want to run this on this; let me just fire it up because it’s not my money.” [laugh]. But we also want to be fairly concerned about you know, how we do things. You don’t want to run a fleet of z1ds, obviously.But there is a certain amount of tire-kicking and infrastructure spinning up that you have to do in order to maintain freshness, right? And it’s not a thing where I’m going to say, “Oh, I know YAML off the top of my head, and I need to do—you know, I’m up to speed on every single possible API call that you can make.” No. My technical prowess has always been in architecture and operations. So, I think when we have these conversations, folks mostly tend to be impressed by not only business acumen and strategy, but also being able to get down to the weeds and talking with the developers and the engineers about the minutia. And you will have seen you know, the feedback that I’ve gotten about my technical prowess has always been good. You know, I can hang with anybody, I feel like.Corey: I would agree wholeheartedly. It’s been really interesting watching you in conversations, internally and with our clients, where you will just idly bust out something fricking brilliant out of left field. And most of the time, I don’t think you even realize it. It’s just one of those things that makes intuitive and instinctive sense to you. And you basically just leave people stunned and their scribbling notes and trying to wrap their heads around what you just said.And it’s adorable because sometimes you wind up almost, like, looking embarrassed, like, “Did I say something rude and not realize it? Like, I wasn’t trying to be insulting.” It’s like, “Nope, nope. You’re just doing your thing, Tim. Just keep on doing it. That’s fine.”Tim: Yeah, it’s funny because, like you, one of the things that I’ve really enjoyed about it is, like, we’ll just start bouncing ideas off of each other and come up with something brilliant. “Yeah, let’s do that.” And then, “Okay, this is now a thing.” And it’s like, you know, there’s something to be said about being around smart people. So, it’s not just me coming up with something brilliant; these are almost always fruits of a conversation and discussion being had, and then you formulate something great in your head.But again, this is why I love the aspect of talking and having conversations with people, so that way you can come up with something kind of brilliant. None of this is done in a silo. Like we’re not really, really good at what we do because we don’t rely or talk to or have conversations with other people.Corey: One thing that you did that I think is one of the most transformative things that has happened in company history in some respects has been when you started, and for the first half of your tenure here, we had two engagement types that we would wind up giving our consulting clients. There’s contract negotiation, where we help companies negotiate their long-term commitment contracts with AWS—and we’re effective at it and that’s fun; that’s basically what you would more or less expected to be—and the other is our cost optimization project engagements. And those tend to look six to eight weeks where we wind up going in deep-dives into the intricacies of an organization’s AWS accounts, bills, strategy, growth plan, et cetera, et cetera, et cetera, to an exhaustive level of detail. And in an interest of being probably overly transparent here, I didn’t like working on those engagements myself. I like coming in, finding the big things that will be transformative to reduce the bills—it’s like solving a puzzle—and then the relatively in-depth analysis for things that are a relatively paltry portion of the AWS bill does not really lead me to enjoying the work very much.And I beat my head against that one for years. And you busted out one day with an idea that became our third type of engagement, which is the first pass, where we charge significantly less for the engagement and it essentially distills down into you get us to talk to your engineering teams for a day. Bring us any questions, give us access in advance to these things, and we will basically go on a whirlwind guided tour and lay waste to your AWS bill and highlight different opportunities that we see to optimize these things. And it has been an absolute smash success. People love the engagements.Very often, it leads to that second full-bore engagement that I was describing earlier, but it also aligns very well with the way that I like to think about these things. I’m a great consultant, specifically because once I’ve delivered the value, I like to leave. Whereas as an employee, I just sort of linger around, and then I go cause problems and other people’s departments—ideally, not on purpose, but you know, I am me—and this really emphasizes that and keeps me moving quickly. I really, really like that engagement style and I have you to thank for coming up with the idea and finding a way to do it that didn’t either not resonate with the market—in which case, we’re not selling a damn thing—or wound up completely eviscerating the value of the longer-term deep-dive engagements, and you threaded that needle perfectly.Tim: I thank you; I appreciate that. There was this kind of vacuum that I saw where, both from a cost and from a resource point where six to eight weeks is a long time for an engineering org to dedicate to any one thing, especially if that one thing isn’t directly making money. But engineering orgs are also very interested in saving money. But it’s especially in smaller orgs where that velocity is very important, they don’t have six to eight weeks for that. They can’t dedicate the resources to those deep-dives all the time, and all the conversations we—and when we do a COP, it is exhaustive. We are exploring every avenue to almost an absurd level, right?And that’s not the right engagement for a lot of orgs, right? So, coming in and saying, “Hey, you know, this is a quick one; these are the things that you can do. This is 90% of the savings you’re going to realize. These things: bam, bam, bam, bam, bam.” Right?And then we give it to the folks and we let them work on it, and then they’re like, “Hey, we need this because we want to negotiate EDP,” or, “We need this because, you know, we’re just trying to make sure that our costs are in line so we can be more agile, so we can do this project, or whatever.” Right? And then there are a lot of other orgs that do need that exhaustive kind of thing, larger orgs especially, right? Larger, more complex orgs, orgs that are trying to maybe—like, if you’re trying to make a play to get acquired, you want to get this very, very in-depth study so you know all your liabilities and all your assets, so that way you can fix those problems and make it very attractive for someone to buy you, right? Or orgs that just have, like, we are not having an impending EDP; we have a lot of time to be able to focus on these things, and we can build this into the roadmap, right?Then we can do a very exhaustive study of those things. But for a lot of times, people are just like, “Look, I just need to save X amount of money on my AWS bill and can you do that?” Well, sure. We can go in there and have those conversations and give you a lot of savings. And I’m very much in the camp of, you know, ‘perfect is the enemy of good.’ I don’t have to save down to the nth penny on your DynamoDB bill. But if I can, shave—cut it in half, that’s great. Most people are very happy about those kinds of things. And that’s a very routine finding for us.Corey: One other aspect that I really liked about it, too, is that it let us move down market a bit, away from companies that are spending millions of dollars a month. Because yeah, the ROI for those customers is a slam dunk on virtually any engagement that we could put together, but what about the smaller companies, the ones that are not spending that much money, yet? They’ve never felt great talk to them and say, “Oh, just go screw up your AWS bill some more. Then, then you will absolutely be able to generate some value. Maybe turn off MFA and post your credentials to GitHub or something. That’ll speed up the process nicely.”That’s terrible advice and we can’t do it. But this enables us to move down to smaller companies that are earlier in their cloud estate build-out or are growing organically rather than trying to do a giant migration as sort of greenfield growth approach. I really, really like our ability to help companies that are a bit earlier in their cloud journey, as well as in smaller environments, just because I guess, on some level, for me, at least, when you see enormous multimillion-dollar levels of spend, the misconfigurations are generally less fun to find; they’re less exciting. Because, yeah at a small scale, you can screw up and your Managed NAT Gateway bill is a third of your spend. When you’re spending $80 million a year, you’re not wasting that kind of money on Managed NAT Gateways because that misconfiguration becomes visible from frickin’ orbit.So, someone has already found that stuff. And it’s always then it’s almost certainly EC2, RDS, and storage. Great. Then there’s some weird data transfer stuff and it starts to look a lot more identical. Smaller accounts, at least from my perspective, tend to have a lot more of interesting things to learn hiding in the shadows.Tim: Oh, absolutely. And I think the impact that you make for the future for small companies much higher, right? You go in there and you have an engagement, you can say, “Okay, I understand the business reason why you did this here, but if you make these changes—bam, bam, bam—12 to 18 months and on, right, this is going to make a huge difference in your business. You’re going to save a tremendous amount of money and you’re going to be much more agile.”You did this thing because it worked for the POC, it worked for the MVP, right? That’s great, but before it gets too big and becomes load-bearing technical debt, let’s make some changes to put you in a better position, both for cost optimization and an architectural future that you don’t have to then break a bone that’s already set to try and fix it. So, getting in there before there’s a tremendous load on their architecture—or rather on their infrastructure, it’s super, super fun because you know that when you’ve done this, you have given that company more runway, or you’ve given them the things they need to actually be more successful, and so they can focus their time and efforts on growth and not on trying to stop the bleeding with their AWS bill.Corey: Tim, it’s been an absolute pleasure to work with you. I’m going to miss working with you, but we are definitely going to remain in touch. Where can people find you to follow along with your continuing adventures?Tim: The best way to find me is on Twitter, I am @elchefe—E-L-C-H-E-F-E. And yeah, I will definitely keep in touch with you, Corey. Again, you have been a tremendous friend and I really appreciate you, your insights, and your honesty. Our partners are friends with each other and I do not think that they will let us ever drift too far apart. So.Corey: No, I think it is pretty clear that we are basically going to be both of their plus-ones forever.Tim: [laugh]. I think so.Corey: I’m just waiting for them when they pulled the prank of dressing us the exact same way because our styles are somewhat different, and I’m pretty sure that there’s not a whole lot of convergence where we both wind up looking great. So, it’s going to be hilarious regardless of what direction it goes in.Tim: Well, you do have velour tracksuits too, right?Corey: Not yet, but please don’t tell that to Bethany.Tim: [laugh].Corey: Tim, it has been an absolute pleasure.Tim: The pleasure has been all mine, Corey. I really appreciate it.Corey: Tim Banks, for one last time, principal cloud economist at The Duckbill Group. I am Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice and an insulting comment that says that we are completely wrong in our approach to management and the real answer is as follows, making sure to keep that answer less than 280 characters.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About AntonDr. Anton Chuvakin is now involved with security solution strategy at Google Cloud, where he arrived via Chronicle Security (an Alphabet company) acquisition in July 2019.Anton was, until recently, a Research Vice President and Distinguished Analyst at Gartner for Technical Professionals (GTP) Security and Risk Management Strategies team. (see chuvakin.org for more)Links Referenced: Google Cloud: https://cloud.google.com/ Cloud Security Podcast: https://cloud.withgoogle.com/cloudsecurity/podcast/ Twitter: https://twitter.com/anton_chuvakin Medium blog: https://medium.com/@anton.chuvakin TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don’t leave managing your database to your cloud vendor because they’re too busy launching another half-dozen managed databases to focus on any one of them that they didn’t build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: Let’s face it, on-call firefighting at 2am is stressful! So there’s good news and there’s bad news. The bad news is that you probably can’t prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: Welcome to Screaming in the Cloud, I’m Corey Quinn. My guest today is Anton Chuvakin, who is a Security Strategy Something at Google Cloud. And I absolutely love the title, given, honestly, how anti-corporate it is in so many different ways. Anton, first, thank you for joining me.Anton: Sure. Thanks for inviting me.Corey: So, you wound up working somewhere else—according to LinkedIn—for two months, which in LinkedIn time is about 20 minutes because their date math is always weird. And then you wound up going—according to LinkedIn, of course—leaving and going to Google. Now, that was an acquisition if I’m not mistaken, correct?Anton: That’s correct, yes. And it kind of explains that timing in a little bit of a title story because my original title was Head of Security Solution Strategy, and it was for a startup called Chronicle. And within actually three weeks, if I recall correctly, I was acquired into Google. So, title really made little sense of Google, so I kind of go with, like, random titles that include the word security, and occasionally strategy if I feel generous.Corey: It’s pretty clear, the fastest way to get hired at Google, given their famous interview process is to just get acquired. Like, “I’m going to start a company and raise it to, like, a little bit of providence, and then do an acquihire because that will be faster than going through the loop, and ideally, there will be less algorithm solving on whiteboards.” But I have to ask, did you have to solve algorithms on whiteboards for your role?Anton: Actually, no, but it did come close to that for some other people who were seen as non-technical and had to join technical roles. I think they were forced to solve coding questions and stuff, but I was somehow grandfathered into a technical role. I don’t know exactly how it happened.Corey: Yeah, how you wound up in a technical role. Let’s be clear, you are Doctor Anton Chuvakin, and you have written multiple books, you were a research VP at Gartner for many years, and once upon a time, that was sort of a punchline in the circles I hung out with, and then I figured out what Gartner actually does. And okay, that actually is something fairly impressive, let’s be clear here. Even as someone who categorically defines himself as not an analyst, I find myself increasingly having a lot of respect for the folks who are actually analysts and the laborious amount of work that they do that remarkably few people understand.Anton: That’s correct. And I don’t want to boost my ego too much. It’s kind of big enough already, obviously, but I actually made it all the way to Distinguished Analyst, which is the next rank after VP.Corey: Ah, my apologies. I did not realize it. This [challenges 00:02:53] the internal structure.Anton: [laugh]. Yeah.Corey: It’s like, “Oh, I went from Senior to Staff,” or Staff to Senior because I’m external; I don’t know the direction these things go in. It almost feels like a half-step away from oh, I went from [SDE3 to SDE4 00:03:02]. It’s like, what do those things mean? Nobody knows. Great.Anton: And what’s the top? Is it 17 or is it 113? [laugh].Corey: Exactly. It’s like, oh okay, so you’re Research VP—or various kinds of VPs—the real question is, how many people have to die before you’re the president? And it turns out that that’s not how companies think. Who knew?Anton: That’s correct. And I think Gartner was a lot of hard work. And it’s the type of work that a lot of people actually don’t understand. Some people understand it wrong, and some people understand it wrong, kind of, for corrupt reasons. So, for example, a lot of Gartner machinery involves soaking insight from the outside world, organizing it, packaging it, writing it, and then giving it as advice to other people.So, there’s nothing offensive about that because there is a lot of insight in the outside world, and somebody needs to be a sponge slash filter slash enrichment facility for that insight. And that, to me, is a good analyst firm, like Gartner.Corey: Yeah. It’s a very interesting world. But you historically have been doing a lot of, well, let’s I don’t even know how to properly describe it because Gardner’s clientele historically has not been startups because let’s face it, Gartner is relatively expensive. And let’s be clear, you’re at Google Cloud now, which is a different kind of expensive, but in a way that works for startups, so good for you; gold star. But what was interesting there is that the majority of the Gartner clientele that I’ve spoken to tend to be big-E Enterprise, which runs legacy businesses, which is a condescending engineering term for ‘it makes money.’And they had the temerity to start their company before 15 years ago, so they built data centers and did things in a data center environment, and now they’re moving in a cloudy direction. Your emphasis has always been on security, so my question for you to start with all this is where do you see security vendors fitting in? Because when I walk the RSA expo hall and find myself growing increasingly depressed, it seems like an awful lot of what vendors are selling looks very little removed from, “We took a box, now we shoved in a virtual machine and here you go; it’s in your cloud environment. Please pay us money.” The end. And it feels, if I’m looking at this from a pure cloud-native, how I would build things in the cloud from scratch perspective, to be the wrong design. Where do you stand on it?Anton: So, this has been one of the agonizing questions. So, I’m going to kind of ignore some of the context. Of course, I’ll come back to it later, but want to kind of frame it—Corey: I love ignoring context. My favorite thing; it’s what makes me a decent engineer some days.Anton: So, the frame was this. One of the more agonizing questions for me as an analyst was, a client calls me and says, “We want to do X.” Deep in my heart, I know that X is absolutely wrong, however given their circumstances and how they got to decided to do X, X is perhaps the only thing they can logically do. So, do you tell them, “Don’t do X; X is bad,” or you tell them, “Here’s how you do X in a manner that aligns with your goals, that’s possible, that’s whatever.”So, cloud comes up a lot in this case. Somebody comes and says, I want to put my on-premise security information management tool or SIM in the cloud. And I say, deep in my heart, I say, “No, get cloud-native tool.” But I tell them, “Okay, actually, here’s how you do it in a less painful manner.” So, this is always hard. Do you tell them they’re on their own path, but you help them tread their own path with least pain? So, as an analyst, I agonized over that. This was almost like a moral decision. What do I tell them?Corey: It makes sense. It’s a microcosm of the architect’s dilemma, on some level, because if you ask a typical Google-style interview whiteboard question, one of my favorites in years past was ‘build a URL shortener.’ Great. And you can scale it out and turn it into different things and design things on the whiteboard, and that’s great. Most mid-level people can wind up building a passable designed for most things in a cloud sense, when you’re starting from scratch.That’s not hard. The problem is that the real world is messy and doesn’t fit on a whiteboard. And when you’re talking about taking a thing that exists in a certain state—for whatever reason, that’s the state that it’s in—and migrating it to a new environment or a new way of operating, there are so many assumptions that have to break, and in most cases, you don’t get the luxury of just taking the thing down for 18 months so you can rework it. And even that it’s never as easy as people think it is, so it’s going to be 36. Great.You have to wind up meeting people where they are as they’re contextualizing these things. And I always feel like the first step of the cloud migration has been to improve your data center environment at the cost of worsening your cloud environment. And that’s okay. We don’t all need to be the absolute vanguard of how everything should be built and pushing the bleeding edge. You’re an insurance company, for God’s sake. Maybe that’s not where you want to spend your innovation energies.Anton: Yeah. And that’s why I tend to lean towards helping them get out of this situation, or maybe build a five-step roadmap of how to become a little bit more cloud-native, rather than tell them, “You’re wrong. You should just rewrite the app in a cloud-native way.” That advice almost never actually works in real world. So, I see a lot of the security people move their security stacks to the cloud.And if I see this, I deepen my heart and say, “Holy cow. What do you mean, you want to IDS every packet between Cloud instances? You want to capture every packet in cloud instances? Why? It’s all encrypted anyway.” But I don’t say that. I say, “Okay, I see how this is the first step for you. Let’s describe the next seven steps.”Corey: The problem I keep smacking into is that very often folks who are pushing a lot of these solutions are, yes, they’re meeting customers where they are, and that makes an awful lot of sense; I’m not saying that there’s anything inherently wrong about that. The challenge is it also feels on the high end, when those customers start to evolve and transform, that those vendors act as a drag. Because if you wind up going in a full-on cloud-native approach, in the fullness of time, there’s an entire swath of security vendors that do not have anything left to sell you.Anton: Yes, that is correct. And I think that—I had a fight with an EDR vendor, Endpoint Detection Response, vendor one day when they said, “Oh, we’re going to be XDR and we’ll do cloud.” And I told them, “You do realize that in a true cloud-native environment, there’s no E? There is no endpoint the way you understand it? There is no OS. There is no server. And 99% of your IP isn’t working on the clients and servers. How are you going to secure a cloud again?”And I get some kind of rambling answer from them, but the point is that you’re right, I do see a lot of vendors that meet clients where they are during their first step in the cloud, and then they may become a drag, or the customer has to show switch to a cloud-native vendor, or to both sometimes, and pay into two mouths. Well, shove money into two pockets.Corey: Well, first, I just want to interject for a second here because when I was walking the RSA expo floor, there were something like 15 different vendors that were trying to sell me XDR. Not a single one of them bothered to expand the acronym—Anton: Just 15? You missed half of them.Corey: Well, yeah—Anton: Holy cow.Corey: As far as I know XDR cable. It’s an audio thing right? I already have a bunch of those for my microphone. What’s the deal here? Like, “I believe that’s XLR.” It’s like, “I believe you should expand your acronyms.” What is XDR?Anton: So, this is where I’m going to be very self-serving and point to a blog that I’ve written that says we don’t know what’s XDR. And I’m going to—Corey: Well, but rather than a spiritual meaning, I’m going to ask, what does the acronym stands for? I don’t actually know the answer to that.Anton: Extended Detection and Response.Corey: Ah.Anton: Extended Detection and Response. But the word ‘extended’ is extended by everybody in different directions. There are multiple camps of opinion. Gartner argues with Forrester. If they ever had a pillow fight, it would look really ugly because they just don’t agree on what XDR is.Many vendors don’t agree with many other vendors, so at this point, if you corner me and say, “Anton, commit to a definition of XDR,” I would not. I will just say, “TBD. Wait two years.” We don’t have a consensus definition of XDR at this point. And RSA notwithstanding, 30 booths with XDRs on their big signs… still, sorry, I don’t have it.Corey: The problem that I keep running into again and again and again, has been pretty consistently that there are vendors willing to help customers in a very certain position, and for those customers, those vendors are spot on the right thing to do.Anton: Mmm, yep.Corey: But then they tried to expand and instead of realizing that the market has moved on and the market that they’re serving is inherently limited and long-term is going to be in decline, they instead start trying to fight the tide and saying, “Oh, no, no, no, no. Those new cloud things, can’t trust them.” And they start out with the FU, the Fear, Uncertainty, and Doubt marketing model where, “You can’t trust those newfangled cloud things. You should have everything on-prem,” ignoring entirely the fact that in their existing data centers, half the time the security team forgets to lock the door.Anton: Yeah, yeah.Corey: It just feels like there is so much conflict of interest about in the space. I mean, that’s the reason I started my Thursday Last Week in AWS newsletter that does security round-ups, just because everything else I found was largely either community-driven where it understood that it was an InfoSec community thing—and InfoSec community is generally toxic—or it was vendor-captured. And I wanted a round-up of things that I had to care about running an infrastructure, but security is not in my job title, even if the word something is or is not there. It’s—I have a job to do that isn’t security full time; what do I need to know? And that felt like an underserved market, and I feel like there’s no equivalent of that in the world of the emerging cloud security space.Anton: Yes, I think so. But it has a high chance of being also kind of captured by legacy vendors. So, when I was at Gartner, there was a lot of acronyms being made with that started with a C: Cloud. There was CSPM, there was CWBP, and after I left the coined SNAPP with double p at the end. Cloud-Native Application Protection Platform. And you know, in my time at Gartner, five-letter acronyms are definitely not very popular. Like, you shouldn’t have done a five-letter acronym if you can help yourself.So, my point is that a lot of these vendors are more from legacy vendors. They are not born in the cloud. They are born in the 1990s. Some are born in the cloud, but it’s a mix. So, the same acronym may apply to a vendor that’s 2019, or—wait for it—1989.Corey: That is… well, I’d say on the one hand, it’s terrifying, but on the other, it’s not that far removed from the founding of Google.Anton: True, true. Well, ’89, kind of, it’s another ten years. I think that if you’re from the ’90s, maybe you’re okay, but if you’re from the ’80s… you really need to have superpowers of adaptation. Again, it’s possible. Funny aside: at Gartner, I met somebody who was an analyst for 32 years.So, he was I think, at Gartner for 32 years. And how do you keep your knowledge current if you are always in an ivory tower? The point is that this person did do that because he had a unique ability to absorb knowledge from the outside world. You can adapt; it’s just hard.Corey: It always is. I’m going to pivot a bit and put you in a little bit of a hot seat here. Not intentionally so. But it is something that I’ve been really kicking around for a while. And I’m going to basically focus on Google because that’s where you work.I yeah, I want you to go and mouth off about other cloud companies. Yeah, that’s—Anton: [laugh]. No.Corey: Going to go super well and no one will have a problem with that. No, it’s… we’ll pick on Google for a minute because Google Cloud offers a whole bunch of services. I think it’s directionally the right number of services because there are areas that you folks do not view as a core competency, and you actually—imagine that—partner with third parties to wind up delivering something great rather than building this shitty knockoff version that no one actually wants. Ehem, I might be some subtweeting someone here with this, only out loud.Anton: [laugh].Corey: The thing that resonates with me though, is that you do charge for a variety of security services. My perspective, by and large, is that the cloud vendors should not be viewing security as a profit center but rather is something that comes baked into the platform that winds up being amortized into the cost of everything else, just because otherwise you wind up with such a perverse set of incentives.Anton: Mm-hm.Corey: Does that sound ridiculous or is that something that aligns with your way of thinking. I’m willing to take criticism that I’m wrong on this, too.Anton: Yeah. It’s not that. It’s I almost start to see some kind of a magic quadrant in my mind that kind of categorizes some things—Corey: Careful, that’s trademarked.Anton: Uhh, okay. So, some kind of vis—Corey: It’s a mystical quadrilateral.Anton: Some kind of visual depiction, perhaps including four parts—not quadrants, mind you—that is focused on things that should be paid and aren’t, things that should be paid and are paid, and whatever else. So, the point is that if you’re charging for encryption, like basic encryption, you’re probably making a mistake. And we don’t, and other people, I think, don’t as well. If you’re charging for logging, then it’s probably also wrong—because charging for log retention, keeping logs perhaps is okay because ultimately you’re spending resources on this—charging for logging to me is kind of in the vile territory. But how about charging for a tool that helps you secure your on-premise environment? That’s fair game, right?Corey: Right. If it’s something you’re taking to another provider, I think that’s absolutely fair. But the idea—and again, I’m okay with the reality of, “Okay, here’s our object storage costs for things, and by the way, when you wind up logging things, yeah, we’ll charge you directionally what it costs to store that an object store,” that’s great, but I don’t have the Google Cloud price list shoved into my head, but I know over an AWS land that CloudWatch logs charge 50 cents per gigabyte, for ingress. And the defense is, “Well, that’s a lot less expensive than most other logging vendors out there.” It’s, yeah, but it’s still horrifying, and at scale, it makes me want to do some terrifying things like I used to, which is build out a cluster of Rsyslog boxes and wind up having everything logged to those because I don’t have an unbounded growth problem.This gets worse with audit logs because there’s no alternative available for this. And when companies start charging for that, either on a data plane or a management plane level, that starts to get really, really murky because you can get visibility into what happened and reconstruct things after the fact, but only if you pay. And that bugs me.Anton: That would bug me as well. And I think these are things that I would very clearly push into the box of this is security that you should not charge for. But authentication is free. But, like, deeper analysis of authentication patterns, perhaps costs money. This to me is in the fair game territory because you may have logs, you may have reports, but what if you want some kind of fancy ML that analyzes the logs and gives you some insights? I don’t think that’s offensive to charge for that.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it’s still somebody’s problem. And a big part of that responsibility is app security from code to cloud. And that’s where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you’re actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That’s S-N-Y-K.co/screamCorey: I think it comes down to what you’re doing with it. Like, the baseline primitives, the things that no one else is going to be in a position to do because honestly, if I can get logging and audit data out of your control plane, you have a different kind of security problem, and—Anton: [laugh].Corey: That is a giant screaming fire in the building, as it should be. The other side of it, though, is that if we take a look at how much all of this stuff can cost, and if you start charging for things that are competitive to other log analytics tools, great because at that point, we’re talking about options. I mean, I’d like to see, in an ideal world, that you don’t charge massive amounts of money for egress but ingress is free. I’d like to see that normalized a bit.But yeah, okay, great. Here’s the data; now I can run whatever analytics tools I want on it and then you’re effectively competing on a level playing field, as opposed to, like, okay, this other analytics tool is better, but it’ll cost me over ten times as much to migrate to it, so is it ten times better? Probably not; few things are, so I guess I’m sticking with the stuff that you’re offering. It feels like the cloud provider security tools never quite hit the same sweet spot that third-party vendors tend to as far as usability, being able to display things in a way that aligns with various stakeholders at those companies. But it still feels like a cash grab and I have to imagine without having insight into internal costing structures, that the security services themselves are not a significant revenue driver for any of the cloud companies. And the rare times where they are is almost certainly some horrifying misconfiguration that should be fixed.Anton: That’s fair, but so to me, it still fits into the bucket of some things you shouldn’t charge for and most people don’t. There is a bucket of things that you should not charge for, but some people do. And there’s a bucket of things where it’s absolutely fair to charge for I don’t know the amount I’m not a pricing person, but I also seen things that are very clearly have cost to a provider, have value to a client, have margins, so it’s very clear it’s a product; it’s not just a feature of the cloud to be more secure. But you’re right if somebody positions as, “I got cloud. Hey, give me secure cloud. It costs double.” I’d be really offended because, like, what is your first cloud is, like, broken and insecure? Yeah. Replace insecure with broken. Why are you selling broken to me?Corey: Right. You tried to spin up a service in Google Cloud, it’s like, “Great. Do you want the secure version or the shitty one?”Anton: Yeah, exactly.Corey: Guess which one of those costs more. It’s… yeah, in the fullness of time, of course, the shitty one cost more because you find out about security breaches on the front page of The New York Times, and no one’s happy, except maybe The Times. But the problem that you hit is that I don’t know how to fix that. I think there’s an opportunity there for some provider—any provider, please—to be a trendsetter, and, “Yeah, we don’t charge for security services on our own stuff just because it’d be believed that should be something that is baked in.” Like, that becomes the narrative of the secure cloud.Anton: What about tiers? What about some kind of a good, better, best, or bronze, gold, platinum, where you have reasonable security, but if you want superior security, you pay money? How do you feel, what’s your gut feel on this approach? Like, I can’t think of example—log analysis. You’re going to get some analytics and you’re going to get fancy ML. Fancy ML costs money; yay, nay?Corey: You’re bringing up an actually really interesting point because I think I’m conflating too many personas at once. Right now, just pulling up last months bill on Google Cloud, it fits in the free tier, but my Cloud Run bill was 13 cents for the month because that’s what runs my snark.cloud URL shortener. And it’s great. And I wound up with—I think my virtual machine costs dozen times that much. I don’t care.Over in AWS-land, I was building out a serverless nonsense thing, my Last Tweet In AWS client, and that cost a few pennies a month all told, plus a whopping 50 cents for a DNS zone. Whatever. But because I was deploying it to all regions and the way that configural evaluations work, my config bill for that was 16 bucks. Now, I don’t actually care about the dollar figures on this. I assure you, you could put zeros on the end of that for days and it doesn’t really move the needle on my business until you get to a very certain number there, and then suddenly, I care a lot.Anton: [laugh]. Yeah.Corey: And large enterprises, this is expected because even the sheer cost of people’s time to go through these things is valuable. What I’m thinking of is almost a hobby-level side project instead, where I’m a student, and I’m learning this in a dorm room or in a bootcamp or in my off hours, or I’m a career switcher and I’m doing this on my own dime out of hours. And I wind up getting smacked with the bill for security services that, for a company, don’t even slightly matter. But for me, they matter, so I’m not going to enable them. And when I transition into the workforce and go somewhere, I’m going to continue to work the same way that I did when I was an independent learner, like, having a wildly generous free tier for small-scale accounts, like, even taking a perspective until you wind up costing, I don’t know, five, ten—whatever it is—thousand dollars a month, none of the security stuff is going to be billable for you because it’s it is not aimed at you and we want you comfortable with and using these things.This is a whole deep dive into the weeds of economics and price-driven behavior and all kinds of other nonsense, but every time I wind up seeing that, like, in my actual production account over at AWS land for The Duckbill Group, all things wrapped up, it’s something like 1100 bucks a month. And over a third of it is monitoring, audit, and observability services, and a few security things as well. And on the one hand, I’m sitting here going, “I don’t see that kind of value coming from it.” Now, the day there’s an incident and I have to look into this, yeah, it’s absolutely going to be worth having, but it’s insurance. But it feels like a disproportionate percentage of it. And maybe I’m just sitting here whining and grousing and I sound like a freeloader who doesn’t want to pay for things, but it’s one of those areas where I would gladly pay more for a just having this be part of the cost and not complain at all about it.Anton: Well, if somebody sells me a thing that costs $1, and then they say, “Want to make it secure?” I say yes, but I’m already suspicious, and they say, “Then it’s going to be 16 bucks.” I’d really freak out because, like, there are certain percentages, certain ratios of the actual thing plus security or a secure version of it; 16x is not the answer expect. 30%, probably still not the answer I expect, frankly. I don’t know. This is, like, an ROI question [crosstalk 00:23:46]—Corey: Let’s also be clear; my usage pattern is really weird. You take a look at most large companies at significant scale, their cloud environments from a billing perspective look an awful lot like a crap ton of instances—or possibly containers running—and smattering of other things. Yeah, you also database and storage being the other two tiers and because of… reasons data transfer loves to show up too, but by and large, everything else was more or less a rounding error. I have remarkably few of those things, just given the weird way that I use services inappropriately, but that is the nature of me, so don’t necessarily take that as being gospel. Like, “Oh, you’ll spend a third of your bill.”Like, I’ve talked to analyst types previously—not you, of course—who will hear a story like this and that suddenly winds up as a headline in some report somewhere. And it’s, “Yeah, if your entire compute is based on Lambda functions and you get no traffic, yeah, you’re going to see some weird distortions in your bill. Welcome to the conversation.” But it’s a problem that I think is going to have to be addressed at some point, especially we talked about earlier, those vendors who are catering to customers who are not born in the cloud, and they start to see their business erode as the cloud-native way of doing things continues to accelerate, I feel like we’re in for a time where they’re going to be coming at the cloud providers and smacking them for this way harder than I am with my, “As a customer, wouldn’t it be nice to have this?” They’re going to turn this into something monstrous. And that’s what it takes, that’s what it takes. But… yeah.Anton: It will take more time than than we think, I think because again, back in the Gartner days, I loved to make predictions. And sometimes—I’ve learned that predictions end up coming true if you’re good, but much later.Corey: I’m learning that myself. I’m about two years away from the end of it because three years ago, I said five years from now, nobody will care about Kubernetes. And I didn’t mean it was going to go away, but I meant that it would slip below the surface level of awareness to point where most people didn’t have to think about it in the same way. And I know it’s going to happen because it’s too complex now and it’s going to be something that just gets handled in the same way that Linux kernels do today, but I think I was aggressive on the timeline. And to be clear, I’ve been misquoted as, “Oh, I don’t think Kubernetes is going to be relevant.”It is, it’s just going to not be something that you need to spend the quarter million bucks an engineer on to run in production safely.Anton: Yeah.Corey: So, we’ll see. I’m curious. One other question I had for you while I’ve got you here is you run a podcast of your own: the Cloud Security Podcast if I’m not mistaken, which is—Anton: Sadly, you are not. [laugh].Corey: —the Cloud Se—yeah. Interesting name on that one, yeah. It’s like what the Cloud Podcast was taken?Anton: Essentially, we had a really cool name [Weather Insecurity 00:26:14]. But the naming team here said, you must be descriptive as everybody else at Google, and we ended up with the name, Cloud Security Podcast. Very, very original.Corey: Naming is challenging. I still maintain that the company is renamed Alphabet, just so it could appear before Amazon in the yellow pages, but I don’t know how accurate that one actually is. Yeah, to be clear, I’m not dunking on your personal fun podcast, for those without context. This is a corporate Google Cloud podcast and if you want to make the argument that I’m punching down by making fun of Google, please, I welcome that debate.Anton: [laugh]. Yes.Corey: I can’t acquire companies as a shortcut to hire people. Yet. I’m sure it’ll happen someday, but I can aspire to that level of budgetary control. So, what are you up to these days? You spent seven years at Gartner and now you’re doing a lot of cloud security… I’ll call it storytelling, and I want to be clear that I mean that as a compliment, not the, “Oh, you just tell stories rather than build things?”Anton: [laugh].Corey: Yeah, it turns out that you have to give people a reason to care about what you’ve built or you don’t have your job for very long. What are you talking about these days? What narratives are you looking at going forward?Anton: So, one of the things that I’ve been obsessed with lately is a lot of people from more traditional companies come in in the cloud with their traditional on-premise knowledge, and they’re trying to do cloud the on-premise way. On our podcast, we do dedicate quite some airtime to people who do cloud as if it were a rented data center, and sometimes we say, the opposite is called—we don’t say cloud-native, I think; we say you’re doing the cloud the cloudy way. So, if you do cloud, the cloudy way, you’re probably doing it right. But if you’re doing the cloud is rented data center, when you copy a security stack, you lift and shift your IDS, and your network capture devices, and your firewalls, and your SIM, you maybe are okay, as a first step. People here used to be a little bit more enraged about it, but to me, we meet customers where they are, but we need to journey with them.Because if all you do is copy your stack—security stack—from a data center to the cloud, you are losing effectiveness, you’re spending money, and you’re making other mistakes. I sometimes joke that you copy mistakes, not just practices. Why copy on-prem mistakes to the cloud? So, that’s been bugging me quite a bit and I’m trying to tell stories to guide people out of a situation. Not away, but out.Corey: A lot of people don’t go for the idea of the lift and shift migration and they say that it’s a terrible pattern and it causes all kinds of problems. And they’re right. The counterpoint is that it’s basically the second-worst approach and everything else seems to tie itself for first place. I don’t mean to sound like I’m trying to pick a fight on these things, but we’re going to rebuild an application while we move it. Great.Then it doesn’t work or worse works intermittently and you have no idea whether it’s the rewrite, the cloud provider, or something else you haven’t considered. It just sounds like a recipe for disaster.Anton: For sure. And so, imagine that you’re moving the app, you’re doing cut-and-paste to the cloud of the application, and then you cut-and-paste security, and then you end up with sizeable storage costs, possibly egress costs, possibly mistakes you used to make beyond five firewalls, now you make this mistake straight on the edge. Well, not on the edge edge, but on the edge of the public internet. So, some of the mistakes do become worse when you copy them from the data center to the cloud. So, we do need to, kind of, help people to get out of the situation but not by telling them don’t do it because they will do it. We need to tell them what step B; what’s step 1.5 out of this?Corey: And cost doesn’t drive it and security doesn’t drive it. Those are trailing functions. It has to be a capability story. It has to be about improving feature velocity or it does not get done. I have learned this the painful way.Anton: Whatever 10x cost if you do something in the data center-ish way in the cloud, and you’re ten times more expensive, cost will drive it.Corey: To an extent, yes. However, the problem is that companies are looking at this from the perspective of okay, we can cut our costs by 90% if we make these changes. Okay, great. It cuts the cloud infrastructure cost that way. What is the engineering time, what is the opportunity cost that they gets baked into that, and what are the other strategic priorities that team has been tasked with this year? It has to go along for the ride with a redesign that unlocks additional capability because a pure cost savings play is something I have almost never found to be an argument that carries the day.There are always exceptions, to be clear, but the general case I found is that when companies get really focused on cost-cutting, rather than expanding into new markets, on some level, it feels like they are not in the best of health, corporately speaking. I mean, there’s a reason I’m talking about cost optimization for what I do and not cost-cutting.It’s not about lowering the bill to zero at all cost. “Cool. Turn everything off. Your bill drops to zero.” “Oh, you don’t have a company anymore? Okay, so there’s a constraint. Let’s talk more about that.” Companies are optimized to increase revenue as opposed to reduce costs. And engineers are always more expensive than the cloud provider resources they’re using, unless you’ve done something horrifying.Anton: And some people did, by replicating their mistakes for their inefficient data centers straight into the cloud, occasionally, yeah. But you’re right, yeah. It costs the—we had the same pattern of Gartner. It’s like, it’s not about doing cheaper in the cloud.Corey: I really want to thank you for spending so much time talking to me. If people want to learn more about what you’re up to, how you view the world, and what you’re up to next, where’s the best place for them to find you?Anton: At this point, it’s probably easiest to find me on Twitter. I was about to say Podcast, I was about to say my Medium blog, but frankly, all of it kind of goes into Twitter at some point. And so, I think I am twitter.com/anton_chuvakin, if I recall correctly. Sorry, I haven’t really—Corey: You are indeed. It’s always great; it’s one of those that you have a sizable audience, and you’re like, “What is my Twitter handle, again? That’s a good question. I don’t know.” And it’s your name. Great. Cool. “So, you’re going to spell that for you, too, while you’re at it?” We will, of course, put a link to that in the [show notes 00:32:09]. I really want to thank you for being so generous with your time. I appreciate it.Anton: Perfect. Thank you. It was fun.Corey: Anton Chuvakin, Security Strategy Something at Google Cloud. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment because people are doing it wrong, but also tell me which legacy vendor you work for.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About Anadelia Anadelia is a B2B marketing leader passionate about building tech brands and growing revenue. She is currently the Sr. Director of Demand Generation at Teleport. In her spare time she enjoys live music and craft beer.Links Referenced: Teleport: https://goteleport.com/ @anadeliafadeev: https://twitter.com/anadeliafadeev LinkedIn: https://www.linkedin.com/in/anadeliafadeev/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: Let’s face it, on-call firefighting at 2am is stressful! So there’s good news and there’s bad news. The bad news is that you probably can’t prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. This may surprise some of you to realize, but every once in a while, I mention how these episodes are sponsored by different companies. Well, to peel back a little bit of the mystery behind that curtain, I should probably inform some of you that when I say that, that means that companies have paid me to talk about them. I know, shocking.This is a revelation that will topple the podcast industry if it gets out. That’s why it’s just between us. My guest today knows this better than most. Anadelia Fadeev is the Senior Director of Demand Generation at Teleport, who does in fact sponsor a number of different things that I do, but this is not a sponsored episode in that context. Anadelia, thank you for joining me today.Anadelia: Thank you for having me.Corey: It’s interesting. I always have to double-check where it is that you happen to be working because when we first met you were a Senior Marketing Manager, also in Demand Gen, at InfluxData, then you were a Director of Demand Generation at LightStep, and then you became a Director of Demand Gen and Growth and then a Senior Director of Demand Gen, where you are now at Teleport. And the couple of things that I’ve noticed are, one, you seem to more or less be not only doing the same role, but advancing within it, and also—selfishly—it turns out that every time you wind up working somewhere, that company winds up sponsoring some of my nonsense. So first, thank you for your business. It’s always appreciated. Now, what is demand gen exactly? Because I have to say, when I started podcasting and newslettering and shooting my mouth off on the internet, I had no clue.Anadelia: [laugh]. Well, to put it very simply, demand generation, our goal is to drive awareness and interest in your products or services. It’s as simple as that. Now, how we do that, we could definitely dive into the specifics, but it’s all about generating awareness and interest. Especially when you work for an early-stage startup, it’s all about awareness, right? Just getting your name out there.Corey: Marketing is one of those things that I suspect in some ways is kind of like engineering, where you take a look at, “Oh, what do you do? I’m a software engineer.” Okay, great. For someone who is in that space, does that mean front-end? Does that mean back-end? Does that mean security? Oh, wait, you’re crying and awake at weird hours and you’re angry all the time. You’re a DevOps, aren’t you?And you start to realize that there are these breakdowns within engineering. And we realize this and we get offended when people in some cases miscategorize us as, “I am not that kind of engineer. How dare you?” Which I think is unwarranted and ridiculous, but it also sort of slips under our notice in the engineering space that marketing is every bit as divided into different functions, different roles, and the rest. For those of us who think of marketing in the naive approach, like I did when I started this place—“Oh, marketing. So basically, you do Super Bowl ads, right?” And it turns out, there might be more than one or two facets to marketing. What’s your journey been like in the wide world of marketing? Where did you start? Where does it stop?Anadelia: Yeah. I have not gotten to the Super Bowl ads phase yet but on my way there. No, but when you think about the different core areas within marketing, right, you have your product marketing team, and this is the team that sets the positioning, the messaging, and the information about who your ideal audience is, what pain points are they having, and how is your product solving those pain points? Right, so they sort of set the direction for the rest of the team, you have another core function, which is the content team, right? So, with the direction from Product Marketing, now that we know what the pain points are and what our value prop for our product is, how do we tell that to the world in a compelling way, right? So, this is where content marketing really comes into play.And then you have your demand generation teams. And some companies might call it growth or revenue or… I guess those two are the ones that come to mind. But this team is taking the direction from Product Marketing, taking the content produced by the content team, and then just making sure that people actually see it, right? And across all those teams, you have a lot of support from operations making sure that there’s processes and systems in place to support all of those marketing efforts, you have teams that help support web development and design, and brand.Corey: One of the challenges that I think people have when they don’t really understand what marketing is they think back on what they know—maybe they’ve seen Mad Men, which to my understanding does not much resemble modern all workplaces, but then again, I’ve been on my own for five years, so one wonders—and they also see things in the context of companies that are targeting more mass-market, in some respects. If you’re trying to advertise Coca-Cola, every person on the planet—give or take—knows what Coca-Cola is. And the job is just to resurface it, on some level, in people’s awareness, so the correct marketing answer there apparently, is to slap the logo on a bunch of things, be it a stadium, be it a billboard, be it almost anything, whereas when we’re talking about earlier stage companies—oh, I don’t know Teleport, for example—if you were to slap the Teleport logo on a stadium somewhere for some sports game, I have the impression that most people looking at that, if they notice it at all, would instead respond to some level of confusion of, “Teleport, what is that exactly? Have scientists cracked the way of getting me to Miami from San Francisco in less than ten seconds? Because I feel like I would have heard about that.”There’s a matter of targeting beyond just the general public or human beings walking around and starting to target people who might have a problem that you know how to solve. And then, of course, figuring out where those people are gathering and how to get in front of them in a way that resonates instead of being annoying. At least that has been my lived experience of watching the challenges that marketing people have talked to me about over the years. Is that directionally correct or are they all just shining me on and, like, “Oh, Corey, you’re adorable, you almost understand how this stuff works. Now, go insult some more things on Twitter. It’ll be fine.”Anadelia: [laugh]. The reality is that advertising is a big part of a demand generation program, but it’s not all, right? So, good demand generation is meeting people where they are. So, the right channels, the right mediums, the right physical places. So, when you look at it from an inbound and outbound approach, inbound, you have a sign outside of your door inviting people to your house, right, and this is in the form of your website. And outbound is you go out to where people are and you knock on their door to introduce yourself.So, when we look at it from that approach, so on the inbound side, right, the goal is to get people to come to your website because that is where you are telling them what you do and giving them the option to start using your product. So, what reason are you giving people to come to you, right? How are you helping them become better at something or achieve certain results, right? So, understanding the motivations behind it is extremely important.And how are you driving people to you? Well, that’s where SEO comes in, right? Search engine optimization.So, what content are you producing that is driving the right search results to get your website to show up and get people to come to you, right? There’s also SEM or Search Engine Marketing. So, when people are searching for certain keywords that are relevant to you, are you showing up in those search results?And on the outbound side of things is, what do you do to contribute to existing communities, right? So, this is where things like advertising comes into play. So, I know you have a huge following and I want to be where you are. So, of course, I’m going to sponsor your podcast and your newsletters. And similarly, I’m looking for what events are out there where I know that our potential customers are spending their time and what can we do to join that conversation in a way that adds value?So, that can be in the form of supporting community events and meetups, giving community members a platform to share their experiences, and even supporting local businesses, right, it’s all about adding value, and by doing so, you are building trust that will allow you to then talk about how your product can help these communities solve their problems.Corey: It’s interesting because when we look at the places that you have been, you were at InfluxData, they are a time-series database company; you were at LightStep, which was effectively an observability company, and now you’re at Teleport where you are an authentication and access company. And forgive me, none of these are your terms. These are my understandings of having talked to these folks. And on the one hand, from a product perspective, it sounds like you’re hopping between this and that and doing all those other things, and yet, we had conversations about all three of those products and how the companies around them are structured and built, and you’ve advertised all three of those on this show and others and all three of those companies and products speak specifically to problems that I have dealt with personally in the way I go through my engineering existence as well. So, instead of specializing on a particular product or on a particular niche, it almost feels like you’re specializing on a particular audience. Is that how you think about it, or is that just one of those happy accident, or in retrospect, we’re just going to retcon everything, and, “Yeah, that’s exactly why I did it.” And you’re like, “Let we jot that down. That belongs on my resume somewhere.”Anadelia: [laugh]. No, so prior to me joining InfluxData, I was at other companies that were marketing to sales, HR, finance, different audiences, right? And the moment I joined Influx, it was really eye-opening for me to be part of a product that has an open-source community, and between that and marketing to a highly technical audience that probably very likely doesn’t want to hear from marketers, I found that to be a really good challenge for myself because it challenged me to elevate my own technical knowledge. And also personally, I just want to be surrounded by people that are smarter than me, and so I know that by being part of a community that markets to a developer audience, I am putting myself in a position where I’m having to constantly continue to learn. So, it’s a good challenge for a marketer in our industry. Just like in any others, there’s always the latest buzzword or the latest trend, and so it’s really easy to get caught up in those things. And I think that being a marketer whose audience is developers really forces you to kind of look at what you’re doing and sort of remove the fluff. This happens everywhere.Corey: Well, I have to be careful about selling yourself too short on this because I’ve talked to a lot of different people who want to wind up promoting what it is that their companies do, and people come from all kinds of different places, and some of the less likely to be successful—in many cases, I turn the business down—are, “Well, this is our first real experience with marketing.” And the reason for that is people expect unrealistic things. I describe what I do as top-of-funnel where we get people’s attention and we give them a glimpse and a hook of what it is the product does. And I do that by talking about the painful problem that the product solves. So, when people hear their pain reflected in what we talk about, then that gives them the little bit of a push to go and take a look and see if this solves it.And that’s great, but there has to be a process on the other side, where oh, a prospect comes in and starts looking at what it is we do. Do we have a sales funnel that moves them from someone just idly browsing to someone who might sign up for a trial, or try this in their own time, or start to understand how the community views it and the rest because just dropping a bunch of traffic on someone’s website doesn’t, in isolation, achieve anything without a means to convert that traffic into something that’s a bit more meaningful and material to the business? I’ve talked to other folks who are big on oh, well, we want to wind up just instrument in the living crap out of everything we put out there, so I want to know, when someone clicks on the ad, who they are, what they do for a living, what their signing authority is, et cetera, et cetera, et cetera. And my answer, that’s super easy, “Cool. We don’t do any of that.”Part of the reason that people like hearing from me, is because I generally tend to respect their time, I’m not supporting invasive tracking of what they do, they don’t see my dumb face smiling with a open mouth grin as they travel across the internet on every property. Although one of these days I will see myself on the side of a bus; I’m just waiting for it. And it’s really nice to be able to talk to people who get the nuances and the peculiarities of the audience that I tend to speak to the most. You’ve always had that unlocked, even since our first conversation.Anadelia: Yeah, well, first of all, thank you. And yeah, the reality is that, especially within my world, right—and demand generation, we are very metrics-driven because our goal [tends 00:13:00] to be pipeline, right? Pipeline for the sales team, so we want to generate sales opportunities, and in order to do that, we need to be able to measure what’s working and what is not working. But the reality is that good marketing is all about building trust, right? So, that’s why I stress the importance of providing something of value to your prospect so that you’re not wasting their time, right? The message that you have for them is something that can help them in the future.And if building trust sometimes means I’m not able to measure the direct results of the activity that you’re doing, then that is okay, right? Because when you’re driving people to your website, there are things that you can measure, like, you have some web visits, and you know that percentage of those visitors might be interested in continue further, right? So, when you look at the journey across the buyer stages, you have to have a compelling offer for a person on each of the possible stages, right? So, if they are just learning about you today because this is the first time that heard your ad, it’s probably not expected that they would immediately go to your website and fill out your form, right? They’ve just heard about you, and now you start building that recognition.Now, if all the stars align, and I actually have a need for a solution that’s like yours today, then, of course, you can expect a conversion to happen in that time point. But the reality is that having offers that are aimed at every stage of the buyer's journey is important.Corey: I’m glad to hear you say this. And the reason is that I often feel like when I say it, it sounds incredibly self-serving. But if you imagine the ideal buyer and their journey, they have the exact problem that your product does and there’s an ad on my podcast that mentions it. Well, I imagine—and maybe this isn’t accurate, but it’s how I engage with podcasts myself—I’m probably not sitting in front of a computer ready to type in whatever it is that gets talked about.I’m probably doing dishes or outside harassing a dog or something. And if it resonates is, “Oh, I should look into that.” In an ideal world. I’ll remember the short URL that I can go to, but in practice, I might just Google the company name. And oh, this does solve the problem.If it’s not just me and there’s a team I have to have a buy-in on, I might very well mention it in our next group meeting. And, “Okay, we’re going to go ahead and try it out with an open-source version or whatnot.” And, “Oh, this seems to be working. We’ll have procurement reach out and see what it takes to wind up generating a longer-term deal.” And the original attribution of the engineer who heard it on a podcast, or the DevOps director who read it in my newsletter, or whatever it is, is long since lost. I’ve commiserated with marketing people over this, and the adage that I picked up that I love quoting is half your marketing budget is wasted, but you can spend an entire career trying to figure out which half and get nowhere by the end of it.Anadelia: And this sort of touches on the buyer's journey is not linear. On the other side of that ad, or that marketing offer is a human, right? So, of course, as marketers, we’re going to try to build this path of once you landed on our website, we want to guide you through all the steps until you do the thing that we want you to do, but the reality is, that does not happen in your example, right? You see something, you come back to it later through another channel, there’s no way for us to measure those. And that’s okay because that’s just the reality of how humans behave.And also, I think it’s worth noting that it takes multiple touch points until a person is ready to even hear what you have to say, right? And it sort of goes back to that point of building trust, right? It takes many times until you’ve gained that person’s trust enough for them to listen to what you have to say.Corey: Building trust is important.Anadelia: SIt is very important. And that’s why I think that running brand awareness programs are an extremely important part of a marketing mix. And sometimes there’s not going to be any direct attribution, and we just have to be okay with it.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it’s still somebody’s problem. And a big part of that responsibility is app security from code to cloud. And that’s where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you’re actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That’s S-N-Y-K.co/screamCorey: I tend to take a perspective that trust is paramount, on some level, where we have our standard rules of, you know, don’t break the law, et cetera, et cetera, that we do require our sponsors to conform to, but there are really two rules that I have that I care about. The first is you’re not allowed to lie to the audience. Because if I wind up saying something is true in an ad or whatnot, and it’s not, that damages my credibility. And I take this old world approach of, well, I believe trust is built over time, and you continually demonstrate a pattern of doing the right thing, and people eventually are willing to extend a little bit of credulousness when you say something that sounds that might be a little bit beyond their experience.The other is, and this is very nebulous, and difficult to define so I don’t think we even have this in writing, but you have to be able to convince me if you’re going to advertise something in one of my shows, that it will not, when used as directed, leave the user worse off than they were when they started. And that is a very strange thing. Like, a security product that has a bunch of typos on its page and is rolling its own crypto, for example—if you want an easy example—is one of those things that I will very gracefully decline not to wind up engaging with, just because I have the sneaking suspicion that if you trust that thing, you might very well live to regret it. In other cases, though—and this is almost never a problem because most companies that you have heard of and have established themselves as brands in this space already instinctively get that you’re not able to build a lasting business by lying to people and then ripping them off.So, it’s a relatively straightforward approach, but every once in a while, I see something that makes me raise an eyebrow. And it’s not always bad. Sometimes I think that’s a little odd. Teleport is a good example of this because, “Oh, really? You wound up doing access and authentication? That sounds exactly like the kind of thing I want something old and boring, not new and exciting, around, so let’s dig into this and figure out whether this might be the one company you work at that doesn’t get to sponsor stuff that I do.”But of course you do. You’re absolutely focusing on an area that is relevant, useful, and having talked to people on your side of the world, you’re doing the right thing. And okay, I would absolutely not be opposed to deploying this in the right production environment. But having that credulousness, having that exploratory conversation, makes it clear that I’m talking to people who know what they’re doing and not effectively shilling for the highest bidder, which is not really a position I ever want to find myself in.Anadelia: And look, you have only one opportunity to make a first impression, right? So, being clear about what it is that you can do, and also being clear about what it is that you cannot do is extremely important, right? It kind of goes back to the point of just be a good human, don’t waste people’s time. You want to provide something of value to your audience. And so, setting those expectations early on is extremely important.And I don’t know anyone that does this, but if your goal is only to drive people to your website, you can do that, probably very easily, but nothing will come out of it unless you have the right message.Corey: Oh, all you do is write something incendiary and offensive, and you’ll have a lot of traffic. They won’t buy anything and they’ll hate you, but you’ll get traffic, so maybe you want to be a little bit more intentional. It’s the same reason that the companies that advertise on what I do pick me to advertise with as opposed to other things. It is more expensive than the mass-market podcasts and whatnot that speak to everyone. But you take a look at those podcasts and the things that they’re advertising are things that actually apply to an awful lot more people, things like mattresses, and click-and-design website services, and the baseline stuff that a lot of people would be interested in, whereas the things that advertise on what I do tend to look a lot more like B2B SaaS companies where they’re talking to folks who spend a lot of time working in cloud computing.And one of the weird things to think about from that perspective, at least for me, is if one person is listening to a show that I’m putting out and they go through the journey and become a customer, well, at the size of some of these B2B contracts between large companies, that one customer has basically paid for everything I can sell for advertising for the next decade and change, just because the long-term value of some of these customers is enormous. But it’s why, for example—and I kept expecting it to happen, but it didn’t—I’ve never been subjected to outreach from the mattress companies of, “Hey, you want to go talk about that to your guests?” No, because for those folks, it is pure raw numbers: how many millions of subscribers do you have? Here, it’s—the newsletter is the easy one to get numbers on because lies, damned lies, and podcast statistics. I have 31,000 people that receive emails. Great, that’s not the biggest newsletter in the world by a longshot, but the people who are the type of person to sign up for cloud computing-style newsletters, that alone says something very specific about them and it doesn’t require anyone do anything creepy to wind up reaching out from that perspective.It doesn’t require spying on customers to intuit that, hmm, maybe people who care about what AWS is up to and have big AWS-sized problems might sign up to a newsletter called Last Week in AWS. That’s the sort of easy thinking about advertising that I tend to go for, which yeah, admittedly sounds a lot like something out of that Mad Men era. But I think that we got a lot right back then, and everything’s new all the time.Anadelia: [laugh]. And actually, that’s exactly what demand generation is, right? We want to find the right channels to reach our audience. And so, for a consumer company that sells mattresses, right, anyone might be on the market for a mattress, right? You want to go as broad as possible. But for something that’s more specific, you want to find what are the right channels to reach that audience where you know that there’s—it might be a smaller audience size, but it’s the right people.And we’ve talked about the other core areas of marketing. So, with demand generation, it’s all about finding people where they are, right, and providing them their message to you and attracting them to come to you, right? It kind of goes back to that inbound and outbound motion that I mentioned earlier. But at the end of the day also, if you don’t have the right messaging to keep them engaged, once you got them to your website, then that’s a different problem, right? So, demand gen alone cannot be successful without really strong product marketing and without really strong content, and everything else that’s needed to support that, right? I mentioned the—if your website is not loading fast enough, then you’re losing people if your form is not working. So, there’s so many, so many different factors that come into play.Corey: Oh, God, the forms. Don’t get me started on the forms. Hey, we have a great report that’s super useful. Okay, cool. I’ll click the link and I’ll follow that. I talk to sponsors about this all the time. And it’s, you have 30 mandatory fields on that website that I need to fill out. I am never going to do that.What is the absolute bare minimum that you need in an ideal world? Don’t put any sort of gateway in front of it and just make it that good that I will reach out to thank you for it or something, but just make it an email address or something and that’s it. You don’t need to know the size of my company, the industry we’re in, the level of my signing authority, et cetera, et cetera, et cetera. Because if this is good, I might very well be in touch. And if it’s not, all you’re going to do is harass me forever with pointless calls and emails and whatnot, and I don’t want to deal with that. There’s something to be said for adding value early in the conversation and letting other people sometimes make the first move. But this is also, to be clear, a very inbound type of approach.Anadelia: It’s a never-ending debate, to gate or not to gate. And I don’t know if there is a right answer. My approach is that if your content is good, people will come back to you. They’ll keep coming back, and they’ll want to take the next step with you. And so, I have some gated assets, and I have some that are not, and—but—Corey: But your gates have also never been annoying of the type that I’m talking about where it’s the, “Oh, great. You need to, like, put in, like, how big is your company? What’s the budget?” It feels like I’m answering a survey at some point. AWS is notorious for this.I counted once; there are 19 mandatory fields I had to fill out in order to watch a webinar that AWS was putting on.Anadelia: [laugh].Corey: And the worst part is they asked me the same questions every time I want to watch a different webinar. It’s like, for a company that says the data is so valuable, you’d really think they’d be better at managing it.Anadelia: You know, like, some of the questions keep getting stranger. Like, I would not be surprised if people start asking what’s your favorite color, or what’s the answer to your—Corey: The one they always ask now for, like, big data seminars and whatnot, is where this really gets me, is this in relation to your professional interests or your personal interests? It’s… “What do you think my hobbies are over there? Oh, yeah, I like big enterprise software. That’s my hobby.” “Okay, I guess.” But I really do wonder what happens if someone checks the personal interest [vibe 00:25:33]. Do they wind up just with various AWS employees showing up want to hang out on the weekends and go surfing or something? I don’t know.Anadelia: As somebody who has been on the receiving end of lists like this—for example, we sponsor a conference and we get people stop by to talk to us, and now we get the list of those people. And there’s 25 columns. Like, honestly, that data does not come in helpful because at the end of the day, whatever you’ve marked on the required question is not going to change how I am going to communicate to you after, right, because we just had a conversation in person at this event.Corey: My budget is not material to the reason I let you scan my badge. The reason I let you scan my badge because I really wanted one of those fun plastic toy things, so I waited in line for 45 minutes to get it. But that doesn’t mean that I’m going to be a buyer; it just means that now I’m in your funnel, although I could not possibly care less about what you do. One thing I do at re:Invent and a couple other conferences, for example, is I will have swag at a booth—because I don’t tend to get booths myself, I don’t have the staff to man it and I’m bad at that type of thing. But when people come up to get a sticker for Last Week in AWS or when of our data transfer diagram things or whatnot, the rule that we’ve always put in place is, you’re not going to mandate a badge scan for that.And the kind of company I like doing that with gets it because the people who walk by and are interested will say, “Hey, can you scan my badge as well?” But they don’t want to pollute their own lead lists with a bunch of people who are only there to get a sticker featuring a sarcastic platypus, as opposed to getting them confused with people actually care about what it is that they’re solving for. And that’s a delicate balance to strike sometimes, but the nice thing about being me is I have customers who come back again and again and again. Although I will argue that I probably got better at being a service provider when I started also being a customer at the same time, where I hired out a marketing department here because it turns out that fixing the AWS bill is something that does a fair bit of marketing work. It’s not something people talk about at large scale in public, so you have to be noisy enough so that inbound finds its path to you a bunch of times. That’s always tricky.And learning about how no matter what it is you do, in the case of my consulting work, we are quite honestly selling money, bring us in for an engagement, you will turn a profit on that engagement and we don’t come back with a whole bunch of extra add-ons after the fact to basically claw back more things. It’s one of the easiest sales in the world. And it’s still nuanced, and challenging, and finding the right way to talk about it to the right people at the right time explains why marketing is the industry that it is. It’s hard. None of this is easy.Anadelia: It is. And you know, in your example, you’re not scanning that badge, but giving the person the sticker, right? Like, it’s all about making a good first impression, and if the person’s not ready to talk to you, that is okay. But there are ways that you can stay top-of-mind so that the moment that they have a need, they’ll come to you. It kind of goes back again to my earlier points of adding value in supporting existing communities, right? So, what are you doing to stay top-of-mind with that person that wasn’t quite ready back then, but the moment they have a need, they’ll think of you first because you made a good first impression.Corey: And that’s really what it comes down to. It’s nice to talk to people who actually work in marketing because a lot of what I do in the marketing space, I’ve got to be honest, is terrible. Because I’ve done the old engineering thing of, well, I’m no marketer, but I know how to write code, so how hard could marketing really be and I invent this theory of marketing from first principles, which not only is mostly wrong, but also has a way of being incredibly insulting to people who have actually made this their profession and excel at it. But it’s an evolutionary process and trying to figure out the right way to do things and how to think about things from particular point of view has been transformative. Really easy example of this: when I first started selling sponsorships, I was constantly worried that a sponsor was going to reach out and say, “Well, hang on a second. We didn’t get the number of clicks that we expected to on this campaign. What do you have to say about that?”Because I’m a consultant. I am used to clients not getting results that they expected having some harsh words for me. In practice, I don’t believe I’ve ever had a deep conversation about that with a marketing person. I’ve talked to them and they’ve said, “Well, some of these things worked. Some of these things didn’t. Here’s what works; here’s what didn’t, and for our next round, here’s what we want to try instead.” Those are the great constructive conversations.The ones that I was fearing somehow would assume that I held this iron grip of control over exactly how many people would be clicking on a thing in a newsletter, and I’m not. We barely provide click-tracking at this point in the aggregate, let alone anything more specific, just because it’s so hard to actually tell and get value out of it. You talk as well, about there being brand awareness. Even if someone doesn’t click an ad, they’re potentially reading it, they’re starting to associate your company with the problem space. That’s one of those things that are effectively impossible to track, but it does pay dividends.When you suddenly have a problem in a particular area. And there’s one or two companies off the top of your mind that you know work in that space. Well, what do you think marketing is? There has been huge money put into making that association in your mind. It’s not just about click the link; it’s not just about buy the thing; it’s about shaping the way that we think about different things.Anadelia: And I spend a lot of time thinking about how people think we talk about what are the things that motivate you. When you have a problem, where do you go to look for a solution, or who do you go to, right? So, just understanding what the thought process is when someone is trying to solve a problem or making a purchasing decision, I think that a lot of demand generation is what are the different ways by which someone is trying to solve a problem that they’re having? And I had an interest in psychology growing up; both my parents are psychologists, and I think that marketing tends to bring some aspects of that in business and creativity, which is what led me to a career in marketing.And you ended up being sort of a connector, right? Like your job was to connect to people who would benefit from meeting each other. Just one of them happens to be a product, or you know, it depends on your company, right, but you’re just introducing people and making sure they know about each other because there’s going to be a mutually beneficial relationship between them.Corey: That seems to be what so many jobs ultimately distilled down to in the final analysis of things. I really want to thank you for being so generous with your time and talking about how you view the world slash industry in which we live. If people want to learn more about what you’re up to and how you think about these things, where’s the best place to find you?Anadelia: You can follow me on Twitter at @anadeliafadeev, or connect with me on LinkedIn.Corey: Oh, you’re one of the LinkedIn peoples. I used to do that a bit, and then I just started getting deluged with all kinds of nonsense, and let me adjust my notification settings, and there are 600 of them. And no, no, no, no, no. And I basically have quit the field, by and large, on LinkedIn. But power to you for not having done that. Links to that will of course be in the [show notes 00:32:38]. Thank you so much for being so generous with your time.Anadelia: Thank you for having me. I appreciate it.Corey: Anadelia Fadeev, Senior Director of Demand Generation at Teleport. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry ranting comment about how we got it completely wrong and that marketing does not work on you in the least. And by the way, when you close out that ranting comment, tell me what kind of brand of shoes you’re wearing today.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About JeffJeff Smith has been in the technology industry for over 20 years, oscillating between management and individual contributor. Jeff currently serves as the Director of Production Operations for Basis Technologies (formerly Centro), an advertising software company headquartered in Chicago, Illinois. Before that he served as the Manager of Site Reliability Engineering at Grubhub.Jeff is passionate about DevOps transformations in organizations large and small, with a particular interest in the psychological aspects of problems in companies. He lives in Chicago with his wife Stephanie and their two kids Ella and Xander.Jeff is also the author of Operations Anti-Patterns, DevOps Solutions with Manning publishing. (https://www.manning.com/books/operations-anti-patterns-devops-solutions) Links Referenced: Basis Technologies: https://basis.net/ Operations Anti-Patterns: https://attainabledevops.com/book Personal Site: https://attainabledevops.com LinkedIn: https://www.linkedin.com/in/jeffery-smith-devops/ Twitter: https://twitter.com/DarkAndNerdy Medium: https://medium.com/@jefferysmith duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored by our friends at Fortinet. Fortinet’s partnership with AWS is a better-together combination that ensures your workloads on AWS are protected by best-in-class security solutions powered by comprehensive threat intelligence and more than 20 years of cybersecurity experience. Integrations with key AWS services simplify security management, ensure full visibility across environments, and provide broad protection across your workloads and applications. Visit them at AWS re:Inforce to see the latest trends in cybersecurity on July 25-26 at the Boston Convention Center. Just go over to the Fortinet booth and tell them Corey Quinn sent you and watch for the flinch. My thanks again to my friends at Fortinet.Corey: Let’s face it, on-call firefighting at 2am is stressful! So there’s good news and there’s bad news. The bad news is that you probably can’t prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. One of the fun things about doing this show for long enough is that you eventually get to catch up with people and follow up on previous conversations that you’ve had. Many years ago—which sounds like I’m being sarcastic, but is increasingly actually true—Jeff Smith was on the show talking about a book that was about to release. Well, time has passed and things have changed. And Jeff Smith is back once again. He’s the Director of Product Operations at Basis Technologies, and the author of DevOps Anti-Patterns? Or what was the actual title of the book it was—Jeff: Operations Anti-Patterns.Corey: I got hung up in the anti-patterns part because it’s amazing. I love the title.Jeff: Yeah, Operations Anti-Patterns, DevOps Solutions.Corey: Got you. Usually in my experience, alway been operations anti-patterns, and here I am to make them worse, probably by doing something like using DNS as a database or some godforsaken thing. But you were talking about the book aspirationally a few years ago, and now it’s published and it has been sent out to the world. And it went well enough that they translated it to Japanese, I believe, and it has seen significant uptick. What was your experience of it? How did it go?Jeff: You know, it was a great experience. This is definitely the first book that I’ve written. And the Manning process was extremely smooth. You know, they sort of hold your hand through the entire process. But even after launch, just getting feedback from readers and hearing how it resonated with folks was extremely powerful.I was surprised to find out that they turned it into an audiobook as well. So, everyone reaches out and says, “Did you read the audiobook? I was going to buy it, but I wasn’t sure.” I was like, “No, unfortunately, I don’t read it.” But you know, still cool to have it out there.Corey: My theory has been for a while now that no one wants to actually write a book; they want to have written a book. Now that you’re on the other side, how accurate is that? Are you in a position of, “Wow, sure glad that’s done?” Or are you, “That was fun. Let’s do it again because I like being sad all the time.” I mean, you do work Kubernetes for God’s sake. I mean, there’s a bit of masochism inherent to all of us in this space.Jeff: Yeah. Kubernetes makes me cry a little bit more than the writing process. But it’s one of the things when you look back on it, you’re like, “Wow, that was fun,” but not in the heat of the moment, right? So, I totally agree with the sentiment that people want to have written a book but not actually gone through the process. And that’s evident by the fact that how many people try to start a book on their own without a publisher behind them, and they end up writing it for 15 years. The process is pretty grueling. The feedback is intense at first, but you start to get into a groove and you—I could see, you know, in a little while wanting to write another book. So, I can see the appeal.Corey: And the last time you were on the show, I didn’t really bother to go in a particular topical direction because, what’s the point? It didn’t really seem like it was a top-of-mind issue to really bring up because what’s it matter; it’s a small percentage of the workforce. Now I feel like talking about remote work is suddenly taking on a bit of a different sheen than it was before the dark times arrived. Where do you land on the broad spectrum of opinions around the idea of remote work, given that you have specialized in anti-patterns, and well, as sarcastic as I am, I tend to look at almost every place I’ve ever worked is expressing different anti-patterns from time to time. So, where do you land on the topic?Jeff: So, it’s funny, I started as a staunch office supporter, right? I like being in the office. I like collaborating in person; I thought we were way more productive. Since the pandemic, all of us are forced into remote work, I’ve hired almost half of my team now as remote. And I am somewhat of a convert, but I’m not on the bandwagon of remote work is just as good or is better as in person work.I’ve firmly landed in the camp of remote work is good. It’s got its shortcomings, but it’s worth the trade off. And I think acknowledging what those trade-offs are important to keeping the team afloat. We just recently had a conversation with the team where we were discussing, like, you know, there’s definitely been a drop in productivity over the past six months to a year. And in that conversation, a lot of the things that came up were things that are different remote that were better in person, right, Slack etiquette—which is something, you know, I could talk a little bit about as well—but, you know, Slack etiquette in terms of getting feedback quickly, just the sort of camaraderie and the lack of building that camaraderie with new team members as they come on board and not having those rituals to replace the in-person rituals. But through all that, oddly enough, no one suggested going back into the office. [laugh].Corey: For some strange reason, yeah. I need to be careful what I say here, I want to disclaim the position that I’m in. There is a power imbalance and nothing I say is going to be able to necessarily address that because I own the company and if my team members are listening to this, they’re going to read a lot into what I say that I might not necessarily intend. But The Duckbill Group, since its founding, has been a fully distributed company. My business partner lives in a different state than I do so there’s never been the crappy version of remote, which is, well, we’re all going to be in the same city, except for Theodore. Theodore is going to be timezones away and then wonder why he doesn’t get to participate in some of the conversations where the real decisions get made.Like that’s crappy. I don’t like that striated approach to things. We don’t have many people who are co-located in any real sense, nor have we for the majority of the company’s life. But there are times when I am able to work on a project in a room with one of my colleagues, and things go a lot more smoothly. As much as we want to pretend that video is the same, it quite simply isn’t.It is a somewhat poor substitute for the very high bandwidth of a face-to-face interaction. And yes, I understand this is also a somewhat neurotypical perspective, let’s be clear with that as well, and it’s not for everyone. But I think that for the base case, a lot of the remote work advocates are not being fully, I guess, honest with themselves about some of the shortcomings remote has. That is where I’ve mostly landed on this. Does that generally land with where you are?Jeff: Yeah, that’s exactly where I’m at. I completely agree. And when we take work out of the equation, I think the shortcomings lay themselves bare, right? Like I was having a conversation with a friend and we were like, well, if you had a major breakup, right, I would never be like, “Oh, man. Grab a beer and hop on Zoom,” right? [laugh]. “Let’s talk it out.”No, you’re like, hey, let’s get in person and let’s talk, right? We can do all of that conversation over Zoom, but the magic of being in person and having that personal connection, you know, can’t be replaced. So, you know, if it’s not going to work, commiserating over beers, right? I can’t imagine it’s going to work, diagramming some complex workflows and trying to come to an answer or a solution on that. So again, not to say that, you know, remote work is not valuable, it’s just different.And I think organizations are really going to have to figure out, like, okay, if I want to entice people back into the office, what are the things that I need to do to make this realistic? We’ve opened the floodgates on remote hiring, right, so now it’s like, okay, everyone’s janky office setup needs to get fixed, right? So, I can’t have a scenario where it’s like, “Oh, just point your laptop at the whiteboard, right?” [laugh]. Like that can’t exist, we have to have office spaces that are first-class citizens for our remote counterparts as well.Corey: Right because otherwise, the alternative is, “Great, I expect you to take the home that you pay for and turn it into an area fit for office use. Of course, we’re not going to compensate you for that, despite the fact that, let’s be realistic, rent is often larger than the AWS bill.” Which I know, gasp, I’m as shocked as anyone affected by that, but it’s true. “But oh, you want to work from home? Great. That just means you can work more hours.”I am not of the school of thought where I consider time in the office to be an indicator of anything meaningful. I care if the work gets done and at small-scale, this works. Let me also be clear, we’re an 11-person company. A lot of what I’m talking about simply will not scale to companies that are orders of magnitude larger than this. And from where I sit, that’s okay. It doesn’t need to.Jeff: Right. And I think a lot of the things that you talk about will scale, right? Because in most scenarios, you’re not scaling it organizationally so much as you are with a handful of teams, right? Because when I think about all the different teams I interact with, I never really interact with the organization as a whole, I interact with my little neighborhood in the organization. So, it is definitely something that scales.But again, when it comes to companies, like, enticing people back into the office, now that I’m talking about working from home five days a week, I’ve invested in my home setup. I’ve got the monitor I want, I’ve got the chair that I want, I’ve got the mouse and keyboard that I want. So, you’re going to bring me back to the office so I can have some standard Dell keyboard and mouse with some janky, you know—maybe—21-inch monitor or something like that, right? Like, you really have to decide, like, okay, we’re going to make the office a destination, we’re going to make it where people want to go there where it’s not just even about the collaboration aspect, but people can still work and be effective.And on top of that, I think how we look at what the office delivers is going to change, right? Because now when I go to the office now, I do very little work. It’s connections, right? It’s like, you know, “Oh, I haven’t seen you in forever. Let’s catch up.” And a lot of that stuff is valuable. You know, there’s these hallway conversations that exist that just weren’t happening previously because how do I accidentally bump into you on Slack? [laugh]. Right, it has to be much more it of a—Corey: Right. It takes some contrivance to wind up making that happen. I remember back in the days of working in offices, I remember here in San Francisco where we had unlimited sick time and unlimited PTO, I would often fake a sick day, but just stay home and get work done. Because I knew if I was in the office, I’d be constantly subjected to drive-bys the entire time of just drive-by requests, people stopping by to ask, “Oh, can you just help me with this one thing,” that completely derails my train of thought. Then at the end of the day, they’d tell me, “You seem distractible and you didn’t get a lot of work done.”It’s, “Well, no kidding. Of course not. Are you surprised?” And one of the nice things about starting your own company—because there are a lot of downsides, let me be very clear—one of the nice things is you get to decide how you want to work. And that was a study in, first, amazement, and then frustration.It was, “All right, I just landed a big customer. I’m off to the races and going to take this seriously for a good six to twelve months. Great sky’s the limit, I’m going to do up my home office.” And then you see how little money it takes to have a nice chair, a good standing desk, a monitor that makes sense and you remember fighting tooth-and-nail for nothing that even approached this quality at companies and they acted like it was going to cost them 20-grand. And here, it’s two grand at most, when I decorated this place the first time.And it was… “What the hell?” Like, it feels like the scales fall away from your eyes, and you start seeing things that you didn’t realize were a thing. Now I worry that five years in, there’s no way in the world I’m ever fit to be an employee again, so this is probably the last job I’ll ever have. Just because I’ve basically made myself completely unemployable across six different axes.Jeff: [laugh]. And I think one of the things when it comes to, like, furniture, keyboard, stuff like that, I feel like part of it was just, like, this sort of enforced conformity, right, that the office provided us the ability to do. We can make sure everyone’s got the same monitor, the same keyboard that way, when it breaks, we can replace it easily. In a lot of organizations that I’ve been in, you know, that sort of like, you know, even if it was the same amount or ordering a custom keyboard was a big exception process, right? Like, “Oh, we’ve got to do a whole thing.” And it’s just like, “Well, it doesn’t have to be that complicated.”And like you said, it doesn’t cost much to allow someone to get the tools that they want and prefer and they’re going to be more productive with. But to your point really quickly about work in the office, until the pandemic, I personally didn’t recognize how difficult it actually was to get work done in the office. I don’t think I appreciated it. And now that I’m remote, I’m like, wow, it is so much easier for me to close this door, put my headphones on, mute Slack and go heads down. You know, the only drive-by I’ve got is my wife wondering if I want to go for a walk, and that’s usually a text message that I can ignore and come back to later.Corey: The thing that just continues to be strange for me and breaks in some of the weirdest ways has just been the growing awareness of how much of office life is unnecessary and ridiculous. When you’re in the office every day, you have to find a way to make it work and be productive and you have this passive-aggressive story of this open office, it’s for collaboration purposes. Yeah, I can definitively say that is not true. I had a boss who once told me that there was such benefits to working in an open plan office that if magically it were less expensive to give people individual offices, he would spare the extra expense for open plan. That was the day I learned he would lie to me while looking me in the eye. Because of course you wouldn’t.And it’s for collaboration. Yeah, it means two loud people—often me—are collaborating and everyone else wears noise-canceling headphones trying desperately to get work done, coming in early, hours before everyone else to get things done before people show up and distracted me. What the hell kind of day-to-day work environment is that?Jeff: What’s interesting about that, though, is those same distractions are the things that get cited as being missed from the perspective of the person doing the distracting. So, everyone universally hates that sort of drive-by distractions, but everyone sort of universally misses the ability to say like, “Hey, can I just pull on your ear for a second and get your feedback on this?” Or, “Can we just walk through this really quickly?” That’s the thing that people miss, and I don’t think that they ever connect it to the idea that if you’re not the interruptee, you’re the interruptor, [laugh] and what that might do to someone else’s productivity. So, you would think something like Slack would help with that, but in reality, what ends up happening is if you don’t have proper Slack etiquette, there’s a lot of signals that go out that get misconstrued, misinterpreted, internalized, and then it ends up impacting morale.Corey: And that’s the most painful part of a lot of that too. Is that yeah, I want to go ahead and spend some time doing some nonsense—as one does; imagine that—and I know that if I’m going to go into an office or meet up with my colleagues, okay, that afternoon or that day, yeah, I’m planning that I’m probably not going to get a whole lot of deep coding done. Okay, great. But when that becomes 40 hours a week, well, that’s a challenge. I feel like being full remote doesn’t work out, but also being in the office 40 hours a week also feels a little sadistic, more than almost anything else.I don’t know what the future looks like and I am privileged enough that I don’t have to because we have been full remote the entire time. But what we don’t spend on office space we spend on plane tickets back and forth so people can have meetings. In the before times, we were very good about that. Now it’s, we’re hesitant to do it just because it’s we don’t want people traveling before the feel that it’s safe to do so. We’ve also learned, for example, when dealing with our clients, that we can get an awful lot done without being on site with them and be extraordinarily effective.It was always weird have traveled to some faraway city to meet with the client, and then you’re on a Zoom call from their office with the rest of the team. It’s… I could have done this from my living room.Jeff: Yeah. I find those sorts of hybrid meetings are often worse than if we were all just remote, right? It’s just so much easier because now it’s like, all right, three of us are going to crowd around one person’s laptop, and then all of the things that we want to do to take advantage of being in person are excluding the people that are remote, so you got to do this careful dance. The way we’ve been sort of tackling it so far—and we’re still experimenting—is we’re not requiring anyone to come back into the office, but some people find it useful to go to the office as a change of scenery, to sort of, like break things up from their typical routine, and they like the break and the change. But it’s something that they do sort of ad hoc.So, we’ve got a small group that meets, like, every Thursday, just as a day to sort of go into the office and switch things up. I think the idea of saying everyone has to come into the office two or three days a week is probably broken when there’s no purpose behind it. So, my wife technically should go into the office twice a week, but her entire team is in Europe. [laugh]. So, what point does that make other than I am a body in a chair? So, I think companies are going to have to get flexible with this sort of hybrid environment.But then it makes you wonder, like, is it worth the office space and how many people are actually taking advantage of it when it’s not mandated? We find that our office time centers around some event, right? And that event might be someone in town that’s typically remote. That might be a particular project that we’re working on where we want to get ideas and collaborate and have a workshop. But the idea of just, like, you know, we’re going to systematically require people to be in the office x many days, I don’t see that in our future.Corey: No, and I hope you’re right. But it also feels like a lot of folks are also doing some weird things around the idea of remote such as, “Oh, we’re full remote but we’re going to pay you based upon where you happen to be sitting geographically.” And we find that the way that we’ve done this—and again, I’m not saying there’s a right answer for everyone—but we wind up paying what the value of the work is for us. In many cases, that means that we would be hard-pressed to hire someone in the Bay Area, for example. On the other hand, it means that when we hire people who are in places with relatively low cost of living, they feel like they’ve just hit the lottery, on some level.And yeah, some of them, I guess it does sort of cause a weird imbalance if you’re a large Amazon-scale company where you want to start not disrupting local economies. We’re not hiring that many people, I promise. So, there’s this idea of figuring out how that works out. And then where does the headquarters live? And well, what state laws do we wind up following on what we’re doing? Just seems odd.Jeff: Yeah. So, you know, one thing I wanted to comment on that you’d mentioned earlier, too, was the weird things that people are doing, and organizations are doing with this, sort of, remote work thing, especially the geographic base pay. And you know, a lot of it is, how can we manipulate the situation to better us in a way that sounds good on paper, right? So, it sounds perfectly reasonable. Like, oh, you live in New York, I’m going to pay you in New York rates, right?But, like, you live in Des Moines, so I’m going to pay you Des Moines rates. And on the surface, when you just go you’re like, oh, yeah, that makes sense, but then you think about it, you’re like, “Wait, why does that matter?” Right? And then, like, how do I, as a manager, you know, level that across my employees, right? It’s like, “Oh, so and so is getting paid 30 grand less. Oh, but they live in a cheaper area, right?” I don’t know what your personal situation is, and how much that actually resonates or matters.Corey: Does the value that they provide to your company materially change based upon where they happen to be sitting that week?Jeff: Right, exactly. But it’s a good story that you can tell, it sounds fair at first examination. But then when you start to scratch the surface, you’re like, “Wait a second, this is BS.” So, that’s one thing.Corey: It’s like tipping on some level. If you can’t afford the tip, you can’t afford to eat out. Same story here. If you can’t afford to compensate people the value that they’re worth, you can’t afford to employ people. And figure that out before you wind up disappointing people and possibly becoming today’s Twitter main character.Jeff: Right. And then the state law thing is interesting. You know, when you see states like California adopting laws similar to, like, GDPR. And it’s like, do you have to start planning for the most stringent possibility across every hire just to be safe and to avoid having to have this sort of patchwork of rules and policies based on where someone lives? You might say like, “Okay, Delaware has the most stringent employer law, so we’re going to apply Delaware’s laws across the board.” So, it’ll be interesting to see how that sort of plays out in the long run. Luckily, that’s not a problem I have to solve, but it’ll be interesting to see how it shakes out.Corey: It is something we had to solve. We have an HR consultancy that helps out with a lot of these things, but the short answer is that we make sure that we obey with local laws, but the way that we operate is as if everyone were a San Francisco employee because that is—so far—the locale that, one, I live here, but also of every jurisdiction we’ve looked at in the United States, it tends to have the most advantageous to the employee restrictions and requirements. Like one thing we do is kind of ridiculous—and we have to do for me and one other person, but almost no one else, but we do it for everyone—is we have to provide stipends every month for electricity, for cellphone usage, for internet. They have to be broken out for each one of those categories, so we do 20 bucks a month for each of those. It adds up to 100 bucks, as I recall, and we call it good. And employees say, “Okay. Do we just send you receipts? Please don’t.”I don’t want to look at your cell phone bill. It’s not my business. I don’t want to know. We’re doing this to comply with the law. I mean, if it were up to me, it would be this is ridiculous. Can we just give everyone $100 a month raise and call it good? Nope. The forms must be obeyed. So, all right.We do the same thing with PTO accrual. If you’ve acquired time off and you leave the company, we pay it out. Not every state requires that. But paying for cell phone access and internet access as well, is something Amazon is currently facing a class action about because they didn’t do that for a number of their California employees. And even talking to Amazonians, like, “Well, they did, but you had to jump through a bunch of hoops.”We have the apparatus administratively to handle that in a way that employees don’t. Why on earth would we make them do it unless we didn’t want to pay them? Oh, I think I figured out this sneaky, sneaky plan. I’m not here to build a business by exploiting people. If that’s the only way to succeed, and the business doesn’t deserve to exist. That’s my hot take of the day on that topic.Jeff: No, I totally agree. And what’s interesting is these insidious costs that sneak up that employees tend to discount, like, one thing I always talk about with my team is all that time you’re thinking about a problem at work, right, like when you’re in the shower, when you’re at dinner, when you’re talking it over with your spouse, right? That’s work. That’s work. And it’s work that you’re doing on your time.But we don’t account for it that way because we’re not typing; we’re not writing code. But, like, think about how much more effective as people, as employees, we would be if we had time dedicated to just sit and think, right? If I could just sit and think about a problem without needing to type but just critically think about it. But then it’s like, well, what does that look like in the office, right? If I’m just sitting there in my chair like this, it doesn’t look like I’m doing anything.But that’s so important to be able to, like, break down and digest some of the complex problems that we’re dealing with. And we just sort of write it off, right? So, I’m like, you know, you got to think about how that bleeds into your personal time and take that into account. So yeah, maybe you leave three hours early today, but I guarantee you, you’re going to spend three hours throughout the week thinking about work. It’s the same thing with these cellphone costs that you’re talking about, right? “Oh, I’ve got a cell phone anyways; I’ve got internet anyways.” But still, that’s something that you’re contributing to the business that they’re not on the hook for, so it seems fair that you get compensated for that.Corey: I just think about that stuff all the time from that perspective, and now that I you know, own the place, it’s one of those which pocket of mine does it come out of? But I hold myself to a far higher standard about that stuff than I do the staff, where it’s, for example, I could theoretically justify paying my internet bill here because we have business-class internet and an insane WiFi system because of all of the ridiculous video production I do. Now. It’s like, like, if anyone else on the team was doing this, yes, I will insist we pay it, but for me, it just it feels a little close to the edge. So, it’s one of those areas where I’m very conservative around things like that.The thing that also continues to just vex me, on some level, is this idea that time in a seat is somehow considered work. I’ll never forget one of the last jobs I had before I started this place. My boss walked past me and saw that I was on Reddit. And, “Is that really the best use of your time right now?” May I use the bathroom when I’m done with this, sir?Yeah, of course it is. It sounds ridiculous, but one of the most valuable things I can do for The Duckbill Group now is go on the internet and start shit posting on Twitter, which sounds ridiculous, but it’s also true. There’s a brand awareness story there, on some level. And that’s just wild to me. It’s weird, we start treating people like adults, they start behaving that way. And if you start micromanaging them, they live up or down to the expectations you tend to hold. I’m a big believer in if I have to micromanage someone, I should just do the job myself.Jeff: Yeah. The Reddit story makes me think of, like, how few organizations have systematic ways of getting vital information. So, the first thing I think about is, like, security and security vulnerabilities, right? So, how does Basis Technologies, as an organization, know about these things? Right now, it’s like, well, my team knows because we’re plugged into Reddit and Twitter, right, but if we were gone Basis, right, may not necessarily get that information.So, that’s something we’re trying to correct, but it just sort of highlights the importance of freedom for these employees, right? Because yeah, I’m on Reddit, but I’m on /r/sysadmin. I’m on /r/AWS, right, I’m on /r/Atlassian. Now I’m finding out about this zero-day vulnerability and it’s like, “Oh, guys, we got to act. I just heard about this thing.” And people are like, “Oh, where did this come from?” And it’s like it came from my network, right? And my network—Corey: Mm-hm.Jeff: Is on Twitter, LinkedIn, Reddit. So, the idea that someone browsing the internet on any site, really, is somehow not a productive use of their time, you better be ready to itemize exactly what that means and what that looks like. “Oh, you can do this on Reddit but you can’t do that on Reddit.”Corey: I have no boss now, I have no oversight, but somehow I still show up with a work ethic and get things done.Jeff: Right. [laugh].Corey: Wow, I guess I didn’t need someone over my shoulder the whole time. Who knew?Jeff: Right. That’s all that matters, right? And if you do it in 30 hours or 40 hours, that doesn’t really matter to me, you know? You want to do it at night because you’re more productive there, right, like, let’s figure out a way to make that happen. And remote work is actually empowering us ways to really retain people that wasn’t possible before I had an employee that was like, you know, I really want to travel. I’m like, “Dude, go to Europe. Work from Europe. Just do it. Work from Europe,” right? We’ve got senior leaders on the C-suite that are doing it. One of the chief—Corey: I’m told they have the internet, even there. Imagine that?Jeff: Yeah. [laugh]. So, our chief program officer, she was in Greece for four weeks. And it worked. It worked great. They had a process. You know, she would spent one week on and then one week off on vacation. But you know, she was able to have this incredible, long experience, and still deliver. And it’s like, you know, we can use that as a model to say, like—Corey: And somehow the work got done. Wow, she must be amazing. No, that’s the baseline expectation that people can be self-managing in that respect.Jeff: Right.Corey: They aren’t toddlers.Jeff: So, if she can do that, I’m sure you can figure out how to code in China or wherever you want to visit. So, it’s a great way to stay ahead of some of these companies that have a bit more lethargic policies around that stuff, where it’s like, you know, all right, I’m not getting that insane salary, but guess what, I’m going to spend three weeks in New Zealand hanging out and not using any time off or anything like that, and you know, being able to enjoy life. I wish this pandemic had happened pre-kids because—Corey: Yeah. [laugh].Jeff: —you know, we would really take advantage of this.Corey: You and me both. It would have very different experience.Jeff: Yeah. [laugh]. Absolutely, right? But with kids in school, and all that stuff, we’ve been tethered down. But man, I you know, I want to encourage the young people or the single people on my team to just, like, hey, really, really embrace this time and take advantage of it.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it’s still somebody’s problem. And a big part of that responsibility is app security from code to cloud. And that’s where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you’re actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That’s S-N-Y-K.co/screamCorey: One last topic I want to get into before we call it an episode is, I admit, I read an awful lot of books, it’s a guilty pleasure. And it’s easy to fall into the trap, especially when you know the author, of assuming that snapshot of their state of mind at a very fixed point in time is somehow who they are, like a fly frozen in amber, and it’s never true. So, my question for you is, quite simply, what have you learned since your book came out?Jeff: Oh, man, great question. So, when I was writing the book, I was really nervous about if my audience was as big as I thought it was, the people that I was targeting with the book.Corey: Okay, that keeps me up at night, too. I have no argument there.Jeff: Yeah. You know what I mean?Corey: Please, continue.Jeff: I’m surrounded, you know, by—Corey: Is anyone actually listening to this? Yeah.Jeff: Right. [laugh]. So, after the book got finished and it got published, I would get tons of feedback from people that so thoroughly enjoyed the book, they would say things like, you know, “It feels like you were in our office like a fly on the wall.” And that was exciting, one, because I felt like these were experiences that sort of resonated, but, two, it sort of proved this thesis that sometimes you don’t have to do something revolutionary to be a positive contribution to other people, right? So, like, when I lay out the tips and things that I do in the book, it’s nothing earth-shattering that I expect Google to adopt. Like, oh, my God, this is the most unique view ever.But being able to talk to an audience in a way that resonates with them, that connects with them, that shows that I understand their problem and have been there, it was really humbling and enlightening to just see that there are people out there that they’re not on the bleeding edge, but they just need someone to talk to them in a language that they understand and resonate with. So, I think the biggest thing that I learned was this idea that your voice is important, your voice matters, and how you tell your story may be the difference between someone understanding a concept and someone not understanding a concept. So, there’s always an audience for you out there as you’re writing, whether it be your blog post, the videos that you produce, the podcasts that you make, somewhere there’s someone that needs to hear what you have to say, and the unique way that you can say it. So, that was extremely powerful.Corey: Part of the challenge that I found is when I start talking to other people, back in the before times, trying to push them into conference talks and these days, write blog posts, the biggest objection I get sometimes is, “Well, I don’t have anything worth saying.” That is provably not true. One of my favorite parts about writing Last Week in AWS is as I troll the internet looking for topics about AWS that I find interesting, I keep coming across people who are very involved in one area or another of this ecosystem and have stories they want to tell. And I love, “Hey, would you like to write a guest post for Last Week in AWS?” It’s always invite only and every single one of them has been paid because people die of exposure and I’m not about that exploitation lifestyle.A couple have said, “Oh, I can’t accept payment for a variety of reasons.” Great. Pick a charity that you would like it to go to instead because we do not accept volunteer work, we are a for-profit entity. That is the way it works here. And that has been just one of the absolute favorite parts about what I do just because you get to sort of discover new voices.And what I find really neat is that for a lot of these folks, this is their start to writing and telling the story, but they don’t stop there, they start telling their story in other areas, too. It leads to interesting career opportunities for them, it leads to interesting exposure that they wouldn’t have necessarily had—again, not that they’re getting paid in exposure, but the fact that they are able to be exposed to different methodologies, different ways of thinking—I love that. It’s one of my favorite parts about doing what I do. And it seems to scale a hell of a lot better than me sitting down with someone for two hours to help them build a CFP that they wind up not getting accepted or whatnot.Jeff: Right. It’s a great opportunity that you provide folks, too, because of, like, an instant audience, I think that’s one of the things that has made Medium so successful as, like, a blogging platform is, you know, everyone wants to go out and build their own WordPress site and launch it, but then it like, you write your blog post and it’s crickets. So, the ability for you to, you know, use your platform to also expose those voices is great and extremely powerful. But you’re right, once they do it, it lights a fire in a way that is admirable to watch. I have a person that I’m mentoring and that was my biggest piece of advice I can give. It was like, you know, write. Just write.It’s the one thing that you can do without anyone else. And you can reinforce your own knowledge of a thing. If you just say, you know, I’m going to teach this thing that I just learned, just the writing process helps you solidify, like, okay, I know this stuff. I’m demonstrating that I know it and then four years from now, when you’re applying for a job, someone’s like, “Oh, I found your blog post and I see that you actually do know how to set up a Kubernetes cluster,” or whatever. It’s just extremely great and it—Corey: It’s always fun. You’re googling for how to do something and you find something you wrote five years ago.Jeff: Right, yeah. [laugh]. And it’s like code where you’re like, “Oh, man, I would do that so much differently now.”Corey: Since we last spoke, one of the things I’ve been doing is I have been on the hook to write between a one to two-thousand-word blog post every week, and I’ve done that like clockwork, for about a year-and-a-half now. And I was no slouch at storytelling before I started doing that. I’ve given a few hundred conference talks in the before times. And I do obviously long Twitter threads in the past and I write reports a lot. But forcing me to go through that process every week and then sit with an editor and go ahead and get it improved, has made me a far better writer, it’s made me a better storyteller, I am far better at articulating my point of view.It is absolutely just unlocking a host of benefits that I would have thought I was, oh, I passed all this. I’m already good at these things. And I was, but I’m better now. I think that writing is one of those things that people need to do a lot more of.Jeff: Absolutely. And it’s funny that you mentioned that because I just recently, back in April, started to do the same thing I said, I’m going to write a blog post every week, right? I’m going to get three or four in the can, so that if life comes up and I miss a beat, right, I’m not actually missing the production schedule, so I have a steady—and you’re right. Even after writing a book, I’m still learning stuff through the writing process, articulating my point of view.It’s just something that carries over, and it carries over into the workforce, too. Like, if you’ve ever read a bad piece of documentation, right, that comes from—Corey: No.Jeff: Right? [laugh]. That comes from an inability to write. Like, you know, you end up asking these questions like who’s the audience for this? What is ‘it’ in this sentence? [laugh].Corey: Part of it too, is that people writing these things are so close to the problem themselves that the fact that, “Well, I’m not an expert in this.” That’s why you should write about it. Talk about your experience. You’re afraid everyone’s going to say, “Oh, you’re a fool. You didn’t understand how this works.”Yeah, my lived experiences instead—and admittedly, I have the winds of privilege of my back on this—but it’s also yeah, I didn’t understand that either. It turns out that you’re never the only person who has trouble with a concept. And by calling it out, you’re normalizing it and doing a tremendous service for others in your shoes.Jeff: Especially when you’re not an expert because I wrote some documentation about the SSL process and it didn’t occur to me that these people don’t use the AWS command line, right? Like, you know, in our organization, we sort of mask that from them through a bunch of in-house automation. Now we’re starting to expose it to them and simple things like oh, you need to preface the AWS command with a profile name. So, then when we’re going through the setup, we’re like, “Oh. What if they already have an existing profile, right?” Like, we don’t want to clobber that.SSo, it just changed the way you write the documentation. But like, that’s not something that initially came to mind for me. It wasn’t until someone went through the docs, and they’re like, “Uh, this is blowing up in a weird way.” And I was like, “Oh, right. You know, like, I need to also teach you about profile management.”Corey: Also, everyone has a slightly different workflow for the way they interact with AWS accounts, and their shell prompts, and the way they set up local dev environments.Jeff: Yeah, absolutely. So, not being an expert on a thing is key because you’re coming to it with virgin eyes, right, and you’re able to look at it from a fresh perspective.Corey: So, much documentation out there is always coming from the perspective of someone who is intimately familiar with the problem space. Some of the more interesting episodes that I have, from a challenge perspective, are people who are deep technologists in a particular area and they love they fallen in love with the thing that they are building. Great. Can you explain it to the rest of us mere mortals so that we can actually we can share your excitement on this? And it’s very hard to get them to come down to a level where it’s coherent to folks who haven’t spent years thinking deeply about that particular problem space.Jeff: Man, the number one culprit for that is, like, the AWS blogs where they have, like, a how-to article. You follow that thing and you’re like, “None of this is working.” [laugh]. Right? And then you realize, oh, they made an assumption that I knew this, but I didn’t right?So, it’s like, you know, I didn’t realize this was supposed to be, like, a handwritten JSON document just jammed into the value field. Because I didn’t know that, I’m not pulling those values out as JSON. I’m expecting that just to be, like, a straight string value. And that has happened more and more times on the AWS blog than I can count. [laugh].Corey: Oh, yeah, very often. And then there’s other problems, too. “Oh, yeah. Set up your IAM permissions properly.” That’s left as an exercise for the reader. And then you wonder why everything’s full of stars. Okay.Jeff: Right. Yep, exactly, exactly.Corey: Ugh. It’s so great to catch up with you and see what you’ve been working on. If people want to learn more, where’s the best place to find you?Jeff: So, the best place is probably my website, attainabledevops.com. That’s a place where you can find me on all the other places. I don’t really update that site much, but you can find me on LinkedIn, Twitter, from that jumping off point, links to the book are there if anyone’s interested in that. Perfect stocking stuffers. Mom would love it, grandma would love it, so definitely, definitely buy multiple copies of that.Corey: Yeah, it’s going to be one of my two-year-old’s learning to read books, it’d be great.Jeff: Yeah, it’s perfect. You know, you just throw it in the crib and walk away, right? They’re asleep at no time. Like I said, I’ve also been taking to, you know, blogging on Medium, so you can catch me there, the links will be there on Attainable DevOps as well.Corey: Excellent. And that link will of course, be in the show notes. Thank you so much for being so generous with your time. I really do appreciate it. And it’s great to talk to you again.Jeff: It was great to catch up.Corey: Really was. Jeff Smith, Director of Product Operations at Basis Technologies. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice or smash the like and subscribe buttons on the YouTubes, whereas if you’ve hated this podcast, do the exact same thing—five-star review, smash the buttons—but also leave an angry, incoherent comment that you’re then going to have edited and every week you’re going to come back and write another incoherent comment that you get edited. And in the fullness of time, you’ll get much better at writing angry, incoherent comments.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About BenjaminBenjamin Anderson is CTO, Cloud at EDB, where he is responsible for developing and driving strategy for the company’s Postgres-based cloud offerings. Ben brings over ten years’ experience building and running distributed database systems in the cloud for multiple startups and large enterprises. Prior to EDB, he served as chief architect of IBM’s Cloud Databases organization, built an SRE practice at database startup Cloudant, and founded a Y Combinator-funded hardware startup.Links Referenced: EDB: https://www.enterprisedb.com/ BigAnimal: biganimal.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it’s still somebody’s problem. And a big part of that responsibility is app security from code to cloud. And that’s where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you’re actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That’s S-N-Y-K.co/screamCorey: This episode is sponsored by our friends at Fortinet. Fortinet’s partnership with AWS is a better-together combination that ensures your workloads on AWS are protected by best-in-class security solutions powered by comprehensive threat intelligence and more than 20 years of cybersecurity experience. Integrations with key AWS services simplify security management, ensure full visibility across environments, and provide broad protection across your workloads and applications. Visit them at AWS re:Inforce to see the latest trends in cybersecurity on July 25-26 at the Boston Convention Center. Just go over to the Fortinet booth and tell them Corey Quinn sent you and watch for the flinch. My thanks again to my friends at Fortinet.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. This promoted guest episode is brought to us by our friends at EDB. And not only do they bring us this promoted episode, they bring me their CTO for Cloud, Benjamin Anderson. Benjamin, thank you so much for agreeing to suffer the slings and arrows that I will no doubt throw at you in a professional context, because EDB is a database company, and I suck at those things.Benjamin: [laugh]. Thanks, Corey. Nice to be here.Corey: Of course. So, databases are an interesting and varied space. I think we can all agree—or agree to disagree—that the best database is, of course, Route 53, when you misuse TXT records as a database. Everything else is generally vying for number two. EDB was—back in the days that I was your customer—was EnterpriseDB, now rebranded as EDB, which is way faster to say, and I approve of that.But you were always the escalation point of last resort. When you’re stuck with a really weird and interesting Postgres problem, EDB was where you went because if you folks couldn’t solve the problem, it was likely not going to get solved. I always contextualized you folks as a consulting shop. That’s not really what you do. You are the CTO for Cloud.And, ah, interesting. Do databases behave differently in cloud environments? Well, they do when you host them as a managed service, which is an area you folks have somewhat recently branched into. How’d you get there?Benjamin: Ah, that’s interesting. So, there’s a bunch of stuff to unpack there. I think EDB has been around for a long time. It’s something like 13, 14, 15 years, something like that, and really it's just been kind of slowly growing, right? We did start very much as a product company. We built some technology to help customers get from Oracle database on to Postgres, way back in 2007, 2008.That business has just slowly been growing. It’s been going quite well. Frankly, I only joined about 18 months ago, and it’s really cool tech, right? We natively understand some things that Oracle is doing. Customers don’t have to change their schemas to migrate from Oracle to Postgres. There’s some cool technology in there.But as you point out, I think a lot of our position in the market has not been that product focused. There’s been a lot of people seeing us as the Postgres experts, and as people who can solve Postgres problems, in general. We have, for a long time, employed a lot of really sharp Postgres people. We still employ a lot of really sharp Postgres people. That’s very much, in a lot of ways, our bread and butter. That we’re going to fix Postgres problems as they come up.Now, over the past few years, we’ve definitely tried to shift quite a bit into being more of a product company. We’ve brought on a bunch of people who’ve been doing more enterprise software product type development over the past few years, and really focusing ourselves more and more on building products and investing in ourselves as a product company. We’re not a services company. We’re not a consulting company. We do, I think, provide the best Postgres support in the market. But it’s been a journey. The cloud has been a significant part of that as well, right? You can’t get away.Corey: Oh, yeah. These days, when someone’s spinning up a new workload, it’s unlikely—in most cases—they’re going to wind up spinning up a new data center, if they don’t already have one. Yes, there’s still a whole bunch of on-prem workloads. But increasingly, the default has become cloud. Instead of, “Why cloud?” The question’s become, “Why not?”Benjamin: Right, exactly. Then, as people are more and more accepting of managed services, you have to be a product company. You have to be building products in order to support your database customers because what they want his managed services. I was working in managed databases and service, something like, ten years ago, and it was like pulling teeth. This is after RDS launched. This was still pulling teeth trying to get people to think about, oh, I’m going to let you run my database. Whereas, now obviously, it’s just completely different. We have to build great products in order to succeed in the database business, in general.Corey: One thing that jumped out at me when you first announced this was the URL is enterprisedb.com. That doesn’t exactly speak to, you know, non-large companies, and EDB is what you do. You have a very corporate logo, but your managed service is called BigAnimal, which I absolutely love. It actually expresses a sense of whimsy and personality that I can no doubt guess that a whole bunch of people argued against, but BigAnimal, it is. It won through. I love that. Was that as contentious as I’m painting it to be, or people actually have a sense of humor sometimes?Benjamin: [laugh]. Both, it was extremely contentious. I, frankly, was one of the people who was not in favor of it at first. I was in favor of something that was whimsical, but maybe not quite that whimsical.Corey: Well, I call it Postgres-squeal, so let’s be very clear here that we’re probably not going to see eye-to-eye on most anything in pronunciation things. But we can set those differences aside and have a conversation.Benjamin: Absolutely, no consider that. It was deliberate, though, to try to step away a little bit from the blue-suit-and-tie, enterprise, DB-type branding. Obviously, a lot of our customers are big enterprises. We’re good at that. We’re not trying to be the hip, young startup targeting business in a lot of ways. We have a wide range of customers, but we want to branch out a little bit.Corey: One of the challenges right now is if I spin up an environment inside of AWS, as one does, and I decide I certainly don’t want to take the traditional approach of running a database on top of an EC2 instance—the way that we did in the olden days—because RDS was crappy. Now that it’s slightly less crappy, that becomes a not ideal path. I start looking at their managed database offerings, and there are something like 15 distinct managed databases that they offer, and they never turn anything off. And they continue to launch things into the far future. And it really feels, on some level, like 20 years from now—what we call a DBA today—their primary role is going to look a lot more like helping a company figure out which of Amazon’s 40 managed databases is the appropriate fit for this given workload. Yet, when I look around at what the industry has done, it seems that when we’re talking about relational databases. Postgres has emerged back when I was, more or less, abusing servers in person in my data center days, it was always MySQL. These days, Postgres is the de facto standard, full stop. I admit that I was mostly keeping my aura away from any data that was irreplaceable at that time. What happened? What did I miss?Benjamin: It’s a really good question. And I certainly am not a hundred percent on all the trends that went on there. I know there’s a lot of folks that are not happy about the MySQL acquisition by Oracle. I think there’s a lot of energy that was adopted by the NoSQL movement, as well. You have people who didn’t really care about transactional semantics that were using MySQL because they needed a place to store their data. And then, things like MongoDB and that type of system comes along where it’s significantly easier than MySQL, and that subset of the population just sort of drifts away from MySQL.Corey: And in turn, those NoSQL projects eventually turn into something where, okay, now we’re trying to build a banking system on top of it, and it’s, you know, I guess you can use a torque wrench as a hammer if you’re really creative about it, but it seems like there’s a better approach.Benjamin: Yeah, exactly. And those folks are coming back around to the relational databases, exactly. At the same time, the advancements in Postgres from the early eight series to today are significant, right? We shouldn’t underestimate how much Postgres has really moved forward. It wasn’t that long ago that replication was hardly a thing and Postgres, right? It’s been a journey.Corey: One thing that your website talks about is that you accelerate your open-sourced database transformation. And this is a bit of a hobby horse I get on from time to time. I think that there are a lot of misunderstandings when people talk about this. You have the open-source purists—of which I shamefully admit I used to be one—saying that, “Oh, it’s about the idea of purity and open and free as in software.” Great. Okay, awesome. But when I find that corporate customers are talking about when they say open-source database, they don’t particularly care if they have access to the source code because they’re not going to go in and patch a database engine, we hope. But what they do care about is regardless of where they are today—even if they’re perfectly happy there—they don’t want to wind up beholden to a commercial database provider, and/or they don’t want to wind up beholden to the environment that is running within. There’s a strategic Exodus that’s available in theory, which on some level serves to make people feel better about not actually Exodus-ing, but it also means if they’re doing a migration at some point, they don’t also have to completely redo their entire data plan.Benjamin: Yeah, I think that’s a really good point. I mean, I like to talk—there’s a big rat’s nest of questions and problems in here—but I generally like talk to about open APIs, talk about standards, talk about how much is going to have to change if you eliminate this vendor. We’re definitely not open-source purists. Well, we employ a lot of open-source purists. I also used to be an open—Corey: Don’t let them hear you say that, then. Fair enough. Fair enough.Benjamin: [laugh] we have proprietary software at EDB, as well. There’s a kind of wide range of businesses that we participate in. Glad to hear you also mention this where-it’s-hosted angle, as well. I think there’s some degree to which people are—they figured out that having at least open APIs or an open-source-ish database is a good idea rather than being beholden to proprietary database. But then, immediately forget that when they’re picking a cloud vendor, right? And realizing that putting their data in Cloud Vendor A versus Cloud Vendor B is also putting them in a similar difficult situation. They need to be really wary of when they’re doing that. Now, obviously, I work at an independent software company, and I have some incentive to say this, but I do think it’s true. And you know, there’s meaningful data gravity risk.Corey: I assure you, I have no incentive. I don’t care what cloud provider you’re on. My guidance has been, for years, to—as a general rule—pick a provider, I care about which one, and go all in until there’s a significant reason to switch. Trying to build an optionality, “Oh, everything we do should be fully portable at an instance notice.” Great. Unless you’re actually doing it, you’re more or less, giving up a whole bunch of shortcuts and feature velocity you could otherwise have, in the hopes of one day you’ll do a thing, but all the assumptions you’re surrounded by baked themselves in regardless. So, you’re more or less just creating extra work for yourself for no defined benefit. This is not popular in some circles, where people try to sell something that requires someone to go multi-cloud, but here we are.Benjamin: No, I think you’re right. I think people underestimate the degree to which the abstractions are just not very good, right, and the degree to which those cloud-specific details are going to leak in if you’re going to try to get anything done, you end up in kind of a difficult place. What I see more frequently is situations where we have a big enterprise—not even big, even medium-sized companies where maybe they’ve done an acquisition or two, they’ve got business units that are trying to do things on their own. And they end up in two or three clouds, sort of by happenstance. It’s not like they’re trying to do replication live between two clouds, but they’ve got one business unit in AWS and one business unit and Azure, and somebody in the corporate—say enterprise architect or something like that—really would like to make things consistent between the two so they get a consistent security posture and things like that. So, there are situations where the multi-cloud is a reality at a certain level, but maybe not at a concrete technical level. But I think it’s still really useful for a lot of customers.Corey: You position your cloud offering in two different ways. One of them is the idea of BigAnimal, and the other—well, it sort of harkens back to when I was in sixth grade going through the American public school system. They had a cop come in and talk to us and paint to this imaginary story of people trying to push drugs. “Hey, kid. You want to try some of this?” And I’m reading this and it says EDB, Postgres for Kubernetes. And I’m sent back there, where it’s like, “Hey, kid. You want to run your stateful databases on top of Kubernetes?” And my default answer to that is good lord, no. What am I missing?Benjamin: That’s a good question. Kubernetes has come a long way—I think is part of that.Corey: Oh, truly. I used to think of containers as a pure story for stateless things. And then, of course, I put state into them, and then, everything exploded everywhere because it turns out, I’m bad at computers. Great. And it has come a long way. I have been tracking a lot of that. But it still feels like the idea being that you’d want to have your database endpoints somewhere a lot less, I guess I’ll call it fickle, if that makes sense.Benjamin: It’s an interesting problem because we are seeing a lot of people who are interested in our Kubernetes-based products. It’s actually based on—we recently open-sourced the core of it under a project called cloud-native PG. It’s a cool piece of technology. If you think about sort of two by two. In one corner, you’ve got self-managed on-premise databases. So, you’re very, very slow-moving, big-iron type, old-school database deployments. And on the opposite corner, you’ve got fully-managed, in the cloud, BigAnimal, Amazon RDS, that type of thing. There’s a place on that map where you’ve got customers that want a self-service type experience. Whether that’s for production, or maybe it’s even for dev tests, something like that. But you don’t want to be giving the management capability off to a third party.For folks that want that type of experience, trying to build that themselves by, like, wiring up EC2 instances, or doing something in their own data center with VMware, or something like that, can be extremely difficult. Whereas if you’ve go to a Kubernetes-based product, you can get that type of self-service experience really easily, right? And customers can get a lot more flexibility out of how they run their databases and operate their databases. And what sort of control they give to, say application developers who want to spin up a new database for a test or for some sort of small microservice, that type of thing. Those types of workloads tend to work really well with this first-party Kubernetes-based offering. I’ve been doing databases on Kubernetes in managed services for a long time as well. And I don’t, frankly, have any concerns about doing it. There are definitely some sharp edges. And if you wanted to do to-scale, you need to really know what you’re doing with Kubernetes because the naive thing will shoot you in the foot.Corey: Oh, yes. So, some it feels almost like people want to cosplay working for Google, but they don’t want to pass the technical interview along the way. It’s a bit of a weird moment for it.Benjamin: Yeah, I would agree.Corey: I have to go back to my own experiences with using RDS back at my last real job before I went down this path. We were migrating from EC2-Classic to VPC. So, you could imagine what dates me reasonably effectively. And the big problem was the database. And the joy that we had was, “Okay, we have to quiesce the application.” So, the database is now quiet, stop writes, take a snapshot, restore that snapshot into the environment. And whenever we talk to AWS folks, it’s like, “So, how long is this going to take?” And the answer was, “Guess.” And that was not exactly reassuring. It went off without a hitch because every migration has one problem. We were sideswiped in an Uber on the way home. But that’s neither here nor there. This was two o’clock in the morning, and we finished in half the maintenance time we had allotted. But it was the fact that, well, guess we’re going to have to take the database down for many hours with no real visibility, and we hope it’ll be up by morning. That wasn’t great. But that was the big one going on, on an ongoing basis, there were maintenance windows with a database. We just stopped databasing for a period of time during a fairly broad maintenance window. And that led to a whole lot of unfortunate associations in my mind with using relational databases for an awful lot of stuff. How do you handle maintenance windows and upgrading and not tearing down someone’s application? Because I have to assume, “Oh, we just never patch anything. It turns out that’s way easier,” is in fact, the wrong answer.Benjamin: Yeah, definitely. As you point out, there’s a bunch of fundamental limitations here, if we start to talk about how Postgres actually fits together, right? Pretty much everybody in RDS is a little bit weird. The older RDS offerings are a little bit weird in terms of how they do replication. But most folks are using Postgres streaming replication, to do high availability, Postgres in managed services. And honestly, of course—Corey: That winds up failing over, or the application’s aware of both endpoints and switches to the other one?Benjamin: Yeah—Corey: Sort of a database pooling connection or some sort of proxy?Benjamin: Right. There’s a bunch of subtleties that get into their way. You say, well, did the [vit 00:16:16] failover too early, did the application try to connect and start making requests before the secondaries available? That sort of thing.Corey: Or you misconfigure it and point to the secondary, suddenly, when there’s a switchover of some database, suddenly, nothing can write, it can only read, then you cause a massive outage on the weekend?Benjamin: Yeah. Yeah.Corey: That may have been of an actual story I made up.Benjamin: [laugh] yeah, you should use a managed service.Corey: Yeah.Benjamin: So, it’s complicated, but even with managed services, you end up in situations where you have downtime, you have maintenance windows. And with Postgres, especially—and other databases as well—especially with Postgres, one of the biggest concerns you have is major version upgrades, right? So, if I want to go from Postgres 12 to 13, 13 to 14, I can’t do that live. I can’t have a single cluster that is streaming one Postgres version to another Postgres version, right?So, every year, people want to put things off for two years, three years sometimes—which is obviously not to their benefit—you have this maintenance, you have some sort of downtime, where you perform a Postgres upgrade. At EDB, we’ve got—so this is a big problem, this is a problem for us. We’re involved in the Postgres community. We know this is challenging. That’s just a well-known thing. Some of the folks that are working EDB are folks who worked on the Postgres logical replication tech, which arrived in Postgres 10. Logical replication is really a nice tool for doing things like change data capture, you can do Walter JSON, all these types of things are based on logical replication tech.It’s not really a thing, at least, the code that’s in Postgres itself doesn’t really support high availability, though. It’s not really something that you can use to build a leader-follower type cluster on top of. We have some techs, some proprietary tech within EDB that used to be called bi-directional replication. There used to be an open-source project called bi-directional replication. This is a kind of a descendant of that. It’s now called Postgres Distributed, or EDB Postgres Distributed is the product name. And that tech actually allows us—because it’s based on logical replication—allows us to do multiple major versions at the same time, right? So, we can upgrade one node in a cluster to Postgres 14, while the other nodes in the clusters are at Postgres 13. We can then upgrade the next node. We can support these types of operations in a kind of wide range of maintenance operations without taking a cluster down from maintenance.So, there’s a lot of interesting opportunities here when we start to say, well, let’s step back from what your typical assumptions are for Postgres streaming replication. Give ourselves a little bit more freedom by using logical replication instead of physical streaming replication. And then, what type of services, and what type of patterns can we build on top of that, that ultimately help customers build, whether it’s faster databases, more highly available databases, so on and so forth.Corey: Let’s face it, on-call firefighting at 2am is stressful! So there’s good news and there’s bad news. The bad news is that you probably can’t prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: One approach that I took for, I guess you could call it backup sort of, was intentionally staggering replication between the primary and the replica about 15 minutes or so. So, if I drop a production table or something like that, I have 15 short minutes to realize what has happened and sever the replication before it is now committed to the replica and now I’m living in hell. It felt like this was not, like, option A, B, or C, or the right way to do things. But given that meeting customers where they are as important, is that the sort of thing that you support with BigAnimal, or do you try to talk customers into not being ridiculous?Benjamin: That’s not something we support now. It’s not actually something that I hear that many asks for these days. It’s kind of interesting, that’s a pattern that I’ve run into a lot in the past.Corey: I was an ancient, grumpy sysadmin. Again, I’m dating myself here. These days, I just store everything at DNS text records, and it’s way easier. But I digress.Benjamin: [laugh] yeah, it’s something that we see a lot for and we had support for a point-in-time restore, like pretty much anybody else in the business at this point. And that’s usually the, “I fat-fingered something,” type response. Honestly, I think there’s room to be a bit more flexible and room to do some more interesting things. I think RDS is setting a bar and a lot of database services out there and kind of just meeting that bar. And we all kind of need to be pushing a little bit more into more interesting spaces and figuring out how to get customers more value, get customers to get more out of their money for the database, honestly.Corey: One of the problems we tend to see, in the database ecosystem at large, without naming names or companies or anything like that, is that it’s a pretty thin and blurry line between database advocate, database evangelist, and database zealot. Where it feels like instead, we’re arguing about religion more than actual technical constraints and concerns. So, here’s a fun question that hopefully isn’t too much of a gotcha. But what sort of workloads would you actively advise someone not to use BigAnimal for in the database world? But yes, again, if you try to run a DNS server, it’s probably not fit for purpose without at least a shim in the way there. But what sort of workloads are you not targeting that a customer is likely to have a relatively unfortunate time with?Benjamin: Large-scale analytical workloads is the easy answer to that, right? If you’ve got a problem where you’re choosing between Postgres and Snowflake, you’re seriously considering—you actually have as much data that you seriously be considering Snowflake? You probably don’t want to be using Postgres, right? You want to be using something that’s column, or you want to be using a query planner that really understands a columnar layout that’s going to get you the sorts of performance that you need for those analytical workloads. We don’t try to touch that space.Corey: Yeah, we’re doing some of that right now with just the sheer volume of client AWS bills we have. We don’t really need a relational model for a lot of it. And Athena is basically fallen down on the job in some cases, and, “Oh, do you want to use Redshift, that’s basically Postgres.” It’s like, “Yeah, it’s Postgres, if it decided to run on bars of gold.” No, thank you. It just becomes this ridiculously overwrought solution for what feels like it should be a lot similar. So, it’s weird, six months ago or so I wouldn’t have had much of an idea what you’re talking about. I see it a lot better now. Generally, by virtue of trying to do something the precise wrong way that someone should.Benjamin: Right. Yeah, exactly. I think there’s interesting room for Postgres to expand here. It’s not something that we’re actively working on. I’m not aware of a lot happening in the community that Postgres is, for better or worse, extremely extensible, right? And if you see the JSON-supported Postgres, it didn’t exist, I don’t know, five, six years ago. And now it’s incredibly powerful. It’s incredibly flexible. And you can do a lot of quote-unquote, schemaless stuff straight in Postgres. Or you look at PostGIS, right, for doing GIS geographical data, right? That’s really a fantastic integration directly in the database.Corey: Yeah, before that people start doing ridiculous things almost looks similar to a graph database or a columnar store somehow, and yeah.Benjamin: Yeah, exactly. I think sometimes somebody will do a good column store that’s an open-source deeply integrated into Postgres, rather than—Corey: I’ve seen someone build one on top of S3 bucket with that head, a quarter of a trillion objects in it. Professional advice, don’t do that.Benjamin: [laugh]. Unless you’re Snowflake. So, I mean, it’s something that I’d like to see Postgres expand into. I think that’s an interesting space, but not something that, at least especially for BigAnimal, and frankly, for a lot of EDB customers. It’s not something we’re trying to push people toward.Corey: One thing that I think we are seeing a schism around is the idea that some vendors are one side of it, some are on the other, where on the one side, you have, oh, every workload should have a bespoke, purpose-built database that is exactly for this type of workload. And the other school of thought is you should generally buy us for a general-purpose database until you have a workload that is scaled and significant to a point where running that on its own purpose-built database begins to make sense. I don’t necessarily think that is a binary choice, where do you tend to fall on that spectrum?Benjamin: I think everybody should use Postgres. And I say not just because I work in a Postgres company.Corey: Well, let’s be clear. Before this, you were at IBM for five years working on a whole bunch of database stuff over there, not just Postgres. And you, so far, have not struck me as the kind of person who’s like, “Oh, so what’s your favorite database?” “The one that pays me.” We’ve met people like that, let’s be very clear. But you seem very even-handed in those conversations.Benjamin: Yeah, I got my start in databases, actually, with Apache CouchDB. I am a committer on CouchDB. I worked on a managed at CouchDB service ten years ago. At IBM, I worked on something in nine different open-source databases and managed services. But I love having conversations about, like, well, I’ve got this workload, should I use Postgres, rr should I use Mongo, should I use Cassandra, all of those types of discussions. Frankly, though, I think in a lot of cases people are—they don’t understand how much power they’re missing out on if they don’t choose a relational database. If they don’t understand the relational model well enough to understand that they really actually want that. In a lot of cases, people are also just over-optimizing too early, right? It’s just going to be much faster for them to get off the ground, get product in customers hands, if they start with something that they don’t have to think twice about. And they don’t end up with this architecture with 45 different databases, and there’s only one guy in the company that knows how to manage the whole thing.Corey: Oh, the same story of picking a cloud provider. It’s, “Okay, you hire a team, you’re going to build a thing. Which cloud provider do you pick?” Every cloud provider has a whole matrix and sales deck, and the rest. The right answer, of course, is the one your team’s already familiar with because learning a new cloud provider while trying not to run out of money at your startup, can’t really doesn’t work super well.Benjamin: Exactly. Yeah.Corey: One thing that I think has been sort of interesting, and when I saw it, it was one of those, “Oh, I sort of like them.” Because I had that instinctive reaction and I don’t think I’m alone in this. As of this recording a couple of weeks ago, you folks received a sizable investment from private equity. And default reaction to that is, “Oh, well, I guess I put a fork in the company, they’re done.” Because the narrative is that once private equity takes an investment, well, that company’s best days are probably not in front of it. Now, the counterpoint is that this is not the first time private equity has invested in EDB, and you folks from what I can tell are significantly better than you were when I was your customer a decade ago. So clearly, there is something wrong with that mental model. What am I missing?Benjamin: Yeah. Frankly, I don’t know. I’m no expert in funding models and all of those sorts of things. I will say that my experience has been what I’ve seen at EDB, has definitely been that maybe there’s private equity, and then there’s private equity. We’re in this to build better products and become a better product company. We were previously owned by a private equity firm for the past four years or so. And during the course of those four years, we brought on a bunch of folks who were very product-focused, new leadership. We made a significant acquisition of a company called 2ndQuadrant, which they employed a lot of the European best Postgres company. Now, they’re part of EDB and most of them have stayed with us. And we built the managed cloud service, right? So, this is a pretty significant—private equity company buying us to invest in the company. I’m optimistic that that’s what we’re looking at going forward.Corey: I want to be clear as well, I’m not worried about what I normally would be in a private equity story about this, where they’re there to save money and cut costs, and, “Do we really need all these database replicas floating around,” and, “These backups, seems like that’s something we don’t need.” You have, at last count, 32 Postgres contributors, 7 Postgres committers, and 3 core members. All of whom would run away screaming loudly and publicly, in the event that such a thing were taking place. Of all the challenges and concerns I might have about someone running a cloud service in the modern day. I do not have any fear that you folks are not doing what will very clearly be shown to be the right thing by your customers for the technology that you’re building on top of. That is not a concern. There are companies I do not have that confidence in, to be clear.Benjamin: Yeah, I’m glad to hear that. I’m a hundred percent on board as well. I work here, but I think we’re doing the right thing, and we’re going to be doing great stuff going forward.Corey: One last topic I do want to get into a little bit is, on some level, launching in this decade, a cloud-hosted database offering at a time when Amazon—whose product strategy of yes is in full display—it seems like something ridiculous, that is not necessarily well thought out that why would you ever try to do this? Now, I will temper that by the fact that you are clearly succeeding in this direction. You have customers who say nice things about you, and the reviews have been almost universally positive anywhere I can see things. The negative ones are largely complaining about databases, which I admit might be coming from me.Benjamin: Right, it is a crowded space. There’s a lot of things happening. Obviously, Amazon, Microsoft, Google are doing great things, both—Corey: Terrible things, but great, yes. Yes.Benjamin: [laugh] right, there’s good products coming in. I think AlloyDB is not necessarily a great product. I haven’t used it myself yet, but it’s an interesting step in the direction. I’m excited to see development happening. But at the end of the day, we’re a database company. Our focus is on building great databases and supporting great databases. We’re not entering this business to try to take on Amazon from an infrastructure point of view. In fact, the way that we’re structuring the product is really to try to get the strengths of both worlds. We want to give customers the ability to get the most out of the AWS or Azure infrastructure that they can, but come to us for their database.Frankly, we know Postgres better than anybody else. We have a greater ability to get bugs fixed in Postgres than anybody else. We’ve got folks working on the database in the open. We got folks working on the database proprietary for us. So, we give customers things like break/fix support on that database. If there is a bug in Postgres, there’s a bug in the tech that sits around Postgres. Because obviously, Postgres is not a batteries-included system, really. We’re going to fix that for you. That’s part of the contract that we’re giving to our customers. And I know a lot of smaller companies maybe haven’t been burned by this sort of thing very much. We start to talk about enterprise customers and medium, larger-scale customers, this starts to get really valuable. The ability to have assurance on top of your open-source product. So, I think there’s a lot of interesting things there, a lot of value that we can provide there.I think also that I talked a little bit about this earlier, but like the box, this sort of RDS-shaped box, I think is a bit too small. There’s an opportunity for smaller players to come in and try to push the boundaries of that. For example, giving customers more support by default to do a good job using their database. We have folks on board that can help consult with customers to say, “No, you shouldn’t be designing your schemas that way. You should be designing your schemas this way. You should be using indexes here,” that sort of stuff. That’s been part of our business for a long time. Now, with a managed service, we can bake that right into the managed service. And that gives us the ability to kind of make that—you talk about shared responsibility between the service writer and the customer—we can change the boundaries of that shared responsibility a little bit, so that customers can get more value out of the managed database service than they might expect otherwise.Corey: There aren’t these harsh separations and clearly defined lines across which nothing shall pass, when it makes sense to do that in a controlled responsible way.Benjamin: Right, exactly. Some of that is because we’re a database company, and some of that is because, frankly, we’re much smaller.Corey: I’ll take it a step further beyond that, as well, that I have seen this pattern evolve a number of times where you have a customer running databases on EC2, and their AWS account managers suggests move to RDS. So, they do. Then, move to Aurora. So, they do. Then, I move this to DynamoDB. At which point, it’s like, what do you think your job is here, exactly? Because it seems like every time we move databases, you show up in a nicer car. So, what exactly is the story here, and what are the incentives? Where it just feels like there is a, “Whatever you’re doing is not the way that it should be done. So, it’s time to do, yet, another migration.”There’s something to be said for companies who are focused around a specific aspect of things. Then once that is up and working and running, great. Keep on going. This is fine. As opposed to trying to chase the latest shiny, on some level. I have a big sense of, I guess, affinity for companies that wind up knowing where they start, and most notably, where they stop.Benjamin: Yeah, I think that’s a really good point. I don’t think that we will be building an application platform anytime soon.Corey: “We’re going to run Lambda functions on top of a database.” It’s like, “Congratulations. That is the weirdest stored procedure I can imagine this week, but I’m sure we can come up with a worse one soon.”Benjamin: Exactly.Corey: I really want to thank you for taking the time to speak with me so much about how you’re thinking about this, and what you’ve been building over there. If people want to learn more, where’s the best place to go to find you?Benjamin: biganimal.com.Corey: Excellent. We will throw a link to that in the show notes and it only just occurred to me that the Postgres mascot is an elephant, and now I understand why it’s called BigAnimal. Yeah, that’s right. He who laughs last, thinks slowest, and today, that’s me. I really want to thank you for being so generous with your time. I appreciate it.Benjamin: Thank you. I really appreciate it.Corey: Benjamin Anderson, CTO for Cloud at EDB. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that you then wind up stuffing into a SQLite database, converting to Base64, and somehow stuffing into the comment field.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About BrandonBrandon West was raised in part by video games and BBSes and has been working on web applications since 1999. He entered the world of Developer Relations in 2011 as an evangelist for a small startup called SendGrid and has since held leadership roles at companies like AWS. At Datadog, Brandon is focused on helping developers improve the performance and developer experience of the things they build. He lives in Seattle where enjoys paddle-boarding, fishing, and playing music.Links Referenced: Datadog: https://www.datadoghq.com/ Twitter: https://twitter.com/bwest TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it’s hard to know where problems originate. Is it your application code, users, or the underlying systems? I’ve got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it’s more than just hipster monitoring.Corey: This episode is sponsored in part by our friends at Fortinet. Fortinet’s partnership with AWS is a better-together combination that ensures your workloads on AWS are protected by best-in-class security solutions powered by comprehensive threat intelligence and more than 20 years of cybersecurity experience. Integrations with key AWS services simplify security management, ensure full visibility across environments, and provide broad protection across your workloads and applications. Visit them at AWS re:Inforce to see the latest trends in cybersecurity on July 25-26 at the Boston Convention Center. Just go over to the Fortinet booth and tell them Corey Quinn sent you and watch for the flinch. My thanks again to my friends at Fortinet.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. My guest today is someone I’ve been trying to get on the show for years, but I’m very bad at, you know, following up and sending the messages and all the rest because we all struggle with our internal demons. My guest instead struggles with external demons. He is the team lead for developer experience and tools advocacy at what I can only assume is a Tinder for Pets style company, Date-A-Dog. Brendon West, thank you for joining me today.Brandon: Hey, Corey, thanks for having me. I’m excited to be here. Finally, like you said, it’s been a couple of years. But glad that it’s happening. And yeah, I’m on the DevRel team at Datadog.Corey: Yes, I’m getting a note here in the headset of breaking news coming in. Yes, you’re not apparently a dog dating company, you are a monitoring slash observability slash whatever the cool kids are calling it today telemetry outputer dingus nonsense. Anyone who has ever been to a community or corporate event has no doubt been tackled by one of the badge scanners that you folks have orbiting your booth, but what is it that you folks do?Brandon: Well, the observability, the monitoring, the distributed tracing, all that stuff that you mentioned. And then a lot of other interesting things that are happening. Security is a big focus—InfoSec—so we’re adding some products around that, automated security monitoring, very cool. And then the sort of stuff that I’m representing is stuff that helps developers provide a better experience to their end-users. So, things like front-end monitoring, real-time user monitoring, synthetic testing of your APIs, whatever it might be.Corey: Your path has been somewhat interesting because you—well, everyone’s path has been somewhat interesting; yours has been really interesting because back in 2011, you entered the world of developer relations, or being a DevReloper as I insist on calling it. And you were in a—you call it a small startup called SendGrid. Which is, on some level, hilarious from my point of view. I’ve been working with you folks—you folks being SendGrid—for many years now. I cared a lot about email once upon a time.And now I send an email newsletter every week, that deep under the hood, through a couple of vendor abstraction layers is still SendGrid, and I don’t care about email because that’s something that I can pay someone else to worry about. You went on as well to build out DevRel teams at AWS. You decided okay, you’re going to take some time off after that. You went to a small scrappy startup and ah, nice. You could really do things right and you have a glorious half of the year and then surprise, you got acquired by Datadog. Congratu-dolances on that because now you’re right back in the thick of things at big company-style approaches. Have I generally nailed the trajectory of the past decade for you?Brandon: Yeah, I think the broad strokes are all correct there. SendGrid was a small company when I joined, you know? There were 30 of us or so. So, got to see that grow into what it is today, which was super, super awesome. But other than that, yeah, I think that’s the correct path.Corey: It’s interesting to me, in that you were more or less doing developer relations before that was really a thing in the ecosystem. And I understand the challenge that you would have in a place like SendGrid because that is large-scale email sending, transactional or otherwise, and that is something that by and large, has slipped below the surface level of awareness for an awful lot of folks in your target market. It’s, “Oh, okay, and then we’ll just have the thing send an email,” they say, hand-waving over what is an incredibly deep and murky pool. And understanding that is a hard thing requires a certain level of technical sophistication. So, you started doing developer relations for something that very clearly needed some storytelling chops. How did you fall into it originally?Brandon: Well, I wanted to do something that let me use those storytelling chops, honestly. I had been writing code at an agency for coal mines and gold mines and really actively inserting evil into the world, power plants, and that sort of thing. And, you know, I went to school for English literature. I loved writing. I played in thrash metal bands when I was a kid, so I’ve been up on stage being cussed at and told that I suck. So I—Corey: Oh, I get that conference talks all the time.Brandon: Yeah, right? So, that’s why when people ask me to speak, I’m like, “Absolutely.” There’s no way I can bomb harder than I’ve bombed before. No fear, right? So yeah, I wanted to use those skills. I wanted to do something different.And one of my buddies had a company that he had co-founded that was going through TechStars in Boulder. SendGrid was the first accelerator-backed company to IPO which is pretty cool. But they had gone through TechStars in 2009. They were looking for a developer evangelist. So, SendGrid was looking for developer evangelist and my friend introduced me said, “I think you’d be good at this. You should have a conversation.” My immediate thought was what the hell is a developer evangelist?Corey: And what might a SendGrid be? And all the rest. Yes, it’s that whole, “Oh, how do I learn to swim?” Someone throws you off the end of the dock and then retrospect, it’s, “I don’t think they were trying to teach me how to swim.” Yeah. Hindsight.Brandon: Yeah. It worked out great. I will say, though, that I think DevRel has been around for a long time, you know? The title has been around since the original Macintosh at Apple in 1980-ish. There’s a whole large part of the tech world that would like you to think that it’s new because of all the terrible things that their DevRel team did at Microsoft in the late-90s.And you can go read all about this. There were trials about it. These documents were released to the public, James Plamondon is the lead architect of all of this nastiness. But I think there was then a concerted effort to memory-hole that and say, “No, DevRel is new and shiny.” And then Google came along and said, “Well, it’s not evangelism anymore. It’s advocacy.”Corey: It’s not sysadmin work anymore. It’s SRE. It’s not on-prem, it’s Sparkling Kubernetes, et cetera, et cetera.Brandon: Yeah, so there’s this sense in a lot of places that DevRel is new, but it’s actually been around a long time. And you can learn a lot from reading about the history and understanding it, something I’ve given a talk on and written a bit about. So.Corey: My philosophy around developer relations for a while has been that in many cases, its biggest obstacle is the way that it is great at telling stories about fantastically complex, deeply technical things; it can tell stories about almost anything except itself. And I keep seeing similar expressions of the same problem again, and again, and again. I mean, AWS, where you worked, as an example: they love to talk about their developer advocates, and you read the job descriptions and these are high-level roles with sweeping responsibilities, broad basis of experience being able to handle things at a borderline executive level. And then they almost neuter the entire thing by slapping a developer advocate title on top of those people, which means that some of the people that would be most effectively served by talking to them will dismiss them as, “Well, I’m a director”—or a VP—“What am I going to do talking to a developer advocate?” It feels like there’s a swing and a miss as far as encapsulating the value that the function provides.I want to be clear, I am not sitting here shitting on DevRel or its practitioners, I see a problem with how it [laugh] is being expressed. Now, feel free to argue with me and just scream at me for the next 20 minutes, and this becomes a real short show. But—Brandon: [laugh].Corey: —It’ll be great. Hit me.Brandon: No, you’re correct in many ways, which makes me sad because these are the same conversations that I’ve been having for the 11, 12 years that I’ve been in DevRel now. And I thought we would have moved past this at some point, but the problem is that we are bad at advocating for advocacy. We do a bad job of relating to people about DevRel because we spend so much time worried about stuff that doesn’t really matter. And we get very loud voices in the echo chamber screaming about titles and evangelism versus advocate versus community manager, and which department you should report up to, and all of these things that ultimately don’t matter. And it just seems like bickering from the outside. I think that the core of what we do is super awesome. And I don’t think it’s very hard to articulate. It’s just that we don’t spend the time to do that.Corey: It’s always odd to me when I talk to someone like, “Oh, you’re in DevRel. What does that mean?” And their immediate response is, “Well, it’s not marketing, I’ll tell you that.” It’s feels like there might be some trauma that is being expressed in some strange ways. I do view it as marketing, personally, and people who take umbrage at that don’t generally tend to understand what marketing is.Yeah, you can look at any area of business or any function and judge it by some of the worst examples that we’ve all seen, but when someone tells me they work in sales, I don’t automatically assume that they are sending me horrifyingly passive-aggressive drip campaigns, or trying to hassle me in a car lot. It’s no, there’s a broad spectrum of people. Just like I don’t assume that you’re an engineer. And I immediately think, oh, you can’t solve FizzBuzz on a whiteboard. No, there’s always going to be a broad spectrum of experience.Marketing is one of those awesome areas of business that’s dramatically misunderstood a lot. Similarly to the fact that, you know, DevRel can’t tell stories, you think marketing could tell stories about itself, but it’s still struggles, too, in a bunch of ways. But I do believe that even if they’re not one of the same, developer relations and marketing are aligned around an awful lot of things like being able to articulate value that is hard to quantify.Brandon: I completely agree with that. And if I meet someone in DevRel that starts off the conversation by saying that they’re not in marketing, then I know they’re probably not that great at their job. I mean, I think there’s a place of tech hubris, where we want to disrespect anything that’s not a hard skill where it’s not putting zeros and ones into a chip—Corey: And spoiler, they’re all very hard skills.Brandon: [laugh]. Yeah. And so, first off, like, stop disrespecting marketing. It’s important; your business probably wouldn’t survive if you didn’t have it. And second of all, you’re not immune to it, right?Like, Heartbleed had a logo and a name for vulnerability because tech people are so susceptible to it, right? People don’t just wake up and wait in line for three days for a new iPhone because tech marketing doesn’t work, right?Corey: “Oh, tech marketing doesn’t work on me,” says someone who’s devoted last five years of their life to working on Kubernetes. Yeah, sure it doesn’t.Brandon: Yeah exactly. So, that whole perspective is silly. I think part of the problem is that they don’t want to invest in learning how to communicate what they do to a marketing org. They don’t want to spend the time to say, “Here’s how the marketing world thinks, and here’s how we can fit into that perspective.” They want to come in and say, “Well, you don’t understand DevRel. Let me define DevRel for you and tell you what we do.” And all those sorts of things. It’s too prescriptive and less collaborative.Corey: Anytime you start getting into the idea of metrics around how do you measure someone in a developer advocacy role, the answer is, “Well, your metrics that you’re using are wrong, and any metrics you use are wrong, and there’s no good way to do it.” And I am sympathetic to that. When I started this place, I knew that if I went to a bunch of events and did my thing, good things would happen for the business. And how did I articulate that? Gut feel, but when you own the place, you can do that.Whereas when you are a function inside of another org, inside of another org, and you start looking at from the executive leadership position at these things, it’s, “Okay, so let me get this straight. You cost as much as an engineer, you cost as much as that again, in your expenses because you’re traveling all the time, you write zero production code, whenever people ask you what it is you do here, you have a very strange answer, and from what we can tell, it looks like you hang out with your friends in exotic locations, give a 15-minute talk from time to time that mentions our name at the beginning, and nothing else relevant to our business, and then you go around and the entire story is ‘just trust me, I’m adding value.’” Yeah, when it’s time to tighten belts and start cutting back, is it any wonder that the developer advocacy is often one of the first departments hit from that perspective?Brandon: It doesn’t surprise me. I mean, I’ve been a part of DevRel teams where we had some large number of events that we had attended for the year—I think 450-something—and the director of the team was very excited to show that off, right, you should have seen the CFOs face when he heard that, right, because all he sees is outgoing dollar signs. Like, how much expense? What’s the ROI on 450 events?Corey: Yeah, “450 events? That’s more than one a day. Okay, great. That’s a big number and I already know what we’re spending. Great. How much business came out of that?”And that’s when the hemming and hawing starts. Like, well, sort of, and yadda—and yeah, it doesn’t present well in the language that they are prepared to speak. But marketing can tell those stories because they have for ages. Like, “Okay, how much business came from our Superbowl ad?” “I dunno. The point is, is that there’s a brand awareness play, there’s the chance to remain top of the mental stack when people think about this space. And over the next few months, we can definitely see there’s been a dramatic uptick in our business. Now, how do we attribute that back? Well, I don’t know.”There’s a saying in marketing, that half of your marketing budget is wasted. Now, figuring out which half will spend the rest of your career, you’ll never get even close. Because people don’t know the journey that customers go through, not really. Even customers don’t often see it.Take this podcast, for example. I have sponsors that I do love and appreciate who say things from time to time on this show. And people will hear it and occasionally will become customers of those sponsors. But very often, it’s, “Oh, I heard about that on the podcast. I’ll Google it when I get to work and then I’ll have a conversation with my team and we’ll agree to investigate that.”And any UTM tracking has long since fallen by the wayside. You might get to that from discussions with users in their interview process, but very often, they won’t remember where it came up. And it’s one of those impossible to quantify things. Now, I sound like one of those folks where I’m trying to say, “Oh, buy sponsorships that you can never prove add value.” But that is functionally how advertising tends to work, back in the days before it spied on you.Brandon: Yeah, absolutely. And we’ve added a bunch of instrumentation to allow us to try and put that multi-touch attribution model together after the fact, but I’m still not sure that that’s worth the squeeze, right? You don’t get much juice out. One of the problems with metrics in DevRel is that the things that you can measure are very production-focused. It’s how many talks did you give? How many audience members did you reach?Some developer relations folks do actually write production code, so it might be how many of the official SDK that you support got downloaded? That can be more directly attributed to business impact, those sorts of things are fantastic. But a lot of it is kind of fuzzy and because it’s production-focused, it can lead to burnout because it’s disconnected from business impact. “It’s how many widgets did your line produce today?” “Well, we gave all these talks and we had 150,000 engaged developer hours.” “Well, cool, what was the business outcome?” And if you can’t answer that for your own team and for your own self in your role, that leads pretty quickly to burnout.Corey: Anytime you start measuring something and grading people based on it, they’re going to optimize for what you measure. For example, I send an email newsletter out, at time of this recording, to 31,000 people every week and that’s awesome. I also periodically do webinars about the joys of AWS bill optimization, and you know, 50 people might show up to one of those things. Okay, well, from a broad numbers perspective, yeah, I’d much rather go and send something out to those 31,000, folks until you realize that the kind of person that’s going to devote half an hour, forty-five minutes to having a discussion with you about AWS bill optimization is far likelier to care about this to the point where they become a customer than someone who just happens to be in an audience for something that is orthogonally-related. And that is the trick because otherwise, we would just all be optimizing for the single biggest platforms out there if oh, I’m going to go talk at this conference and that conference, not because they’re not germane to what we do, but because they have more people showing up.And that doesn’t work. When you see that even on the podcast world, you have Joe Rogan, as the largest podcast in the world—let’s not make too many comparisons in different ways because I don’t want to be associated with that kind of tomfoolery—but there’s a reason that his advertisers, by and large, are targeting a mass-market audience, whereas mine are targeting B2B SaaS, by and large. I’m not here shilling for various mattress companies. I’m instead talking much more about things that solve the kind of problem that listeners to this show are likely to have. It’s the old-school of thought of advertising, where this is a problem that is germane to a certain type of audience, and that certain type of audience listens to shows like this. That was my whole school of thought.Brandon: Absolutely. I mean, the core value that you need to do DevRel, in my opinion is empathy. It’s all about what Maya Angelou said, right? “People may not remember what you said, but they’ll definitely remember how you made them feel.” And I found that to be incredibly true.Like, the moments that I regret the most in DevRel are the times when someone that I’ve met and spent time with before comes up to have a conversation and I don’t remember them because I met 200 people that night. And then I feel terrible, right? So, those are the metrics that I use internally. It’s hearts and minds. It’s how do people feel? Am I making them feel empowered and better at their craft through the work that I do?That’s why I love DevRel. If I didn’t get that fulfillment, I’d go write code again. But I don’t get that sense of satisfaction, and wow, I made an impact on this person’s trajectory through their career that I do from DevRel. So.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it’s still somebody’s problem. And a big part of that responsibility is app security from code to cloud. And that’s where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you’re actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That’s S-N-Y-K.co/screamCorey: The way that I tend to see it, too, is that there’s almost a bit of a broadening of DevRel. And let’s be clear, it’s a varied field with a lot of different ways to handle that approach. I’m have a terrible public speaker, so I’m not going to ever succeed in DevRel. Well, that’s certainly not true. People need to write blog posts; people need to wind up writing some of the sample code, in some cases; people need to talk to customers in a small group environment, as opposed to in front of 3000 people and talk about the things that they’re seeing, and the rest.There’s a broad field and different ways that it applies. But I also see that there are different breeds of developer advocate as well. There are folks, like you for example. You and I have roughly the same amount of time in the industry working on different things, whereas there’s also folks who it seems like they graduate from a boot camp, and a year later, they’re working in a developer advocacy role. Does that mean that they’re bad developer advocates?I don’t think so, but I think that if they try and present things the same way that you were I do from years spent in the trenches working on these things, they don’t have that basis of experience to fall back on, so they need to take a different narrative path. And the successful ones absolutely do.Brandon: Yeah.Corey: I think it’s a nuanced and broad field. I wish that there was more acceptance and awareness of that.Brandon: That’s absolutely true. And part of the reason people criticize DevRel and don’t take it seriously, as they say, “Well, it’s inconsistent. This org, it reports to product; or, this org, it reports up to marketing; this other place, it’s part of engineering.” You know, it’s poorly defined. But I think that’s true of a lot of roles in tech.Like, engineering is usually done a different way, very differently at some orgs compared to others. Product teams can have completely different methodologies for how they track and manage and estimate their time and all of those things. So, I would like to see people stop using that as a cudgel against the whole profession. It just doesn’t make any sense. At the same time, two of the best evangelist I ever hired were right out of university, so you’re completely correct.The key thing to keep in mind there is, like, who’s the audience, right, because ultimately, it’s about building trust with the audience. There’s a lot of rooms where if you and I walk into the room; if it’s like a college hackathon, we’re going to have a—[laugh], we’re going to struggle.Corey: Yeah, we have some real, “Hello fellow kids,” energy going on when we do that.Brandon: Yeah. Which is also why I think it’s incredibly important for developer relations teams to be aware of the makeup of their team. Like, how diverse is your team, and how diverse are the audiences you’re speaking to? And if you don’t have someone who can connect, whether it’s because of age or lived experience or background, then you’re going to fail because like I said that the number one thing you need to be successful in this role is empathy, in my opinion.Corey: I think that a lot of the efforts around a lot of this—trying to clarify what it is—some cases gone in well, I guess I’m going to call it the wrong direction. And I know that sounds judgy and I’m going to have to live with that, I suppose, but talk to me a bit about the, I guess, rebranding that we’ve seen in some recent years around developer advocates. Specifically, like, I like calling folks DevRelopers because it’s cutesy, it’s a bit of a portmanteau. Great. But it’s also not something I seriously suggest most people put on business cards.But there are people who are starting to, I think, take a similar joke and actually identify with it where they call themselves developer avocados, which I don’t fully understand. I have opinions on it, but again, having opinions that are not based in data is something I try not to start shouting from the rooftops wherever I can. You live in that world a lot more posted than I do, where do you stand?Brandon: So, I think it was well-intentioned and it was an attempt to do some of the awareness and brand building for DevRel, broadly, that we had lacked. But I see lots of problems with it. One, we already struggle to be taken seriously in many instances, as we’ve been discussing, and I don’t think we do ourselves any favors by giving ourselves cutesy nicknames that sort of infantilize the role like I can’t think of any other job that has a pet name for the work that they do.Corey: Yeah. The “ooh-woo accounting”. Yeah, I sort of don’t see that happening very often in most business orgs.Brandon: Yeah. It’s strange to me at the same time, a lot of the people who came up with it and popularized it are people that I consider friends and good colleagues. So hopefully, they won’t be too offended, but I really think that it kind of set us back in many ways. I don’t want to represent the work that I do with an emoji.Corey: Funny, you bring that up. As we record this through the first recording, I have on my new ridiculous desktop computer thing from Apple, which I have named after a—you know, the same naming convention that you would expect from an AWS region—it’s us-shitpost-one. Instead of the word shit, it has the poop emoji. And you’d be amazed at the number of things that just melt when you start trying to incorporate that. GitHub has a problem with that being the name of an SSH key, for example.I don’t know if I’ll keep it or I’ll just fall back to just spelling words out, but right now, at least, it really is causing all kinds of strange computer problems. Similarly, it causes strange cultural problems when you start having that dissonance and seeing something new and different like that in a business context. Because in some cases, yeah, it helps you interact with your audience and build rapport; in many others, it erodes trust and confidence that you know what you’re talking about because people expect things to be cast a certain way. I’m not saying they’re right. There’s a shitload of bias that bakes into that, but at the same time, I’d like to at least bias for choosing when and where I’m going to break those expectations.There’s a reason that increasingly, my Duckbillgroup.com website speaks in business terms, rather than in platypus metaphors, whereas lastweekinaws.com, very much leans into the platypus. And that is the way that the branding is breaking down, just because people expect different things in different places.Brandon: Yeah and, you know, this framing matters. And I’ve gone through two exercises now where I’ve helped rename an evangelism team to an advocacy team, not because I think it’s important to me—it’s a bunch of bikeshedding—but it has external implications, right? Especially evangelism, in certain parts of the world, has connotations. It’s just easier to avoid those. And how we present ourselves, the titles that we choose are important.I wish we would spend way less time arguing about them, you know, advocacy has won evangelism, don’t use it. DevRel, if you don’t want to pick one, great. DevRel is broader umbrella. If you’ve got community managers, people who can’t write code that do things involving your events or whatever, program managers, if they’re on your team, DevRel, great description. I wish we could just settle that. Lots of wasted air discussing that one.Corey: Constantly. It feels like this is a giant distraction that detracts from the value of DevRel. Because I don’t know about you, but when I pick what I want to do next in my career, the things I want to explain to people and spend that energy on are never, I want to explain what it is that I do. Like I’ve never liked those approaches where you have to first educate someone before they’re going to be in a position where they want to become your customer.I think, honestly, that’s one of the things that Datadog has gotten very right. One of the early criticisms lobbed against Datadog when it first came out was, “Oh, this is basically monitoring by Fisher-Price.” Like, “This isn’t the deep-dive stuff.” Well yeah, but it turns out a lot of your buying audience are fundamentally toddlers with no visibility into what’s going on. For an awful lot of what I do, I want it to be click, click, done.I am a Datadog customer for a reason. It’s not because I don’t have loud and angry opinions about observability; it’s because I just want there to be a dashboard that I can look at and see what’s working, what’s not, and do I need to care about things today? And it solves that job admirably because if I have those kinds of opinions about every aspect, I’m never going to be your customer anyway, or anyone’s customer. I’m going to go build my own and either launch a competitor or realize this is my what I truly love doing and go work at a company in this space, possibly yours. There’s something to be said for understanding the customer journey that those customers do not look like you.And I think that’s what’s going on with a lot of the articulation around what developer relations is or isn’t. The people on stage who go to watch someone in DevRel give a talk, do not care, by and large, what DevRel is. They care about the content that they’re about to hear about, and when the first half of it is explaining what the person’s job is or isn’t, people lose interest. I don’t even like intros at the beginning of a talk. Give me a hook. Talk for 45 seconds. Give me a story about why I should care before you tell me who you are, what your credentials are, what your job title is, who you work for. Hit me with something big upfront and then we’ll figure it out from there.Brandon: Yeah, I agree with you. I give this speaking advice to people constantly. Do not get up on stage and introduce yourself. You’re not a carnival hawker. You’re not trying to get people to roll up and see the show.They’re already sitting in the seat. You’ve established your credibility. If they had questions about it, they read your abstract, and then they went and checked you out on LinkedIn, right? So, get to the point; make it engaging and entertaining.Corey: I have a pet theory about what’s going on in some cases where, I think, on some level, it’s an outgrowth of an impostor-syndrome-like behavior, where people don’t believe that they deserve to be onstage talking about things, so they start backing up their bona fides to almost reassure themselves because they don’t believe that they should be up there and if they don’t believe it, why would anyone else. It’s the wrong approach. By holding the microphone, you inherently deserve to hold the microphone. And go ahead and tell your story. If people care enough to dig into you and who you are and well, “What is this person’s background, really?” Rest assured the internet is pretty easy to use these days, people will find out. So, let them do that research if they care. If they don’t, then there’s an entire line of people in this world who are going to dislike you or say you’re not qualified for what it is you’re doing or you don’t deserve it. Don’t be in that line, let alone at the front of it.Brandon: So, you mentioned imposter syndrome and it got me thinking a little bit. And hopefully this doesn’t offend anyone, but I kind of starting to think that imposter syndrome is in many ways invented by people to put the blame on you for something that’s their fault. It’s like a carbon footprint to the oil and gas industry, right? These companies can’t provide you psychological safety and now they’ve gone and convinced you that it’s your fault and that you’re suffering from this syndrome, rather than the fact that they’re not actually making you feel prepared and confident and ready to get up on that stage, even if it’s your first time giving a talk, right?Corey: I hadn’t considered it like that before. And again, I do tend to avoid straying into mental health territory on this show because I’m not an—Brandon: Yes.Corey: Expert. I’m a loud, confident white guy in tech. My failure mode is a board seat and a book deal, but I am not board-certified, let’s be clear. But I think you’re onto something here because early on in my career, I was very often faced with a whole lot of nebulous job description-style stuff and I was never sure if I was working on the right thing. Now that I’m at this stage of my career, and as you become more senior, you inherently find yourselves in roles, most of the time, that are themselves mired in uncertainty. That is, on some level, what seniority leads to.And that’s fine, but early on in your career, not knowing if you’re succeeding or failing, I got surprise-fired a number of times when I thought I was doing great. There are also times that I thought I was about to be fired on the spot and, “Come on in; shut the door.” And yeah, “Here’s a raise because you’re just killing it.” And it took me a few years after that point to realize, wait a minute. They were underpaying me. That’s what that was, and they hope they didn’t know.But it’s that whole approach of just trying to understand your place in the world. Do I rock? Do I suck? And it’s that constant uncertainty and unknowing. And I think companies do a terrible job, by and large, of letting people know that they’re okay, they’re safe, and they belong.Brandon: I completely agree. And this is why I would strongly encourage people—if you have the privilege—please do not work at a company that does not want you to bring your whole self to work, or that bans politics, or however they want to describe it. Because that’s just a code word for we won’t provide you psychological safety. Or if they’re going to, it ends at a very hard border somewhere between work and life. And I just don’t think anyone can be successful in those environments.Corey: I’m sure it’s possible, but it does bias for folks who, frankly, have a tremendous amount of privilege in many respects where I mentioned about, like, I’m a white dude in tech—you are too—and when we say things, we are presumed competent and people don’t argue with us by default. And that is a very easy to forget thing. Not everyone who looks like us is going to have very similar experiences. I have gotten it hilariously wrong before when I gave talks on how to wind up negotiating for salaries, for example, because well, it worked for me, what’s the problem? Yeah, I basically burned that talk with fire, redid the entire thing and wound up giving it with a friend of mine who was basically everything that I am not.She was an attorney, she was a woman of color, et cetera, et cetera. And suddenly, it was a much stronger talk because it wasn’t just, “How to Succeed for White Guys.” There’s value in that, but you also have to be open to hearing that and acknowledging that you were born on third; you didn’t hit a triple. There’s a difference. And please forgive the sports metaphor. They do not sound natural coming from me.Brandon: [laugh]. I don’t think I have anything more interesting to add on that topic.Corey: [laugh]. So, I really want to thank you for taking the time to speak with me today. If people want to learn more about what you’re up to and how you view the world, what’s the best place to find you.Brandon: So, I’m most active on Twitter at @bwest, but you know, it’s a mix of things so you may or may not just get tech. Most recently, I’ve been posting about a—Corey: Oh, heaven forbid you bring your whole self to school.Brandon: Right? I think most recently, I’ve been posting about a drill press that I’m restoring. So, all kinds of fun stuff on there.Corey: I don’t know it sounds kind of—wait for it—boring to me. Bud-dum-tiss.Brandon: [laugh]. [sigh]. I can’t believe I missed that one.Corey: You’re welcome.Brandon: Well, done. Well, done. And then I also will be hiring for a couple of developer relations folks at Datadogs soon, so if that’s interesting and you like the words I say about how to do DevRel, then reach out.Corey: And you can find all of that in the show notes, of course. I want to thank you for being so generous with your time. I really appreciate it.Brandon: Hey, thank you, Corey. I’m glad that we got to catch up after all this time. And hopefully get to chat with you again sometime soon.Corey: Brandon West, team lead for developer experience and tools advocacy at Datadog. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry and insulting comment that is talking about how I completely misunderstand the role of developer advocacy. And somehow that rebuttal features no fewer than 400 emoji shoved into it.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About ChrisChris Short has been a proponent of open source solutions throughout his over two decades in various IT disciplines, including systems, security, networks, DevOps management, and cloud native advocacy across the public and private sectors. He currently works on the Kubernetes team at Amazon Web Services and is an active Kubernetes contributor and Co-chair of OpenGitOps. Chris is a disabled US Air Force veteran living with his wife and son in Greater Metro Detroit. Chris writes about Cloud Native, DevOps, and other topics at ChrisShort.net. He also runs the Cloud Native, DevOps, GitOps, Open Source, industry news, and culture focused newsletter DevOps’ish.Links Referenced: DevOps’ish: https://devopsish.com/ EKS News: https://eks.news/ Containers from the Couch: https://containersfromthecouch.com opengitops.dev: https://opengitops.dev ChrisShort.net: https://chrisshort.net Twitter: https://twitter.com/ChrisShort TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. Coming back to us since episode two—it’s always nice to go back and see the where are they now type of approach—I am joined by Senior Developer Advocate at AWS Chris Short. Chris, been a few years. How has it been?Chris: Ha. Corey, we have talked outside of the podcast. But it’s been good. For those that have been listening, I think when we recorded I wasn’t even—like, when was season two, what year was that? [laugh].Corey: Episode two was first pre-pandemic and the rest. I believe—Chris: Oh. So, yeah. I was at Red Hat, maybe, when I—yeah.Corey: Yeah. You were doing Red Hat stuff, back when you got to work on open-source stuff, as opposed to now, where you’re not within 1000 miles of that stuff, right?Chris: Actually well, no. So, to be clear, I’m on the EKS team, the Kubernetes team here at AWS. So, when I joined AWS in October, they were like, “Hey, you do open-source stuff. We like that. Do more.” And I was like, “Oh, wait, do more?” And they were like, “Yes, do more.” “Okay.”So, since joining AWS, I’ve probably done more open-source work than the three years at Red Hat that I did. So, that’s kind of—you know, like, it’s an interesting point when I talk to people about it because the first couple months are, like—you know, my friends are like, “So, are you liking it? Are you enjoying it? What’s going on?” And—Corey: Do they beat you with reeds? Like, all the questions people have about companies? Because—Chris: Right. Like, I get a lot of random questions about Amazon and AWS that I don’t know the answer to.Corey: Oh, when I started telling people, I fixed Amazon bills, I had to quickly pivot that to AWS bills because people started asking me, “Well, can you save me money on underpants?” It’s I—Chris: Yeah.Corey: How do you—fine. Get the prime credit card. It docks 5% off the bill, so there you go. But other than that, no, I can’t.Chris: No.Corey: It’s—Chris: Like, I had to call my bank this morning about a transaction that I didn’t recognize, and it was from Amazon. And I was like, that’s weird. Why would that—Corey: Money just flows one direction, and that’s the wrong direction from my employer.Chris: Yeah. Like, what is going on here? It shouldn’t have been on that card kind of thing. And I had to explain to the person on the phone that I do work at Amazon but under the Web Services team. And he was like, “Oh, so you’re in IT?”And I’m like, “No.” [laugh]. “It’s actually this big company. That—it’s a cloud company.” And they’re like, “Oh, okay, okay. Yeah. The cloud. Got it.” [laugh]. So, it’s interesting talking to people about, “I work at Amazon.” “Oh, my son works at Amazon distribution center,” blah, blah, blah. It’s like, cool. “I know about that, but very little. I do this.”Corey: Your son works in Amazon distribution center. Is he a robot? Is normally my next question on that? Yeah. That’s neither here nor there.So, you and I started talking a while back. We both write newsletters that go to a somewhat similar audience. You write DevOps’ish. I write Last Week in AWS. And recently, you also have started EKS News because, yeah, the one thing I look at when I’m doing these newsletters every week is, you know what I want to do? That’s right. Write more newsletters.Chris: [laugh].Corey: So, you are just a glutton for punishment? And, yeah, welcome to the addiction, I suppose. How’s it been going for you?Chris: It’s actually been pretty interesting, right? Like, we haven’t pushed it very hard. We’re now starting to include it in things. Like we did Container Day; we made sure that EKS news was on the landing page for Container Day at KubeCon EU. And you know, it’s kind of just grown organically since then.But it was one of those things where it’s like, internally—this happened at Red Hat, right—when I started live streaming at Red Hat, the ultimate goal was to do our product management—like, here’s what’s new in the next version thing—do those live so anybody can see that at any point in time anywhere on Earth, the second it’s available. Similar situation to here. This newsletter actually is generated as part of a report my boss puts together to brief our other DAs—or developer advocates—you know, our solutions architects, the whole nine yards about new EKS features. So, I was like, why can’t we just flip that into a weekly newsletter, you know? Like, I can pull from the same sources you can.And what’s interesting is, he only does the meeting bi-weekly. So, there’s some weeks where it’s just all me doing it and he ends up just kind of copying and pasting the newsletter into his document, [laugh] and then adds on for the week. But that report meeting for that team is now getting disseminated to essentially anyone that subscribes to eks.news. Just go to the site, there’s a subscribe thing right there. And we’ve gotten 20 issues in and it’s gotten rave reviews, right?Corey: I have been a subscriber for a while. I will say that it has less Chris Short personality—Chris: Mm-hm.Corey: —to it than DevOps’ish does, which I have to assume is by design. A lot of The Duckbill Group’s marketing these days is no longer in my voice, rather intentionally, because it turns out that being a sarcastic jackass and doing half-billion dollar AWS contracts can not to be the most congruent thing in the world. So okay, we’re slowly ameliorating that. It’s professional voice versus snarky voice.Chris: Well, and here’s the thing, right? Like, I realized this year with DevOps’ish that, like, if I want to take a week off, I have to do, like, what you did when your child was born. You hired folks to like, do the newsletter for you, or I actually don’t do the newsletter, right? It’s binary: hire someone else to do it, or don’t do it. So, the way I structured this newsletter was that any developer advocate on my team could jump in and take over the newsletter so that, you know, if I’m off that week, or whatever may be happening, I, Chris Short, am not the voice. It is now the entire developer advocate team.Corey: I will challenge you on that a bit. Because it’s not Chris Short voice, that’s for sure, but it’s also not official AWS brand voice either.Chris: No.Corey: It is clearly written by a human being who is used to communicating with the audience for whom it is written. And that is no small thing. Normally, when oh, there’s a corporate newsletter; that’s just a lot of words to say it’s bad. This one is good. I want to be very clear on that.Chris: Yeah, I mean, we have just, like, DevOps’ish, we have sections, just like your newsletter, there’s certain sections, so any new, what’s new announcements, those go in automatically. So, like, that can get delivered to your inbox every Friday. Same thing with new blog posts about anything containers related to EKS, those will be in there, then Containers from the Couch, our streaming platform, essentially, for all things Kubernetes. Those videos go in.And then there’s some ecosystem news as well that I collect and put in the newsletter to give people a broader sense of what’s going on out there in Kubernetes-land because let’s face it, there’s upstream and then there’s downstream, and sometimes those aren’t in sync, and that’s normal. That’s how Kubernetes kind of works sometimes. If you’re running upstream Kubernetes, you are awesome. I appreciate you, but I feel like that would cause more problems and it’s worse sometimes.Corey: Thank you for being the trailblazers. The rest of us can learn from your misfortune.Chris: [laugh]. Yeah, exactly. Right? Like, please file your bugs accordingly. [laugh].Corey: EKS is interesting to me because I don’t see a lot of it, which is, probably, going to get a whole lot of, “Wait, what?” Moments because wait, don’t you deal with very large AWS bills? And I do. But what I mean by that is that EKS, until you’re using its Fargate expression, charges for the control plane, which rounds to no money, and the rest is running on EC2 instances running in a company’s account. From the billing perspective, there is no difference between, “We’re running massive fleets of EKS nodes.” And, “We’re managing a whole bunch of EC2 instances by hand.”And that feels like an interesting allegory for how Kubernetes winds up expressing itself to cloud providers. Because from a billing perspective, it just looks like one big single-tenant application that has some really strange behaviors internally. It gets very chatty across AZs when there’s no reason to, and whatnot. And it becomes a very interesting study in how to expose aspects of what’s going on inside of those containers and inside of the Kubernetes environment to the cloud provider in a way that becomes actionable. There are no good answers for this yet, but it’s something I’ve been seeing a lot of. Like, “Oh, I thought you’d be running Kubernetes. Oh, wait, you are and I just keep forgetting what I’m looking at sometimes.”Chris: So, that’s an interesting point. The billing is kind of like, yeah, it’s just compute, right? So—Corey: And my insight into AWS and the way I start thinking about it is always from a billing perspective. That’s great. It’s because that means the more expensive the services, the more I know about it. It’s like, “IAM. What is that?” Like, “Oh, I have no idea. It’s free. How important could it be?” Professional advice: do not take that philosophy, ever.Chris: [laugh]. No. Ever. No.Corey: Security: it matters. Oh, my God. It’s like you’re all stars. Your IAM policy should not be. I digress.Chris: Right. Yeah. Anyways, so two points I want to make real quick on that is, one, we’ve recently released an open-source project called Carpenter, which is really cool in my purview because it looks at your Kubernetes file and says, “Oh, you want this to run on ARM instance.” And you can even go so far as to say, right, here’s my limits, and it’ll find an instance that fits those limits and add that to your cluster automatically. Run your pod on that compute as long as it needs to run and then if it’s done, it’ll downsize—eventually, kind of thing—your cluster.So, you can basically just throw a bunch of workloads at it, and it’ll auto-detect what kind of compute you will need and then provision it for you, run it, and then be done. So, that is one-way folks are probably starting to save money running EKS is to adopt Carpenter as your autoscaler as opposed to the inbuilt Kubernetes autoscaler. Because this is instance-aware, essentially, so it can say, like, “Oh, your massive ARM application can run here,” because you know, thank you, Graviton. We have those processors in-house. And you know, you can run your ARM64 instances, you can run all the Intel workloads you want, and it’ll right size the compute for your workloads.And I’ll look at one container or all your containers, however you want to configure it. Secondly, the good folks over at Kubecost have opencost, which is the open-source version of Kubecost, basically. So, they have a service that you can run in your clusters that will help you say, “Hey, maybe this one notes too heavy; maybe this one notes too light,” and you know, give you some insights into Kubernetes spend that are a little bit more granular as far as usage and things like that go. So, those two projects right there, I feel like, will give folks an optimal savings experience when it comes to Kubernetes. But to your point, it’s just compute, right? And that’s really how we treat it, kind of, here internally is that it’s a way to run… compute, Kubernetes, or ECS, or any of those tools.Corey: A fairly expensive one because ignoring entirely for a second the actual raw cost of compute, you also have the other side of it, which is in every environment, unless you are doing something very strange or pre-funding as a one-person startup in your spare time, your payroll costs will it—should—exceed your AWS bill by a fairly healthy amount. And engineering time is always more expensive than services time. So, for example, looking at EKS, I would absolutely recommend people use that rather than rolling their own because—Chris: Rolling their own? Yeah.Corey: —get out of that engineering space where your time is free. I assure you from a business context, it is not. So, there’s always that question of what you can do to make things easier for people and do more of the heavy lifting.Chris: Yeah, and to your rather cheeky point that there’s 17 ways to run a container on AWS, it is answering that question, right? Like those 17 ways, like, how much of this do you want to run yourself, you could run EKS distro on EC2 instances if you want full control over your environment.Corey: And then run IoT Greengrass core on top within that cluster—Chris: Right.Corey: So, I can run my own Lambda function runtime, so I’m not locked in. Also, DynamoDB local so I’m not locked into AWS. At which point I have gone so far around the bend, no one can help me.Chris: Well—Corey: Pro tip, don’t do that. Just don’t do that.Chris: But to your point, we have all these options for compute, and specifically containers because there’s a lot of people that want to granularly say, “This is where my engineering team gets involved. Everything else you handle.” If I want EKS on Spot Instances only, you can do that. If you want EKS to use Carpenter and say only run ARM workloads, you can do that. If you want to say Fargate and not have anything to manage other than the container file, you can do that.It’s how much does your team want to manage? That’s the customer obsession part of AWS coming through when it comes to containers is because there’s so many different ways to run those workloads, but there’s so many different ways to make sure that your team is right-sized, based off the services you’re using.Corey: I do want to change gears a bit here because you are mostly known for a couple of things: the DevOps’ish newsletter because that is the oldest and longest thing you’ve been doing the time that I’ve known you; EKS, obviously. But when prepping for this show, I discovered you are now co-chair of the OpenGitOps project.Chris: Yes.Corey: So, I have heard of GitOps in the context of, “Oh, it’s just basically your CI/CD stuff is triggered by Git events and whatnot.” And I’m sitting here going, “Okay, so from where you’re sitting, the two best user interfaces in the world that you have discovered are YAML and Git.” And I just have to start with the question, “Who hurt you?”Chris: [laugh]. Yeah, I share your sentiment when it comes to Git. Not so much with YAML, but I think it’s because I’m so used to it. Maybe it’s Stockholm Syndrome, maybe the whole YAML thing. I don’t know.Corey: Well, it’s no XML. We’ll put it that way.Chris: Thankfully, yes because if it was, I would have way more, like, just template files laying around to build things. But the—Corey: And rage. Don’t forget rage.Chris: And rage, yeah. So, GitOps is a little bit more than just Git in IaC—infrastructure as Code. It’s more like Justin Garrison, who’s also on my team, he calls it infrastructure software because there’s four main principles to GitOps, and if you go to opengitops.dev, you can see them. It’s version one.So, we put them on the website, right there on the page. You have to have a declared state and that state has to live somewhere. Now, it’s called GitOps because Git is probably the most full-featured thing to put your state in, but you could use an S3 bucket and just version it, for example. And make it private so no one else can get to it.Corey: Or you could use local files: copy-of-copy-of-this-thing-restored-parentheses-use-this-one-dot-final-dot-doc-dot-zip. You know, my preferred naming convention.Chris: Ah, yeah. Wow. Okay. [laugh]. Yeah.Corey: Everything I touch is terrifying.Chris: Yes. Geez, I’m sorry. So first, it’s declarative. You declare your state. You store it somewhere. It’s versioned and immutable, like I said. And then pulled automatically—don’t focus so much on pull—but basically, software agents are applying the desired state from source. So, what does that mean? When it’s—you know, the fourth principle is implemented, continuously reconciled. That means those software agents that are checking your desired state are actually putting it back into the desired state if it’s out of whack, right? So—Corey: You’re talking about agents running it persistently on instances, validating—Chris: Yes.Corey: —a checkpoint on a cron. How is this meaningfully different than a Puppet agent running in years past? Having spent I learned to speak publicly by being a traveling trainer for Puppet; same type of model, and in fact, when I was at Pinterest, we wound up having a fair bit—like, that was their entire model, where they would have—the Puppet’s code would live in an S3 bucket that was then copied down, I believe, via Git, and then applied to the instance on a schedule. Like, that sounds like this was sort of a early days GitOps.Chris: Yeah, exactly. Right? Like so it’s, I like to think of that as a component of GitOps, right? DevOps, when you talk about DevOps in general, there’s a lot of stuff out there. There’s a lot of things labeled DevOps that maybe are, or maybe aren’t sticking to some of those DevOps core things that make you great.Like the stuff that Nicole Forsgren writes about in books, you know? Accelerate is on my desk for a reason because there’s things that good, well-managed DevOps practices do. I see GitOps as an actual implementation of DevOps in an open-source manner because all the tooling for GitOps these days is open-source and it all started as open-source. Now, you can get, like, Flux or Argo—Argo, specifically—there’s managed services out there for it, you can have Flux and not maintain it, through an add-on, on EKS for example, and it will reconcile that state for you automatically. And the other thing I like to say about GitOps, specifically, is that it moves at the speed of the Kubernetes Audit Log.If you’ve ever looked at a Kubernetes audit log, you know it’s rather noisy with all these groups and versions and kinds getting thrown out there. So, GitOps will say, “Oh, there’s an event for said thing that I’m supposed to be watching. Do I need to change anything? Yes or no? Yes? Okay, go.”And the change gets applied, or, “Hey, there’s a new Git thing. Pull it in. A change has happened inGit I need to update it.” You can set it to reconcile on events on time. It’s like a cron or it’s like an event-driven architecture, but it’s combined.Corey: How does it survive the stake through the heart of configuration management? Because before I was doing all this, I wasn’t even a T-shaped engineer: you’re broad across a bunch of things, but deep in one or two areas, and one of mine was configuration management. I wrote part of SaltStack, once upon a time—Chris: Oh.Corey: —due to a bunch of very strange coincidences all hitting it once, like, I taught people how to use Puppet. But containers ultimately arose and the idea of immutable infrastructure became a thing. And these days when we were doing full-on serverless, well, great, I just wind up deploying a new code bundle to the Lambdas function that I wind up caring about, and that is a immutable version replacement. There is no drift because there is no way to log in and change those things other than through a clear deployment of this as the new version that goes out there. Where does GitOps fit into that imagined pattern?Chris: So, configuration management becomes part of your approval process, right? So, you now are generating an audit log, essentially, of all changes to your system through the approval process that you set up as part of your, how you get things into source and then promote that out to production. That’s kind of the beauty of it, right? Like, that’s why we suggest using Git because it has functions, like, requests and issues and things like that you can say, “Hey, yes, I approve this,” or, “Hey, no, I don’t approve that. We need changes.” So, that’s kind of natively happening with Git and, you know, GitLab, GitHub, whatever implementation of Git. There’s always, kind of—Corey: Uh, JIF-ub is, I believe, the pronunciation.Chris: JIF-ub? Oh.Corey: Yeah. That’s what I’m—Chris: Today, I learned. Okay.Corey: Exactly. And that’s one of the things that I do for my lasttweetinaws.com Twitter client that I build—because I needed it, and if other people want to use it, that’s great—that is now deployed to 20 different AWS commercial regions, simultaneously. And that is done via—because it turns out that that’s a very long to execute for loop if you start down that path—Chris: Well, yeah.Corey: I wound up building out a GitHub Actions matrix—sorry a JIF-ub—actions matrix job that winds up instantiating 20 parallel builds of the CDK deploy that goes out to each region as expected. And because that gets really expensive with native GitHub Actions runners for, like, 36 cents per deploy, and I don’t know how to test my own code, so every time I have a typo, that’s another quarter in the jar. Cool, but that was annoying for me so I built my own custom runner system that uses Lambda functions as runners running containers pulled from ECR that, oh, it just runs in parallel, less than three minutes. Every time I commit something between I press the push button and it is out and running in the wild across all regions. Which is awesome and also terrifying because, as previously mentioned, I don’t know how to test my code.Chris: Yeah. So, you don’t know what you’re deploying to 20 regions sometime, right?Corey: But it also means I have a pristine, re-composable build environment because I can—Chris: Right.Corey: Just automatically have that go out and the fact that I am making a—either merging a pull request or doing a direct push because I consider main to be my feature branch as whenever something hits that, all the automation kicks off. That was something that I found to be transformative as far as a way of thinking about this because I was very tired of having to tweak my local laptop environment to, “Oh, you didn’t assume the proper role and everything failed again and you broke it. Good job.” It wound up being something where I could start developing on more and more disparate platforms. And it finally is what got me away from my old development model of everything I build is on an EC2 instance, and that means that my editor of choice was Vim. I use the VS Code now for these things, and I’m pretty happy with it.Chris: Yeah. So, you know, I’m glad you brought up CDK. CDK gives you a lot of the capabilities to implement GitOps in a way that you could say, like, “Hey, use CDK to declare I need four Amazon EKS clusters with this size, shape, and configuration. Go.” Or even further, connect to these EKS clusters to RDS instances and load balancers and everything else.But you put that state into Git and then you have something that deploys that automatically upon changes. That is infrastructure as code. Now, when you say, “Okay, main is your feature branch,” you know, things happen on main, if this were running in Kubernetes across a fleet of clusters or the globe-wide in 20 regions, something like Flux or Argo would kick in and say, “There’s been a change to source, main, and we need to roll this out.” And it’ll start applying those changes. Now, what do you get with GitOps that you don’t get with your configuration?I mean, can you rollback if you ever have, like, a bad commit that’s just awful? I mean, that’s really part of the process with GitOps is to make sure that you can, A, roll back to the previous good state, B, roll forward to a known good state, or C, promote that state up through various environments. And then having that all done declaratively, automatically, and immutably, and versioned with an audit log, that I think is the real power of GitOps in the sense that, like, oh, so-and-so approve this change to security policy XYZ on this date at this time. And that to an auditor, you just hand them a log file on, like, “Here’s everything we’ve ever done to our system. Done.” Right?Like, you could get to that state, if you want to, which I think is kind of the idea of DevOps, which says, “Take all these disparate tools and processes and procedures and culture changes”—culture being the hardest part to adopt in DevOps; GitOps kind of forces a culture change where, like, you can’t do a CAB with GitOps. Like, those two things don’t fly. You don’t have a configuration management database unless you absolutely—Corey: Oh, you CAB now but they’re all the comments of the pull request.Chris: Right. Exactly. Like, don’t push this change out until Thursday after this other thing has happened, kind of thing. Yeah, like, that all happens in GitHub. But it’s very democratizing in the sense that people don’t have to waste time in an hour-long meeting to get their five minutes in, right?Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: So, would it be overwhelmingly cynical to suggest that GitOps is the means to implement what we’ve all been pretending to have implemented for the last decade when giving talks at conferences?Chris: Ehh, I wouldn’t go that far. I would say that GitOps is an excellent way to implement the things you’ve been talking about at all these conferences for all these years. But keep in mind, the technology has changed a lot in the, what 11, 12 years of the existence of DevOps, now. I mean, we’ve gone from, let’s try to manage whole servers immutably to, “Oh, now we just need to maintain an orchestration platform and run containers.” That whole compute interface, you go from SSH to a Docker file, that’s a big leap, right?Like, you don’t have bespoke sysadmins; you have, like, a platform team. You don’t have DevOps engineers; they’re part of that platform team, or DevOps teams, right? Like, which was kind of antithetical to the whole idea of DevOps to have a DevOps team. You know, everybody’s kind of in the same boat now, where we see skill sets kind of changing. And GitOps and Kubernetes-land is, like, a platform team that manages the cluster, and its state, and health and, you know, production essentially.And then you have your developers deploying what they want to deploy in when whatever namespace they’ve been given access to and whatever rights they have. So, now you have the potential for one set of people—the platform team—to use one set of GitOps tooling, and your applications teams might not like that, and that’s fine. They can have their own namespaces with their own tooling in it. Like, Argo, for example, is preferred by a lot of developers because it has a nice UI with green and red dots and they can show people and it looks nice, Flux, it’s command line based. And there are some projects out there that kind of take the UI of Argo and try to run Flux underneath that, and those are cool kind of projects, I think, in my mind, but in general, right, I think GitOps gives you the choice that we missed somewhat in DevOps implementations of the past because it was, “Oh, we need to go get cloud.” “Well, you can only use this cloud.” “Oh, we need to go get this thing.” “Well, you can only use this thing in-house.”And you know, there’s a lot of restrictions sometimes placed on what you can use in your environment. Well, if your environment is Kubernetes, how do you restrict what you can run, right? Like you can’t have an easily configured say, no open-source policy if you’re running Kubernetes. [laugh] so it becomes, you know—Corey: Well, that doesn’t stop some companies from trying.Chris: Yeah, that’s true. But the idea of, like, enabling your developers to deploy at will and then promote their changes as they see fit is really the dream of DevOps, right? Like, same with production and platform teams, right? I want to push my changes out to a larger system that is across the globe. How do I do that? How do I manage that? How do I make sure everything’s consistent?GitOps gives you those ways, with Kubernetes native things like customizations, to make consistent environments that are robust and actually going to be reconciled automatically if someone breaks the glass and says, “Oh, I need to run this container immediately.” Well, that’s going to create problems because it’s deviated from state and it’s just that one region, so we’ll put it back into state.Corey: It’ll be dueling banjos, at some point. You’ll try and doing something manually, it gets reverted automatically. I love that pattern. You’ll get bored before the computer does, always.Chris: Yeah. And GitOps is very new, right? When you think about the lifetime of GitOps, I think it was coined in, like, 2018. So, it’s only four years old, right? When—Corey: I prefer it to ChatOps, at least, as far as—Chris: Well, I mean—Corey: —implementation and expression of the thing.Chris: —ChatOps was a way to do DevOps. I think GitOps—Corey: Well, ChatOps is also a way to wind up giving whoever gets access to your Slack workspace root in production.Chris: Mmm.Corey: But that’s neither here nor there.Chris: Mm-hm.Corey: It’s yeah, we all like to pretend that’s not a giant security issue in our industry, but that’s a topic for another time.Chris: Yeah. And that’s why, like, GitOps also depends upon you having good security, you know, and good authorization and approval processes. It enforces that upon—Corey: Yeah, who doesn’t have one of those?Chris: Yeah. If it’s a sole operation kind of deal, like in your setup, your case, I think you kind of got it doing right, right? Like, as far as GitOps goes—Corey: Oh, to be clear, we are 11 people and we do have dueling pull requests and all the rest.Chris: Right, right, right.Corey: But most of the stuff I talk about publicly is not our production stuff, so it really is just me. Just as a point of clarity there. I’ve n—the 11 people here do not all—the rest of you don’t just sit there and clap as I do all the work.Chris: Right.Corey: Most days.Chris: No, I’m sure they don’t. I’m almost certain they don’t clap… for you. I mean, they would—Corey: No. No, they try and talk me out of it in almost every case.Chris: Yeah, exactly. So, the setup that you, Corey Quinn, have implemented to deploy these 20 regions is kind of very GitOps-y, in the sense that when main changes, it gets updated. Where it’s not GitOps-y is what if the endpoint changes? Does it get reconciled? That’s the piece you’re probably missing is that continuous reconciliation component, where it’s constantly checking and saying, “This thing out there is deployed in the way I want it. You know, the way I declared it to be in my source of truth.”Corey: Yeah, when you start having other people getting involved, there can—yeah, that’s where regressions enter. And it’s like, “Well, I know where things are so why would I change the endpoint?” Yeah, it turns out, not everyone has the state of the entire application in their head. Ideally it should live in—Chris: Yeah. Right. And, you know—Corey: —you know, Git or S3.Chris: —when I—yeah, exactly. When I think about interactions of the past coming out as a new DevOps engineer to work with developers, it’s always been, will developers have access to prod or they don’t? And if you’re in that environment with—you’re trying to run a multi-billion dollar operation, and your devs have direct—or one Dev has direct access to prod because prod is in his brain, that’s where it’s like, well, now wait a minute. Prod doesn’t have to be only in your brain. You can put that in the codebase and now we know what is in your brain, right?Like, you can almost do—if you document your code, well, you can have your full lifecycle right there in one place, including documentation, which I think is the best part, too. So, you know, it encourages approval processes and automation over this one person has an entire state of the system in their head; they have to go in and fix it. And what if they’re not on call, or in Jamaica, or on a cruise ship somewhere kind of thing? Things get difficult. Like, for example, I just got back from vacation. We were so far off the grid, we had satellite internet. And let me tell you, it was hard to write an email newsletter where I usually open 50 to 100 tabs.Corey: There’s a little bit of internet out Californ-ie way.Chris: [laugh].Corey: Yeah it’s… it’s always weird going from, like, especially after pandemic; I have gigabit symmetric here and going even to re:Invent where I’m trying to upload a bunch of video and whatnot.Chris: Yeah. Oh wow.Corey: And the conference WiFi was doing its thing, and well, Verizon 5G was there but spotty. And well, yeah. Usual stuff.Chris: Yeah. It’s amazing to me how connectivity has become so ubiquitous.Corey: To the point where when it’s not there anymore, it’s what do I do with myself? Same story about people pushing back against remote development of, “Oh, I’m just going to do it all on my laptop because what happens if I’m on a plane?” It’s, yeah, the year before the pandemic, I flew 140,000 miles domestically and I was almost never hamstrung by my ability to do work. And my only local computer is an iPad for those things. So, it turns out that is less of a real world concern for most folks.Chris: Yeah I actually ordered the components to upgrade an old Nook that I have here and turn it into my, like, this is my remote code server, that’s going to be all attached to GitHub and everything else. That’s where I want to be: have Tailscale and just VPN into this box.Corey: Tailscale is transformative.Chris: Yes. Tailscale will change your life. That’s just my personal opinion.Corey: Yep.Chris: That’s not an AWS opinion or anything. But yeah, when you start thinking about your network as it could be anywhere, that’s where Tailscale, like, really shines. So—Corey: Tailscale makes the internet work like we all wanted to believe that it worked.Chris: Yeah. And Wireguard is an excellent open-source project. And Tailscale consumes that and puts an amazingly easy-to-use UI, and troubleshooting tools, and routing, and all kinds of forwarding capabilities, and makes it kind of easy, which is really, really, really kind of awesome. And Tailscale and Kubernetes—Corey: Yeah, ‘network’ and ‘easy’ don’t belong in the same sentence, but in this case, they do.Chris: Yeah. And trust me, the Kubernetes story in Tailscale, there is a lot of there. I understand you might want to not open ports in your VPC, maybe, but if you use Tailscale, that node is just another thing on your network. You can connect to that and see what’s going on. Your management cluster is just another thing on the network where you can watch the state.But it’s all—you’re connected to it continuously through Tailscale. Or, you know, it’s a much lighter weight, kind of meshy VPN, I would say, if I had to sum it up in one sentence. That was not on our agenda to talk about at all. Anyways. [laugh]Corey: No, no. I love how many different topics we talk about on these things. We’ll have to have you back soon to talk again. I really want to thank you for being so generous with your time. If people want to learn more about what you’re up to and how you view these things, where can they find you?Chris: Go to ChrisShort.net. So, Chris Short—I’m six-four so remember, it’s Short—dot net, and you will find all the places that I write, you can go to devopsish.com to subscribe to my newsletter, which goes out every week. This year. Next year, there’ll be breaks. And then finally, if you want to follow me on Twitter, Chris Short: at @ChrisShort on Twitter. All one word so you see two s’s. Like, it’s okay, there’s two s’s there.Corey: Links to all of that will of course be in the show notes. It’s easier for people to do the clicky-clicky thing as a general rule.Chris: Clicky things are easier than the wordy things, yes.Corey: Says the Kubernetes guy.Chris: Yeah. Says the Kubernetes guy. Yeah, you like that, huh? Like I said, Argo gives you a UI. [laugh].Corey: Thank you [laugh] so much for your time. I really do appreciate it.Chris: Thank you. This has been fun. If folks have questions, feel free to reach out. Like, I am not one of those people that hides behind a screen all day and doesn’t respond. I will respond to you eventually.Corey: I’m right here, Chris. Come on, come on. You’re calling me out in front of myself. My God.Chris: Egh. It might take a day or two, but I will respond. I promise.Corey: Thanks again for your time. This has been Chris Short, senior developer advocate at AWS. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice and if it’s YouTube, click the thumbs-up button. Whereas if you’ve hated this podcast, same thing, smash the buttons five-star review and leave an insulting comment that is written in syntactically correct YAML because it’s just so easy to do.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About SheeriAfter almost 2 decades as a database administrator and award-winning thought leader, Sheeri Cabral pivoted to technical product management. Her super power of “new customer” empathy informs her presentations and explanations. Sheeri has developed unique insights into working together and planning, having survived numerous reorganizations, “best practices”, and efficiency models. Her experience is the result of having worked at everything from scrappy startups such as Guardium – later bought by IBM – to influential tech companies like Mozilla and MongoDB, to large established organizations like Salesforce.Links Referenced: Collibra: https://www.collibra.com WildAid GitHub: https://github.com/wildaid Twitter: https://twitter.com/sheeri Personal Blog: https://sheeri.org TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored by our friends at Fortinet. Fortinet’s partnership with AWS is a better-together combination that ensures your workloads on AWS are protected by best-in-class security solutions powered by comprehensive threat intelligence and more than 20 years of cybersecurity experience. Integrations with key AWS services simplify security management, ensure full visibility across environments, and provide broad protection across your workloads and applications. Visit them at AWS re:Inforce to see the latest trends in cybersecurity on July 25-26 at the Boston Convention Center. Just go over to the Fortinet booth and tell them Corey Quinn sent you and watch for the flinch. My thanks again to my friends at Fortinet.Corey: Let’s face it, on-call firefighting at 2am is stressful! So there’s good news and there’s bad news. The bad news is that you probably can’t prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: Welcome to Screaming in the Cloud, I’m Corey Quinn. My guest today is Sheeri Cabral, who’s a Senior Product Manager of ETL lineage at Collibra. And that is an awful lot of words that I understand approximately none of, except maybe manager. But we’ll get there. The origin story has very little to do with that.I was following Sheeri on Twitter for a long time and really enjoyed the conversations that we had back and forth. And over time, I started to realize that there were a lot of things that didn’t necessarily line up. And one of the more interesting and burning questions I had is, what is it you do, exactly? Because you’re all over the map. First, thank you for taking the time to speak with me today. And what is it you’d say it is you do here? To quote a somewhat bizarre and aged movie now.Sheeri: Well, since your listeners are technical, I do like to match what I say with the audience. First of all, hi. Thanks for having me. I’m Sheeri Cabral. I am a product manager for technical and ETL tools and I can break that down for this technical audience. If it’s not a technical audience, I might say something—like if I’m at a party, and people ask what I do—I’ll say, “I’m a product manager for technical data tool.” And if they ask what a product manager does, I’ll say I helped make sure that, you know, we deliver a product the customer wants. So, you know, ETL tools are tools that transform, extract, and load your data from one place to another.Corey: Like AWS Glue, but for some of them, reportedly, you don’t have to pay AWS by the gigabyte-second.Sheeri: Correct. Correct. We actually have an AWS Glue technical lineage tool in beta right now. So, the technical lineage is how data flows from one place to another. So, when you’re extracting, possibly transforming, and loading your data from one place to another, you’re moving it around; you want to see where it goes. Why do you want to see where it goes? Glad you asked. You didn’t really ask. Do you care? Do you want to know why it’s important?Corey: Oh, I absolutely do. Because it’s—again, people who are, like, “What do you do?” “Oh, it’s boring, and you won’t care.” It’s like when people aren’t even excited themselves about what they work on, it’s always a strange dynamic. There’s a sense that people aren’t really invested in what they do.I’m not saying you have to have this overwhelming passion and do this in your spare time, necessarily, but you should, at least in an ideal world, like what you do enough to light up a bit when you talk about it. You very clearly do. I’m not wanting to stop you. Please continue.Sheeri: I do. I love data and I love helping people. So, technical lineage does a few things. For example, a DBA—which I used to be a DBA—can use technical lineage to predict the impact of a schema update or migration, right? So, if I’m going to change the name of this column, what uses it downstream? What’s going to be affected? What scripts do I need to change? Because if the name changes other thing—you know, then I need to not get errors everywhere.And from a data governance perspective, which Collibra is data governance tool, it helps organizations see if, you know, you have private data in a source, does it remain private throughout its journey, right? So, you can take a column like email address or government ID number and see where it’s used down the line, right? GDPR compliance, CCPA compliance. The CCPA is a little newer; people might not know that acronym. It’s California Consumer Privacy Act.I forget what GDPR is, but it’s another privacy act. It also can help the business see where data comes from so if you have technical lineage all the way down to your reports, then you know whether or not you can trust the data, right? So, you have a report and it shows salary ranges for job titles. So, where did the data come from? Did it come from a survey? Did it come from job sites? Or did it come from a government source like the IRS, right? So, now you know, like, what you get to trust the most.Corey: Wait, you can do that without a blockchain? I kid, I kid, I kid. Please don’t make me talk about blockchains. No, it’s important. The provenance of data, being able to establish a almost a chain-of-custody style approach for a lot of these things is extraordinarily important.Sheeri: Yep.Corey: I was always a little hazy on the whole idea of ETL until I started, you know, working with large-volume AWS bills. And it turns out that, “Well, why do you have to wind up moving and transforming all of these things?” “Oh, because in its raw form, it’s complete nonsense. That’s why. Thank you for asking.” It becomes a problem—Sheeri: [laugh]. Oh, I thought you’re going to say because AWS has 14 different products for things, so you have to move it from one product to the other to use the features.Corey: And two of them are good. It’s a wild experience.Sheeri: [laugh].Corey: But this is also something of a new career for you. You were a DBA for a long time. You’re also incredibly engaging, you have a personality, you’re extraordinarily creative, and that—if I can slander an entire profession for a second—does not feel like it is a common DBA trait. It’s right up there with an overly creative accountant. When your accountant has done a stand-up comedy, you’re watching and you’re laughing and thinking, “I am going to federal prison.” It’s one of those weird things that doesn’t quite gel, if we’re speaking purely in terms of stereotypes. What has your career been like?Sheeri: I was a nerd growing up. So, to kind of say, like, I have a personality, like, my personality is very nerdish. And I get along with other nerdy people and we have a lot of fun, but when I was younger, like, when I was, I don’t know, seven or eight, one of the things I really love to do is I had a penny collection—you know, like you do—and I love to sort it by date. So, in the states anyway, we have these pennies that have the date that they were minted on it. And so, I would organize—and I probably had, like, five bucks worth a pennies.So, you’re talking about 500 pennies and I would sort them and I’d be like, “Oh, this is 1969. This was 1971.” And then when I was done, I wanted to sort things more, so I would start to, like, sort them in order how shiny the pennies were. So, I think that from an early age, it was clear that I wanted to be a DBA from that sorting of my data and ordering it, but I never really had a, like, “Oh, I want to be this when I grew up.” I kind of had a stint when I was in, like, middle school where I was like, maybe I’ll be a creative writer and I wasn’t as creative a writer as I wanted to be, so I was like, “Ah, whatever.”And I ended up actually coming to computer science just completely through random circumstance. I wanted to do neuroscience because I thought it was completely fascinating at how the brain works and how, like, you and I are, like, 99.999—we’re, like, five-nines the same except for, like, a couple of genetic, whatever. But, like, how our brain wiring right how the neuron, how the electricity flows through it—Corey: Yeah, it feels like I want to store a whole bunch of data, that’s okay. I’ll remember it. I’ll keep it in my head. And you’re, like, rolling up the sleeves and grabbing, like, the combination software package off the shelf and a scalpel. Like, “Not yet, but you’re about to.” You’re right, there is an interesting point of commonality on this. It comes down to almost data organization and the—Sheeri: Yeah.Corey: —relationship between data nodes if that’s a fair assessment.Sheeri: Yeah. Well, so what happened was, so I went to university and in order to take introductory neuroscience, I had to take, like, chemistry, organic chemistry, biology, I was basically doing a pre-med track. And so, in the beginning of my junior year, I went to go take introductory neuroscience and I got a D-minus. And a D-minus level doesn’t even count for the major. And I’m like, “Well, I want to graduate in three semesters.”And I had this—I got all my requirements done, except for the pesky little major thing. So, I was already starting to take, like, a computer science, you know, basic courses and so I kind of went whole-hog, all-in did four or five computer science courses a semester and got my degree in computer science. Because it was like math, so it kind of came a little easy to me. So taking, you know, logic courses, and you know, linear algebra courses was like, “Yeah, that’s great.” And then it was the year 2000, when I got my bachelor’s, the turn of the century.And my university offered a fifth-year master’s degree program. And I said, I don’t know who’s going to look at me and say, conscious bias, unconscious bias, “She’s a woman, she can’t do computer science, so, like, let me just get this master’s degree.” I, like, fill out a one page form, I didn’t have to take a GRE. And it was the year 2000. You were around back then.You know what it was like. The jobs were like—they were handing jobs out like candy. I literally had a friend who was like, “My company”—that he founded. He’s like, just come, you know, it’s Monday in May—“Just start, you will just bring your resume the first day and we’ll put it on file.” And I was like, no, no, I have this great opportunity to get a master’s degree in one year at 25% off the cost because I got a tuition reduction or whatever for being in the program. I was like, “What could possibly go wrong in one year?”And what happened was his company didn’t exist the next year, and, like, everyone was in a hiring freeze in 2001. So, it was the best decision I ever made without really knowing because I would have had a job for six months had been laid off with everyone else at the end of 2000 and… and that’s it. So, that’s how I became a DBA is I, you know, got a master’s degree in computer science, really wanted to use databases. There weren’t any database jobs in 2001, but I did get a job as a sysadmin, which we now call SREs.Corey: Well, for some of the younger folks in the audience, I do want to call out the fact that regardless of how they think we all rode dinosaurs to school, databases did absolutely exist back in that era. There’s a reason that Oracle is as large as it is of a company. And it’s not because people just love doing business with them, but technology was head and shoulders above everything else for a long time, to the point where people worked with them in spite of their reputation, not because of it. These days, it seems like in the database universe, you have an explosion of different options and different ways that are great at different things. The best, of course, is Route 53 or other DNS TXT records. Everything else is competing for second place on that. But no matter what it is, you’re after, there are options available. This was not the case back then. It was like, you had a few options, all of them with serious drawbacks, but you had to pick your poison.Sheeri: Yeah. In fact, I learned on Postgres in university because you know, that was freely available. And you know, you’d like, “Well, why not MySQL? Isn’t that kind of easier to learn?” It’s like, yeah, but I went to college from ’96 to 2001. MySQL 1.0 or whatever was released in ’95. By the time I graduated, it was six years old.Corey: And academia is not usually the early adopter of a lot of emerging technologies like that. That’s not a dig on them any because otherwise, you wind up with a major that doesn’t exist by the time that the first crop of students graduates.Sheeri: Right. And they didn’t have, you know, transactions. They didn’t have—they barely had replication, you know? So, it wasn’t a full-fledged database at the time. And then I became a MySQL DBA. But yeah, as a systems administrator, you know, we did websites, right? We did what web—are they called web administrators now? What are they called? Web admins? Webmaster?Corey: Web admins, I think that they became subsumed into sysadmins, by and large and now we call them DevOps, or SRE, which means the exact same thing except you get paid 60% more and your primary job is arguing about which one of those you’re not.Sheeri: Right. Right. Like we were still separated from network operations, but database stuff that stuff and, you know, website stuff, it’s stuff we all did, back when your [laugh] webmail was your Horde based on PHP and you had a database behind it. And yeah, it was fun times.Corey: I worked at a whole bunch of companies in that era. And that’s where basically where I formed my early opinion of a bunch of DBA-leaning sysadmins. Like the DBA in and a lot of these companies, it was, I don’t want to say toxic, but there’s a reason that if I were to say, “I’m writing a memoir about a career track in tech called The Legend of Surly McBastard,” people are going to say, “Oh, is it about the DBA?” There’s a reason behind this. It always felt like there was a sense of elitism and a sense of, “Well, that’s not my job, so you do your job, but if anything goes even slightly wrong, it’s certainly not my fault.” And to be fair, all of these fields have evolved significantly since then, but a lot of those biases that started early in our career are difficult to shake, particularly when they’re unconscious.Sheeri: They are. I’d never ran into that person. Like, I never ran into anyone who—like a developer who treated me poorly because the last DBA was a jerk and whatever, but I heard a lot of stories, especially with things like granting access. In fact, I remember, my first job as an actual DBA and not as a sysadmin that also the DBA stuff was at an online gay dating site, and the CTO rage-quit. Literally yelled, stormed out of the office, slammed the door, and never came back.And a couple of weeks later, you know, we found out that the customer service guys who were in-house—and they were all guys, so I say guys although we also referred to them as ladies because it was an online gay dating site.Corey: Gals works well too, in those scenarios. “Oh, guys is unisex.” “Cool. So’s ‘gals’ by that theory. So gals, how we doing?” And people get very offended by that and suddenly, yeah, maybe ‘folks’ is not a terrible direction to go in. I digress. Please continue.Sheeri: When they hired me, they were like, are you sure you’re okay with this? I’m like, “I get it. There’s, like, half-naked men posters on the wall. That’s fine.” But they would call they’d be, like, “Ladies, let’s go to our meeting.” And I’m like, “Do you want me also?” Because I had to ask because that was when ladies actually might not have included me because they meant, you know.Corey: I did a brief stint myself as the director of TechOps at Grindr. That was a wild experience in a variety of different ways.Sheeri: Yeah.Corey: It’s over a decade ago, but it was still this… it was a very interesting experience in a bunch of ways. And still, to this day, it remains the single biggest source of InfoSec nightmares that kept me awake at night. Just because when I’m working at a bank—which I’ve also done—it’s only money, which sounds ridiculous to say, especially if you’re in a regulated profession, but here in reality where I’m talking about it, it’s I’m dealing instead, with cool, this data leaks, people will die. Most of what I do is not life or death, but that was and that weighed very heavily on me.Sheeri: Yeah, there’s a reason I don’t work for a bank or a hospital. You know, I make mistakes. I’m human, right?Corey: There’s a reason I work on databases for that exact same reason. Please, continue.Sheeri: Yeah. So, the CTO rage-quit. A couple of weeks later, the head of customer service comes to me and be like, “Can we have his spot as an admin for customer service?” And I’m like, “What do you mean?” He’s like, “Well, he told us, we had, like, ten slots of permission and he was one of them so we could have have, like, nine people.”And, like, I went and looked, and they put permission in the htaccess file. So, this former CTO had just wielded his power to be like, “Nope, can’t do that. Sorry, limitations.” When there weren’t any. I’m like, “You could have a hundred. You want every customer service person to be an admin? Whatever. Here you go.” So, I did hear stories about that. And yeah, that’s not the kind of DBA I was.Corey: No, it’s the more senior you get, the less you want to have admin rights on things. But when I leave a job, like, the number one thing I want you to do is revoke my credentials. Not—Sheeri: Please.Corey: Because I’m going to do anything nefarious; because I don’t want to get blamed for it. Because we have a long standing tradition in tech at a lot of places of, “Okay, something just broke. Whose fault is it? Well, who’s the most recent person to leave the company? Let’s blame them because they’re not here to refute the character assassination and they’re not going to be angling for a raise here; the rest of us are so let’s see who we can throw under the bus that can’t defend themselves.” Never a great plan.Sheeri: Yeah. So yeah, I mean, you know, my theory in life is I like helping. So, I liked helping developers as a DBA. I would often run workshops to be like, here’s how to do an explain and find your explain plan and see if you have indexes and why isn’t the database doing what you think it’s supposed to do? And so, I like helping customers as a product manager, right? So…Corey: I am very interested in watching how people start drifting in a variety of different directions. It’s a, you’re doing product management now and it’s an ETL lineage product, it is not something that is directly aligned with your previous positioning in the market. And those career transitions are always very interesting to me because there’s often a mistaken belief by people in their career realizing they’re doing something they don’t want to do. They want to go work in a different field and there’s this pervasive belief that, “Oh, time for me to go back to square one and take an entry level job.” No, you have a career. You have experience. Find the orthogonal move.Often, if that’s challenging because it’s too far apart, you find the half-step job that blends the thing you do now with something a lot closer, and then a year or two later, you complete the transition into that thing. But starting over from scratch, it’s why would you do that? I can’t quite wrap my head around jumping off the corporate ladder to go climb another one. You very clearly have done a lateral move in that direction into a career field that is surprisingly distant, at least in my view. How’d that happen?Sheeri: Yeah, so after being on call for 18 years or so, [laugh] I decided—no, I had a baby, actually. I had a baby. He was great. And then I another one. But after the first baby, I went back to work, and I was on call again. And you know, I had a good maternity leave or whatever, but you know, I had a newborn who was six, eight months old and I was getting paged.And I was like, you know, this is more exhausting than having a newborn. Like, having a baby who sleeps three hours at a time, like, in three hour chunks was less exhausting than being on call. Because when you have a baby, first of all, it’s very rare that they wake up and crying in the midnight it’s an emergency, right? Like they have to go to the hospital, right? Very rare. Thankfully, I never had to do it.But basically, like, as much as I had no brain cells, and sometimes I couldn’t even go through this list, right: they need to be fed; they need to be comforted; they’re tired, and they’re crying because they’re tired, right, you can’t make them go to sleep, but you’re like, just go to sleep—what is it—or their diaper needs changing, right? There’s, like, four things. When you get that beep of that pager in the middle of the night it could be anything. It could be logs filling up disk space, you’re like, “Alright, I’ll rotate the logs and be done with it.” You know? It could be something you need snoozed.Corey: “Issue closed. Status, I no longer give a shit what it is.” At some point, it’s one of those things where—Sheeri: Replication lag.Corey: Right.Sheeri: Not actionable.Corey: Don’t get me started down that particular path. Yeah. This is the area where DBAs and my sysadmin roots started to overlap a bit. Like, as the DBA was great at data analysis, the table structure and the rest, but the backups of the thing, of course that fell to the sysadmin group. And replication lag, it’s, “Okay.”“It’s doing some work in the middle of the night; that’s normal, and the network is fine. And why are you waking me up with things that are not actionable? Stop it.” I’m yelling at the computer at that point, not the person—Sheeri: Right,right.Corey: —to be very clear. But at some point, it’s don’t wake me up with trivial nonsense. If I’m getting woken up in the middle of the night, it better be a disaster. My entire business now is built around a problem that’s business-hours only for that explicit reason. It’s the not wanting to deal with that. And I don’t envy that, but product management. That’s a strange one.Sheeri: Yeah, so what happened was, I was unhappy at my job at the time, and I was like, “I need a new job.” So, I went to, like, the MySQL Slack instance because that was 2018, 2019. Very end of 2018, beginning of 2019. And I said, “I need something new.” Like, maybe a data architect, or maybe, like, a data analyst, or data scientist, which was pretty cool.And I was looking at data scientist jobs, and I was an expert MySQL DBA and it took a long time for me to be able to say, “I’m an expert,” without feeling like oh, you’re just ballooning yourself up. And I was like, “No, I’m literally a world-renowned expert DBA.” Like, I just have to say it and get comfortable with it. And so, you know, I wasn’t making a junior data scientist’s salary. [laugh].I am the sole breadwinner for my household, so at that point, I had one kid and a husband and I was like, how do I support this family on a junior data scientist’s salary when I live in the city of Boston? So, I needed something that could pay a little bit more. And a former I won’t even say coworker, but colleague in the MySQL world—because is was the MySQL Slack after all—said, “I think you should come at MongoDB, be a product manager like me.”Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it’s hard to know where problems originate. Is it your application code, users, or the underlying systems? I’ve got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it’s more than just hipster monitoring. Corey: If I’ve ever said, “Hey, you should come work with me and do anything like me,” people will have the blood drain from their face. And like, “What did you just say to me? That’s terrible.” Yeah, it turns out that I have very hard to explain slash predict, in some ways. It’s always fun. It’s always wild to go down that particular path, but, you know, here we are.Sheeri: Yeah. But I had the same question everybody else does, which was, what’s a product manager? What does the product manager do? And he gave me a list of things a product manager does, which there was some stuff that I had the skills for, like, you have to talk to customers and listen to them.Well, I’ve done consulting. I could get yelled at; that’s fine. You can tell me things are terrible and I have to fix it. I’ve done that. No problem with that. Then there are things like you have to give presentations about how features were okay, I can do that. I’ve done presentations. You know, I started the Boston MySQL Meetup group and ran it for ten years until I had a kid and foisted it off on somebody else.And then the things that I didn’t have the skills in, like, running a beta program were like, “Ooh, that sounds fascinating. Tell me more.” So, I was like, “Yeah, let’s do it.” And I talked to some folks, they were looking for a technical product manager for MongoDB’s sharding product. And they had been looking for someone, like, insanely technical for a while, and they found me; I’m insanely technical.And so, that was great. And so, for a year, I did that at MongoDB. One of the nice things about them is that they invest in people, right? So, my manager left, the team was like, we really can’t support someone who doesn’t have the product management skills that we need yet because you know, I wasn’t a master in a year, believe it or not. And so, they were like, “Why don’t you find another department?” I was like, “Okay.”And I ended up finding a place in engineering communications, doing, like, you know, some keynote demos, doing some other projects and stuff. And then after—that was a kind of a year-long project, and after that ended, I ended up doing product management for developer relations at MongoDB. Also, this was during the pandemic, right, so this is 2019, until ’21; beginning of 2019, to end of 2020, so it was, you know, three full years. You know, I kind of like woke up from the pandemic fog and I was like, “What am I doing? Do I want to really want to be a content product manager?” And I was like, “I want to get back to databases.”One of the interesting things I learned actually in looking for a job because I did it a couple of times at MongoDB because I changed departments and I was also looking externally when I did that. I had the idea when I became a product manager, I was like, “This is great because now I’m product manager for databases and so, I’m kind of leveraging that database skill and then I’ll learn the product manager stuff. And then I can be a product manager for any technical product, right?”Corey: I like the idea. Of some level, it feels like the product managers likeliest to succeed at least have a grounding or baseline in the area that they’re in. This gets into the age-old debate of how important is industry-specific experience? Very often you’ll see a bunch of job ads just put that in as a matter of course. And for some roles, yeah, it’s extremely important.For other roles it’s—for example, I don’t know, hypothetically, you’re looking for someone to fix the AWS bill, it doesn’t necessarily matter whether you’re a services company, a product company, or a VC-backed company whose primary output is losing money, it doesn’t matter because it’s a bounded problem space and that does not transform much from company to company. Same story with sysadmin types to be very direct. But the product stuff does seem to get into that industry specific stuff.Sheeri: Yeah, and especially with tech stuff, you have to understand what your customer is saying when they’re saying, “I have a problem doing X and Y,” right? The interesting part of my folly in that was that part of the time that I was looking was during the pandemic, when you know, everyone was like, “Oh, my God, it’s a seller’s market. If you’re looking for a job, employers are chomping at the bit for you.” And I had trouble finding something because so many people were also looking for jobs, that if I went to look for something, for example, as a storage product manager, right—now, databases and storage solutions have a lot in common; databases are storage solutions, in fact; but file systems and databases have much in common—but all that they needed was one person with file system experience that had more experience than I did in storage solutions, right? And they were going to choose them over me. So, it was an interesting kind of wake-up call for me that, like, yeah, probably data and databases are going to be my niche. And that’s okay because that is literally why they pay me the literal big bucks. If I’m going to go niche that I don’t have 20 years of experience and they shouldn’t pay me as big a bucks right?Corey: Yeah, depending on what you’re doing, sure. I don’t necessarily believe in the idea that well you’re new to this particular type of role so we’re going to basically pay you a lot less. From my perspective it’s always been, like, there’s a value in having a person in a role. The value to the company is X and, “Well, I have an excuse now to pay you less for that,” has never resonated with me. It’s if you’re not, I guess, worth—the value-added not worth being paid what the stated rate for a position is, you are probably not going to find success in that role and the role has to change. That has always been my baseline operating philosophy. Not to yell at people on this, but it’s, uh, I am very tired of watching companies more or less dunk on people from a position of power.Sheeri: Yeah. And I mean, you can even take the power out of that and take, like, location-based. And yes, I understand the cost of living is different in different places, but why do people get paid differently if the value is the same? Like if I want to get a promotion, right, my company is going to be like, “Well, show me how you’ve added value. And we only pay your value. We don’t pay because—you know, you don’t just automatically get promoted after seven years, right? You have to show the value and whatever.” Which is, I believe, correct, right?And yet, there are seniority things, there are this many years experience. And you know, there’s the old caveat of do you have ten years experience or do you have two years of experience five times?Corey: That is the big problem is that there has to be a sense of movement that pushes people forward. You’re not the first person that I’ve had on the show and talked to about a 20 year career. But often, I do wind up talking to folks as I move through the world where they basically have one year of experience repeated 20 times. And as the industry continues to evolve and move on and skill sets don’t keep current, in some cases, it feels like they have lost touch, on some level. And they’re talking about the world that was and still is in some circles, but it’s a market in long-term decline as opposed to keeping abreast of what is functionally a booming industry.Sheeri: Their skills have depreciated because they haven’t learned more skills.Corey: Yeah. Tech across the board is a field where I feel like you have to constantly be learning. And there’s a bit of an evolve-or-die dinosaur approach. And I have some, I do have some fallbacks on this. If I ever decide I am tired of learning and keeping up with AWS, all I have to do is go and work in an environment that uses GovCloud because that’s, like, AWS five years ago.And that buys me the five years to find something else to be doing until a GovCloud catches up with the modern day of when I decided to make that decision. That’s a little insulting and also very accurate for those who have found themselves in that environment. But I digress.Sheeri: No, and I find it to with myself. Like, I got to the point with MySQL where I was like, okay, great. I know MySQL back and forth. Do I want to learn all this other stuff? Literally just today, I was looking at my DMs on Twitter and somebody DMed me in May, saying, “Hi, ma’am. I am a DBA and how can I use below service: Lambda, Step Functions, DynamoDB, AWS Session Manager, and CloudWatch?”And I was like, “You know, I don’t know. I have not ever used any of those technologies. And I haven’t evolved my DBA skills because it’s been, you know, six years since I was a DBA.” No, six years, four or five? I can’t do math.Corey: Yeah. Which you think would be a limiting factor to a DBA but apparently not. One last question that [laugh] I want to ask you, before we wind up calling this a show. You’ve done an awful lot across the board. As you look at all of it, what is it you would say that you’re the most proud of?Sheeri: Oh, great question. What I’m most proud of is my work with WildAid. So, when I was at MongoDB—I referenced a job with engineering communications, and they hired me to be a product manager because they wanted to do a collaboration with a not-for-profit and make a reference application. So, make an application using MongoDB technology and make it something that was going to be used, but people can also see it. So, we made this open-source project called o-fish.And you know, we can give GitHub links: it’s github.com/wildaid, and it has—that’s the organization’s GitHub which we created, so it only has the o-fish projects in it. But it is a mobile and web app where governments who patrol waters, patrol, like, marine protected areas—which are like national parks but in the water, right, so they are these, you know, wildlife preserves in the water—and they make sure that people aren’t doing things they shouldn’t do: they’re not throwing trash in the ocean, they’re not taking turtles out of the Galapagos Island area, you know, things like that. And they need software to track that and do that because at the time, they were literally writing, you know, with pencil on paper, and, you know, had stacks and stacks of this paper to do data entry.And MongoDB had just bought the Realm database and had just integrated it, and so there was, you know, some great features about offline syncing that you didn’t have to do; it did all the foundational plumbing for you. And then the reason though, that I’m proud of that project is not just because it’s pretty freaking cool that, you know, doing something that actually makes a difference in the world and helps fight climate change and all that kind of stuff, the reason I was proud of it is I was the sole product manager. It was the first time that I’d really had sole ownership of a product and so all the mistakes were my own and the credit was my own, too. And so, it was really just a great learning experience and it turned out really well.Corey: There’s a lot to be said for pitching in and helping out with good causes in a way that your skill set winds up benefitting. I found that I was a lot happier with a lot of the volunteer stuff that I did when it was instead of licking envelopes, it started being things that I had a bit of proficiency in. “Hey, can I fix your AWS bill?” It turns out as some value to certain nonprofits. You have to be at a certain scale before it makes sense, otherwise it’s just easier to maybe not do it that way, but there’s a lot of value to doing something that puts good back into the world. I wish more people did that.Sheeri: Yeah. And it’s something to do in your off-time that you know is helping. It might feel like work, it might not feel like work, but it gives you a sense of accomplishment at the end of the day. I remember my first job, one of the interview questions was—no, it wasn’t. [laugh]. It wasn’t an interview question until after I was hired and they asked me the question, and then they made it an interview question.And the question was, what video games do you play? And I said, “I don’t play video games. I spend all day at work staring at a computer screen. Why would I go home and spend another 12 hours till three in the morning, right—five in the morning—playing video games?” And they were like, we clearly need to change our interview questions. This was again, back when the dinosaurs roamed the earth. So, people are are culturally sensitive now.Corey: These days, people ask me, “What’s your favorite video game?” My answer is, “Twitter.”Sheeri: Right. [laugh]. Exactly. It’s like whack-a-mole—Corey: Yeah.Sheeri: —you know? So, for me having a tangible hobby, like, I do a lot of art, I knit, I paint, I carve stamps, I spin wool into yarn. I know that’s not a metaphor for storytelling. That is I literally spin wool into yarn. And having something tangible, you work on something and you’re like, “Look. It was nothing and now it’s this,” is so satisfying.Corey: I really want to thank you for taking the time to speak with me today about where you’ve been, where you are, and where you’re going, and as well as helping me put a little bit more of a human angle on Twitter, which is intensely dehumanizing at times. It turns out that 280 characters is not the best way to express the entirety of what makes someone a person. You need to use a multi-tweet thread for that. If people want to learn more about you, where can they find you?Sheeri: Oh, they can find me on Twitter. I’m @sheeri—S-H-E-E-R-I—on Twitter. And I’ve started to write a little bit more on my blog at sheeri.org. So hopefully, I’ll continue that since I’ve now told people to go there.Corey: I really want to thank you again for being so generous with your time. I appreciate it.Sheeri: Thanks to you, Corey, too. You take the time to interview people, too, so I appreciate it.Corey: I do my best. Sheeri Cabral, Senior Product Manager of ETL lineage at Collibra. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice or smash the like and subscribe buttons on the YouTubes, whereas if you’ve hated it, do exactly the same thing—like and subscribe, hit those buttons, five-star review—but also leave a ridiculous comment where we will then use an ETL pipeline to transform it into something that isn’t complete bullshit.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About ChrisChris is the Co-founder and Chief Product Officer at incident.io, where they're building incident management products that people actually want to use. A software engineer by trade, Chris is no stranger to gnarly incidents, having participated (and caused!) them at everything from early stage startups through to enormous IT organizations.Links Referenced: incident.io: https://incident.io Practical Guide to Incident Management: https://incident.io/guide/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: Let’s face it, on-call firefighting at 2am is stressful! So there’s good news and there’s bad news. The bad news is that you probably can’t prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. Today’s promoted guest is Chris Evans, who’s the CPO and co-founder of incident.io. Chris, first, thank you very much for joining me. And I’m going to start with an easy question—well, easy question, hard answer, I think—what is an incident.io exactly?Chris: Incident.io is a software platform that helps entire organizations to respond to recover from and learn from incidents.Corey: When you say incident, that means an awful lot of things. And depending on where you are in the ecosystem in the world, that means different things to different people. For example, oh, incident. Like, “Are you talking about the noodle incident because we had an agreement that we would never speak about that thing again,” style, versus folks who are steeped in DevOps or SRE culture, which is, of course, a fancy way to say those who are sad all the time, usually about computers. What is an incident in the context of what you folks do?Chris: That, I think, is the killer question. I think if you look at organizations in the past, I think incidents were those things that happened once a quarter, maybe once a year, and they were the thing that brought the entirety of your site down because your big central database that was in a data center sort of disappeared. The way that modern companies run means that the definition has to be very, very different. So, most places now rely on distributed systems and there is no, sort of, binary sense of up or down these days. And essentially, in the general case, like, most companies are continually in a sort of state of things being broken all of the time.And so, for us, when we look at what an incident is, it is essentially anything that takes you away from your planned work with a sense of urgency. And that’s the sort of the pithy definition that we use there. Generally, that can mean anything—it means different things to different folks, and, like, when we talk to folks, we encourage them to think carefully about what that threshold is, but generally, for us at incident.io, that means basically a single error that is worthwhile investigating that you would stop doing your backlog work for is an incident. And also an entire app being down, that is an incident.So, there’s quite a wide range there. But essentially, by sort of having more incidents and lowering that threshold, you suddenly have a heap of benefits, which I can go very deep into and talk for hours about.Corey: It’s a deceptively complex question. When I talk to folks about backups, one of the biggest problems in the world of backup and building a DR plan, it’s not building the DR plan—though that’s no picnic either—it’s okay. In the time of cloud, all your planning figures out, okay. Suddenly the site is down, how do we fix it? There are different levels of down and that means different things to different people where, especially the way we build apps today, it’s not is the service or site up or down, but with distributed systems, it’s how down is it?And oh, we’re seeing elevated error rates in us-tire-fire-1 region of AWS. At what point do we begin executing on our disaster plan? Because the worst answer, in some respects is, every time you think you see a problem, you start failing over to other regions and other providers and the rest, and three minutes in, you’ve irrevocably made the cutover and it’s going to take 15 minutes to come back up. And oh, yeah, then your primary site comes back up because whoever unplugged something, plugged it back in and now you’ve made the wrong choice. Figuring out all the things around the incident, it’s not what it once was.When you were running your own blog on a single web server and it’s broken, it’s pretty easy to say, “Is it up or is it down?” As you scale out, it seems like that gets more and more diffuse. But it feels to me that it’s also less of a question of how the technology has scaled, but also how the culture and the people have scaled. When you’re the only engineer somewhere, you pretty much have no choice but to have the entire state of your stack shoved into your head. When that becomes 15 or 20 different teams of people, in some cases, it feels like it’s almost less than a technology problem than it is a problem of how you communicate and how you get people involved. And the issues in front of the people who are empowered and insightful in a certain area that needs fixing.Chris: A hundred percent. This is, like, a really, really key point, which is that organizations themselves are very complex. And so, you’ve got this combination of systems getting more and more complicated, more and more sort of things going wrong and perpetually breaking but you’ve got very, very complicated information structures and communication throughout the whole organization to keep things up and running. The very best orgs are the ones where they can engage the entire, sort of, every corner of the organization when things do go wrong. And lived and breathed this firsthand when various different previous companies, but most recently at Monzo—which is a bank here in the UK—when an incident happened there, like, one of our two physical data center locations went down, the bank wasn’t offline. Everything was resilient to that, but that required an immediate response.And that meant that engineers were deployed to go and fix things. But it also meant the customer support folks might be required to get involved because we might be slightly slower processing payments. And it means that risk and compliance folks might need to get involved because they need to be reporting things to regulators. And the list goes on. There’s, like, this need for a bunch of different people who almost certainly have never worked together or rarely worked together to come together, land in this sort of like empty space of this incident room or virtual incident room, and figure out how they’re going to coordinate their response and get things back on track in the sort of most streamlined way and as quick as possible.Corey: Yeah, when your bank is suddenly offline, that seems like a really inopportune time to be introduced to the database team. It’s, “Oh, we have one of those. Wonderful. I feel like you folks are going to come in handy later today.” You want to have those pathways of communication open well in advance of these issues.Chris: A hundred percent. And I think the thing that makes incidents unique is that fact. And I think the solution to that is this sort of consistent, level playing field that you can put everybody on. So, if everybody understands that the way that incidents are dealt with is consistent, we declare it like this, and under these conditions, these things happen. And, you know, if I flag this kind of level of impact, we have to pull in someone else to come and help make a decision.At the core of it, there’s this weird kind of duality to incidents where they are both kind of semi-formulaic and that you can basically encode a lot of the processes that happen, but equally, they are incredibly chaotic and require a lot of human impact to be resilient and figure these things out because stuff that you have never seen happen before is happening and failing in ways that you never predicted. And so, this is where incident.io plays into this is that we try to take the first half of that off of your hands, which is, we will help you run your process so that all of the brain capacity you have, it goes on to the bit that humans are uniquely placed to be able to do, which is responding to these very, very chaotic, sort of, surprise events that have happened.Corey: I feel as well—because I played around in this space a bit before I used to run ops teams—and, more or less I really should have had a t-shirt then that said, “I am the root cause,” because yeah, I basically did a lot of self-inflicted outages in various environments because it turns out, I’m not always the best with computers. Imagine that. There are a number of different companies that play in the space that look at some part of the incident lifecycle. And from the outside, first, they all look alike because it’s, “Oh, so you’re incident.io. I assume you’re PagerDuty. You’re the thing that calls me at two in the morning to make sure I wake up.”Conversely, for folks who haven’t worked deeply in that space, as well, of setting things on fire, what you do sounds like it’s highly susceptible to the Hacker News problem. Where, “Wait, so what you do is effectively just getting people to coordinate and talk during an incident? Well, that doesn’t sound hard. I could do that in a weekend.” And no, no, you can’t.If this were easy, you would not have been in business as long as you have, have the team the size that you do, the customers that you do. But it’s one of those things that until you’ve been in a very specific set of a problem, it doesn’t sound like it’s a real problem that needs solving.Chris: Yeah, I think that’s true. And I think that the Hacker News point is a particularly pertinent one and that someone else, sort of, in an adjacent area launched on Hacker News recently, and the amount of feedback they got around, you know, “You’re a Slack bot. How is this a company?” Was kind of staggering. And I think generally where that comes from is—well, first of all that bias that engineers have, which is just everything you look at as an engineer is like, “Yeah, I can build that in a weekend.” I think there’s often infinite complexity under the hood that just gets kind of brushed over. But yeah, I think at the core of it, you probably could build a Slack bot in a weekend that creates a channel for you in Slack and allows you to post somewhere that some—Corey: Oh, good. More channels in Slack. Just when everyone wants.Chris: Well, there you go. I mean, that’s a particular pertinent one because, like, our tool does do that. And one of the things—so I built at Monzo, a version of incident.io that we used at the company there, and that was something that I built evenings and weekends. And among the many, many things I never got around to building, archiving and cleaning up channels was one of the ones that was always on that list.And so, Monzo did have this problem of littered channels everywhere, I think that sort of like, part of the problem here is, like, it is easy to look at a product like ours and sort of assume it is this sort of friendly Slack bot that helps you orchestrate some very basic commands. And I think when you actually dig into the problems that organizations above a certain size have, they’re not solved by Slack bots. They’re solved by platforms that help you to encode your processes that otherwise have to live on a Google Doc somewhere which is five pages long and when it’s 2 a.m. and everything’s on fire, I guarantee you not a single person reads that Google Doc, so your process is as good as not in place at all. That’s the beauty of a tool like ours. We have a powerful engine that helps you basically to encode that and take some load off of you.Corey: To be clear, I’m also not coming at this from a position of judging other people. I just look right now at the Slack workspace that we have The Duckbill Group, and we have something like a ten-to-one channel-to-human ratio. And the proliferation of channels is a very real thing. And the problem that I’ve seen across the board with other things that try to address incident management has always been fanciful at best about what really happens when something breaks. Like, you talk about, oh, here’s what happens. Step one: you will pull up the Google Doc, or you will pull up the wiki or the rest, or in some aspirational places, ah, something seems weird, I will go open a ticket in Jira.Meanwhile, here in reality, anyone who’s ever worked in these environments knows that step one, “Oh shit, oh shit, oh shit, oh shit, oh shit. What are we going to do?” And all the practices and procedures that often exist, especially in orgs that aren’t very practiced at these sorts of things, tend to fly out the window and people are going to do what they’re going to do. So, any tool or any platform that winds up addressing that has to accept the reality of meeting people where they are not trying to educate people into different patterns of behavior as such. One of the things I like about your approach is, yeah, it’s going to be a lot of conversation in Slack that is a given we can pretend otherwise, but here in reality, that is how work gets communicated, particularly in extremis. And I really appreciate the fact that you are not trying to, like, fight what feels almost like a law of nature at this point.Chris: Yeah, I think there’s a few things in that. The first point around the document approach or the clearly defined steps of how an incident works. In my experience, those things have always gone wrong because—Corey: The data center is down, so we’re going to the wiki to follow our incident management procedure, which is in the data center just lost power.Chris: Yeah.Corey: There’s a dependency problem there, too. [laugh].Chris: Yeah, a hundred percent. [laugh]. A hundred percent. And I think part of the problem that I see there is that very, very often, you’ve got this situation where the people designing the process are not the people following the process. And so, there’s this classic, I’ve heard it through John Allspaw, but it’s a bunch of other folks who talk about the difference between people, you know, at the sharp end or the blunt end of the work.And I think the problem that people are facing the past is you have these people who sit in the, sort of, metaphorical upstairs of the office and think that they make a company safe by defining a process on paper. And they ship the piece of paper and go, “That is a good job for me done. I’m going to leave and know that I’ve made the bank—the other whatever your organization does—much, much safer.” And I think this is where things fall down because—Corey: I want to ambush some of those people in their performance reviews with, “Cool. Just for fun, all the documentation here, we’re going to pull up the analytics to see how often that stuff gets viewed. Oh, nobody ever sees it. Hmm.”Chris: It’s frustrating. It’s frustrating because that never ever happens, clearly. But the point you made around, like, meeting people where you are, I think that is a huge one, which is incidents are founded on great communication. Like, as I said earlier, this is, like, a form of team with someone you’ve never ever worked with before and the last thing you want to do is be, like, “Hey, Corey, I’ve never met you before, but let’s jump out onto this other platform somewhere that I’ve never been or haven’t been for weeks and we’ll try and figure stuff out over there.” It’s like, no, you’re going to be communicating—Corey: We use Slack internally, but we have a WhatsApp chat that we wind up using for incident stuff, so go ahead and log into WhatsApp, which you haven’t done in 18 months, and join the chat. Yeah, in the dawn of time, in the mists of antiquity, you vaguely remember hearing something about that your first week and then never again. This stuff has to be practiced and it’s important to get it right. How do you approach the inherent and often unfortunate reality that incident response and management inherently becomes very different depending upon the specifics of your company or your culture or something like that? In other words, how cookie-cutter is what you have built versus adaptable to different environments it finds itself operating in?Chris: Man, the amount of time we spent as a founding team in the early days deliberating over how opinionated we should be versus how flexible we should be was staggering. The way we like to describe it as we are quite opinionated about how we think incidents should be run, however we let you imprint your own process into that, so putting some color onto that. We expect incidents to have a lead. That is something you cannot get away from. However, you can call the lead whatever makes sense for you at your organization. So, some folks call them an incident commander or a manager or whatever else.Corey: There’s overwhelming militarization of these things. Like, oh, yes, we’re going to wind up taking a bunch of terms from the military here. It’s like, you realize that your entire giant screaming fire is that the lights on the screen are in the wrong pattern. You’re trying to make them in the right pattern. No one dies here in most cases, so it feels a little grandiose for some of those terms being tossed around in some cases, but I get it. You’ve got to make something that is unpleasant and tedious in many respects, a little bit more gripping. I don’t envy people. Messaging is hard.Chris: Yeah, it is. And I think if you’re overly virtuoustic and inflexible, you’re sort of fighting an uphill battle here, right? So, folks are going to want to call things what they want to call things. And you’ve got people who want to import [ITIL 00:15:04] definitions for severity ease into the platform because that’s what they’re familiar with. That’s fine.What we are opinionated about is that you have some severity levels because absent academic criticism of severity levels, they are a useful mechanism to very coarsely and very quickly assess how bad something is and to take some actions off of it. So yeah, we basically have various points in the product where you can customize and put your own sort of flavor on it, but generally, we have a relatively opinionated end-to-end expectation of how you will run that process.Corey: The thing that I find that annoys me—in some cases—the most is how heavyweight the process is, and it’s clearly built by people in an ivory tower somewhere where there’s effectively a two-day long postmortem analysis of the incident, and so on and so forth. And okay, great. Your entire site has been blown off the internet, yeah, that probably makes sense. But as soon as you start broadening that to things like okay, an increase in 500 errors on this service for 30 minutes, “Great. Well, we’re going to have a two-day postmortem on that.” It’s, “Yeah, sure would be nice if we could go two full days without having another incident of that caliber.” So, in other words, whose foot—are we going to hire a new team whose full-time job it is, is to just go ahead and triage and learn from all these incidents? Seems to me like that’s sort of throwing wood behind the wrong arrows.Chris: Yeah, I think it’s very reductive to suggest that learning only happens in a postmortem process. So, I wrote a blog, actually, not so long ago that is about running postmortems and when it makes sense to do it. And as part of that, I had a sort of a statement that was [laugh] that we haven’t run a single postmortem when I wrote this blog at incident.io. Which is probably shocking to many people because we’re an incident company, and we talk about this stuff, but we were also a company of five people and when something went wrong, the learning was happening and these things were sort of—we were carving out the time, whether it was called a postmortem, or not to learn and figure out these things. Extrapolating that to bigger companies, there is little value in following processes for the sake of following processes. And so, you could have—Corey: Someone in compliance just wound up spitting their coffee over their desktop as soon as you said that. But I hear you.Chris: Yeah. And it's those same folks who are the ones who care about the document being written, not the process and the learning happening. And I think that’s deeply frustrating to me as—Corey: All the plans, of course, assume that people will prioritize the company over their own family for certain kinds of disasters. I love that, too. It’s divorced from reality; that’s ridiculous, on some level. Speaking of ridiculous things, as you continue to grow and scale, I imagine you integrate with things beyond just Slack. You grab other data sources and over in the fullness of time.For example, I imagine one of your most popular requests from some of your larger customers is to integrate with their HR system in order to figure out who’s the last engineer who left, therefore everything immediately their fault because lord knows the best practice is to pillory whoever was the last left because then they’re not there to defend themselves anymore and no one’s going to get dinged for that irresponsible jackass’s decisions, even if they never touched the system at all. I’m being slightly hyperbolic, but only slightly.Chris: Yeah. I think [laugh] that's an interesting point. I am definitely going to raise that feature request for a prefilled root cause category, which is, you know, the value is just that last person who left the organization. That it’s a wonderful scapegoat situation there. I like it.To the point around what we do integrate with, I think the thing is actually with incidents that’s quite interesting is there is a lot of tooling that exists in this space that does little pockets of useful, valuable things in the shape of incidents. So, you have PagerDuty is this system that does a great job of making people’s phone making noise, but that happens, and then you’re dropped into this sort of empty void of nothingness and you’ve got to go and figure out what to do. And then you’ve got things like Jira where clearly you want to be able to track actions that are coming out of things going wrong in some cases, and that’s a great tool for that. And various other things in the middle there. And yeah, our value proposition, if you want to call it that, is to bring those things together in a way that is massively ergonomic during an incident.So, when you’re in the middle of an incident, it is really handy to be able to go, “Oh, I have shipped this horrible fix to this thing. It works, but I must remember to undo that.” And we put that at your fingertips in an incident channel from Slack, that you can just log that action, lose that cognitive load that would otherwise be there, move on with fixing the thing. And you have this sort of—I think it’s, like, that multiplied by 1000 in incidents that is just what makes it feel delightful. And I cringe a little bit saying that because it’s an incident at the end of the day, but genuinely, it feels magical when some things happen that are just like, “Oh, my gosh, you’ve automatically hooked into my GitHub thing and someone else merged that PR and you’ve posted that back into the channel for me so I know that that happens. That would otherwise have been a thing where I jump out of the incident to go and figure out what was happening.”Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don’t leave managing your database to your cloud vendor because they’re too busy launching another half-dozen managed databases to focus on any one of them that they didn’t build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: The problem with the cloud, too, is the first thing that, when there starts to be an incident happening is the number one decision—almost the number one decision point is this my shitty code, something we have just pushed in our stuff, or is it the underlying provider itself? Which is why the AWS status page being slow to update is so maddening. Because those are two completely different paths to go down and you are having to pursue both of them equally at the same time until one can be ruled out. And that is why time to identify at least what side of the universe it’s on is so important. That has always been a bit of a tricky challenge.I want to talk a bit about circular dependencies. You target a certain persona of customer, but I’m going to go out on a limb and assume that one explicit company that you are not going to want to do business with in your current iteration is Slack itself because a tool to manage—okay, so our service is down, so we’re going to go to Slack to fix it doesn’t work when the service is Slack itself. So, that becomes a significant challenge. As you look at this across the board, are you seeing customers having problems where you have circular dependency issues with this? Easy example: Slack is built on top of AWS.When there’s an underlying degradation of, huh, suddenly us-east-1 is not doing what it’s supposed to be doing, now, Slack is degraded as well, as well as the customer site, it seems like at that point, you’re sort of in a bit of tricky positioning as a customer. Counterpoint, when neither Slack nor your site are working, figuring out what caused that issue doesn’t seem like it’s the biggest stretch of the imagination at that point.Chris: I’ve spent a lot of my career working in infrastructure, platform-type teams, and I think you can end up tying yourself in knots if you try and over-optimize for, like, avoiding these dependencies. I think it’s one of those, sort of, turtles all the way down situations. So yes, Slack are unlikely to become a customer because they are clearly going to want to use our product when they are down.Corey: They reach out, “We’d like to be your customer.” Your response is, “Please don’t be.” None of us are going to be happy with this outcome.Chris: Yeah, I mean, the interesting thing that is that we’re friends with some folks at Slack, and they believe it or not, they do use Slack to navigate their incidents. They have an internal tool that they have written. And I think this sort of speaks to the point we made earlier, which is that incidents and things failing or not these sort of big binary events. And so—Corey: All of Slack is down is not the only kind of incident that a company like Slack can experience.Chris: I’d go as far as that it’s most commonly not that. It’s most commonly that you’re navigating incidents where it is a degradation, or some edge case, or something else that’s happened. And so, like, the pragmatic solution here is not to avoid the circular dependencies, in my view; it’s to accept that they exist and make sure you have sensible escape hatches so that when something does go wrong—so a good example, we use incident.io at incident.io to manage incidents that we’re having with incident.io. And 99% of the time, that is absolutely fine because we are having some error in some corner of the product or a particular customer is doing something that is a bit curious.And I could count literally on one hand the number of times that we have not been able to use our products to fix our product. And in those cases, we have a fallback which is jump into—Corey: I assume you put a little thought into what happened. “Well, what if our product is down?” “Oh well, I guess we’ll never be able to fix it or communicate about it.” It seems like that’s the sort of thing that, given what you do, you might have put more than ten seconds of thought into.Chris: We’ve put a fair amount of thought into it. But at the end of the day, [laugh] it’s like if stuff is down, like, what do you need to do? You need to communicate with people. So, jump on a Google Chat, jump on a Slack huddle, whatever else it is we have various different, like, fallbacks in different order. And at the core of it, I think this is the thing is, like, you cannot be prepared for every single thing going wrong, and so what you can be prepared for is to be unprepared and just accept that humans are incredibly good at being resilient, and therefore, all manner of things are going to happen that you’ve never seen before and I guarantee you will figure them out and fix them, basically.But yeah, I say this; if my SOC 2 auditor is listening, we also do have a very well-defined, like, backup plan in our SOC 2 [laugh] in our policies and processes that is the thing that we will follow that. But yeah.Corey: The fact that you’re saying the magic words of SOC 2, yes, exactly. Being in a responsible adult and living up to some baseline compliance obligations is really the sign of a company that’s put a little thought into these things. So, as I pull up incident.io—the website, not the company to be clear—and look through what you’ve written and how you talk about what you’re doing, you’ve avoided what I would almost certainly have not because your tagline front and center on your landing page is, “Manage incidents at scale without leaving Slack.” If someone were to reach out and say, well, we’re down all the time, but we’re using Microsoft Teams, so I don’t know that we can use you, like, the immediate instinctive response that I would have for that to the point where I would put it in the copy is, “Okay, this piece of advice is free. I would posit that you’re down all the time because you’re the kind of company to use Microsoft Teams.” But that doesn’t tend to win a whole lot of friends in various places. In a slightly less sarcastic bent, do you see people reaching out with, “Well, we want to use you because we love what you’re doing, but we don’t use Slack.”Chris: Yeah. We do. A lot of folks actually. And we will support Teams one day, I think. There is nothing especially unique about the product that means that we are tied to Slack.It is a great way to distribute our product and it sort of aligns with the companies that think in the way that we do in the general case but, like, at the core of what we’re building, it’s a platform that augments a communication platform to make it much easier to deal with a high-stress, high-pressure situation. And so, in the future, we will support ways for you to connect Microsoft Teams or if Zoom sought out getting rich app experiences, talk on a Zoom and be able to do various things like logging actions and communicating with other systems and things like that. But yeah, for the time being very, very deliberate focus mechanism for us. We’re a small company with, like, 30 people now, and so yeah, focusing on that sort of very slim vertical is working well for us.Corey: And it certainly seems to be working to your benefit. Every person I’ve talked to who is encountered you folks has nothing but good things to say. We have a bunch of folks in common listed on the wall of logos, the social proof eye chart thing of here’s people who are using us. And these are serious companies. I mean, your last job before starting incident.io was at Monzo, as you mentioned.You know what you’re doing in a regulated, serious sense. I would be, quite honestly, extraordinarily skeptical if your background were significantly different from this because, “Well, yeah, we worked at Twitter for Pets in our three-person SRE team, we can tell you exactly how to go ahead and handle your incidents.” Yeah, there’s a certain level of operational maturity that I kind of just based upon the name of the company there; don’t think that Twitter for Pets is going to nail. Monzo is a bank. Guess you know what you’re talking about, given that you have not, basically, been shut down by an army of regulators. It really does breed an awful lot of confidence.But what’s interesting to me is the number of people that we talk to in common are not themselves banks. Some are and they do very serious things, but others are not these highly regulated, command-and-control, top-down companies. You are nimble enough that you can get embedded at those startup-y of startup companies once they hit a certain point of scale and wind up helping them arrive at a better outcome. It’s interesting in that you don’t normally see a whole lot of tools that wind up being able to speak to both sides of that very broad spectrum—and most things in between—very effectively. But you’ve somehow managed to thread that needle. Good work.Chris: Thank you. Yeah. What else can I say other than thank you? I think, like, it’s a deliberate product positioning that we’ve gone down to try and be able to support those different use cases. So, I think, at the core of it, we have always tried to maintain the incident.io should be installable and usable in your very first incident without you having to have a very steep learning curve, but there is depth behind it that allows you to support a much more sophisticated incident setup.So, like, I mean, you mentioned Monzo. Like, I just feel incredibly fortunate to have worked at that company. I joined back in 2017 when they were, I don’t know, like, 150,000 customers and it was just getting its banking license. And I was there for four years and was able to then see it scale up to 6 million customers and all of the challenges and pain that goes along with that both from building infrastructure on the technical side of things, but from an organizational side of things. And was, like, front-row seat to being able to work with some incredibly smart people and sort of see all these various different pain points.And honestly, it feels a little bit like being in sort of a cheat mode where we get to this import a lot of that knowledge and pain that we felt at Monzo into the product. And that happens to resonate with a bunch of folks. So yeah, I feel like things are sort of coming out quite well at the moment for folks.Corey: The one thing I will say before we wind up calling this an episode is just how grateful I am that I don’t have to think about things like this anymore. There’s a reason that the problem that I chose to work on of expensive AWS bills being very much a business-hours only style of problem. We’re a services company. We don’t have production infrastructure that is externally facing. “Oh, no, one of our data analysis tools isn’t working internally.”That’s an interesting curiosity, but it’s not an emergency in the same way that, “Oh, we’re an ad network and people are looking at ads right now because we’re broken,” is. So, I am grateful that I don’t have to think about these things anymore. And also a little wistful because there’s so much that you do it would have made dealing with expensive and dangerous outages back in my production years a lot nicer.Chris: Yep. I think that’s what a lot of folks are telling us essentially. There’s this curious thing with, like, this product didn’t exist however many years ago and I think it’s sort of been quite emergent in a lot of companies that, you know, as sort of things have moved on, that something needs to exist in this little pocket of space, dealing with incidents in modern companies. So, I’m very pleased that what we’re able to build here is sort of working and filling that for folks.Corey: Yeah. I really want to thank you for taking so much time to go through the ethos of what you do, why you do it, and how you do it. If people want to learn more, where’s the best place for them to go? Ideally, not during an incident.Chris: Not during an incident, obviously. Handily, the website is the company name. So, incident.io is a great place to go and find out more. We’ve literally—literally just today, actually—launched our Practical Guide to Incident Management, which is, like, a really full piece of content which, hopefully, will be useful to a bunch of different folks.Corey: Excellent. We will, of course, put a link to that in the [show notes 00:29:52]. I really want to thank you for being so generous with your time. Really appreciate it.Chris: Thanks so much. It’s been an absolute pleasure.Corey: Chris Evans, Chief Product Officer and co-founder of incident.io. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this episode, please leave a five-star review on your podcast platform of choice along with an angry comment telling me why your latest incident is all the intern’s fault.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About MaishMaish Saidel-Keesing is a Senior Enterprise Developer Advocate @AWS working on containers and has been working in IT for the past 20 years and with a stronger focus on cloud and automation for the past 7.He has extensive experience with AWS Cloud technologies, DevOps and Agile practices and implementations, containers, Kubernetes, virtualization, and a number of fun things he has done along the wayHe is constantly trying to bridge the gap between Developers and Operators to allow all of us provide a better service for our customers (and not wake up from pages in the middle of the night). He is an avid practitioner of dissolving silos - educating Ops how to code and explaining to Devs what the hell is OperationsLinks Referenced: @maishsk: https://twitter.com/maishsk duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored by our friends at Fortinet. Fortinet’s partnership with AWS is a better-together combination that ensures your workloads on AWS are protected by best-in-class security solutions powered by comprehensive threat intelligence and more than 20 years of cybersecurity experience. Integrations with key AWS services simplify security management, ensure full visibility across environments, and provide broad protection across your workloads and applications. Visit them at AWS re:Inforce to see the latest trends in cybersecurity on July 25-26 at the Boston Convention Center. Just go over to the Fortinet booth and tell them Corey Quinn sent you and watch for the flinch. My thanks again to my friends at Fortinet.Corey: Let’s face it, on-call firefighting at 2am is stressful! So there’s good news and there’s bad news. The bad news is that you probably can’t prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. I’m a cloud economist at The Duckbill Group, and that was a fun thing for me to become because when you’re starting to set out to solve a problem, well, what do you call yourself? I find that if you create a job title for yourself, well, no one knows quite how to categorize you and it leads to really interesting outcomes as a result. My guest today did something very similar. Maish Saidel-Keesing is an EntReloper, or Enterprise Developer Advocate, specifically for container services at AWS. Maish, thank you for joining me.Maish: Thank you for having me on the show, Corey. It’s great to be here.Corey: So, how did you wind up taking a whole bunch of words such as enterprise, developer, advocate because I feel like the way you really express seniority at big companies, almost as a display of dominance, is to have additional words in your job title, which all those words are very enterprise-y, very business-y, and very serious. And in container services to boot, which is a somewhat interesting culture, just looking at the enterprise adoption of the pattern. And then at AWS, whose entire sense of humor can be distilled down into, “That’s not funny.” You have the flexibility to refer to yourself as an EntReloper in public. I love it. Is it just something you started doing? Was there, like, 18 forms of approval you had to go through to do it? How did this happen? I love it.Maish: So, no. I didn’t have to go through approval, of course. Same way, you didn’t call yourself a cloud economist with anybody else’s approval. But I got the idea mostly from you because I love your term of coining everybody who’s in developer advocacy or developer relations as a DevReloper. And specifically, the reason that I coined the term of an EntReloper—and actually looked it up on Google to see if anybody had actually used that term before, and no they haven’t—it’s the fact of I came into the position on the premise of trying to bring the enterprise voice of the customer into developer advocacy.When we speak about developer advocates today, most of them are the people who are the small startups, developers who write the code, and we kind of forget that there is a whole big world outside of, besides small startups, which are these big, massive, behemoth sort of enterprise companies who kind of do things differently because they’ve been around for many, many years; they have many, many silos inside their organizations. And it’s not the most simple thing to open up your laptop, and install whatever software you want on, because some of these people don’t even give you admin rights on your laptop, or you’re allowed to ssh out to a computer in the cloud because also the same thing: everything is blocked by corporate firewalls where you have to put in a ticket in order to get access to the outside world. I worked in companies like that when I was—before I moved to Amazon. So, I want to bring that perspective to the table on behalf of our customers.Corey: Bias is a very funny thing. I spent the overwhelming majority of my career in small environments like you describe. To me a big company is one that has 200 people there, and it turns out that there’s a whole ‘nother sense of scale that goes beyond that. And there’s, like, 18 different tiers beyond. But I still bias based upon my own experiences when I talk about how I do things and how I think about things to a certain persona that closely resembles my own experiences where, “Just install this thing as a tool and it’ll be great,” ignoring entirely, the very realistic fact that you’ve got an entire universe out there of people who are not empowered to install things on their own laptop, for example.How is developer advocacy different within enterprises than in the common case of, “We’re a startup. We’re going to change the world with our amazing SaaS.” Great, maybe you will. Statistically, you won’t. But enterprises have different concerns, different challenges, and absolutely a different sense of scale. How is the practice of advocacy different in those environments?Maish: So, I think the fact is, mostly working on standardization from the get-go that these big enterprises want things to work in a standard way where they can control it, they can monitor it, they can log everything, they can secure it mostly, of course, the most important thing. But it’s also the fact that as a developer advocate, you don’t always talk to developers within the enterprise. You also have to talk to the security team and to the network team and to the business itself or the C-level to understand. And as you also probably have found out as well in your job, you connect the people with inside the business one to another, these different groups, and get them talking to each other to make these decisions together. So, we act as kind of a bridge in between the people with inside their own company where they don’t really talk to each other, or don’t have the right connections, or the right conduit in order for them to start that conversation and make things better for themselves.Corey: On some level, my line about developer relations, developer advocacy, has generally taken the track of, “What does that mean? Well, it means you work in marketing, but they’re scared to tell you that.” Do you view what you do as being within marketing, aligned with marketing, subtly different and I’m completely wrong, et cetera, et cetera? All positions are legitimate, by the way.Maish: So, I think, at the position that I’m currently in, which is a developer advocate but for the service team, is slightly different than a marketing developer advocate. The marketing developer advocate—and we have many of them which are amazing people and doing amazing work within AWS—their job is to teach everybody about the services and the capabilities available within AWS. That is also part of my job, but I would think that is the 40% of my job. I also go on stage, I go on podcasts like this, I present at conferences, I write blog posts. I also do the kind of marketing work as well.But the other 60% of my job as a service developer advocate is to seek out the feedback, or the signals, or the sentiment from our customers, and bring that back into the service teams, into the product management, into the engineering teams. And, as I said, sit as the enterprise customer in the chair in those meetings, to voice their concerns… their opinions, how they would like the products to go, how we can make the products better. So, the 60% is mostly what we call inbound, which is taking feedback from our customers back into the service teams directly in order to have some influence on the roadmap. And 40% is the outbound work, which we do, as I said, conferences, blog posts, and things like this.Corey: I have a perception. And I am thrilled to be corrected on this because it’s not backed by data; it’s backed by my own biases—and some people tend to conflate the two; I strive not to—that there’s a—I think the term that I heard bandied around at one point was ‘the dark matter developers.’ These are folks that primarily work in .NET or Java. They work for companies that are not themselves tech companies, but rather tech is a supporting function, usually in a central IT-style organization, that supports what the business actually does, and they generally are not visible to a lot of traditional developer advocacy approaches.They, by and large, don’t go to conferences, they don’t go on Twitter to yell at people about things, they commit the terrible sin—according to many startup folks—of daring to view the craft of writing software as this artistic thing, and they just view it as a job and a thing to make money for—filthy casuals—as opposed to this higher calling that’s changing the world. Which I think is wild take. But there are a tremendous number of people out there who do fit the profile of they show up, they do their jobs working on this stuff, they don’t go to conferences, they don’t go out into the community, and they just do their job and go home. The end. Is that an accurate perception? Are there large swaths of folks like that in the industry, and if so, do they centralize or congregate more around enterprises than they do around smaller companies?Maish: I think that your perception is correct. Specifically, for my experience, when I worked, for example, my first two years before I was a developer advocate, I was an enterprise solutions architect which I worked with financial institutions, which are banks, which usually have software which are older than me, which are written in languages, which are older than I am. So, there are people which, as they say, they come there to—they do their job. They’re not interested in looking at Twitter, or writing blog posts, or participating in any kind of thing which is outgoing. And they just, they’re there to write the code. They go home at the end of the day.They also usually don’t have pagers that page them in middle of the night because that’s what you have operations teams for, not developers because they’re completely different entities. So, I do think your perception might be correct, yes. There are people like that when you say, these dark matter people, dark matter developers.Corey: And I don’t have any particular problem. I’m not here to cast shade on anything that they’re doing, to be very clear.Maish: Not at all.Corey: Everyone makes different choices and that’s great. I don’t think necessarily everyone should have a job that is all-consuming, that eats them alive. I wish I didn’t, some days. [laugh]. The challenge I have for you then is, as an EntReloper, how do you reach folks in positions like that? Or don’t you?Maish: I think the way to reach those people is to firstly, expose them to technology, expose them to the capabilities that they can use in AWS in the cloud, specifically with my position in container services, and gain their trust because that’s one of the LPs in Amazon itself: customer obsession. And we work consistently in order to—with our customers to gain their trust and help them along their journey, whatever it may be. If it might be the fact, okay, I only want to write software for nine to five and go home and do everything afterwards, which most normal people do without having to worry about work, or they still want to continue working and adopt the full model of you build it, you own it; manage everything in production on their own and go into the new world of modern software, which many enterprises, unfortunately, are not all the way there yet, but hopefully, they will get there sooner than later.Corey: There’s a misguided perception in many corners that you have to be able to reach everyone at all times; wherever they are, you have to be able to go there. I don’t think that’s true. I think that showing up and badgering people who are just trying to get a job done into, “Hey, have you heard the good word of cloud?” It’s like, evangelists knocking on your door at seven o’clock in the morning on a weekend and you’re trying to sleep in because the kids are somewhere else for the week. Yeah, I might be projecting a little bit on that.I think that is the wrong direction to go. And I find that being able and willing to meet people where they are is key to success on this. I’m also a big believer in the idea that in any kind of developer advocacy role, regardless whether their targets are large, small, or in my case, patently ridiculous because my company is in fact ridiculous in some ways, you have to meet them where they are. There’s no choice around that. Do you find that there are very different concerns that you have to wind up addressing with your audience versus a more, “Mainstream,” quote-unquote, developer advocacy role?Maish: For the enterprise audience, they need to, I would say, relate to what we’re talking to. For an example, I gave a talk a couple of weeks ago on the AWS Summit here in Tel Aviv, of how to use App Runner. So, instead of explaining to the audiences how you use the console, this is what it does, you can deploy here, this is how the deployments work, blue, green, et cetera, et cetera, I made up an imaginary company and told the story of how the three people in the startup of this company would start working using App Runner in order to make the thing more relatable, something which people can hopefully remember and understand, okay, this is something which I would do as a startup, or this is what my project, which I’m doing or starting to work on, something I can use. So, to answer your question, in two words, tell stories instead of demo products.Corey: It feels like that’s a… heavy lift, in many cases, because I guess it’s also partially a perception issue on my part, where I’m looking at this across the board, where I see a company that has 5000 developers working there and, like, how do you wind up getting them to adopt cloud, or adopt new practices, or change anything? It feels like it’s a Herculean, impossible task. But in practice, I feel like you don’t try and do all of that at once. You start with small teams, you start with specific projects, and move on. Is that directionally accurate?Maish: Completely accurate. There’s no way to move a huge mothership in one direction at one time. You have to do, as you say, start small, find the projects, which are going to bring value to the company or the business, and start small with those projects and those small teams, and continue that education within the organization and help the people with your teaching or introducing them to the cloud, to help others within inside their own organization. Make them, or enable them, or empower them to become leaders within their own organization. That’s what I tried to do, at least.Corey: You and I have a somewhat similar background, which is weird given that we’ve just spent a fair bit of time talking about how different our upbringings were in tech at scales of companies and whatnot, but we’re alike in that we are both fairly crusty, old operations-side folks, sysadmins—Maish: [laugh]. Yep.Corey: —grumpy people.Maish: Grumpy old sysadmins. Yeah, exactly.Corey: Exactly. Because do you ever notice there’s never a happy one? Imagine that. And DevOps was always a meeting of the development and operations, meaning everyone’s unhappy. And there’s a school of thought that—like, I used to think that, “Oh, this is just what we call sysadmins once they want a better title and more money, but it’s still the same job.”But then I started meeting a bunch of DevOps types who had come from the exact opposite of our background, where they were software developers and then they wound up having to learn not so much how the code stuff works the way that we did, but rather how systems work, how infrastructure works. Compare and contrast those for me. Who makes, I guess, the more successful DevOps engineer when you look at it through that lens?Maish: So, I might be crucified for this on the social media from a number of people from the other side of the fence, but I have the firm belief that the people who make the best DevOps engineers—and I hate that term—but people who move DevOps initiatives or changes or transformations with organization is actually the operations people because they usually have a broader perspective of what is going on around them besides writing code. Too many times in my career, I’ve been burned by DNS, by a network cable, by a power outage, by somebody making a misconfiguration in the Puppet module, or whatever it might have been, somebody wrote it to deploy to 15,000 machines, whatever it may be. These are things where developers, at least my perception of what developers have been doing up until now, don’t really do that. In a previous organization I used to work for, the fact was, there was a very, very clear delineation about between the operations people, and the developers who wrote the software. We had very hard times getting them into rotations for on-call, we had very hard times educating them about the fact that not every single log line has to be written to the log because it doesn’t interest anybody.But from developer perspective, of course, we need that log because we need to know what’s happening in the end. But there are 15, different thousand… turtles all the way down, which have implications about the number of log lines which are written into a piece of software. So, I am very much of the belief that the people that make the best DevOps engineers—if we can use that term still today—are actually people which come from an operations background because it’s easier to teach them how to write code or become a programmer than the other way round of teaching a developer how to become an operations person. So, the change or the move from one direction from operations to adding the additional toil of writing software is much easier to accomplish than the other way around, from a developer learning how to run infrastructure at scale.Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don’t leave managing your database to your cloud vendor because they’re too busy launching another half-dozen managed databases to focus on any one of them that they didn’t build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: I once believed much the same because—and it made sense coming from the background that I was in. Everyone intellectually knows that if you’re having trouble with a piece of equipment, have you made sure that it’s plugged in? Yes, everyone knows that intellectually. But there’s something about having worked on a thing for three hours that wasn’t working and only discover it wasn’t plugged in, that really sears that lesson into your bones. The most confidence-inspiring thing you can ever hear from someone an operations role is, “Oh, I’ve seen this problem before. Here’s how we fixed it.”It feels like there are no junior DevOps engineers, for lack of a better term. And for a long time, I believed that the upcoming and operational side of the world were in fact, the better DevOps types. And in the fullness of time, I think a lot of that—at least my position on it—was rooted in some level of insecurity because I didn’t know how to write code and the thing that I saw happening was my job that I had done historically was eroding. Today, I don’t know that it’s possible to be in the operation space and not be at least basically conversant with how code works. There’s a reason most of these job interviews turn into algorithm hazing.And my articulation of it was rooted, for me at least—at least in a small way—in a sense of defensiveness and wanting to validate the thing that I had done with my career that I defined myself by, I was under threat. And obviously, the thing that I do is the best thing because otherwise it’s almost a tacit admission that I made poor career choices at some point. And I don’t think that’s true, either. But for me, at least psychologically, it was very much centered in that. And honestly, I found that the right answer for me was, in fact, neither of those two things because I have met a couple of people in my life that I would consider to be full-stack engineers.And there’s a colloquialism these days, that means oh, you do front-end and back-end. Yeah. The people I’m thinking of did front-end, they did back-end, they did mobile software, they did C-level programming, they wrote their own freaking device drivers at one point. Like, they have done basically everything. And they were the sort of person you could throw any technical issue whatsoever at and get out of their way because it was going to get solved. Those people are, as it turns out, the best. Like, who does a better job developer or operations, folks? Yes. Specifically, both of those things together.Maish: Exactly.Corey: And I think that is a hard thing to talk about. I think that it’s a hard—it’s certainly a hard thing to find. It turns out that there’s a reason that I only know two or three of those folks in the course of my entire career. They’re out there, but they’re really, really hard to track down.Maish: I completely agree.Corey: A challenge that I hear articulated in some cases—and while we’re saying things that are going to get us yelled out on social media, let’s go for the fences on this one—a concern that comes up when talking about enterprises moving to cloud is that they have a bunch of existing sysadmin types—while we’re on the topic—and well, those people need to learn to work within cloud. And the reality is, in many cases that first, that’s a whole new skill set that not everyone is going to be willing or able to pick up. For those who can they have just found that their market rate has effectively doubled. And that seems, on some level, to pose a significant challenge to companies undergoing this, and the larger the company, the more significant the challenge.Because it’s my belief that you pay market rate for the talent you have whether you want to or not. And if companies don’t increase compensation, these people will leave for things that double their income. And if they raise compensation internally, good for them, but that does have a massive drag on their budget that may not have been accounted for in a lot of the TCO analyses. How do you find that the companies you talk to wind up squaring that circle?Maish: I don’t think I have a correct answer for that. I do completely agree—Corey: Oh, I’m not convinced there’s a correct answer at all. I’m just trying [laugh] to figure out how to even think about it.Maish: I… have seen this as well in companies which I used to work for and companies that were customers that I have also worked with as part of my tenure in AWS. It’s the fact of, when companies are trying to move to the cloud and they start upskilling their people, there’s always the concern in the back of their mind of the fact, “Okay, I’m now training this person with new technology. I’m investing time, I’m investing money. And why would I do this if I know that, for example, as soon as I finish this, I’m going to have to just say, I have to pay them more because they can go somewhere else and get the same job with a better pay? So, why would we invest amount of time and resources into upscaling the people?”And these are questions which I have received and conversations which I’ve had with customers many times over the last two, three years. And the answer, from my perspective always, is the fact is because, number one, you’re making the world a better place. Number two, you’re making your employees feel more appreciated, giving them better knowledge. And if you’re afraid of the fact of teaching somebody to become better is going to have negative effects on your organization then, unfortunately, you deserve to have that person leave and let them find a better job because you’re not taking good care of your people. And it’s sometimes hard for companies to hear that.Sometimes we get, “You know what? You’re completely right.” Sometimes I don’t agree with you because I need to compete there, get to the bottom line, and make sure that I stay within my budget or my TCO. But the most important thing is to have the conversation, let people hear different ideas, see how it can benefit them, not only by giving people more options to maybe leave the company, but it can actually make their whole organization a lot better in the long run.Corey: I think that you have to do right by people because reputations last a long time. Even at big companies it becomes a very slow thing to change and almost impossible to do in the short term. So, people tell stories when they feel wronged. That becomes a problem. I do want to pivot a little bit because you’re not merely an EntReloper; you are an EntReloper specifically focusing on container services.Maish: Correct.Corey: Increasingly, I am viewing containers as what amounts to effectively a packaging format. That is the framework through which I am increasingly seeing. How are you seeing customers use containers? Is that directionally correct? Is it completely moonbat stuff compared to what you’re seeing in the wild, or something else?Maish: I don’t think it’s a packaging format; I think it’s more as an accelerator to enable the customers to develop in a more modern way with using twelve-factor apps with modern technology and not necessarily have their own huge, sticky, big monolith of whatever it might be, written in C# or whatever, or C++ whatever it may be, as they’ve been using up until now, but they now have the option and the technology and the background in order to split it up into smaller services and develop in the way that most of the modern world—or at least, the what we perceive as the modern world—is developing and creating applications today.Corey: I feel like on some level, containers were a radical change to how companies envisioned software. They definitely provide a path of modernizing things that were very tied to hardware previously. It let some companies even just leapfrog the virtualization migration that they’d been considering doing. But, on some level, I also feel like it runs counter to the ideas of DevOps, where you have development and operations working in partnership, where now it’s like, welp, inside the container is a development thing and outside the container, ops problem now. It feels almost, on some level, like, it reinforces a wall. But in a lot of cultures and a lot of companies, that wall is there and there’s no getting rid of it anytime soon. So, I confess that I’m conflicted on that.Maish: I think you might be right, and it depends, of course, on the company and the company culture, but what I think that companies need to do is understand that there will never be one hundred percent of people writing software that want to know one hundred percent of how the underlying infrastructure works. And the opposite direction as well: that there will never be people which maintain infrastructure and understand how computers and CPUs and memory buses and NUMA works on motherboards, that they don’t need to know how to write the most beautiful enticing and wonderful software for programs, for the world. There’s always going to have to be a compromise of who’s going to be doing this or who’s going to be doing that, and how comfortable they are with taking at least part of the responsibility of the other side into their own realm of what they should be doing. So, there’s going to be a compromise on both sides, but there is some kind of divide today of separating, okay, you just write the Helm chart for your Kubernetes Pod spec, or your ECS task, or whatever task definition, whatever you would like. And don’t worry about the things in the background because they’re just going to magically happen in the end. But they do have to understand exactly what is happening at the background in the end because if something goes wrong, and of course, something will go wrong, eventually, one day somewhere, somehow, they’re going to have to know how to take care of it.Corey: I really want to thank you for taking the time to speak with me today about, well, I guess a wide ranging variety of topics, some of which will absolutely inspire people to take to their feet—or at least their Twitter accounts—and tell us, “You know what your problem is?” And I honestly live for that. If you don’t evoke that kind of reaction on some level, have you ever really had an opinion in the first place? So, I’m looking forward to that. If people want to learn more about you, your beliefs, call set beliefs misguided, et cetera, et cetera, where’s the best place to find you?Maish: So, I’m on Twitter under @maishsk. I assume that will be in the [show notes 00:26:31]. I pontificate some time on technology, on cooking every now and again, on Friday before the end of the weekend, a little bit of politics, but you can find me @maishsk on Twitter. Or maishsk everywhere else social that’s possible.Corey: Excellent. We will toss links to that, of course, in the [show notes 00:26:50]. Thank you so much for being so generous with your time. I appreciate it.Maish: Thank you very much, Corey. It was fun.Corey: Maish Saidel-Keesing, EntReloper of container services at AWS. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated it, please leave a five-star review on your podcast platform of choice along with an angry comment that your 5000 enterprise developer colleagues can all pile on.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About ChrisChris is a robotics engineer turned cloud security practitioner. From building origami robots for NASA, to neuroscience wearables, to enterprise software consulting, he is a passionate builder at heart. Chris is a cofounder of Common Fate, a company with a mission to make cloud access simple and secure.Links: Common Fate: https://commonfate.io/ Granted: https://granted.dev Twitter: https://twitter.com/chr_norm TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Let’s face it, on-call firefighting at 2am is stressful! So there’s good news and there’s bad news. The bad news is that you probably can’t prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it’s hard to know where problems originate. Is it your application code, users, or the underlying systems? I’ve got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it’s more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. It doesn’t matter where you are on your journey in cloud—you could never have heard of Amazon the bookstore—and you encounter AWS and you spin up an account. And within 20 minutes, you will come to the realization that everyone in this space does. “Wow, logging in to AWS absolutely blows goats.”Today, my guest, obviously had that reaction, but unlike most people I talked to, decided to get up and do something about it. Chris Norman is the co-founder of Common Fate and most notably to how I know him is one of the original authors of the tool, Granted. Chris, thank you so much for joining me.Chris: Hey, Corey, thank you for having me.Corey: I have done podcasts before; I have done a blog post on it; I evangelize it on Twitter constantly, and even now, it is challenging in a few ways to explain holistically what Granted is. Rather than trying to tell your story for you, when someone says, “Oh, Granted, that seems interesting and impossible to Google for in isolation, so therefore, we know it’s going to be good because all the open-source projects with hard to find names are,” what is Granted and what does it do?Chris: Granted is a command-line tool which makes it really easy for you to get access and assume roles when you’re working with AWS. For me, when I’m using Granted day-to-day, I wake up, go to my computer—I’m working from home right now—crack open the MacBook and I log in and do some development work. I’m going to go and start working in the cloud.Corey: Oh, when I start first thing in the morning doing development work and logging into the cloud, I know. All right, I’m going to log in to AWS and now I know that my day is going downhill from here.Chris: [laugh]. Exactly, exactly. I think maybe the best days are when you don’t need to log in at all. But when you do, I go and I open my terminal and I run this command. Using Granted, I ran this assume command and it authenticates me with single-sign-on into AWS, and then it opens up a console window in a particular account.Now, you might ask, “Well, that’s a fairly standard thing.” And in fact, that’s probably the way that the console and all of the tools work by default with AWS. Why do you need a third-party tool for this?Corey: Right. I’ve used a bunch of things that do varying forms of this and unlike Granted, you don’t see me gushing about them. I want to be very clear, we have no business relationship. You’re not sponsoring anything that I do. I’m not entirely clear on what your day job entails, but I have absolutely fallen in love with the Granted tool, which is why I’m dragging you on to this show, kicking and screaming, mostly to give me an excuse to rave about it some more.Chris: [laugh]. Exactly. And thank you for the kind words. And I’d say really what makes it special or why I’ve been so excited to be working on it is that it makes this access, particularly when you’re working with multiple accounts, really, really easy. So, when I run assume and I open up that console window, you know, that’s all fine and that’s very similar to how a lot of the other tools and projects that are out there work, but when I want to open that second account and that second console window, maybe because I’m looking at like a development and a staging account at the same time, then Granted allows me to view both of those simultaneously in my browser. And we do that using some platform sort of tricks and building into the way that the browser works.Corey: Honestly, one of the biggest differences in how you describe what Granted is and how I view it is when you describe it as a CLI application because yes, it is that, but one of the distinguishing characteristics is you also have a Firefox extension that winds up leveraging the multi-container functionality extension that Firefox has. So, whenever I wind up running a single command—assume with a-c’ flag, then I give it the name of my AWS profile, it opens the web console so I can ClickOps my heart’s content inside of a tab that is locked to a container, which means I can have one or two or twenty different AWS accounts and/or regions up running simultaneously side-by-side, which is basically impossible any other way that I’ve ever looked at it.Chris: Absolutely, yeah. And that’s, like, the big differentiating factor right now between Granted and between this sort of default, the native experience, if you’re just using the AWS command line by itself. With Granted, you can—with these Firefox containers, all of your cookies, your profile, everything is all localized into that one container. It’s actually it’s a privacy features that are built into Firefox, which keeps everything really separate between your different profiles. And what we’re doing with Granted is that we make it really easy to open a specific profiles that correspond with different AWS profiles that you’re using.So, you’d have one which could be your development account, one which could be production or staging. And you can jump between these and navigate between them just as separate tabs in your browser, which is a massive improvement over, you know, what I’ve previously had to use in the past.Corey: The thing that really just strikes me about this is first, of course, the functionality and the rest, so I saw this—I forget how I even came across it—and immediately I started using it. On my Mac, it was great. I started using it when I was on the road, and it was less great because you built this thing in Go. It can compile and install on almost anything, but there were some assumptions that you had built into this in its early days that did not necessarily encompass all of the use cases that I use. For example, it hadn’t really occurred to you that some lunatic would try and only use an iPad when they’re on the road, so they have to be able to run this to get federated login links via SSHing into an EC2 instance running somewhere and not have it open locally.You seemed almost taken aback when I brought it up. Like, “What lunatic would do that?” Like, “Hi, I’m such a lunatic. Let’s talk about this.” And it does that now, and it’s awesome. It does seem to me though, and please correct me if I’m wrong on this assumption slash assessment that this is first and foremost aimed at desktop users, specifically people running Mac on the desktop, is that the genesis of it?Chris: It is indeed. And I think part of the cause behind that is that we originally built a tool for ourselves. And as we were building things and as we were working using the cloud, we were running things—you know, we like to think that we’re following best practices when we’re using AWS, and so we’d set up multiple accounts, we’d have a special account for development, a separate one for staging, a separate one for production, even internal tools that we would build, we would go and spin up an individual account for those. And then you know, we had lots of accounts. and to go and access those really easily was quite difficult.So, we definitely, we built it for ourselves first and I think that that’s part of when we released it, it actually a little bit of cause for some of the initial problems. And some of the feedback that we had was that it’s great to build tools for yourself, but when you’re working in open-source, there’s a lot of different diversity with how people are using things.Corey: We take different approaches. You want to try to align with existing best practices, whereas I am a loudmouth white guy who works in tech. So, what I do definitionally becomes a best practice in the ecosystem. It’s easier to just comport with the ones that are already existing that smart people put together rather than just trying to competence your way through it, so you took a better path than I did.But there’s been a lot of evolution to Granted as I’ve been using it for a while. I did a whole write-up on it and that got a whole bunch of eyes onto the project, which I can now admit was a nefarious plan on my part because popping into your community Slack and yelling at you for features I want was all well and good, but let’s try and get some people with eyes on this who are smarter than me—which is not that high of a bar when it comes to SSO, and IAM, and federated login, and the rest—and they can start finding other enhancements that I’ll probably benefit from. And sure enough, that’s exactly what happened. My sneaky plan has come to fruition. Thanks for being a sucker, I guess. I mean—[laugh] it worked. I’m super thrilled by the product.Chris: [laugh]. I guess it’s a great thing I think that the feedback and particularly something that’s always been really exciting is just seeing new issues come through on GitHub because it really shows the kinds of interesting use cases and the kinds of interesting teams and companies that are using Granted to make their lives a little bit easier.Corey: When I go to the website—which again is impossible to Google—the website for those wondering is granted.dev. It’s short, it’s concise, I can say it on a podcast and people automatically know how to spell it. But at the top of the website—which is very well done by the way—it mentions that oh, you can, “Govern access to breakglass roles with Common Fate Cloud,” and it also says in the drop shadow nonsense thing in the upper corner, “Brought to you by Common Fate,” which is apparently the name of your company.So, the question I’ll get to in a second is what does your company do, but first and foremost, is this going to be one of those rug-pull open-source projects where one day it’s, “Oh, you want to log into your AWS accounts? Insert quarter to continue.” I’m mostly being a little over the top with that description, but we’ve all seen things that we love turn into molten garbage. What is the plan around this? Are you about to ruin this for the rest of us once you wind up raising a round or something? What’s the deal?Chris: Yeah, it’s a great question, Corey. And I think that to a degree, releasing anything like this that sits in the access workflow and helps you assume roles and helps you day-to-day, you know, we have a responsibility to uphold stability and reliability here and to not change things. And I think part of, like, not changing things includes not [laugh] rug-pulling, as you’ve alluded to. And I think that for some companies, it ends up that open-source becomes, like, a kind of a lead-generation tool, or you end up with, you know, now finally, let’s go on add another login so that you have to log into Common Fate to use Granted. And I think that, to be honest, a tool like this where it’s all about improving the speed of access, the incentives for us, like, it doesn’t even make sense to try and add another login for to try to get people to, like, to say, login to Common Fate because that would make your signing process for AWS take even longer than it already does.Corey: Yeah, you decided that you know, what’s the biggest problem? Oh, you can sleep at night, so let’s go ahead and make it even worse, by now I want you to be this custodian of all my credentials to log into all of my accounts. And now you’re going to be critical path, so if you’re down, I’m not able to log into anything. And oh, by the way, I have to trust you with full access to my bank stuff. I just can’t imagine that is a direction that you would be super excited about diving head-first into.Chris: No, no. Yeah, certainly not. And I think that the, you know, building anything in this space, and with what we’re doing with Common Fate, you know, we’re building a cloud platform to try to make IAM a little bit easier to work with, but it’s really sensitive around granting any kind of permission and I think that you really do need that trust. So, trying to build trust, I guess, with our open-source projects is really important for us with Granted and with this project, that it’s going to continue to be reliable and continue to work as it currently does.Corey: The way I see it, one of the dangers of doing anything that is particularly open-source—or that leans in the direction of building in Amazon’s ecosystem—it leads to the natural question of, well, isn’t this just going to be some people say stolen—and I don’t think those people understand how open-source works—by AWS themselves? Or aren’t they going to build something themselves at AWS that’s going to wind up stomping this thing that you’ve built? And my honest and remarkably cynical answer is that, “You have built a tool that is a joy to use, that makes logging into AWS accounts streamlined and efficient in a variety of different patterns. Does that really sound like something AWS would do?” And followed by, “I wish they would because everyone would benefit from that rising tide.”I have to be very direct and very clear. Your product should not exist. This should be something the provider themselves handles. But nope. Instead, it has to exist. And while I’m glad it does, I also can’t shake the feeling that I am incredibly annoyed by the fact that it has to.Chris: Yeah. Certainly, certainly. And it’s something that I think about a little bit. I like to wonder whether there’s maybe like a single feature flag or some single sort of configuration setting in AWS where they’re not allowing different tabs to access different accounts, they’re not allowing this kind of concurrent access. And maybe if we make enough noise about Granted, maybe one of the engineers will go and flick that switch and they’ll just enable it by default.And then Granted itself will be a lot less relevant, but for everybody who’s using AWS, that’ll be a massive win because the big draw of using Granted is mainly just around being able to access different accounts at the same time. If AWS let you do that out of the box, hey, that would be great and, you know, I’d have a lot less stuff to maintain.Corey: Originally, I had you here to talk about Granted, but I took a glance at what you’re actually building over at Common Fate and I’m about to basically hijack slash derail what probably is going to amount the rest of this conversation because you have a quick example on your site for by developers, for developers. You show a quick Python script that tries to access a S3 bucket object and it’s denied. You copy the error message, you paste it into what you’re building over a Common Fate, and in return, it’s like, “Oh. Yeah, this is the policy that fixes it. Do you want us to apply it for you?”And I just about fell out of my chair because I have been asking for this explicit thing for a very long time. And AWS doesn’t do it. Their IAM access analyzer claims to. Like, “Oh, just go look at CloudTrail and see what permissions it uses and we’ll build a policy to scope it down.” “Okay. So, it’s S3 access. Fair enough. To what object or what bucket?” “Guess,” is what it tells you there.And it’s, this is crap. Who thinks this is a good user experience? You have built the thing that I wish AWS had built in natively. Because let’s be honest here, I do what an awful lot of people do and overscope permissions massively just because messing around with the bare minimum set of permissions in many cases takes more time than building the damn thing in the first place.Chris: Oh, absolutely. Absolutely. And in fact, this—was a few years ago when I was consulting—I had a really similar sort of story where one of the clients that we were working with, the CTO of this company, he was needing to grant us access to AWS and we were needing to build a particular service. And he said, “Okay, can you just let me know the permissions that you will need and I’ll go and deploy the role for this.” And I came back and I said, “Wait. I don’t even know the permissions that I’m going to need because the damn thing isn’t even built yet.”So, we went sort of back and forth around this. And the compromise ended up just being you know, way too much access. And that was sort of part of the inspiration for, you know, really this whole project and what we’re building with Common Fate, just trying to make that feedback loop around getting to the right level of permissions a lot faster.Corey: Yeah, I am just so overwhelmingly impressed by the fact that you have built—and please don’t take this as a criticism—but a set of very simple tools. Not simple in the terms of, “Oh, that’s, like, three lines of bash, and a fool could write that on a weekend.” No. Simple in the sense of it solves a problem elegantly and well and it’s straightforward—well, straightforward as anything in the world of access control goes—to wrap your head around exactly what it does. You don’t tend to build these things by sitting around a table brainstorming with someone you met at co-founder dating pool or something and wind up figuring out, “Oh, we should go and solve that. That sounds like a billion-dollar problem.”This feels very much like the outcome of when you’re sitting around talking to someone and let’s start by drinking six beers so we become extraordinarily honest, followed immediately by let’s talk about what sucks. What pisses you off the most? It feels like this is sort of the low-hanging fruit of things that upset people when it comes to AWS. I mean, if things had gone slightly differently, instead of focusing on AWS bills, IAM was next on my list of things to tackle just because I was tired of smacking my head into it.This is very clearly a problem space that you folks have analyzed deeply, worked within, and have put a lot of thought into. I want to be clear, I’ve thrown a lot of feature suggestions that you for Granted from start to finish. But all of them have been around interface stuff and usability and expanding use cases. None of them have been, “Well, that seems screamingly insecure.” Because it hasn’t been.Chris: [laugh].Corey: It has been effective, start to finish, I think that from a security posture, you make terrific choices, in many cases better than ones I would have made a starting from scratch myself. Everything that I’m looking at in what you have built is from a position of this is absolutely amazing and it is transformative to my own workflows. Now, how can we improve it?Chris: Mmm. Thank you, Corey. And I’ll say as well, maybe around the security angle, that one of the goals with Granted was to try and do things a little bit better than the default way that AWS does them when it comes to security. And it’s actually been a bit of a source for challenges with some of the users that we’ve been working with with Granted because one of the things we wanted to do was encrypt the SSO token. And this is the token that when you sign in to AWS, kind of like, it allows you to then get access to all of the rest of the accounts.So, it’s like a pretty—it’s a short-lived token, but it’s a really sensitive one. And you know, by default, it’s just stored in plain text on your disk. So, we dump to a file and, you know, anything that can go and read that, they can go and get it. It’s also a little bit hard to revoke and to lock people out. There’s not really great workflows around that on AWS’s side.So, we thought, “Okay, great. One of the goals for Granted can be that we will go and store this in your keychain in your system and we’ll work natively with that.” And that’s actually been a cause for a little bit of a hassle for some users, though, because by doing that and by storing all of this information in the keychain, it’s actually broken some of the integrations with the rest of the tooling, which kind of expects tokens and things to be in certain places. So, we’ve actually had to, as part of dealing with that with Granted, we’ve had to give users the ability to opt out for that.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: That’s why I find this so, I think, just across the board, fantastic. It’s you are very clearly engaged with your community. There’s a community Slack that you have set up for this. And I know, I know, too many Slacks; everyone has this problem. This is one of those that is worth hanging in, at least from my perspective, just because one of the problems that you have, I suspect, is on my Mac it’s great because I wind up automatically updating it to whatever the most recent one is every time I do a brew upgrade.But on the Linux side of the world, you’ve discovered what many of us have discovered, and that is that packaging things for Linux is a freaking disaster. The current installation is, “Great. Here’s basically a curl bash.” Or, “Here, grab this tarball and install it.” And that’s fine, but there’s no real way of keeping that updated and synced.So, I was checking the other day, oh wow, I’m something like eight versions behind on this box. But it still just works. I upgraded. Oh, wow. There’s new functionality here. This is stuff that’s actually really handy. I like this quite a bit. Let’s see what else we can do.I’m just so impressed, start to finish, by just how receptive you’ve been to various community feedbacks. And as well—I want to be very clear on this point, too—I’ve had folks who actually know what they’re doing in an InfoSec sense look at what you’re up to, and none of them had any issues of note. I’m sure that they have a pile of things like, with that curl bash, they should really be doing a GPG check. Yes, yes, fine. Whatever. If that’s your target threat model, okay, great. Here in reality-land for what I do, this is awesome.And they don’t seem to have any problems with, “Oh, yeah. By the way, sending analytics back up”—which, okay, fine, whatever. “And it’s not disclosing them.” Okay, that’s bad. “And it’s including the contents of your AWS credentials.”Ahhhh. I did encounter something that was doing that on the back-end once. [cough]—Serverless Framework—sorry, something caught in my throat for a second.Chris: [laugh].Corey: No faster way I can think of to erode trust in that. But everything you’re doing just makes sense.Chris: Oh, I do remember that. And that was a little bit of a fiasco, really, around all of that, right? And it’s great to hear actually around that InfoSec folks and security people being, you know, not unhappy, I guess, with a tool like this. It’s been interesting for me personally. We’ve really come from a practitioner’s background.You know, I wouldn’t call myself a security engineer at all. I would call myself as a sometimes a software developer, I guess. I have been hacking my way around Go and definitely learning a lot about how the cloud has worked over the past seven, eight years or so, but I wouldn’t call myself a security engineer, so being very cautious around how all of these things work. And we’ve really tried to defer to things like the system keychain and defer to things that we know are pretty safe and work.Corey: The thing that I also want to call out as well is that your licensing is under the MIT license. This is not one of those, “Oh, you’re required to wind up doing a bunch of branding stuff around it.” And, like some people say, “Oh, you have to own the trademark for all of these things.” I mean, I’m not an expert in international trademark law, let’s be very clear, but I also feel that trademarking a term that is already used heavily in the space such as the word ‘Granted,’ feels like kind of an uphill battle. And let’s further be clear that it doesn’t matter what you call this thing.In fact, I will call attention to an oddity that I’ve encountered a fair bit. After installing it, the first thing you do is you run the command ‘granted.’ That sets it up, it lets you configure your browser, what browser you want to use, and it now supports standard out for that headless, EC2 use case. Great. Awesome. Love it. But then the other binary that ships with it is Assume. And that’s what I use day-to-day. It actually takes me a minute sometimes when it’s been long enough to remember that the tool is called Granted and not Assume what’s up with that?Chris: So, part of the challenge that we ran into when we were building the Granted project is that we needed to export some environment variables. And these are really important when you’re logging into AWS because you have your access key, your secret key, your session token. All of those, when you run the assume command, need to go into the terminal session that you called it. This doesn’t matter so much when you’re using the console mode, which is what we mentioned earlier where you can open 100 different accounts if you want to view all of those at the same time in your browser. But if you want to use it in your terminal, we wanted to make it look as really smooth and seamless as possible here.And we were really inspired by this approach from—and I have to shout them out and kind of give credit to them—a tool called AWSume—they’re spelled A-W-S-U-M-E—Python-based tool that they don’t do as much with single-sign-on, but we thought they had a really nice, like, general approach to the way that they did the scripting and aliasing. And we were inspired by that and part of that means that we needed to have a shell script that called this executable, which then will export things back out into the shell script. And we’re doing all this wizardry under the hood to make the user experience really smooth and seamless. Part of that meant that we separated the commands into granted and assume and the other part of the naming for everything is that I felt Granted had a far better ring to it than calling the whole project Assume.Corey: True. And when you say assume, is it AWS or not? I’ve used the AWSume project before; I’ve used AWS Vault out of 99 Designs for a while. I’ve used—for three minutes—the native AWS SSO config, and that is just trash. Again, they’re so good at the plumbing, so bad at the porcelain, I think is the criticism that I would levy toward a lot of this stuff.Chris: Mmm.Corey: And it’s odd to think there’s an entire company built around just smoothing over these sharp, obnoxious edges, but I’m saying this as someone who runs a consultancy and have five years that just fixes the bill for this one company. So, there’s definitely a series of cottage industries that spring up around these things. I would be thrilled, on some level, if you wound up being completely subsumed by their product advancements, but it’s been 15 years for a lot of this stuff and we’re still waiting. My big failure mode that I’m worried about is that you never are.Chris: Yeah, exactly, exactly. And it’s really interesting when you think about all of these user experience gaps in AWS being opportunities for, I guess, for companies like us, I think, trying to simplify a lot of the complexity for things. I’m interested in sort of waiting for a startup to try and, like, rebuild the actual AWS console itself to make it a little bit faster and easier to use.Corey: It’s been done and attempted a bunch of different times. The problem is that the console is a lot of different things to a lot of different people, and as you step through that, you can solve for your use case super easily. “Yeah, what do I care? I use RDS, I use some VPC nonsense, and I use EC2. The end.” “Great. What about IAM?”Because I promise you’re using that whether you know it or not. And okay, well, I’m talking to someone else who’s DynamoDB, and someone else is full-on serverless, and someone else has more money than sense, so they mostly use SageMaker, and so on and so forth. And it turns out that you’re effectively trying to rebuild everything. I don’t know if that necessarily works.Chris: Yeah, and I think that’s a good point around maybe while we haven’t seen anything around that sort of space so far. You go to the console, and you click down, you see that list of 200 different services and all of those have had teams go and actually, like, build the UI and work with those individual APIs. Yeah.Corey: Any ideas as far as what’s next for features on Granted?Chris: I think that, for us, it’s continuing to work with everybody who’s using it, and with a focus of stability and performance. We actually had somebody in the community raise an issue because they have an AWS config file that’s over 7000 lines long. And I kind of pity that person, potentially, for their day-to-day. They must deal with so much complexity. Granted is currently quite slow when the config files get very big. And for us, I think, you know, we built it for ourselves; we don’t have that many accounts just yet, so working to try to, like, make it really performant and really reliable is something that’s really important.Corey: If you don’t mind a feature request while we’re at it—and I understand that this is more challenging than it looks like—I’m willing to fund this as a feature bounty that makes sense. And this also feels like it might be a good first project for a very particular type of person, I would love to get tab completion working in Zsh. You have it—Chris: Oh.Corey: For Fish because there’s a great library that automatically populates that out, but for the Zsh side of it, it’s, “Oh, I should just wind up getting Zsh completion working,” and I fell down a rabbit hole, let me tell you. And I come away from this with the perception of yeah, I’m not going to do it. I have not smart enough to check those boxes. But a lot of people are so that is the next thing I would love to see. Because I will change my browser to log into the AWS console for you, but be damned if I’m changing my shell.Chris: [laugh]. I think autocomplete probably should be higher on our roadmap for the tool, to be honest because it’s really, like, a key metric and what we’re focusing on is how easy is it to log in. And you know, if you’re not too sure what commands to use or if we can save you a few keystrokes, I think that would be the, kind of like, reaching our goals.Corey: From where I’m sitting, you definitely have. I really want to thank you for taking the time to not only build this in the first place, but also speak with me about it. If people want to learn more, where’s the best place to find you?Chris: So, you can find me on Twitter, I’m @chr_norm, or you can go and visit granted.dev and you’ll have a link to join the Slack community. And I’m very active on the Slack.Corey: You certainly are, although I will admit that I fall into the challenge of being in just the perfectly opposed timezone from you and your co-founder, who are in different time zones to my understanding; one of you is on Australia and one of you was in London; you’re the London guy as best I’m aware. And as a result, invariably, I wind up putting in feature requests right when no one’s around. And, for better or worse, in the middle of the night is not when I’m usually awake trying to log into AWS. That is Azure time.Chris: [laugh]. Yeah, no, we don’t have the US time zone properly covered yet for our community support and help. But we do have a fair bit of the world timezone covered. The rest of the team for Common Fate is all based in Australia and I’m out here over in London.Corey: Yeah. I just want to thank you again, for just being so accessible and, like, honestly receptive to feedback. I want to be clear, there’s a way to give feedback and I do strive to do it constructively. I didn’t come crashing into your Slack one day with a, “You know what your problem is?” I prefer to take the, “This is awesome. Here’s what I think would be even better. Does that make sense?” As opposed to the imperious demands and GitHub issues and whatnot? It’s, “I’d love it if it did this thing. Doesn’t do this thing. Can you please make it do this thing?” Turns out that’s the better way to drive change. Who knew?Chris: Yeah. [laugh]. Yeah, definitely. And I think that one of the things that’s been the best around our journey with Granted so far has been listening to feedback and hearing from people how they would like to use the tool. And a big thank you to you, Corey, for actually suggesting changes that make it not only better for you, but better for everybody else who’s using Granted.Corey: Well, at least as long as we’re using my particular byzantine workload patterns in some way, or shape, or form, I’ll hear that. But no, it’s been an absolute pleasure and I really want to thank you for your time as well.Chris: Yeah, thank you for having me.Corey: Chris Norman, co-founder of Common Fate, as well as one of the two primary developers originally behind the Granted project that logs you into AWS without you having to lose your mind. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, incensed, raging comment that talks about just how terrible all of this is once you spend four hours logging into your AWS account by hand first.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
Full Description / Show Notes Corey and Linda talk about Tiktok and the online developer community (1:18) Linda talks about what prompted her to want to work at AWS (5:29) Linda discusses navigating the change from just being part of the developer community to being an employee of AWS (10:37) Linda talks about moving AWS more in the direction of short form content, and Corey and Linda talk about the Tiktok algorithm (15:56) Linda talks about the potential struggle of going from short form to long form content (25:21) About LindaLinda Vivah is a Site Reliability Engineer for a major media organization in NYC, a tech content creator, an AWS community builder member, a part-time wedding singer, and the founder of a STEM jewelry shop called Coding Crystals. At the time of this recording she was about to join AWS in her current position as a Developer Advocate.Linda had an untraditional journey into tech. She was a Philosophy major in college and began her career in journalism. In 2015, she quit her tv job to attend The Flatiron School, a full stack web development immersive program in NYC. She worked as a full-stack developer building web applications for 5 years before shifting into SRE to work on the cloud end internally.Throughout the years, she’s created tech content on platforms like TikTok & Instagram and believes that sometimes the best way to learn is to teach.Links Referenced:lindavivah.com: https://lindavivah.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it’s hard to know where problems originate. Is it your application code, users, or the underlying systems? I’ve got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it’s more than just hipster monitoring.Corey: Let’s face it, on-call firefighting at 2am is stressful! So there’s good news and there’s bad news. The bad news is that you probably can’t prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. We talk a lot about how people go about getting into this ridiculous industry of ours, and I’ve talked a little bit about how I go about finding interesting and varied guests to show up and help me indulge my ongoing love affair on this show with the sound of my own voice. Today, we’re going to be able to address both of those because today I’m speaking to Linda Haviv, who, as of this recording, has accepted a job as a Developer Advocate at AWS, but has not started. Linda, welcome to the show.Linda: Thank you so much for having me, Corey. Happy to be here.Corey: So, you and I have been talking for a while and there’s been a lot of interesting things I learned along the way. You were one of the first people I encountered when I joined the TikToks, as all the kids do these days, and was trying to figure out is there a community of folks who use AWS. Which really boils down to, “So, where are these people that are sad all the time?” Well, it turns out, they’re on TikTok, so there we go. We found my people.And that was great. And we started talking, and it turns out that we were both in the AWS community builder program. And we’ve developed a bit of a rapport. We talk about different things. And then, I guess, weird stuff started happening, in the context of you were—you’re doing very well at building an audience for yourself on TikTok.I tried it, and it was—my sense of humor sometimes works, sometimes doesn’t. I’ve had challenges in finding any reasonable way to monetize it because a 30-second video doesn’t really give nuance for a full ad read, for example. And you’ve been looking at it from the perspective of a content creator looking to build the audience slash platform is step one, and then, eh, step two, you’ll sort of figure out aspects of monetization later. Which, honestly, is a way easier way to do it in hindsight, but, yeah, the things that we learn. Now, that you’re going to AWS, first, you planning to still be on the TikToks and whatnot?Linda: Absolutely. So, I really look at TikTok as a funnel. I don’t think it’s the main place, you’re going to get that deep-dive content but I think it’s a great way, especially for things that excite you or get you into understanding it, especially beginner-type audience, I think there’s a lot of untapped market of people looking to into tech, or technologists that aren’t in the cloud. I mean, even when I worked—I worked as a web developer and then kind of learned more about the cloud, and I started out as a front-end developer and shifted into, like, SRE and infrastructure, so even for people within tech, you can have a huge tech community which there is on TikTok, with a younger community—but not all of them really understand the cloud necessarily, depending on their job function. So, I think it’s a great way to kind of expose people to that.For me, my exposure came from community. I met somebody at a meetup who was working in cloud, and it wasn’t even on the job that I really started getting into cloud because many times in corporations, you might be working on a specific team and you’re not really encountering other ends, and it seems kind of like a mystery. Although it shouldn’t seem like magic, many times when you’re doing certain job functions—especially the DevOps—could end up feeling like magic. So, [laugh] for the good and the bad. So sometimes, if you’re not working on that end, you really sometimes take it for granted.And so, for me, I actually—meetups were the way I got exposed to that end. And then I brought it back into my work and shifted internally and did certifications and started, even, lunch-and-learns where I work to get more people in their learning journey together within the company, and you know, help us as we’re migrating to the cloud, as we’re building on the cloud. Which, of course, we have many more roles down the road. I did it for a few years and saw the shift. But I worked at a media company for many years and now shifting to AWS, and so I’ve seen that happen on different ends.Not—oh, I wasn’t the one doing the migration because I was on the other end of that time, but now for the last two years, I was working on [laugh] the infrastructure end, and so it’s really fascinating. And many people actually—until now I feel like—that will work on maybe the web and mobile and don’t always know as much about the cloud. I think it’s a great way to funnel things in a quick manner. I think also society is getting used to short videos, and our attention span is very low, and I think for—Corey: No argument here.Linda: —[crosstalk 00:04:39] spending so mu—yeah, and we’re spending so much time on these platforms, we might as well, you know, learn something. And I think it depends what content. Some things work well, some things doesn’t. As with anything content creation, you kind of have to do trial and error, but I do find the audience to be a bit different on TikTok versus Twitter versus Instagram versus YouTube. Which is interesting how it’s going to play out on YouTube, too, which is a whole ‘nother topic conversation.Corey: Well, it’s odd to me watching your path. It’s almost the exact opposite of mine where I started off on the back-end, grumpy sysadmin world and, “Oh, why would I ever need to learn JavaScript?” “Well, genius, because as the world progresses, guess what? That’s right. The entire world becomes JavaScript. Welcome.”And it took me a long time to come around to that. You started with the front-end world and then basically approached from the exact opposite end. Let’s be clear, back in my day, mine was the common path. These days, yours is very much the common path.Linda: Yeah.Corey: I also want to highlight that all of those transitions and careers that you spoke about, you were at the same company for nine years, which in tech is closer to 30. So, I have to ask, what was it that inspired you, after nine years, to decide, “I’m going to go work somewhere else. But not just anywhere; I’m going to AWS.” Because normally people don’t almost institutionalized lifers past a certain point.Linda: [laugh].Corey: Like, “Oh, you’ll be there till you retire or die.” Whereas seeing significant career change after that long in one place, even if you’ve moved around internally and experienced a lot of different roles, is not common at all what sparked that?Linda: Yeah. Yeah, no, it’s such a good question. I always think about that, too, especially as I was reflecting because I’m, you know, in the midst of this transition, and I’ve gotten a lot of reflecting over the last two weeks [laugh], or more. But I think the main thing for me is, I always, wherever I was—and this kind of something that—I’m very proactive when it comes to trying to transition. I think, even when I was—right, I held many roles in the same company; I used to work in TV production and actually left for three months to go to a coding boot camp and then came back on the other end, but I understood the product in a different way.So, for that time period, it was really interesting to work on the other end. But, you know, as I kind of—every time I wanted to progress further, I always made a move that was actually new and put me in an uncomfortable place, even within the same company. And I’m at the point now that I’m in my career, I felt like this next step really needs to be, you know, at AWS. It's not, like, the natural progression for me. I worked alongside—on the client end—with AWS and have seen so many projects come through and how much our own workloads have changed.And it’s just been an incredible journey, also dealing with accounts team. On that end, I’ve worked alongside them, so for me, it was kind of a natural progression. I was very passionate about cloud computing at AWS and I kind of wanted to take it to that next place, and I felt like—also, dealing with the community as part of my job is a dream part to me because I was always doing that on the side on social media. So, it wasn’t part of my day-to-day job. I was working as an SRE and an infrastructure engineer, so I didn’t get to do that as part of my day-to-day.I was making videos at 2 a.m. and, you know, kind of trying to, like, do—you know, interact with the community like that. And I think—I come from a performing background, the people background, I was singing since I was four years old. I always go to—I was a wedding singer, so I go into a room and I love making people happy or giving value. And I think, like, education has a huge part of that. And in a way, like making that content and—Corey: You got to get people’s attention—Linda: Yeah.Corey: —you can’t teach them a damn thing.Linda: Right. Exactly. So, it’s kind of a mix of everything. It’s like that performance, the love of learning. You know, between you and I, like, I wanted to be a lawyer before I thought I was going to—before I went to tech.I thought I was going to be a lawyer purely because I loved the concept of going to law school. I never took time to think about the law part, like, being the lawyer part. I always thought, “Oh, school.” I’m a student at heart. I always call myself a professional student. I really think that’s part of what you need to be in this world, in this tech industry, and I think for me, that’s what keeps my fire going.I love to experiment, to learn, to build. And there’s something very fulfilling about building products. If you take a step back, like, you’re kind of—you know, for me that part, every time I look back at that, that always is what kind of keeps me going. When I was doing front-end, it felt a lot more like I was doing smaller things than when I was doing infrastructure, so I felt like that was another reason why I shifted. I love doing the front-end, but I felt like I was spending two days on an Internet Explorer bug and it just drove me—[laugh] it just made it feel unfulfilling versus spending two days on, you know, trying to understand why, you know, something doesn’t run the infrastructure or, like, there’s—you know, it’s failing blindly, you know? Stuff like that. Like, I don’t know, for me that felt more fulfilling because the problem was more macro. But I think I needed both. I have a love for both, but I definitely prefer being back-end. So. [laugh]. Well, I’m saying that now but—[laugh].Corey: This might be a weakness on my part where I’m basically projecting onto others, and this is—I might be completely wrong on this, but I tend to take a bit of a bifurcated view of community. I mean, community is part of the reason that I know the things I know and how I got to this place that I am, so use that as a cautionary tale if you want. But when I talk to someone like you at this moment, where you’re in the community, I’m in the community, and I’m talking to you about a problem I’m having and we’re working on ways to potentially solve that or how to think about that. I view us as basically commiserating on these things, whereas as soon as you start on day one—and yes, it’s always day one—at AWS and this becomes your day job and you work there, on some level, for me, there’s a bit shift that happens and a switch gets flipped in my head where, oh, you actually work at this company. That means you’re the problem.And I’m not saying that in a way of being antagonistic. Please, if you’re watching or listening to this, do not antagonize the developer advocates. They have a very hard job understanding all this so they can explain that to the rest of us. But how do you wind up planning to navigate, or I guess your views on, I guess, handling the shift between, “One of the customers like the rest of us,” to, as I say, “Part of the problem,” for lack of a better term.Linda: Or, like, work because you kind of get the—you know. I love this question and it’s something I’ve been pondering a lot on because I think the messaging will need to be a little different [coming from me 00:10:44] in the sense of, there needs to be—just in anything, you have to kind of create trust. And to create trust, you have to be vulnerable and authentic. And I think I, for example, utilize a lot of things outside of just the AWS cloud topic to do that now, even, when I—you know, kind of building it without saying where I work or anything like that, going into this role and it being my job, it’s going to be different kind of challenge as far as the messaging, but I think it still holds true that part, that just developing trust and authenticity, I might have to do more of that, you know? I might have to really share more of that part, share other things to really—because it’s more like people come, it doesn’t matter how much somet—how many times you explain it, many times, they will see your title and they will judge you for it, and they don’t know what happened before. Every TikTok, for example, you have to act like it’s a new person watching. There is no series, you know? Like, yes, there’s a series but, like, sometimes you can make that but it’s not really the way TikTok functions or a short-form video functions. So, you kind of have to think this is my first time—Corey: It works really terribly when you’re trying to break it out that way on TikTok.Linda: [laugh]. Yeah.Corey: Right. Here’s part 17 of my 80-TikTok-video saga. And it’s, “Could you just turn this into a blog post or put this on YouTube or something? I don’t have four hours to spend learning how all this stuff works in your world.”Linda: Yeah. And you know, I think repeating certain things, too, is really important. So, they say you have to repeat something eight times for people to see it or [laugh] something like that. I learned that in media [crosstalk 00:12:13]—Corey: In a row, or—yeah. [laugh].Linda: I mean, the truth is that when you, kind of like, do a TikTok maybe, like, there’s something you could also say or clarify because I think there’s going to be—and I’m going to have to—there’s going to be a lot of trial and error for me; I don’t know if I have answers—but my plan is going into it very much testing that kind of introduction, or, like, clarifying what that role is. Because the truth is, the role is advocating on behalf of the community and really helping that community, so making sure that—you don’t have to say it as far as a definition maybe, but, like, making sure that comes across when you create a video. And I think that’s going to be really important for me, and more important than the prior even creating content going forward. So, I think that’s one thing that I definitely feel like is key.As well as creating more raw interaction. So, it depends on the platform, too. Instagram, for example, is much more community—how do I put this? Instagram is much more easy to navigate as far as reaching the same community because you have something, like, called Instagram Stories, right? So, on Instagram Stories, you’re bringing those stories, mostly the same people that follow you. You’re able to build that trust through those stories.On TikTok, they just released Stories. I haven’t really tried them much and I don’t play with it a lot, but I think that’s something I will utilize because those are the people that are already follow you, meaning they have seen a piece of content. So, I think addressing it differently and knowing who’s watching what and trying to kind of put yourself in their shoes when you’re trying to, you know, teach something, it’s important for you to have that trust with them. And I think—key to everything—being raw and authentic. I think people see through that. I would hope they do.And I think, uh, [laugh] that’s what I’m going to be trying to do. I’m just going to be really myself and real, and try to help people and I hope that comes through because that’s—I’m passionate about getting more people into the cloud and getting them educated. And I feel like it’s something that could also allow you to build anything, just from anywhere on your computer, brings people together, the world is getting smaller, really. And just being able to meet people through that and there’s just a way to also change your life. And people really could change their life.I changed my life, I think, going into tech and I’m in the United States and I, you know—I’m in New York, you know, but I feel like so many people in the States and outside of the States, you know, all over the world, you know, have access to this, and it’s powerful to be able to build something and contribute and be a part of the future of technology, which AWS is.Corey: I feel like, in three years or whatever it is that you leave AWS in the far future, we’re going to basically pull this video up and MST3k came together. It’s like, “Remember how naive you were talking about these things?” And I’m mostly kidding, but let’s be serious. You are presumably going to be focusing on the idea of short-form content. That is—Linda: Yeah.Corey: What your bread-and-butter of audience-building has been around, and that is something that is new for AWS.Linda: Yeah.Corey: And I’m always curious as to how companies and their cultures continue to evolve. I can only imagine there’s a lot of support structure in place for that. I personally remember giving a talk at an AWS event and I had my slides reviewed by their legal team, as they always do, and I had a slide that they were looking at very closely where I was listing out the top five AWS services that are bullshit. And they don’t really have a framework for that, so instead, they did their typical thing of, “Okay, we need to make sure that each of those services starts with the appropriate AWS or Amazon naming convention and are they capitalized properly?” Because they have a framework for working on those things.I’m really curious as to how the AWS culture and way of bringing messaging to where people are is going to be forced to evolve now that they, like it or not, are going to be having significantly increased presence on TikTok and other short-form platforms.Linda: I mean, it’s really going to be interesting to see how this plays out. There’s so much content that’s put out, but sometimes it’s just not reaching the right audience, so making sure that funnel exists to the right people is important and reaching those audiences. So, I think even YouTube Shorts, for example. Many people in tech use YouTube to search a question.They do not care about the intro, sometimes. It depends what kind of following, it depends if [in gaming 00:16:30], but if you’re coming and you’re building something, it’s like a Stack Overflow sometimes. You want to know the answer to your question. Now, YouTube Shorts is a great solution to that because many times people want the shortest possible answer. Now, of course, if it’s a tutorial on how to build something, and it warrants ten minutes, that’s great.Even ten minutes is considered, now, Shorts because TikTok now has ten-minute videos, but I think TikTok is now searchable in the way YouTube is, and I think let’s say YouTube Shorts is short-form, but very different type of short-form than TikTok is. TikTok, hooks matter. YouTube answers to your questions, especially in chat. I wouldn’t say everything in YouTube is like that; depends on the niche. But I think even within short-form, there’s going to be a different strategy regarding that.So, kind of like having that mix. I guess, depending on platform and audience, that’s there. Again, trial and error, but we’ll see how this plays out and how this will evolve. Corey: This episode is sponsored in part by our friends at Vultr. Optimized cloud compute plans have landed at Vultr to deliver lightning-fast processing power, courtesy of third-gen AMD EPYC processors without the IO or hardware limitations of a traditional multi-tenant cloud server. Starting at just 28 bucks a month, users can deploy general-purpose, CPU, memory, or storage optimized cloud instances in more than 20 locations across five continents. Without looking, I know that once again, Antarctica has gotten the short end of the stick. Launch your Vultr optimized compute instance in 60 seconds or less on your choice of included operating systems, or bring your own. It’s time to ditch convoluted and unpredictable giant tech company billing practices and say goodbye to noisy neighbors and egregious egress forever. Vultr delivers the power of the cloud with none of the bloat. Screaming in the Cloud listeners can try Vultr for free today with a $150 in credit when they visit getvultr.com/screaming. That’s G-E-T-V-U-L-T-R dot com slash screaming. My thanks to them for sponsoring this ridiculous podcast.Corey: I feel like there are two possible outcomes here. One is that AWS—Linda: Yeah.Corey: Nails this pivot into short-form content, and the other is that all your TikTok videos start becoming ten minutes long, which they now support, welcome to my TED Talk. It’s awful, and then you wind up basically being video equivalent for all of your content, of recipes when you search them on the internet where first they circle the point to death 18 times with, “Back when I was a small child growing up in the hinterlands, we wound—my grandmother would always make the following stew after she killed the bison with here bare hands. Why did grandma kill a bison? We don’t know.” And it just leads down this path so they can get, like, long enough content or they can have longer and longer articles to display more ads.And then finally at the end, it’s like ingredient one: butter. Ingredient two, there is no ingredient two. Okay. That explains why it’s delicious. Awesome. But I don’t like having people prolong it. It’s just, give me the answer I’m looking for.Linda: Yeah.Corey: Get to the point. Tell me the story. And—Linda: And this is—Corey: —I’m really hoping that is not the direction your content goes in. Which I don’t think it would, but that is the horrifying thing and if for some chance I’m right, I will look like Nostradamus when we do that MST3k episode.Linda: No, no. I mean, I really am—I always personally—even when I was creating content these last few years and testing different things, I’m really a fan of the shortest way possible because I don’t have the patience to watch long videos. And maybe it’s because I’m a New Yorker that can’t sit down from the life of me—apart from when I code of course—but, you know, I don’t like wasting time, I’m always on the go, I’m with my coffee, I’m like—that’s the kind of style I prefer to bring in videos in the sense of, like, people have no time. [laugh]. You know?The amount of content we’re consuming is just, uh, bonkers. So, I don’t think our mind is really a built for consuming [laugh] this much content every time you open your phone, or every time you look, you know, online. It’s definitely something that is challenging in a whole different way. But I think where my content—if it’s ten minutes, it better be because I can’t shorten it. That’s my thing. So, you can hold me accountable to that because—Corey: Yeah, I want ten minutes of—Linda: I’m not a—Corey: Content, not three minutes of content in a ten-minute bag.Linda: Exactly. Exactly. So, if it’s a ten-minute video, it would have been in one hour that I cut down, like, meaning a tutorial, a very much technical types of content. I think things that are that long, especially in tech, would be something like, on that end—unless, of course, you know, I’m not talking about, like, longer videos on YouTube which are panels or that kind of thing. I’m talking more like if I’m doing something on TikTok specifically.TikTok also cares about your watch time, so if people aren’t interested in it, it’s not going to do well, it doesn’t matter how many followers you have. Which is what I do like about the way TikTok functions as opposed to, let’s say, Instagram. Instagram is more like it gives it to your following—and this is the current state, I don’t know if it always evolves—but the current state is, Instagram Reels kind of functions in a way where it goes first to the people that follow you, but, like, in a way that’s more amplified than TikTok. TikTox tests people that follows you, but if it’s not a good video, it won’t do well. And honestly, they’re many good videos videos that don’t go viral. I’m not talking about that.Sometimes it’s also the topic and the niche and the sound and the title. I mean, there’s so many people who take a topic and do it in three different ways and one of them goes viral. I mean, there’s so many factors that play into it and it’s hard to really, like, always, you know, kind of reverse engineer but I do think that with TikTok, things won’t do well, more likely if it’s not a good piece of content as opposed to—or, like, too long, right? Not—I shouldn’t say not good a good piece of content—it’s too long.Corey: The TikTok algorithm is inscrutable to me. TikTok is firmly convinced, based upon what it shows me, that I am apparently a lesbian. Which okay, fine. Awesome. Whatever. I’m also—it keeps showing me ads for ADHD stuff, and it was like, “Wow, like, how did it know that?” Followed by, “Oh, right. I’m on TikTok. Nevermind.”And I will say at one point, it recommended someone to me who, looking at the profile picture, she’s my nanny. And it’s, I have a strong policy of not, you know, stalking my household employees on social media. We are not Facebook friends, we are not—in a bunch of different areas. Like, how on earth would they have figured this out? I’m filling the corkboard with conspiracy and twine followed by, “Wait a minute. We probably both connect from the same WiFi network, which looks like the same IP address and it probably doesn’t require a giant data science team to put two and two together on those things.” So, it was great. I was all set to do the tinfoil hat conspiracy, but no, no, that’s just very basic correlation 101.Linda: And also, this is why I don’t enable contacts on TikTok. You know, how it says, “Oh, connect your contacts?”Corey: Oh, I never do that. Like, “Can we look at your contacts?”Linda: Never.Corey: “No.” “Can we look at all of your photos?” “Absolutely not.” “Can we track you across apps?” “Why would anyone say yes to this? You’re going to do it anyway, but I’ll say no.” Yeah.Linda: Got to give the least privilege. [laugh]. Definitely not—Corey: Oh absolutely.Linda: Yeah. I think they also help [crosstalk 00:22:40]—Corey: But when I’m looking at—the monetization problem is always a challenge on things like this, too, because when I’m—my guilty TikTok scrolling pleasures hit, it’s basically late at night, I just want to see—I want something to want to wind down and decompress. And I’m not about ready to watch, “Hey, would you like to migrate your enterprise database to this other thing?” It’s, I… no. There’s a reason that the ads that seem to be everywhere and doing well are aimed at the mass market, they’re generally impulse buys, like, “Hey, do you want to set that thing over there on fire, but you’re not close enough to get the job done? But this flame thrower today. Done.”And great, like, that is something everyone can enjoy, but these nuanced database products and anything else is B2B SaaS style stuff, it feels like it’s a very tough sell and no one has quite cracked that nut, yet.Linda: Yeah, and I think the key there—this is, I’m guessing based on, like, what I want to try out a lot—is the hook and the way you’re presenting it has to be very product-focused in the sense that it needs to be very relatable. Even if you don’t know anything about tech, you need to be—like, for example, in the architecture page on AWS, there’s a video about the Emirates going to Mars mission. Space is a very interesting topic, right? I think, a hook, like, “Do want to see how, like, how this is bu—” like, it’s all, like, freely available to see exactly [laugh] how this was built. Like, it might—in the right wording, of course—it might be interesting to someone who’s looking for fun-fact-style content.Now, is it really addressing the people that are building everyday? Not really always, depends who’s on there and the mass market there. But I feel like going on the product and the things that are mass-market, and then working backwards to the tech part of it, even if they learn something and then want to learn more, that’s really where I see TikTok. I don’t think every platform would be, maybe, like this, but that’s where I see getting people: kind of inviting them in to learn more, but making it cool and fun. It’s very important, but it feels cool and fun. [laugh]. So.Because you’re right, you’re scrolling at 2 a.m. who wants to start seeing that. Like, it’s all about how you teach. The content is there, the content has—you know, that’s my thing. It’s like, the content is there. You don’t need to—it’s yes, there’s the part where things are always evolving and you need to keep track of that; that’s whole ‘nother type thing which you do very well, right?And then there’s a part where, like, the content that already exists, which part is evergreen? Meaning, which part is, like, something that could be re—also is not timely as far as update, for example, well-architected framework. Yes, it evolves all the time, you always have new pillars, but the guide, the story, that is an evergreen in some sense because that guide doesn’t, you know, that whole concept isn’t going anywhere. So, you know, why should someone care about that?Corey: Right. How to turn on two-factor authentication for your AWS account.Linda: Right.Corey: That’s evergreen. That’s the sort of thing that—and this is the problem, I think, AWS has had for a long time where they’re talking about new features, new enhancements, new releases. But you look what people are actually doing and so much of it is just the same stuff again and again because yeah, that is how most of the cloud works. It turns out that three-quarters of company’s production infrastructures tends to run on EC2 more frequently than it tends to run on IoT Greengrass. Imagine that.So, there’s this idea of continuing to focus on these things. Now, one of my predictions is that you’re going to have a lot of fun with this and on some level, it’s going to really work for you. In others, it’s going to be hilariously—well, its shortcomings might be predictable. I can just picture now you’re at re:Invent; you have a breakout talk and terrific. And you’ve successfully gotten your talk down to one minute and then you’re sitting there with—Linda: [laugh].Corey: —the remainder of maybe 59. Like, oh, right. Yeah. Turns out not everything is short-form. Are you predicting any—Linda: Yep.Corey: Problems going from short-form to long-form in those instances?Linda: I think it needs to go hand-in-hand, to be honest. I think when you’re creating any short-form content, you have—you know, maybe something short is actually sometimes in some ways, right, harder because you really have to make sure, especially in a technical standpoint, leaving things out is sometimes—leaves, like, a blind spot. And so, making sure you’re kind of—whatever you’re educating, you kind of, to be clear, “Here’s where you learn more. Here’s how I’m going to answer this next question for you: go here.” Now, in a longer-form content, you would cover all that.So, there’s always that longevity. I think even when I write a script, and there’s many scripts I’m still [laugh] I’ve had many ideas until now I’ve been doing this still at 2 a.m. so of course, there’s many that didn’t, you know, get released, but those are the things that are more time consuming to create because you’re taking something that’s an hour-long, and trying to make sure you’re pulling out the things that are most—that are hook-style, that invite people in, that are accurate, okay, that really give you—explain to you clearly where are the blind spots that I’m not explaining on this video are. So, “XYZ here is, like, the high level, but by the way, there’s, like, this and this.” And in a long-form, you kind of have to know the long-form version of it to make the short-form, in some ways, depending on what—you’re doing because you’re funneling them to somewhere. That’s my thing. Because I don’t think there should be [crosstalk 00:27:36]—Corey: This is the curse of Twitter, on some level. It’s, “Well, you forgot about this corner case.” “Yeah, I had 280 characters to get into.” Like, the whole point of short-form content—which I do consider Twitter to be—is a glimpse and a hook, and get people interested enough to go somewhere and learn more.For something like AWS, this makes a lot of sense. When you highlight a capability or something interesting, it’s something relevant, whereas on the other side of it, where it’s this, “Oh, great. Now, here’s an 8000-word blog post on how I did this thing.” Yeah, I’m going to get relatively fewer amounts of traffic through that giant thing, but the people who are they’re going to be frickin’ invested because that’s going to be a slog.Linda: Exactly.Corey: “And now my eight-hour video on how exactly I built this thing with TypeScript.” Badly—Linda: Exactly.Corey: —as it turns out because I’m a bad programmer.Linda: [laugh]. No, you’re not. I love your shit-posting. It’s great.Corey: Challenge accepted.Linda: [laugh]. I love what you just mentioned because I think you’re hitting the nail on the head when it comes to the quality content that’s niche focus, like, there needs to be a good healthy mix. I think always doing that, like, mass-market type video, it doesn’t give you, also, the credibility you need. So, doing those more niche things that might not be relevant to everybody, but here and there, are part of that is really key for your own knowledge and for, like, the com—you know, as far as, like, helping someone specific. Because it’s almost like—right, when you’re selling a service and you’re using social media, right, not everybody’s going to buy your service. It doesn’t matter what business you’re in right? The deep-divers are going to be the people that pay up. It’s just a numbers game, right? The more people you, kind of, address from there, you’ll find—Corey: It’s called a funnel for a reason.Linda: Right. Exactly.Corey: Free content, paid content. Almost anyone will follow me on Twitter; fewer than will sign up for a newsletter; fewer will listen to a podcast; fewer will watch a video, and almost none of them will buy a consulting engagement. But ‘almost’ and ‘actually none of them,’ it turns out is a very different world.Linda: Exactly. [laugh]. So FYI, I think there’s—Corey: And that’s fine. That’s the way it works.Linda: That’s the way it works. And I think there needs to be that niche content that might not be, like, the most viral thing, but viral doesn’t mean quality, you know? It doesn’t. There’s many things that play into what viral is, but it’s important to have the quality content for the people that need that content, and finding those people, you know, it’s easier when you have that kind of mass engagement. Like, who knows? I’m a student. I told you; I’m a professional student. I’m still [laugh] learning every day.Corey: Working with AWS almost makes it a requirement. I wish you luck—Linda: Yeah.Corey: —in the new gig and I also want to thank you for taking time out of your day to speak with me about how you got to this point. And we’re all very eager to see where you go from here.Linda: Thank you so much, Corey, for having me. I’m a huge fan, I love your content, I’m an avid reader of your newsletter and I am looking forward to very much being in touch and on the Twitterverse and beyond. So. [laugh].Corey: If people want to learn more about what you’re up to, and other assorted nonsense, where’s the best place they can go to find you?Linda: So, the best place they could go is lindavivah.com. I have all my different social handles listed on there as well a little bit about me, and I hope to connect with you. So, definitely go to lindavivah.com.Corey: And that link will, of course, be in the [show notes 00:30:39]. Thank you so much for taking the time to speak with me. I really appreciate it.Linda: Thank you, Corey. Have a wonderful rest of the day.Corey: Linda Haviv, AWS Developer Advocate, very soon now anyway. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, smash the like and subscribe buttons, and of course, leave an angry comment that you have broken down into 40 serialized TikTok videos.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
Full Description / Show Notes Steren and Corey talk about how Google Cloud Run got its name (00:49) Corey talks about his experiences using Google Cloud (2:42) Corey and Steren discuss Google Cloud’s cloud run custom domains (10:01) Steren talks about Cloud Run’s high developer satisfaction and scalability (15:54) Corey and Steren talk about Cloud Run releases at Google I/O (23:21) Steren discusses the majority of developer and customer interest in Google’s cloud product (25:33) Steren talks about his 20% projects around sustainability (29:00) About SterenSteren is a Senior Product Manager at Google Cloud. He is part of the serverless team, leading Cloud Run. He is also working on sustainability, leading the Google Cloud Carbon Footprint product.Steren is an engineer from École Centrale (France). Prior to joining Google, he was CTO of a startup building connected objects and multi device solutions.Links Referenced: Google Cloud Run: https://cloud.run sheets-url-shortener: https://github.com/ahmetb/sheets-url-shortener snark.cloud/run: https://snark.cloud/run Twitter: https://twitter.com/steren TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. I’m joined today by Steren Giannini, who is a senior product manager at Google Cloud, specifically on something called Google Cloud Run. Steren, thank you for joining me today.Steren: Thanks for inviting me, Corey.Corey: So, I want to start at the very beginning of, “Oh, a cloud service. What are we going to call it?” “Well, let’s put the word cloud in it.” “Okay, great. Now, it is cloud, so we have to give it a vague and unassuming name. What does it do?” “It runs things.” “Genius. Let’s break and go for work.” Now, it’s easy to imagine that you spent all of 30 seconds on a name, but it never works that way. How easy was it to get to Cloud Run as a name for the service?Steren: [laugh]. Such a good question because originally it was not named Cloud Run at all. The original name was Google Serverless Engine. But a few people know that because they’ve been helping us since the beginning, but originally it was Google Serverless Engine. Nobody liked the name internally, and I think at one point, we wondered, “Hey, can we drop the engine structure and let’s just think about the name. And what does this thing do?” “It runs things.”We already have Cloud Build. Well, wouldn’t it be great to have Cloud Run to pair with Cloud Build so that after you’ve built your containers, you can run them? And that’s how we ended up with this very simple Cloud Run, which today seems so obvious, but it took us a long time to get to that name, and we actually had a lot of renaming to do because we were about to ship with Google Serverless Engine.Corey: That seems like a very interesting last-minute change because it’s not just a find and replace at that point, it’s—Steren: No.Corey: —“Well, okay, if we call it Cloud Run, which can also be a verb or a noun, depending, is that going to change the meaning of some sentences?” And just doing a find and replace without a proofread pass as well, well, that’s how you wind up with funny things on Twitter.Steren: API endpoints needed to be changed, adding weeks of delays to the launch. That is why we—you know, [laugh] announced in 2018 and publicly launched in 2019.Corey: I’ve been doing a fair bit of work in cloud for a while, and I wound up going down a very interesting path. So, the first native Google Cloud service—not things like WP Engine that ride on top of GCP—but my first native Google Cloud Service was done in service of this podcast, and it is built on Google Cloud Run. I don’t think I’ve told you part of this story yet, but it’s one of the reasons I reached out to invite you onto the show. Let me set the stage here with a little bit of backstory that might explain what the hell I’m talking about.As listeners of this show are probably aware, we have sponsors whom we love and adore. In the early days of this show, they would say, “Great, we want to tell people about our product”—which is the point of a sponsorship—“And then send them to a URL.” “Great. What’s the URL?” And they would give me something that was three layers deep, then with a bunch of UTM tracking parameters at the end.And it’s, “You do realize that no one is going to be sitting there typing all of that into a web browser?” At best, you’re going to get three words or so. So, I built myself a URL redirector, snark.cloud. I can wind up redirecting things in there anywhere it needs to go.And for a long time, I did this on top of S3 and then put CloudFront in front of it. And this was all well and good until, you know, things happened in the fullness of time. And now holy crap, I have an operations team involved in things, and maybe I shouldn’t be the only person that knows how to work on all of these bits and bobs. So, it was time to come up with something that had a business user-friendly interface that had some level of security, so I don’t wind up automatically building out a spam redirect service for anything that wants to, and it needs to be something that’s easy to work with. So, I went on an exploration.So, at first it showed that there were—like, I have an article out that I’ve spoken about before that there are, “17 Ways to Run Containers on AWS,” and then I wrote the sequel, “17 More Ways to Run Containers on AWS.” And I’m keeping a list, I’m almost to the third installation of that series, which is awful. So, great. There’s got to be some ways to build some URL redirect stuff with an interface that has an admin panel. And I spent three days on this trying a bunch of different things, and some were running on deprecated versions of Node that wouldn’t build properly and others were just such complex nonsense things that had got really bad. I was starting to consider something like just paying for Bitly or whatnot and making it someone else’s problem.And then I stumbled upon something on GitHub that really was probably one of the formative things that changed my opinion of Google Cloud for the better. And within half an hour of discovering this thing, it was up and running. I did the entire thing, start to finish, from my iPad in a web browser, and it just worked. It was written by—let me make sure I get his name correct; you know, messing up someone’s name is a great way to say that we don’t care about them—Ahmet Balkan used to work at Google Cloud; now he’s over at Twitter. And he has something up on GitHub that is just absolutely phenomenal about this, called sheets-url-shortener.And this is going to sound wild, but stick with me. The interface is simply a Google Sheet, where you have one column that has the shorthand slug—for example, run; if you go to snark.cloud/run, it will redirect to Google Cloud Run’s website. And the second column is where you want it to go. The end.And whenever that gets updated, there’s of course some caching issues, which means it can take up to five seconds from finishing that before it will actually work across the entire internet. And as best I can tell, that is fundamentally magic. But what made it particularly useful and magic, from my perspective, was how easy it was to get up and running. There was none of this oh, but then you have to integrate it with Google Sheets and that’s a whole ‘nother team so there’s no way you’re going to be able to figure that out from our Docs. Go talk to them and then come back in the day.They were the get started, click here to proceed. It just worked. And it really brought back some of the magic of cloud for me in a way that I hadn’t seen in quite a while. So, all which is to say, amazing service, I continue to use it for all of these sponsored links, and I am still waiting for you folks to bill me, but it fits comfortably in the free tier because it turns out that I don’t have hundreds of thousands of people typing it in every week.Steren: I’m glad it went well. And you know, we measure tasks success for Cloud Run. And we do know that most new users are able to deploy their apps very quickly. And that was the case for you. Just so you know, we’ve put a lot of effort to make sure it was true, and I’ll be glad to tell you more about all that.But for that particular service, yes, I suppose Ahmet—who I really enjoyed working with on Cloud Run, he was really helpful designing Cloud Run with us—has open-sourced this side project. And basically, you might even have clicked on a deploy to Cloud Run button on GitHub, right, to deploy it?Corey: That is exactly what I did and it somehow just worked and—Steren: Exactly.Corey: And it knew, even logging into the Google Cloud Console because it understands who I am because I use Google Docs and things, I’m already logged in. None of this, “Oh, which one of these 85 credential sets is it going to be?” Like certain other clouds. It was, “Oh, wow. Wait, cloud can be easy and fun? When did that happen?”Steren: So, what has happened when you click that deploy to Google Cloud button, basically, the GitHub repository was built into a container with Cloud Build and then was deployed to Cloud Run. And once on Cloud Run, well, hopefully, you have forgotten about it because that’s what we do, right? We—give us your code, in a container if you know containers if you don’t just—we support, you know, many popular languages, and we know how to build them, so don’t worry about that. And then we run it. And as you said, when there is low traffic or no traffic, it scales to zero.When there is low traffic, you’re likely going to stay under the generous free tier. And if you have more traffic for, you know, Screaming in the Cloud suddenly becoming a high destination URL redirects, well, Cloud Run will scale the number of instances of this container to be able to handle the load. Cloud Run scales automatically and very well, but only—as always—charging you when you are processing some requests.Corey: I had to fork and make a couple of changes myself after I wound up doing some testing. The first was to make the entire thing case insensitive, which is—you know, makes obvious sense. And the other was to change the permanent redirect to a temporary redirect because believe it or not, in the fullness of time, sometimes sponsors want to change the landing page in different ways for different campaigns and that’s fine by me. I just wanted to make sure people’s browser cache didn’t remember it into perpetuity. But it was easy enough to run—that was back in the early days of my exploring Go, which I’ve been doing this quarter—and in the couple of months this thing has been running it has been effectively flawless.It’s set it; it’s forget it. The only challenges I had with it are it was a little opaque getting a custom domain set up that—which is still in beta, to be clear—and I’ve heard some horror stories of people saying it got wedged. In my case, no, I deployed it and I started refreshing it and suddenly, it start throwing an SSL error. And it’s like, “Oh, that’s not good, but I’m going to break my own lifestyle here and be patient for ten minutes.” And sure enough, it cleared itself and everything started working. And that was the last time I had to think about any of this. And it just worked.Steren: So first, Cloud Run is HTTPS only. Why? Because it’s 2020, right? It’s 2022, but—Corey: [laugh].Steren: —it’s launched in 2020. And so basically, we have made a decision that let’s just not accept HTTP traffic; it’s only HTTPS. As a consequence, we need to provision a cert for your custom domain. That is something that can take some time. And as you said, we keep it in beta or in preview because we are not yet satisfied with the experience or even the performance of Cloud Run custom domains, so we are actively working on fixing that with a different approach. So, expect some changes, hopefully, this year.Corey: I will say it does take a few seconds when people go to a snark.cloud URL for it to finish resolving, and it feels on some level like it’s almost like a cold start problem. But subsequent visits, the same thing also feel a little on the slow and pokey side. And I don’t know if that’s just me being wildly impatient, if there’s an optimization opportunity, or if that’s just inherent to the platform that is not under current significant load.Steren: So, it depends. If the Cloud Run service has scaled down to zero, well of course, your service will need to be started. But what we do know, if it’s a small Go binary, like something that you mentioned, it should really take less than, let’s say, 500 milliseconds to go from zero to one of your container instance. Latency can also be due to the way the code is running. If it occurred is fetching things from Google Sheets at every startup, that is something that could add to the startup latency.So, I would need to take a look, but in general, we are not spinning up a virtual machine anytime we need to scale horizontally. Like, our infrastructure is a multi-tenant, rapidly scalable infrastructure that can materialize a container in literally 300 milliseconds. The rest of the latency comes from what does the container do at startup time?Corey: Yeah, I just ran a quick test of putting time in front of a curl command. It looks like it took 4.83 seconds. So, enough to be perceptive. But again, for just a quick redirect, it’s generally not the end of the world and there’s probably something I’m doing that is interesting and odd. Again, I did not invite you on the show to file a—Steren: [laugh].Corey: Bug report. Let’s be very clear here.Steren: Seems on the very high end of startup latencies. I mean, I would definitely expect under the second. We should deep-dive into the code to take a look. And by the way, building stuff on top of spreadsheets. I’ve done that a ton in my previous lives as a CTO of a startup because well, that’s the best administration interface, right? You just have a CRUD UI—Corey: [unintelligible 00:12:29] world and all business users understand it. If people in Microsoft decided they were going to change Microsoft Excel interface, even a bit, they would revert the change before noon of the same day after an army of business users grabbed pitchforks and torches and marched on their headquarters. It’s one of those things that is how the world runs; it is the world’s most common IDE. And it’s great, but I still think of databases through the lens of thinking about it as a spreadsheet as my default approach to things. I also think of databases as DNS, but that’s neither here nor there.Steren: You know, if you have maybe 100 redirects, that’s totally fine. And by the way, the beauty of Cloud Run in a spreadsheet, as you mentioned is that Cloud Run services run with a certain identity. And this identity, you can grant it permissions. And in that case, what I would recommend if you haven’t done so yet, is to give an identity to your Cloud Run service that has the permission to read that particular spreadsheet. And how you do that you invite the email of the service account as a reader of your spreadsheet, and that’s probably what you did.Corey: The click button to the workflow on Google Cloud automatically did that—Steren: Oh, wow.Corey: —and taught me how to do it. “Here’s the thing that look at. The end.” It was a flawless user-onboarding experience.Steren: Very nicely done. But indeed, you know, there is this built-in security which is the principle of minimal permission, like each of your Cloud Run service should basically only be able to read and write to the backing resources that they should. And by default, we give you a service account which has a lot of permissions, but our recommendation is to narrow those permissions to basically only look at the cloud storage buckets that the service is supposed to look at. And the same for a spreadsheet.Corey: Yes, on some level, I feel like I’m going to write an analysis of my own security approach. It would be titled, “My God, It's Full Of Stars” as I look at the IAM policies of everything that I’ve configured. The idea of least privilege is great. What I like about this approach is that it made it easy to do it so I don’t have to worry about it. At one point, I want to go back and wind up instrumenting it a bit further, just so I can wind up getting aggregate numbers of all right, how many times if someone visited this particular link? It’ll be good to know.And I don’t know… if I have to change permissions to do that yet, but that’s okay. It’s the best kind of problem: future Corey. So, we’ll deal with that when the time comes. But across the board, this has just been a phenomenal experience and it’s clear that when you were building Google Cloud Run, you understood the assignment. Because I was looking for people saying negative things about it and by and large, all of its seem to come from a perspective of, “Well, this isn’t going to be the most cost-effective or best way to run something that is hyperscale, globe-spanning.”It’s yes, that’s the thing that Kubernetes was originally built to run and for some godforsaken reason people run their blog on it instead now. Okay. For something that is small, scales to zero, and has long periods where no one is visiting it, great, this is a terrific answer and there’s absolutely nothing wrong with that. It’s clear that you understood who you were aiming at, and the migration strategy to something that is a bit more, I want to say robust, but let’s be clear what I mean when I’m saying that if you want something that’s a little bit more impressive on your SRE resume as you’re trying a multi-year project to get hired by Google or pretend you got hired by Google, yeah, you can migrate to something else in a relatively straightforward way. But that this is up, running, and works without having to think about it, and that is no small thing.Steren: So, there are two things to say here. The first is yes, indeed, we know we have high developer satisfaction. You know, we measure this—in Google Cloud, you might have seen those small satisfaction surveys popping up sometimes on the user interface, and you know, we are above 90% satisfaction score. We hire third parties to help us understand how usable and what satisfaction score would users get out of Cloud Run, and we are constantly getting very, very good results, in absolute but also compared to the competition.Now, the other thing that you said is that, you know, Cloud Run is for small things, and here while it is definitely something that allows you to be productive, something that strives for simplicity, but it also scales a lot. And contrary to other systems, you do not have any pre-provisioning to make. So, we have done demos where we go from zero to 10,000 container instances in ten seconds because of the infrastructure on which Cloud Run runs, which is fully managed and multi-tenant, we can offer you this scale on demand. And many of our biggest customers have actually not switched to something like Kubernetes after starting with Cloud Run because they value the low maintenance, the no infrastructure management that Cloud Run brings them.So, we have like Ikea, ecobee… for example ecobee, you know, the smart thermostats are using Cloud Run to ingest events from the thermostat. I think Ikea is using Cloud Run more and more for more of their websites. You know, those companies scale, right? This is not, like, scale to zero hobby project. This is actually production e-commerce and connected smart objects production systems that have made the choice of being on a fully-managed platform in order to reduce their operational overhead.[midroll 00:17:54]Corey: Let me be clear. When I say scale—I think we might be talking past each other on a small point here. When I say scale, I’m talking less about oh tens or hundreds of thousands of containers running concurrently. I’m talking in a more complicated way of, okay, now we have a whole bunch of different microservices talking to one another and affinity as far as location to each other for data transfer reasons. And as you start beginning to service discovery style areas of things, where we build a really complicated applications because we hired engineers and failed to properly supervise them, and that type of convoluted complex architecture.That’s where it feels like Cloud Run increasingly, as you move in that direction, starts to look a little bit less like the tool of choice. Which is fine, I want to be clear on that point. The sense that I’ve gotten of it is a great way to get started, it’s a great way to continue running a thing you don’t have to think about because you have a day job that isn’t infrastructure management. And it is clear to—as your needs change—to either remain with the service or pivot to a very close service without a whole lot of retooling, which is key. There’s not much of a lock-in story to this, which I love.Steren: That was one of the key principles when we started to design Cloud Run was, you know, we realized the industry had agreed that the container image was the standard for the deployment artifact of software. And so, we just made the early choice of focusing on deploying containers. Of course, we are helping users build those containers, you know, we have things called build packs, we can continuously deploy from GitHub, but at the end of the day, the thing that gets auto-scaled on Cloud Run is a container. And that enables portability.As you said. You can literally run the same container, nothing proprietary in it, I want to be clear. Like, you’re just listening on a port for some incoming requests. Those requests can be HTTP requests, events, you know, we have products that can push events to Cloud Run like Eventarc or Pub/Sub. And this same container, you can run it on your local machine, you can run it on Kubernetes, you can run it on another cloud. You’re not locked in, in terms of API of the compute.We even went even above and beyond by having the Cloud Run API looks like a Kubernetes API. I think that was an extra effort that we made. I’m not sure people care that much, but if you look at the Cloud Run API, it is actually exactly looking like Kubernetes, Even if there is no Kubernetes at all under the hood; we just made it for portability. Because we wanted to address this concern of serverless which was lock-in. Like, when you use a Function as a Service product, you are worried that the architecture that you are going to develop around this product is going to be only working in this particular cloud provider, and you’re not in control of the language, the version that this provider has decided to offer you, you’re not in control of more of the complexity that can come as you want to scan this code, as you want to move this code between staging and production or test this code.So, containers are really helping with that. So, I think we made the right choice of this new artifact that to build Cloud Run around the container artifact. And you know, at the time when we launched, it was a little bit controversial because back in the day, you know, 2018, 2019, serverless really meant Functions as a Service. So, when we launched, we little bit redefined serverless. And we basically said serverless containers. Which at the time were two worlds that in the same sentence were incompatible. Like, many people, including internally, had concerns around—Corey: Oh, the serverless versus container war was a big thing for a while. Everyone was on a different side of that divide. It’s… containers are effectively increasingly—and I know, I’ll get email for this, and I don’t even slightly care, they’re a packaging format—Steren: Exactly.Corey: —where it solves the problem of how do I build this thing to deploy on Debian instances? And Ubuntu instances, and other instances, God forbid, Windows somewhere, you throw a container over the wall. The end. Its DevOps is about breaking down the walls between Dev and Ops. That’s why containers are here to make them silos that don’t have to talk to each other.Steren: A container image is a glorified zip file. Literally. You have a set of layers with files in them, and basically, we decided to adopt that artifact standard, but not the perceived complexity that existed at the time around containers. And so, we basically merged containers with serverless to make something as easy to use as a Function as a Service product but with the power of bringing your own container. And today, we are seeing—you mentioned, what kind of architecture would you use Cloud Run for?So, I would say now there are three big buckets. The obvious one is anything that is a website or an API, serving public internet traffic, like your URL redirect service, right? This is, you have an API, takes a request and returns a response. It can be a REST API, GraphQL API. We recently added support for WebSockets, which is pretty unique for a service offering to support natively WebSockets.So, what I mean natively is, my client can open a socket connection—a bi-directional socket connection—with a given instance, for up to one hour. This is pretty unique for something that is as fully managed as Cloud Run.Corey: Right. As we’re recording this, we are just coming off of Google I/O, and there were a number of announcements around Cloud Run that were touching it because of, you know, strange marketing issues. I only found out that Google I/O was a thing and featured cloud stuff via Twitter at the time it was happening. What did you folks release around Cloud Run?Steren: Good question, actually. Part of the Google I/O Developer keynote, I pitched a story around how Cloud Run helps developers, and the I/O team liked the story, so we decided to include that story as part of the live developer keynote. So, on stage, we announced Cloud Run jobs. So now, I talked to you about Cloud Run services, which can be used to expose an API, but also to do, like, private microservice-to-microservice communication—because cloud services don’t have to be public—and in that case, we support GRPC and, you know, a very strong security mechanism where only Service A can invoke Service B, for example, but Cloud Run jobs are about non-request-driven containers. So, today—I mean, before Google I/O a few days ago, the only requirement that we imposed on your container image was that it started to listen for requests, or events, or GRPC—Corey: Web requests—Steren: Exactly—Corey: It speaks [unintelligible 00:24:35] you want as long as it’s HTTP. Yes.Steren: That was the only requirement we asked you to have on your container image. And now we’ve changed that. Now, if you have a container that basically starts and executes to completion, you can deploy it on a Cloud Run job. So, you will use Cloud Run jobs for, like, daily batch jobs. And you have the same infrastructure, so on-demand, you can go from zero to, I think for now, the maximum is a hundred tasks in parallel, for—of course, you can run many tasks in sequence, but in parallel, you can go from zero to a hundred, right away to run your daily batch job, daily admin job, data processing.But this is more in the batch mode than in streaming mode. If you would like to use a more, like, streaming data processing, than a Cloud Run service would still be the best fit because you can literally push events to it, and it will auto-scale to handle any number of events that it receives.Corey: Do you find that the majority of customers are using Cloud Run for one-off jobs that barely will get more than a single container, like my thing, or do you find that they’re doing massively parallel jobs? Where’s the lion’s share of developer and customer interest?Steren: It’s both actually. We have both individual developers, small startups—which really value the scale to zero and pay per use model of Cloud Run. Your URL redirect service probably is staying below the free tier, and there are many, many, many users in your case. But at the same time, we have big, big, big customers who value the on-demand scalability of Cloud Run. And for these customers, of course, they will probably very likely not scale to zero, but they value the fact that—you know, we have a media company who uses Cloud Run for TV streaming, and when there is a soccer game somewhere in the world, they have a big spike of usage of requests coming in to their Cloud Run service, and here they can trust the rapid scaling of Cloud Run so they don’t have to pre-provision things in advance to be able to serve that sudden traffic spike.But for those customers, Cloud Run is priced in a way so that if you know that you’re going to consume a lot of Cloud Run CPU and memory, you can purchase Committed Use Discounts, which will lower your bill overall because you know you are going to spend one dollar per hour on Cloud Run, well purchase a Committed Use Discount because you will only spend 83 cents instead of one dollar. And also, Cloud Run and comes with two pricing model, one which is the default, which is the request-based pricing model, which is basically you only have CPU allocated to your container instances if you are processing at least one request. But as a consequence of that, you are not paying outside of the processing of those requests. Those containers might stay up for you, one, ready to receive new requests, but you’re not paying for them. And so, that is—you know, your URL redirect service is probably in that mode where yes when you haven’t used it for a while, it will scale down to zero, but if you send one request to it, it will serve that request and then it will stay up for a while until it decides to scale down. But you the user only pays when you are processing these specific requests, a little bit like a Function as a Service product.Corey: Scales to zero is one of the fundamental tenets of serverless that I think that companies calling something serverless, but it always charges you per hour anyway. Yeah, that doesn’t work. Storage, let’s be clear, is a separate matter entirely. I’m talking about compute. Even if your workflow doesn’t scale down to zero ever as a workload, that’s fine, but if the workload does, you don’t get to keep charging me for it.Steren: Exactly. And so, in that other mode where you decide to always have CPU allocated to your Cloud Run container instances, then you pay for the entire lifecycle of this container instances. You still benefit from the auto-scaling of Cloud Run, but you will pay for the lifecycle and in that case, the price points are lower because you pay for a longer period of time. But that’s more the price model that those bigger customers will take because at their scale, they basically always receive requests, so they already to pay always, basically.Corey: I really want to thank you for taking the time to chat with me. Before you go, one last question that we’ll be using as a teaser for the next episode that we record together. It seems like this is a full-time job being the product manager on Cloud Run, but no Google, contrary to popular opinion, does in fact, still support 20% projects. What’s yours?Steren: So, I’ve been looking to work on Cloud Run since it was a prototype, and you know, for a long time, we’ve been iterating privately on Cloud Run, launching it, seeing it grow, seeing it adopted, it’s great. It’s my full-time job. But on Fridays, I still find the time to have a 20% project, which also had quite a bit of impact. And I work on some sustainability efforts for Google Cloud. And notably, we’ve released two things last year.The first one is that we are sharing some carbon characteristics of Google Cloud regions. So, if you have seen those small leaves in the Cloud Console next to the regions that are emitting the less carbon, that’s something that I helped bring to life. And the second one, which is something quite big, is we are helping customers report and reduce their gross carbon emissions of their Google Cloud usage by providing an out of the box reporting tool called Google Cloud Carbon Footprint. So, that’s something that I was able to bootstrap with a team a little bit on the side of my Cloud Run project, but I was very glad to see it launched by our CEO at the last Cloud Next Conference. And now it is a fully-funded project, so we are very glad that we are able to help our customers better meet their sustainability goals themselves.Corey: And we will be talking about it significantly on the next episode. We’re giving a teaser, not telling the whole story.Steren: [laugh].Corey: I really want to thank you for being as generous with your time as you are. If people want to learn more, where can they find you?Steren: Well, if they want to learn more about Cloud Run, we talked about how simple was that name. It was obviously not simple to find this simple name, but the domain is https://cloud.run.Corey: We will also accept snark.cloud/run, I will take credit for that service, too.Steren: [laugh]. Exactly.Corey: There we are.Steren: And then, people can find me on Twitter at @steren, S-T-E-R-E-N. I’ll be happy—I’m always happy to help developers get started or answer questions about Cloud Run. And, yeah, thank you for having me. As I said, you successfully deployed something in just a few minutes to Cloud Run. I would encourage the audience to—Corey: In spite of myself. I know, I’m as surprised as anyone.Steren: [laugh].Corey: The only snag I really hit was the fact that I was riding shotgun when we picked up my daughter from school and went through a dead zone. It’s like, why is this thing not loading in the Google Cloud Console? Yeah, fix the cell network in my area, please.Steren: I’m impressed that you did all of that from an iPad. But yeah, to the audience give Cloud Run the try. You can really get started connecting your GitHub repository or deploy your favorite container image. And we’ve worked very hard to ensure that usability was here, and we know we have pretty strong usability scores. Because that was a lot of work to simplicity, and product excellence and developer experience is a lot of work to get right, and we are very proud of what we’ve achieved with Cloud Run and proud to see that the developer community has been very supportive and likes this product.Corey: I’m a big fan of what you’ve built. And well, of course, it links to all of that in the show notes. I just want to thank you again for being so generous with your time. And thanks again for building something that I think in many ways showcases the best of what Google Cloud has to offer.Steren: Thanks for the invite.Corey: We’ll talk again soon. Steren Giannini is a senior product manager at Google Cloud, on Cloud Run. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice. If it’s on YouTube, put the thumbs up and the subscribe buttons as well, but in the event that you hated it also include an angry comment explaining why your 20% project is being a shithead on the internet.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
Full Description / Show Notes Gafnit explains how she found a vulnerability in RDS, an Amazon database service (1:40) Gafnit and Corey discuss the concept of not being able to win in cloud security (7:20) Gafnit talks about transparency around security breaches (11:02) Corey and Gafnit discuss effectively communicating with customers about security (13:00) Gafnit answers the question “Did you come at the RDS vulnerability exploration from a perspective of being deeper on the Postgres side or deeper on the AWS side? (18:10) Corey and Gafnit talk about the risk of taking a pre-existing open source solution and offering it as a managed service (19:07) Security measures in cloud-native approaches versus cloud-hosted (22:41) Gafnit and Corey discuss the security community (25:04) About GafnitGafnit Amiga is the Director of Security Research at Lightspin. Gafnit has 7 years of experience in Application Security and Cloud Security Research. Gafnit leads the Security Research Group at Lightspin, focused on developing new methods to conduct research for new cloud native services and Kubernetes. Previously, Gafnit was a lead product security engineer at Salesforce focused on their core platform and a security researcher at GE Digital. Gafnit holds a Bs.c in Computer Science from IDC Herzliya and a student for Ms.c in Data Science.Links Referenced: Lightspin: https://www.lightspin.io/ Twitter: https://twitter.com/gafnitav LinkedIn: https://www.linkedin.com/in/gafnit-amiga-b1357b125/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. We’ve taken a bit of a security bent to the conversations that we’ve been having on this show and over the past year or so and, well, today’s episode is no different. In fact, we’re going a little bit deeper than we normally tend to. My guest today is Gafnit Amiga, who’s the Director of Security Research at Lightspin. Gafnit, thank you for joining me.Gafnit: Hey, Corey. Thank you for inviting me to the show.Corey: You sort of burst onto the scene—and by ‘scene,’ I of course mean the cloud space, at least to the level of community awareness—back, I want to say in April of 2022 when you posted a very in-depth blog post about exploiting RDS and some misconfigurations on AWS’s side to effectively display internal service credentials for the RDS service itself. Now, that sounds like it’s one of those incredibly deep, incredibly murky things because it is, let’s be clear. At a high level, can you explain to me exactly what it is that you found and how you did it? Gafnit: Yes, so, RDS is database service of Amazon. It’s a managed service where you can choose the engine that you prefer. One of them is Postgres. There, I found the vulnerability. The vulnerability was in the extension in the log_fdw—so it’s for—like, stands for Foreign Data Wrapper—where this extension is, therefore reading the logs directly of the engine, and then you can query it using SQL queries, which should be simpler and easy to use.And this extension enables you to provide a path. And there was a path traversal, but the traversal happened only when you dropped a validation of the wrapper. And this is how I managed to read local files from the database EC2 machine, which shouldn’t happen because this is a managed service and you shouldn’t have any access to the underlying host.Corey: It’s always odd when the abstraction starts leaking, from an AWS perspective. I know that a friend of mine was on Aurora during the beta and was doing some high-performance work and suddenly started seeing SQL errors about /var/temp filling up, which is, for those who are not well versed in SQL, and even for those who are, that’s not the sort of thing you tend to expect to show up on there. It feels like the underlying system tends to leak in—particularly in RDS sense—into what is otherwise at least imagined to be a fully-managed service.Gafnit: Yes because sometimes they want to give you an informative error so you will be able to realize what happened and what caused to the error, and sometimes they prefer not to give you too many information because they don’t want you to get to the underlying machine. This is why, for example, you don’t get a regular superuser; you have an RDS superuser in the database.Corey: It seems to me that this is sort of a problem of layering different security models on top of each other. If you take a cloud-native database that they designed, start to finish, themselves, like DynamoDB, the entire security model for Dynamo, as best I can determine, is wrapped up within IAM. So, if you know IAM—spoiler, nobody knows IAM completely, it seems—but if you have that on lock you’ve got it; there’s nothing else you need to think about. Whereas with RDS, you have to layer on IAM to get access to the database and what you’re allowed to do with it.But then there’s an entirely separate user management system, in many respects, of local users for other Postgres or MySQL or any other systems that were using, to a point where even when they started supporting IRM for authentication to RDS at the database user level. It was flagged in the documentation with a bunch of warnings of, “Don’t do this for high-volume stuff; only do this in development style environments.” So, it’s clear that it has been a difficult marriage, for lack of a better term. And then you have to layer on all the other stuff that if God forbid, you’re in a multi-cloud style environment or working with Kubernetes on top of all of this, and it seems like you’re having to pick and choose between four or five different levels of security modeling, as well as understand how all of those things interplay together. How come we don’t see things like this happening four times a day as a result?Gafnit: Well, I guess that there are more issues being found, but not always published but I think that this is what makes it more complex for both sides. Creating managed services with resources and third parties that everybody knows. To make it easy for them to use requires a deep understanding of the existing permission models of the service where you want to integrate it with your permission model and how the combination works. So, you actually need to understand how every change is going to affect the restrictions that you want to have. So, for example, if you don’t want the database users to be able to read-write or do a network activity, so you really need to understand the permission model of the Postgres itself. So, it makes it more complicated for development, but it’s also good for researchers because they already know Postgres and they have a good starting point.Corey: My philosophy has always been when you’re trying to secure something, you need to have at least a topical level of understanding of the entire system, start to finish. One of the problems I’ve had with the idea of microservices as is frequently envisioned is that there’s separation, but not real separation, so you have to hand-wave over a whole bunch of the security model. If you don’t understand something, I believe it’s very difficult to secure it. And let’s be honest, even if you do understand [laugh] something, it can be very difficult to secure it. And the cloud vendors with IAM and similar systems don’t seem to be doing themselves any favors, given the sheer complexity and the capabilities that they’re demanding of themselves, even for having one AWS service talk to another one, but in the right way.And it’s finicky, and it’s nuanced, and debugging it becomes a colossal pain. And finally, at least those of us who are bad at these things, finally say, “The hell with it,” and they just grant full access from Service A to Service B—in the confines of a test environment. I’m not quite that nuts myself, most days. And then it’s the biggest lie we always tell ourselves is once we have something overscoped like that, usually for CI/CD, it’s, “Oh, todo: I’ll go back and fix that later.” Yeah, I’m looking back five years ago and that’s still on my todo list.For some reason, it’s never been the number one priority. And in all likelihood, it won’t be until right after it really should have been my number one priority. It feels like in cloud security particularly, you can’t win, you can only not lose. I always found that to be something of a depressing perspective and I didn’t accept it for the longest time. But increasingly, these days, it started to feel like that is the state of the world. Am I wrong on that? Am I just being too dour?Gafnit: What do you mean by you cannot lose?Corey: There’s no winning in security from my perspective because no one is going to say, “All right. We won the security. Problem solved. The end.” Companies don’t view security as a value-add. It is only about a downside risk mitigation play.It’s, “Yay, another day of not getting breached.” And the failure mode from there is, “Okay, well, we got breached, but we found out about it ourselves immediately internally, rather than reading about it in The New York Times in two weeks.” The winning is just the steady-state, the status quo. It’s just all different flavors of losing beyond that.Gafnit: So, I don’t think it’s quite the case because I can tell that they do do always an active work on securing the services and their structure because I went over other extensions before reaching to the log foreign data wrapper, and they actually excluded high-risk functionalities that could help me to achieve privileged access to the underlying host. And they do it with other services as well because they do always do the security review before having it integrated externally. But you know, it’s an endless zone. You can always have something. Security vulnerabilities are always [arrays 00:09:06]. So everyone, whenever they can help and to search and to give their value, it’s appreciated.Corey: I feel like I need to clarify a bit of nuance. When your blog post first came out talking about this, I was, well let’s say a little irritated toward AWS on Twitter and other places. And Twitter is not a place for nuance, it is easy to look at that and think, “Oh, I was upset at AWS for having a vulnerability.” I am not, I want to be very clear on that. Now, it’s certainly not good, but these are computers; that is the nature of how they work.If you want to completely secure computer, cut the power to it, sink it in concrete and then drop it in the ocean. And even then, there are exceptions to all of that. So, it’s always a question of not blocking all risk; it’s about trade-offs and what risk is acceptable. And to AWS is credit, they do say that they practice defense-in-depth. Being able to access the credentials for the running RDS service on top of the instance that it was running on, while that’s certainly not good, isn’t as if you’d suddenly had keys to everything inside of AWS and all their security model crumbles away before you.They do the right thing and the people working on these things are incredibly good. And they work very hard at these things. My concern and my complaint is, as much as I enjoy the work that you do and reading these blog posts talking about how you did it, it bothers me that I have to learn about a vulnerability in a service for which I pay not small amounts of money—RDS is the number one largest charge in my AWS bill every month—and I have to hear about it from a third-party rather than the vendor themselves. In this case, it was a full day later, where after your blog post went up, and they finally had a small security disclosure on AWS’s site talking about it. And that pattern feels to me like it leads nowhere good.Gafnit: So, transparency is a key word here. And when I wrote the post, I asked if they want to add anything from their side, and they told that they already reached out to the vulnerable customers and they helped them to migrate to their fixed version. So, from their side, it didn’t felt it’s necessary to add it over there. But I did mention the fact that I did the investigation and no customer data was hurt. Yeah, but I think that if there will be maybe a more organized process for any submission of any vulnerability that where all the steps are aligned, it will help everyone and anyone can be informed with everything that happens.Corey: I have always been extraordinarily impressed by people who work at AWS and handle a lot of the triaging of vulnerability reports. Zack Glick, before he left, was doing an awful lot of that Dan [Erson 00:12:05] continues to be a one of the bright lights of AWS, from my perspective, just as far as customer communication and understanding exactly what the customer perspective is. And as individuals, I see nothing but stars over at AWS. To be clear, ‘Nothing but Stars’ is also the name of most of my IAM policies, but that’s neither here nor there.It seems like, on some level, there’s a communications and policy misalignment, on some level, because I look at this and every conversation I ever have with AWS’s security folks, they are eminently reasonable, they’re incredibly intelligent, and they care. There’s no mistaking that they legitimately care. But somewhere at the scale of company they’re at, incentives get crossed, and everyone has a different position they’re looking at these things from, and it feels like that disjointedness leads to almost a misalignment as far as how to effectively communicate things like this to customers.Gafnit: Yes, it looks like this is the case, but if more things will be discovered and published, I think that they will have eventually an organized process for that. Because I guess the researchers do find things over there, but they’re not always being published for several reasons. But yes, they should work on that. [laugh].Corey: And that is part of the challenge as well, where AWS does not have a public vulnerability disclosure program. [unintelligible 00:13:30] hacker one, they don’t have a public bug bounty program. They have a vulnerability disclosure email address, and the people working behind that are some of the hardest working folks in tech, but there is no unified way of building a community of researchers around the idea of exploring this. And that is a challenge because you have reported vulnerabilities, I have reported significantly fewer vulnerabilities, but it always feels like it’s a hurry up and wait scenario where the communication is not always immediate and clear. And at best, it feels like we often get a begrudging, “Thank you.”Versus all right, if we just throw ethics completely out the window and decide instead that now we’re going to wind up focusing on just effectively selling it to the highest bidder, the value of, for example, a hypervisor escape on EC2 for example, is incalculable. There is no amount of money that a bug bounty program could offer for something like that compared to what it is worth to the right bad actor at the right time. So, the vulnerabilities that we hear about are already we’re starting from a basis of people who have a functioning sense of ethics, people who are not deeply compromised trying to do something truly nefarious. What worries me is the story of—what are the stories that we aren’t seeing? What are the things that are being found where instead of fighting against the bureaucracy around disclosure and the rest, people just use them for their own ends? And I’m gratified by the level of response I see from AWS on the things that they do find out about, but I always have to wonder, what aren’t we seeing?Gafnit: That’s a good question. And it really depends on their side if they choose to expose it or not.Corey: Part of the challenge too, is the messaging and the communication around it and who gets credit and the rest. And it’s weird, whenever they release some additional feature to one of their big headline services, there are blog posts, there are keynote speeches, there are customer references, they go on speaking tours, and the emails, oh, God, they never stopped the emails talking about how amazing all of these things are. But whenever there’s a security vulnerability or a disclosure like this—and to be fair, AWS’s response to this speaks very well of them—it’s like you have to go sneak down into the dark sub-basement, with the filing cabinet behind the leopard sign and the rest, to even find out that these things exist. And I feel like they’re not doing themselves any favors by developing that reputation for lack of transparency around these things. “Well, while there was no customer impact, so why would we talk about it?”Because otherwise, you’re setting up a myth that there never is a vulnerability on the side of—what is it that you’re building as a cloud provider. And when there is a problem down the road—because there always is going to be; nothing is perfect—people are going to say, “Hey, wait a minute. You didn’t talk about this. What else haven’t you talked about?”And it rebounds on them with sometimes really unfortunate side effects. With Azure as a counterexample here, we see a number of Azure exploits where, “Yeah, turned out that we had access to other customers’ data and Azure had no idea until we told them.” And Azure does it statements about, “Oh, we have no evidence of any of this stuff being used improperly.” Okay, that can mean that you’ve either check your logs and things are great or you don’t have logging. I don’t know that necessarily is something I trust.Conversely, AWS has said in the past, “We have looked at the audit logs for this service dating back to its launch years ago, and have validated that none of that has never been used like this.” One of those responses breeds an awful lot of customer trust. The other one doesn’t. And I just wish AWS knew a little bit more how good crisis communication around vulnerabilities can improve customer trust rather than erode it.Gafnit: Yes, and I think that, as you said, there will always be vulnerabilities. And I think that we are expecting to find more, so being able to communicate as clearly as you can and to expose things about maybe the fakes and how the investigation is being done, even in a high level, for all the vulnerabilities can gain more trust from the customer side.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: You have experience in your background specifically around application security and cloud security research. You’ve been doing this for seven years at this point. When you started looking into this, did you come at the RDS vulnerability exploration from a perspective of being deeper on the Postgres side or deeper on the AWS side of things?Gafnit: So, it was both. I actually came to the RDS lead from another service where there was something [about 00:18:21] in the application level. But then I reached to an RDS and thought, well, it will be really nice to find thing over here and to reach the underlying machine. And when I entered to the RDS zone, I started to look at it from the application security eyes, but you have to know the cloud as well because there are integrations with S3, you need to understand the IAM model. So, you need a mix of both to exploit specifically this kind of issue. But you can also be database experts because the payload is a pure SQL.Corey: It always seems to me that this is an inherent risk in trying to take something that is pre-existing is an open-source solution—Postgres is one example but there are many more—and offer it as a managed service. Because I think one of the big misunderstandings is that when—well, AWS is just going to take something like Redis and offer that as a managed service, it’s okay, I accept that they will offer a thing that respects the endpoints and then acts as if it were Redis, but under the hood, there is so much in all of these open-source projects that is built for optionality of wherever you want to run this thing, it will run there; whatever type of workload you want to throw at it, it can work. Whereas when you have a cloud provider converting these things into a managed service, they are going to strip out an awful lot of those things. An easy example might be okay, there’s this thing that winds up having to calculate for the way the hard drives on a computer work and from a storage perspective.Well, all the big cloud providers already have interesting ways that they have solved storage. Every team does not reimplement that particular wheel; they use in-house services. Chubby’s file locking, for example, over on Google side is a classic example of this that they’ve talked about an awful lot so every team building something doesn’t have to rediscover all of that. So, the idea that, oh, we’re just going to take up this open-source thing, clone it off a GitHub, fork it, and then just throw it into production as a managed service seems more than a little naive. What’s your experience around seeing, as you get more [laugh] into the weeds of these things than most customers are allowed to get, what’s your take on this?Do you find that this looks an awful lot like the open-source version that we all use? Or is it something that looks like it has been heavily customized to take advantage of what AWS is offering internally as underlying bedrock services?Gafnit: So, from what I saw until now, they do want to save the functionality so you will have the same experience as you’re working with the same service that not on AWS because you’re you are used to that. So, they are not doing dramatic changes, but they do want to reduce the risk in the security space. So, there will be some functionalities that they will not let you to do. And this is because of the managed party in areas where the full workload is deployed in your account and you can access it anyway, so they will not have the same security restrictions because you can access the workload anyway. But when it’s managed, they need to prevent you from accessing the underlying host, for example. And they do the changes, but they’re really picked to the specific actions that can lead you to that.Corey: It also feels like RDS is something of a, I don’t want to call it a legacy service because it is clearly still very much actively developed, but it’s what we’ll call it a ‘classic service.’ When I look at a new AWS launch, I tend to mentally bucket them into two things. There’s the cloud-native approach, and we’ve already talked about DynamoDB. That would be one example of this. And there’s the cloud-hosted model where you have to worry about things like instances and security groups and the networking stuff, and so on and so forth, where it’s basically feels like they’re running their thing on top of a pile of EC2 instances, and that abstraction starts leaking.Part of me wonders if looking at some of these older services like RDS, they made decisions in the design and build out of these things that they might not if they were to go ahead and build it out today. I mean, Aurora is an example of what that might look like. Have you found as you start looking around the various security foibles of different cloud services, that the security posture of some of the more cloud-native approaches is better or worse or the same as the cloud-hosted world?Gafnit: Well, so for example, in the several issues that were found, and also here in the RDS where you can see credentials in a file, this is not a best practice in security space. And so, definitely there are things to improve, even if it’s developed on the provider side. But it’s really hard to answer this question because in a managed area where you don’t have any access, it’s hard to tell how it’s configured and if it’s configured properly. So, you need to have some certification from their side.Corey: This is, on some level, part of the great security challenge, especially for something that is not itself open-source, where they obviously have terrific security teams, don’t get me wrong. At no point do I want to ever come across a saying, “Oh, those AWS people don’t know how security works.” That is provably untrue. But there is something to be said for the value of having a strong community in the security space focusing on this from the outside of looking at these things, of even helping other people contextualize these things. And I’m a little disheartened that none of the major cloud providers seem to have really embraced the idea of a cloud security community, to the point where the one that I’m most familiar with, the cloud security forum Slack team seems to be my default place where I go for context on things.Because I dabble. I keep my hand in when it comes to security, but I’m certainly no expert. That’s what people like you are for. I make fun of clouds and I work on the billing parts of it and that’s about as far as it goes for me. But being able to get context around is this a big deal? Is this description that a company is giving, is it accurate?For example, when your post came out, I had not heard of Lightspin in this context. So, reaching out to a few people I trusted, is this legitimate? The answer was, “Yes. It’s legitimate and it’s brilliant. That’s a company that keep your eye on.” Great. That’s useful context and there’s no way to buy that. It has to come from having those conversations with people in the [broader 00:24:57] sense of the community. What’s your experience been looking at the community side of the world of security?Gafnit: Well, so I think that the cloud security has a great community, and this is one of the things that we at Lightspin really want to increase and push forward. And we see ourselves as a security-driven company. We always do the best to publish a post, even detailed posts, not about vulnerabilities, about how things works in the cloud and how things are being evaluated, to release open-source tools where you can use them to check your environment even if you’re not a customer. And I think that the community is always willing to explain and to investigate together. And it’s a welcome effort, but I think that the messaging should be also for all layers, you know, also for the DevOps and the developers because it can really help if it will start from this point from their side, as well.Corey: It needs to be baked in, from start to finish.Gafnit: Yeah, exactly.Corey: I really want to thank you for taking the time out of your day to speak with me today. If people want to learn more about what you’re up to, where’s the best place for them to find you?Gafnit: So, you can find me on Twitter and on LinkedIn, and feel free to reach out.Corey: We will, of course, put links to that in the [show notes 00:26:25]. Thank you so much for being so generous with your time today. I appreciate it.Gafnit: Thank you, Corey.Corey: Gafnit Amiga, Director of Security Research at Lightspin. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, and if it’s on the YouTubes, smash the like and subscribe buttons, which I’m told are there. Whereas if you’ve hated this podcast, same story, like and subscribe and the buttons, leave a five-star review on a various platform, but also leave an insulting, angry comment about how my observation that our IAM policies are all full of stars is inaccurate. And then I will go ahead and delete that comment later because you didn’t set a strong password.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
Full Description / Show Notes Marie talks about Oki Doki’s primary product, Notion Mastery (2:38) Corey and Marie talk ADHD diagnosis and how it has impacted their lives and work (4:26) Marie and Corey discuss techniques they’ve developed for coping with ADHD (11:22) Corey and Marie talk about workarounds for people with ADHD who want to adopt something like Notion (16:13)  Marie discusses the importance of being excited about the tools you’re employing (18:54) Corey and Marie talk about finding tools that work for you (26:43) Marie and Corey discuss the unique challenge of teaching skills versus dumping knowledge (30:35) About Marie PoulinMarie teaches business owners to level up their digital systems, workflow, and knowledge management processes using Notion.She’s the co-founder of Oki Doki and creator of Notion Mastery, an online program and community that helps creators, entrepreneurs and small teams tame their work + life chaos by building life and business management systems with Notion.Diagnosed with ADHD at age 37, Marie is especially passionate about helping folks customize their workflows and workspaces to meet their unique needs and preferences.She believes that Notion is especially powerful for neurodivergent folks who have long struggled to adhere to traditional or rigid project management processes, and may need a little extra customization and flexibility.When she's not tinkering in Notion or doing live trainings, you can find her in the garden, playing video games, or cooking up some delicious vegetarian tacos.Links Referenced: Oki Doki: https://weareokidoki.com/ Personal website: https://mariepoulin.com Notion Mastery: https://notionmastery.com Twitter: https://twitter.com/mariepoulin TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it’s hard to know where problems originate. Is it your application code, users, or the underlying systems? I’ve got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it’s more than just hipster monitoring.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. that’s snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. Today I’m joined by Marie Poulin, the CEO of Oki Doki. Marie, thank you for joining me.Marie: Thank you for having me. I’m excited.Corey: So, let’s start at the very beginning. What does Oki Doki do? And for folks listening that is O-K-I D-O-K-I, so you might want to have to think about that if you’re doing the Google approach of, “What is this thing?”Marie: Well, at the moment, the majority of our products and services are surrounded by helping people learn how to use Notion to manage their life and business. So, it’s only a pivot that we took in the last couple of years, and so our signature program is a course called Notion Mastery. So, there’s four full-time employees now and that’s what we do. We design live trainings, we have a forum, we have a curriculum. It’s all products and services related to Notion.Corey: That is an interesting pivot that you can wind up going through. Please tell me I’m not the first person to make the observation that you called it Oki Doki and you’ve turned yourself around.Marie: [laugh]. You are the first, Corey? [laugh].Corey: Oh, good. I am broken like that, so that’s kind of awesome. So, you’ve been more or less doing—I don’t know the best way to frame this, so my apologies if I’m getting it wrong—but the idea of well, what are you selling? Knowledge. You’re selling an understanding of how to improve things, you’re selling a better outcome.And it’s easy to look at that and say, “Oh, you’re selling education.” No, you’re selling understanding. Education is the way that you get there because at least at the moment, you can’t just jack gigabytes of data directly into people’s head without going to prison for it. Or raising a whole boatload of VC money.Marie: [laugh]. I mean, you can also say you’re kind of selling an outcome, right? You’re selling this future version of who someone wants to be. And so, we talk a lot about—you know, on our sales page, we get a lot of compliments on our sales page, but just speaking to the scattered mind, you know, feeling like a shitshow, feeling like you don’t really have all your data in one place. You know, it’s learning how to improve your workflow at work but also in life as well.And so, a lot of our language speaks to the sort of future version of yourself. Like, stop feeling scattered, stop feeling stretched thin. Let’s actually get it so that you turn things into a well-oiled machine. So, you could say we’re selling a dream. [laugh].Corey: This is an interesting direction to take this conversation in because I don’t normally talk about this. But why not; we’ll give it a shot. It’s been sufficiently long since the last time. Last year—you’ve been very public about this—you were diagnosed with ADHD. I periodically talk about the fact that I was diagnosed with it myself—back when it was called ADD—when I was five years old.So, growing up I always knew that there was something neurodivergent about me. And the lesson I took away from this, as someone growing up with a lot of the limitations—yes, there are advantages but at the time, all I saw were limitations—about, “Well, what is ADHD?” It’s like, oh, okay. They sat down and explained it to me. And it’s not what they said, but it was, “See, this is the medical reason why you suck.”And that was not the most constructive way of framing it. In adulthood, talking to other people who have been diagnosed with this, especially later in life. There’s a—it’s a spectrum disorder. It winds up impacting an awful lot of people differently, but the universal experience that I hear is, wait, you mean there’s a reason that I am the way that I am? It’s not that I’m lazy. It’s not that I’m shitty at things. It’s not that I’m—Marie: Yeah.Corey: —careless. And that is one of those things that just is transformative. I didn’t realize at the time how fortunate I was to be diagnosed that early on because trying to try to figure out why am I getting fired all the time? Why do I get bored doing the same thing too many days in a row, so I start causing problems for other people? What is going on with this? Why do I have this incredible opposition to anything that remotely resembles authority, et cetera, et cetera?Not all of this might be ADHD traits, but here I am. And my only solution after, you know, deciding that I didn’t really want to set a world record for number of times getting fired was, well, I guess I’ll start my own company because that at least to get fired, it’s going to take some work. You figured this out while you were already self-employed.Marie: Yes.Corey: What was that like?Marie: What was it like to find out that I finally had an answer or reason for, maybe, past behaviors? [laugh].Corey: Right. Because it’s the simultaneous, “Oh, my God, there’s a reason that I am like I am,” and then followed immediately by, “I still am the way that I am. Huh. Okay.” It feels like it helps things, but it also doesn’t help things. But it does, and it comes back around. What was your experience with it?Marie: Yeah, it started because I was doing research to understand my sister better because she had been diagnosed with ADHD for a couple years. It made so much sense once I kind of understood and started researching a little bit more about it. And then, of course, doing my deep-dive research. I’m hearing all these traits that I’m like, “Oh. Wait, that does really sound like me.” The not being able to wake—Corey: What do you [mean 00:07:01]—Marie: —up in the morning—Corey: ADHD trait? Everyone does that. Wait.Marie: [laugh]. Yeah. When you said that enough times, you’re like, “Oh, wait. Maybe this is not normal.” Or you don’t really know what is—what is normal anyway, right? So, in doing that research, trying to connect with her, trying to understand her experience better, I just started learning about more and more of these traits.I also knew a shit ton of people in our course, had mentioned that they had ADHD in their intake form, and I was like, what is it about people that ADHD that are actually drawn to my YouTube videos or my way of explaining things? And I started to learn a little bit more; it’s quite common for folks with ADHD to be drawn to one another, probably because of our communication styles, even the sort of mild interrupting, or kind of the way we banter together. There’s different styles of communicating that I think often folks with ADHD are maybe drawn to one another or have an easier time understanding one another. So, listening to some of these symptoms, I was like, “Wait a second.” Because my sister and I are so different in the way our symptoms present.I thought, “Well, that’s what ADHD looks like.” It’s pure unbridled chaos and unfiltered. And I just had this idea of what it looked like because she was one of the few examples that I had. Meanwhile, I’m skipping grades, I’m in the gifted program, I’m off, you know, doing my own thing. It looked very different.I thought, “Oh, people with ADHD don’t thrive in university,” or whatnot. So, I had a lot of assumptions that I had to unpack. And I think the one, sort of, I don’t know, symptom that kind of twinged something in my brain was extreme difficulty getting up in the morning and even sort of waking up your brain in the morning. This has been a problem with jobs, it’s been a problem was school, getting to school on time, getting to work on time. Similar to you, it has caused job loss, it has caused tension with partners. They don’t understand, like, why can’t you get out of bed and seize the day?And I just thought, “There’s something weird going on there with my body.” But I can be, you know, wide awake at 7 p.m. and I’m, like, ready to go. And I can hyperfocus for days on end. So, just noticing some of these symptoms and kind of unpacking it a bit, I thought, “Okay, there’s something to go a little deeper in here.”Corey: I have trouble getting up, but I’m almost never late. That one does not hit me in quite the same way. In fact—Marie: Well—Corey: —my first consulting clients, and I’d been building—I was independent for two weeks at that point, and I was in an in-person meeting in San Francisco and one day, I showed up 20 minutes late, and he just stared at me. “You’re never late. What’s the deal here?” And it’s like, “Yeah, I had trouble getting up this morning.” That was a lie.I was able to tell him about three or four months later, that morning, I found out I was going to be a father. And that was an—you know, it turns out that I was going to be okay being late, but it was so early, you didn’t want to tell anyone, yet. But it was—yeah, it’s one of those things where that was more important than—Marie: Absolutely.Corey: —doing the work thing. But I still remember, yeah, I feel like I’m always about to be late but apparently my reputation is, I never am, so okay. I’ll take it. That is a—again, it is a spectrum disorder. I also—Marie: Absolutely.Corey: —further there want to call out for viewers, listeners, et cetera, a couple of things. One, this is not mental health advice. If any of the stories we’re telling resonate, talk to a qualified mental health professional. Secondly, I want to be clear as well here, Marie, that you and I both have significant advantages when it comes to dealing with these things. We both run our own companies, we can effectively restructure the way that we work in ways that are more accommodating for what we do.It turns out that in my employment days, that was never really a solution where, “Yeah, I decided I’m not going to wind up doing the on-call checklist every day. It doesn’t resonate with me.”Marie: “Just not feeling like it.”Corey: “It’s doing the same thing too many days in a row. And yeah, I’m not going to check the backups, either. What do you mean ‘I’m fired?’” yeah, it turns out, you’re not able to—you’re empowered to make those kinds of sweeping changes in the same way.Marie: Exactly.Corey: So, this is not advice for people. This is simply a pair of experience reports, the way I view it.Marie: Absolutely. I sort of feel like self-employment wasn’t necessarily a choice, in a way. It just felt like that’s the only way I'm going to be able to operate in this world. I need some more sense of control and say in how I structure my days, how I structure my work, being able to switch things up, being able to pivot quickly. I knew that I was going to need more control over that. So yeah, pretty unemployable over here. [laugh].Corey: So, once you wound up with the diagnosis, what happened next? What changes did you make that wound up resonating for you, things that were actionable? And, yeah, you’ve been very public about it as well. I want to highlight that. I’m not, for the most part.And part of that is because I internalized growing up that it was somehow a shameful thing that we don’t talk about. And the other part of it, too, on some level, was I didn’t want to turn it into a part of my brand identity, where, “Oh, yeah, Corey is very hard to describe.” So, people thrash around and look for labels to slap on me. ‘Shitposter’ seems to have stuck rather well. Because as soon as people feel that they have a label for something, it becomes easier to classify and then dismiss it.It’s aspects of my personality. It’s who I am. I don’t think of it as a disorder so much as it is part and parcel of who and what I am. And it turns out that being me is not—yet—a medically recognized diagnosis. So, I’m cautious to avoid the labeling aspect of it.But you have very publicly not, if not going for the label, you at least embraced it as an aspect of who you are, and you’ve been very vocal about your experiences and telling people how you have overcome aspects of this. It’s admirable. I wish I did more of it, honestly.Marie: I think it’s kind of essential, I think, in the nature of what we’re teaching. Like, when we’re teaching people to become more organized and we know that executive dysfunction is one of the signs or, you know, issues with ADHD, to me it sort of recontextualized why I became so freakin’ obsessed with systems and organization: because I never felt organized. I always felt the sense of what is the stuff come so easy to other people? Why is it taking me so much longer? Why am I spending nights, evenings, taking courses about systems like I’m trying to understand how to give my life structure?And so, in a way, the way I have become organized was trial by fire, just teaching myself, learning, you know, getting coaches. Like, I literally had a systems coach to teach myself how to get my business organized. So, I had kind of obsessed over it, like a hyperfocus. And so, realizing that other people are struggling with this and there’s a reason that people with ADHD are coming to the course seeking that sense of control. And so, learning that I had it, I was like, oh, this actually [laugh] does explain, in a way, my obsession with this or my curiosity about this, of, like, why does this come easy to some other people? Why do some people need to study this and learn this? Like, what is it about that?And so, I sort of felt like it would be doing a disservice if I didn’t kind of name it and talk about it and say, well, this actually colors a lot of my opinions. This actually influences the way I approach organization or even productivity, not from a timing perspective, but from an energy management perspective. I didn’t realize that was something that I’m doing. I’m not managing time, we’re managing Marie’s energy. And even my team is learning how to do that, too.So, I was like, “Oh, that actually makes a ton of sense.” And it also makes sense why some people won’t resonate with this energy management thing or might think I’m going way too far down a rabbit hole on something and they’re like, “Why can’t people just do what they say?” Like, you don’t understand, some of us need to trick ourselves into being productive. And this is how I’ve learned to do that. So, it was just kind of a funny recontextualizing or uncovering, oh, our brains operate very differently. And even within ADHD, people’s brains operate differently, so how do we get people moving toward progress, but knowing that we kind of need different ways of doing that. So, it’s just been kind of an interesting process.Corey: There’s a fairly common experience report from folks who have ADHD that when they’re kids, their memory is generally very good with a number of expressions of it, so we form our self-image in a lot of those times. And then for the rest of our lives, we tell ourselves the same lie, regardless of how many times it has proven to be a lie. And that lie is, “I don’t need to write this down. I’ll remember it.”Marie: Oh yes.Corey: “No, Corey, you will not remember it. You need to write it down. I promise.” And, for example, right now—I finally gave in and technology leapt ahead to the point where my entire life is run by Google Calendar—specifically three or four of them—that all route through Fantastical—which is the app I use—but it winds up grabbing my attention at the right time. It tells me what I need to do, when, and how, and it’s wonderful.Because if it’s not on my calendar, it does not happen.Marie: Yes.Corey: Like, I will forget our anniversary, my kids’ birthdays, to pick my children up from school. We are talking about, if it is not on my calendar, it does not happen. That is the one system that has been forced on me that worked. Then we—let’s talk about Notion for a minute because I looked at it briefly a few years ago, and it is one in the long, long, long list of tools or approaches or systems that I have played with and then discarded to act as basically an auxiliary brain pack. I used Evernote for a while and that sort of worked because I just would do different notes all the time and I’d wind up with 3000 of those things, and then the app gets bloaty and I move on to something else.For the last five years or so I’ve been using Drafts, a Mac slash iOS app, that only does text, which makes image management and attaching things kind of hard, but okay. And that’s great, and now I have 5000 of those in my [back 00:16:25] folder, not categorized or organized anyway, so I focus instead on well, search for terms and hope I use the term I thought I did at the time. And so, every time I’ve tried to use something like Notion, it’s yeah, this requires a way of thinking that I know I will get excited about if I look at it, and in a month, I’ll be right back to where I am now. So, there’s only so many times you go on the same ride before you know how it ends. How do y—like, that feels like a very common experience. How did you fix it?Marie: I think at the core though, you kind of have to be excited about the tool that you’re using. And so, I don’t think—Notion is not going to be an exciting fun tool for everyone. Some people are going to be like, “I don’t want to frickin’ build my productivity system. Are you kidding me? Like, just give me something that works out of the box.” Absolutely.But I think there’s something about the visual components of Notion. Like, I am a designer; I went to design school. I think I’m—it’s almost like something doesn’t click until I see it in the way that I need to see it. And that’s something I’ve learned about my brain is just, sometimes the same information can be presented to me, but if it’s not in a visual way, or whether it’s not spaced in the right way, my brain just kind of ignores it or it gets overwhelmed by it. And so, for me that visual aspect actually helps me learn.I’m priming my brain, I’m making my goals front and center. The fact that I can design it the way I need my brain to see it is part of its appeal to me. But I also recognize that’s not something everyone gets excited about. They’re not drawn to it. I’m all for using the tool that works the way that your brain is going to work.I get excited about making databases. I get excited about building glossaries of information to help me learn things. Like, for me, that’s part of my learning and part of my process and it’s just kind of what I’m used to, but I fully acknowledge, like, that stuff does not get everybody excited.[midroll 00:18:03]Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don’t leave managing your database to your cloud vendor because they’re too busy launching another half-dozen managed databases to focus on any one of them that they didn’t build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: There’s something very key you’re talking about here, which is the idea of having to be excited about what it is that you do. I look at the things that I do professionally, and if I didn’t deeply enjoy them, they would not get done, and I would have pivoted long ago to something else. People wonder why—Marie: Absolutely.Corey: —I make fun of so many things in the tech ecosystem. The honest answer is because if I just tell the dry, boring version of it, I will get bored because it’s a fairly boring field. Whereas instead, okay, someone releases a new thing. Great. How do I keep it interesting for me? How do I find a way to tell that story?How do I find a way to, in turn, build that into something that, in turn, I can start dragging in different directions and opening up to new ways of talking without going too far? It’s always a razor’s edge, it’s always a bit of a mind puzzle, and it’s always different. I love that. That’s why I do it. It’s not for the audience so much as it is for myself. Because if I’m not engaged, no one else is going to care what I have to say.Marie: Absolutely. And I think that’s a huge part of ADHD as well which is that interest-based nervous system, right? It’s like we have to [laugh] trick ourselves into finding the excitement in it or whatever that looks like for each of us. But just if I’m not motivated, if I’m not excited about it—writing email newsletters doesn’t get me excited; I’m like, “Okay, do I need to hire someone to do this?” Or how can I find a way to do it, whether it’s—if making a video is more fun or easy, great.How can I, you know, make content do double-duty in that way? So yeah, I’m always trying to find ways to incentivize myself to do the things that need to get done, even though they may not be the most exciting. But step one is actually run a business that is based on something that you love doing. Which not everyone, maybe, has the privilege to do, but I think everything about the way I’ve designed my business model and the services that we offer is, don’t offer services you don’t really want to offer. Don’t make products that you don’t want to maintain, you’re not excited about. So, it’s definitely a core part of kind of how we design our whole business model.Corey: For me, a big part of it has always been just trying to make sure that I’m doing the things that engage me. And this is where that whole idea of being in a very privileged position enters into it. Take this podcast slash video right now, as a terrific example. I’m having this conversation, I have an entire system when I wind up sending a link to someone, it fires off Calendly, that hides webhooks and gets a whole bunch of other things set up. I show up, we have a conversation before the show to figure out just this is the general ebb and flow of the show. Here’s the generalized topics we want to talk about. Let’s dive in.And we finish the recording session. Great, I wind up closing the window and that’s the last time I generally think about it. Because everything else has been automated. If anything other than me having this conversation with you does not need to be me, I there is no differentiated value in me being the person that does the audio engineering. It turns out, I can pay people who are world’s better than I am at that, who actually enjoy it as opposed to viewing it as unnecessary chore, and I can do things that I find more appealing, like shitposting about a $1.108 trillion—Marie: Exactly.Corey: Company. It comes down to find the thing, the differentiation point, and find ways to make sure you don’t have to do the other parts of it. But that is not a path that’s available to everyone in every context. And again, I’m talking about this in a professional sense. I still have to do a whole bunch of stuff as I go through the course of my life that is not differentiated, but I can’t very well hire someone to get me dressed in the morning. Well, I can but I feel like that becomes a little bit out of the scope of the lived human experience most of the [crosstalk 00:22:29].Marie: [laugh]. Absolutely. I feel like that’s one thing I sort of regret not doing earlier is hiring someone to work with. So, the very first hire that I made was my chief of operations, and oh my gosh, the things that she took on that I used to do that I’m like, how on earth did I do that before? Because now that you do that, and you do it way faster, I just got to wonder, like, how the heck did I ever convince myself to do those activities?I don’t want to do touch spreadsheets, I don’t want to [laugh] deal with that stuff. I don’t want to, you know, email reminders, or whatever it is. There’s so many activities that she handles that I just… I would be happy to never touch again. And so, I sort of wish I had explored that earlier, but I was in that lone wolf, like, I got this. I’m going to run my own business solo forever.And, you know, I just sort of thought it’s difficult to work with me or because of the way that I work, I don’t know how to delegate. Like, it’s all in your head. I just didn’t really know how to do that. So, that process, I think, takes a while. That first hire when you’re going from solo person to okay, now we’re two; how do we work together? Okay, who else can we hire? What other activities can I get other people to do? So, that’s been a process, for sure.Corey: Mike Julian, my business partner who you know, is a very process-driven person. He is very organized. His love language is Microsoft Excel, as I frequently tease him with. And one of the—not the only factor by a landslide, but one of the big early factors of what would—okay, I know what I’d do. What would Mike do here?Part of it is the never-ending litany of mail I get from the state around things like taxes, business registration, the rest. And normally my response when I get those, is I look at it, and it’s like, “Welp, I’m going to fucking prison. That’s the end of it. The end.” Because it’s not that I don’t have the money to pay my taxes, I assure you. What, I don’t have it—because I—financial planning is kind of part and parcel of how we think about cloud economics.But no, it’s the fact that I’m not going to sit there, fill out the form, put a stamp on it—or God forbid, fax it somewhere—and the rest. It’s not the paying of the taxes that bothers me it is the paperwork and the process and the heavy lift associated with getting the executive function necessary to do it. So, it never gets done and deadlines slide by. And Mike was good at that for a time, and then he took the more reasonable approach about this of, “Huh. Seems to me like a lot of this stuff is not differentiated value that I need to be doing either.”So, we have a CFO who handles a lot of that stuff now and other operational folks. And it turns out that yeah, wow, there’s a lot—I can—the quality of what I put out is a lot better because I get to focus on things instead of having to deal with the ebb and flow minutia of running payroll myself every week.Marie: Oh, yeah. All of that is very relatable. And this is why I can’t do paper in the office. I think this is why I just moved my entire brain online. It’s like if there’s paper, stamps, anything related to having to go [laugh] to a post office to mail something. I think I still have the stack of thank you cards from our wedding from, you know, five years ago. So, yeah. [laugh].Corey: That you haven’t sent out yet. Of course.Marie: Yes, exactly.Corey: Exact same—sorry, people 13—11 years ago, whenever it was.Marie: I’m so sorry.Corey: Yeah, one of these years. Yeah, and see, that’s exactly how I treat things like Drafts or Notion, if I were to use it, or something else is great, it’s still going to be the digital equivalent of a giant pile of paper. The thing is that computers can search through the contents of that paper a hell of a lot faster than I can, even with my own, at times, uncanny reading speed. There’s some value to that. So, understanding how the systems work and having them bend to accommodate you, rather than trying to fool yourself in half to work within the confines of an existing system, that seems to be the direction that you’re taking Notion in, specifically in the context of it is not prescriptive.And, on some level, that’s kind of the problem I have with it. Whenever I try the getting started for us, it’s, “Great, you can build your own system.” It’s like, “Isn’t that your job? What am I missing here?” Because the scariest thing I ever see when it’s time for you to write a blog post or whatnot is an empty editor. It’s, where do I get started? Where’s the rest?I even built a template that I wind up sometimes using text expander to autofill, that gets me started. And it’s just get—once I get started, it’s great. It’s hard to get me started; it’s hard to get me to stop, in case no one has been aware of that. But it’s been understanding how I work and how that integrates with it. I’m curious, given that you do talk to people who are trying to build these systems for a living for themselves? How common is my perspective on this? Am I out there completely, this unique, beautiful Snowflake? Is it yeah, that’s basically everyone? Or somewhere in between?Marie: Oh, I definitely don’t think you’re alone with that. And again, I often will dissuade people from taking on Notion. I’m like, “Oh, if you’re just looking for a note-taker, or you’re just looking for something else,” or, “Your tools are already working for you, great. Keep using them.” So, I think it’s quite common. I don’t think Notion is the right tool for everyone.I think it’s great for very visual people like myself, people that it matters how you are seeing your information, and how much information you’re seeing, and you want more control over that, that’s great. For me, I like the integration. I know that as soon as I’m bouncing around to different tools, like, I just already feel kind of scattered, so I was like, how can I pull everything that I need into these, sort of, singular dashboards. So, my approach is very dashboard-focused. Okay, Marie is going into content mode, it’s time to write a blog. Go to the content hub. On the content hub is your list of most recent ideas, your templates for how to write a blog post. There’s resources for creating video. It’s already there for me; I’m not having to start from scratch like you said.But again, it took time to build that up for myself. So, I think you’re not alone, and I think some people get excited about that building process; other people get irritated by it, and I don’t think there’s a right or wrong answer. It’s just how do our brains work? Know thyself. And, yeah, I’ve sort of—I think also in a way, something that’s a little different, maybe, about the way that I use Notion is I think of it as a personal development tool.It is a tool for making me better in different ways. It’s for exploring my interests, it’s for feeding my curiosity, it’s for looking at change over time. I track my feelings every day. I’ve been journaling for 1300 days in a row, which is probably the only thing I’ve done consistently in my life [laugh] in the last couple of years. But now I can look and I can see trends over time in a really beautiful and visual way. And I just, to me, it’s like a curiosity tool, to see, like, where am I going? Where have I been? What do I want more of?Corey: I need to look into this a bit more because my idea of a well-designed user interface is—I’m very opinionated on this—but it comes down to the idea of where do you use nouns versus verbs in command-line arguments to things you’re running in the terminal. Because I was a grumpy Unix sysadmin for the first part of my career—because there’s no other kind of Unix sysadmin—and going down that path was great. Okay, everything I’m interacting with is basically a text file piped together to do different things. And it took a while for me to realize, you know, maybe—just spitballing here—there’s a better way to convey information than a wall of text, sometimes. Blasphemy.And no, no, it turns out that just because it’s hard using the tools I’m used to doesn’t mean that’s the best way to convey information. And even now, these days, I’m spending more time getting the color theme and the font choices and typeface choices of what I’m doing in the terminal to represent something that’s a bit more aesthetically pleasing. Does it actually account for anything? I don’t know, but it feels better and there’s almost a Feng Shui element of it. Similar to work in a—Marie: Yes.Corey: Clean office versus a messy one.Marie: A hundred percent. I think that’s kind of how I think of an approach. I am much more likely to get the things done. If, when I come in and I open Notion, it’s like, “Here’s what’s on today, Marie.” And it’s like speaking nicely to me, there’s little positive messages, there’s beautiful imagery.It just makes me feel good when I’m starting my day. And knowing that how I feel is going to very much influence what I’m likely to accomplish in the day, again, I’m constantly tricking myself into getting [laugh] more excited and amped up about what’s on the schedule for the day. So, I really liked that about it. It feels beautiful to me.Corey: I’m going to have to take another look at it at some point. I think that there’s a lot of interesting directions to go into on this. I also have the privilege of having known you for a little while, back when you were more or less just getting started. One of the things that you said at the time that absolutely resonated with me was the idea of, wait, you mean build a business around teaching people how to use Notion? Like an info product or a training approach?And a lot of your concerns are the ones that I’ve harbored for a while, too, which is the idea of there’s a proliferation of info products in technical and other spaces, and an awful lot of them—without naming any names or talking in any particular direction—are not the highest quality. People are building these courses while learning the thing themselves. And when they tell stories about it, it’s all about, “And this is how I’m making money quickly.” I don’t find that admirable; I don’t necessarily want to learn how to do a thing from someone who does not have themselves at least a decent understanding themselves of what they’re working on so they can address questions that go a bit off into the weeds. And so mu—again, knowing how to do a thing and knowing how to teach a thing are orthogonal concepts. And very often a lot of these info products are being created by people who don’t really know how to do either, as best I can tell.Marie: Yes. So, I think you’ve nailed a point to that, knowing a thing deeply and then knowing how to teach that thing really well are two totally different skills. And I definitely bumped up against that myself. I’m like, I know, Notion inside and out. Like, you know, name something, I can make it, I can optimize it, I can, you know, build a system out of thin air really fast, no problem. I’m a problem solver that way.But to teach someone else how to do that requires very different skills. And I knew [laugh] as I was starting to teach people stuff, I’m like, “You could do this. You could do that.” And I’m like kind of bouncing around and I’m all over the place because I’m so excited about the possibilities. But wait a second.Beginners that are just learning how to use Notion don’t need to know every frickin’ possible way that you could use it. So, knowing that instructional design, curriculum design is a whole other skill, and I care about student results, it’s like, this is a gap that I have, and I want to be an excellent teacher. It matters to me. I actually do want to become a better teacher. I want to have higher quality YouTube videos, I want to make sure that I’m not losing people along the way.I don’t just care about making a shit ton of money with an info product; I care about peoples’ experience and kind of having that, I don’t know, that prestige element. Like, that’s something that does matter in terms of producing quality products. So, I hired experts to help me do that because again, it’s a not necessarily a strength of mine. So, I think I hired three different people in the course of six months to various consultants and people who understand learning design and that sort of thing. And I think that’s something a lot of info product creators. They think of it as just packaging a blog and selling it, right?It’s different. When you’re teaching a course, for example, your formatting matters, how you display information matters, how you design activities matters. What separates a course from a passive income product or blog, right? We need to think about those things, and I think a lot of people are just like, what’s the quickest, you know, buck that I can make on these products and just kind of turn them out. And I don’t think every course creator has maybe done the extra legwork to really understand what makes students actually follow through and complete a course. It’s hard. It’s really hard.Corey: And these are also very different products. There’s what you are teaching, which is here’s how to contextualize these things and how to build a system around it. There’s another offering out there that would be something that would also be very compelling from my perspective where, cool, I appreciate the understanding and the deep systems design approach that goes into this. Can I just give you a brain dump of all the problems that I have with this? You go away and build a system that accounts for all of that.And again, it’s the outcome that I care about. There’s this belief that oh we want consultants to build by the hour and work hard. No. I don’t care. If you listen to this, nod and do the great customer service thing, the Zoom call, and just like, “Okay, that’s template number three with three one-line changes. Done. Now, we’re going to sit on it for a week so it looks hard.”Which we’ve all got that as consultants in the early days. And then you turn that around because it’s the outcome that I really care about. But that’s a different business, that is a different revenue model, that is different—Marie: Yes.Corey: That is not nearly so much a one-to-many, like an info product. That is a one-to-one or one-to-few.Marie: And I did that for the whole first year that the course was being developed and was out there. I was simultaneously consulting with people one-on-one all the time, with teams, with individuals. So, I’m learning about what are all those common challenges that keep popping up over and over again? What are the unique challenges? What are the common ones?And in my experience, what I bumped up against is people think they want to just pay someone to solve that, but then when you give someone a very fleshed out, organized system that they didn’t participate in the building, it’s a lot harder to get somebody to use it, to plug into a ready-made system. So, in our experience, there’s a sort of back and forth. It has to happen in tandem; we do it over time. And you know, in my partner’s case, Ben does consulting with companies as well, so he’ll meet with them on a weekly basis and working with the different members of the team. So, there is some element of we built you a thing. Let’s have you use it, notice where there’s gaps, friction, whatever, because it’s not a one-and-done process.It’s not like, “You gave me all the info. We’re good to go.” It’s not until people are using it that you’re like, “Oh, okay, that’s close, but I’m finding myself doing this, or avoiding this, or clicking around too much.” And so, to me, it’s a really organic process. But that’s not something that I’m as keen to do. And maybe it’s because I did it for, like, two years and kind of burnt out on it. I’m like, “I’m done. Like, I’d rather teach folks to do it themselves.” But so a partner does the consulting; I’m doing more of the teaching.Corey: That’s what happened to an awful lot of our consulting work here at The Duckbill Group where it was exciting and fun for me for years, and at some point it turned into, I am interested in teaching how to do this a little bit more and systematizing it because I’m starting to get bored with aspects of it. And I was thinking, “Well, do I build a course?” It’s, “Well, no. As it turns out that if you have the right starting point, I can hire people who I can teach how to do AWS bill analysis if they have the right starting point.” And it turns out that a lot of those people—read as all of them—are going to be way better at doing the systemic deep-dive across the board, rather than just finding the things that they find personally interesting and significant, and then, “Well, there you go. I did a consulting engagement.” And the output is basically three bullet points scrawled on the back of an envelope.Yeah, turns out that that’s not quite the level of professionalism clients expect. Great, so our product is better, we’re getting better insight into it, and I get to scratch my itch of teaching people how to do things internally without becoming a critical path blocker.Marie: Yeah, absolutely.Corey: I mean, I have shitposting to get back to. Come on.Marie: Yeah exactly. [laugh]. The important things. Love it.Corey: I really want to thank you for taking so much time to speak with me about all of these things. If people want to learn more—Marie: Absolutely.Corey: —where’s the best place to find you?Marie: Yeah, you can find me at mariepoulin.com is where my personal blog, or weareokidoki.com, or notionmastery.com. You can also catch me on Twitter.Corey: And we will put links to—Marie: That’s where I am most active. Yeah.Corey: Oh, of course. And all the links wind up going into the [show notes 00:37:42], as always. Thank you so much for your time. I appreciate it.Marie: Thanks for having me, Corey. It was awesome.Corey: Marie Poulin, CEO of Oki Doki. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice—and if it’s on the YouTubes smash the like and subscribe buttons—whereas if you’ve hated this podcast episode, great, same thing, five-star review on whatever platform, smash the two buttons, but also leave an insulting comment and then turn that comment into an info product that you wind up selling to a whole bunch of people primarily to boost your own Twitter threads about how successful you are as a creator.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
Full Description / Show Notes Guillermo talks about how he came to work at OCI and what it was like helping to pioneer Oracle’s cloud product (1:40) Corey and Guillermo discuss the challenges and realities of multi-cloud (6:00) Corey asks about OCI’s dedicated region approach (8:27) Guillermo discusses the problem of awareness (12:40) Corey and Guillermo talk cloud providers and cloud migration (14:40) Guillermo shares about how OCI’s cost and customer service is unique among cloud providers (16:56) Corey and Guillermo talk about IoT services and 5G (23:58) About Guillermo RuizGuillermo Ruiz gets into trouble more often than he would like. During his career Guillermo has seen many horror stories while building data centers worldwide. In 2007 he dreamed with space-based internet and direct routing between satellites, but he could only reach “the Cloud”. And there he is, helping customer build their business in someone else servers since 2011.Beware of his sense of humor...If you ever see him in a tech event, run, he will get you in problems.Links: Twitter: https://twitter.com/IaaSgeek, https://twitter.com/OracleStartup LinkedIn: https://www.linkedin.com/in/gruizesteban/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: Let’s face it, on-call firefighting at 2am is stressful! So there’s good news and there’s bad news. The bad news is that you probably can’t prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. I’ve been meaning to get a number of folks on this show for a while and today is absolutely one of those episodes. I’m joined by Guillermo Ruiz who is the Director of OCI Developer Evangelism, slash the Director of Oracle for Startups. Guillermo, thank you for joining me, and is Oracle for Startups an oxymoron because it kind of feels like it in some weird way, in the fullness of time.Guillermo: [laugh]. Thanks, Corey. It’s a pleasure being in your show.Corey: Well, thank you. I enjoy having you here. I’ve been trying to get you on for a while. I’m glad I finally wore you down.Guillermo: [laugh]. Thanks. As I said, well, startup, I think, is the future of the industry, so it’s a fundamental piece of our building blocks for the next generation of services.Corey: I have to say that I know that you folks at Oracle Cloud have been a recurring sponsor of the show. Thank you for that, incidentally. This is not a promoted guest episode. I invited you on because I wanted to talk to you about these things, which means that I can say more or less whatever I damn well want. And my experience with Oracle Cloud has been one of constantly being surprised since I started using it a few years ago, long before I was even taking sponsorships for this show. It was, “Oh, Oracle has a cloud. This ought to be rich.”And I started kicking the tires on it and I came away consistently and repeatedly impressed by the technical qualities the platform has. The always-free tier has a model of cloud economics that great. I have a sizable VM running there and have for years and it’s never charged me a dime. Your data egress fees aren’t, you know, a 10th of what a lot of the other cloud providers are charging, also known as, you know, you’re charging in the bounds of reality; good for that. And the platform continues to—although it is different from other cloud providers, in some respects, it continues to impress.Honestly, I keep saying one of the worst problems that has is the word Oracle at the front of it because Oracle has a 40-some-odd-year history of big enterprise systems, being stodgy, being difficult to work with, all the things you don’t generally tend to think of in terms of cloud. It really is a head turn. How did that happen? And how did you get dragged into the mess?Guillermo: Well, this came, like, back in five, six years ago, when they started building this whole thing, they picked people that were used to build cloud services from different hyperscalers. They dropped them into a single box in Seattle. And it’s like, “Guys, knowing what you know, how you would build the next generation cloud platform?” And the guys came up with OCI, which was a second generation. And when I got hired by Oracle, they showed me the first one, that classic.It was totally bullshit. It was like, “Guys, there’s no key differentiator with what’s there in the market.” I didn’t even know Oracle had a cloud, and I’ve been in this space since late-2010. And I had to sign, like, a bunch of NDAs a lot of papers, and they show me what they were cooking in the oven, and oh my gosh, when I saw that SDN out of the box directly in the physical network, CPUs assign, it was [BLEEP] [unintelligible 00:03:45]. It was, like, bare metal. I saw that the future was there. And I think that they built the right solution, so I joined the company to help them leverage the cloud platform.Corey: The thing that continually surprises me is that, “Oh, we have a cloud.” It has a real, “Hello fellow kids,” energy. Yes, yeah, so does IBM; we’ve seen how that played out. But the more I use it, the more impressed I am. Early on in the serverless function days, you folks more or less acquired Iron.io, and you were streets ahead as far as a lot of the event-driven serverless function style of thing tended to go.And one of the challenges that I see in the story that’s being told about Oracle Cloud is, the big enterprise customer wins. These are the typical global Fortune 2000s, who have been around for, you know—which is weird for those of us in San Francisco, but apparently, these companies have been around longer than 18 months and they’ve built for platforms that are not the latest model MacBook Pro running the current version of Chrome. What is that? What is that legacy piece of garbage? What does it do? It’s like, “Oh, it does about $4 billion a quarter so maybe show some respect.”It’s the idea of companies that are doing real-world things, and they absolutely have cloud power. Problems and needs that are being met by a variety of different companies. It’s easy to look at that narrative and overlook the fact that you could come up with some ridiculous Twitter for Pets-style business idea and build it on top of Oracle Cloud and I would not, at this point, call that a poor decision. I’m not even sure how it got there, and I wish that story was being told a little bit better. Given that you are a developer evangelist focusing specifically on startups and run that org, how do you see it?Guillermo: Well, the thing here is, you mentioned, you know, about Oracle, many startup doesn’t even know we have a cloud provider. So, many of the question comes is like, how we can help on your business. It’s more on the experience, you know, what are the challenges, the gaps, and we go in and identify and try to use our cloud. And even though if I’m not able to fill that gap, that’s why we have this partnership with Microsoft. It’s the first time to cloud providers connect both clouds directly without no third party in between, router to router.It’s like, let’s leverage the best of these clouds together. I’m a truly believer of multi-cloud. Non-single cloud is perfect. We are evolving, we’re getting better, we are adding services. I don’t want to get to 500 services like other guys do. It’s like, just have a set of things that really works and works really, really well.Corey: Until you have 40 distinct managed database services and 80 ways to run containers, are you’re really a full cloud provider? I mean, there’s always that question that, at some point, the database Java, the future is going to have to be disambiguating between all the different managed database services on a per workload basis, and that job sounds terrible. I can’t let the multi-cloud advocacy pass unchallenged here because I’m often misunderstood on this, and if I don’t say something, I will get emails, and nobody wants that. I think that the idea of building a workload with the idea that it can flow seamlessly between cloud providers is a ridiculous fantasy that basically no one achieves. The number of workloads that can do that are very small.That said, the idea of independent workloads living on different cloud providers as is the best fit for placement for those is not just a good idea, it is the—whether it’s a good idea or not as irrelevant because that’s the reality in which we all live now. That is the world we have to deal with.Guillermo: If you want distributed system, obviously you need to have multiple cloud providers in your strategy. How you federate things—if you go down to the Kubernetes side, how you federate multi-clusters and stuff, that’s a challenge out there where people have. But you mentioned that having multiple apps and things, we have customers that they’ve been running Google Cloud, for example, and we build [unintelligible 00:07:40] that cloud service out there. And the thing is that when they run the network throughput and the performance test, they were like, “Damn, this is even better than what I have in my data center.” It’s like, “Guys, because we are room by room.” It’s here is Google, here it’s Oracle; we land in the same data center, we can provide better connectivity that what you even have.So, that kind of perception is not well seen in some customers because they realize that they’re two separate clouds, but the reality is that most of us have our infrastructure in the same providers.Corey: It’s kind of interesting, just to look at the way that the industry is misunderstanding a lot of these things. When you folks came out with your cloud at customer initiatives—the one that jumps out to my mind is the dedicated region approach—a lot of people started making fun of that because, “What is this nonsense? You’re saying that you can deploy a region of your cloud on site at the customer with all of the cloud services? That’s ridiculous. You folks don’t understand cloud.”My rejoinder to that is people saying that don’t understand customers. You take a look at for example… AWS has their Outpost which is a rack or racks with a subset of services in them. And that, from their perspective, as best I can tell, solves the real problem that customers have, which is running virtual machines on-premises that do not somehow charge an hourly cost back to AWS—I digress—but it does bring a lot of those services closer to customers. You bring all of your services closer to customers and the fact that is a feasible thing is intensely appealing to a wide variety of customer types. Rather than waiting for you to build a region in a certain geographic area that conforms with some regulatory data requirement, “Well, cool, we can ship some racks. Does that work for you?” It really is a game-changer in a whole bunch of respects and I don’t think that the industry is paying close enough attention to just how valuable that is.Guillermo: Indeed. I’ve been at least hearing since 2010 that next year is the boom; now everybody will move into the cloud. It has been 12 years and still 75% of customers doesn’t have their critical workloads in the cloud. They have developer environments, some little production stuff, but the core business is still relying in the data center. If I come and say, “Hey, what if I build this behind your firewall?”And it’s not just that you have the whole thing. I’m removing all your operational expenses. Now, you don’t need to think about hardware refresh, upgrade staff, just focus on your business. I think when we came up with a dedicated region, it was awesome. It was one of the best thing I’ve seen their Outpost is a great solution, to be honest, but if you lose the one connectivity, the control plane is still in the cloud.In our site, you have the control plane inside your data center so you can still operate and manage your services, even if there is an outage on your one site. One of the common questions we find on that area is, like, “Damn, this is great, but we would like to have a smaller size of this dedicated region.” Well, stay tuned because maybe we come with smaller versions of our dedicated regions so you guys can go and deploy whatever you need there.Corey: It turns out that, in the fullness of time, I like this computer but I want it to be smaller is generally a need that gets met super well. One thing that I’ve looked into recently has been the evolution of companies, in the fullness of time—which this is what completely renders me a terrible analyst in any traditional sense; I think more than one or two quarters ahead, and I look at these things—the average tenure of a company in the S&P 500 index is 21 years or so. Which means that if we take a look at what’s going on 20 years or so from now in the 2040s, roughly half—give or take—of the constituency of the S&P 500 may very well not have been founded yet. So, when someone goes out and founds a company tomorrow as an idea that they’re kicking around, let’s be clear, with a couple of very distinct exceptions, they’re going to build it on Cloud. There’s a lot of reasons to do that until you hit certain inflection points.So, this idea that, oh, we’re going to rent a rack, and we’re going to go build some nonsense, and yadda, yadda, yadda. It’s just, it’s a fantasy. So, the question that I see for a lot of companies is the longtail legacy where if I take that startup and found it tomorrow and drive it all the way toward being a multinational, at what point did they become a customer for whatever these companies are selling? A lot of the big E enterprise vendors don’t have a story for that, which tells me long-term, they have problems. Looking increasingly at what Oracle Cloud is doing, I have to level with you, I viewed Oracle as being very much in that slow-eroding dinosaur perspective until I started using the platform in some depth. I am increasingly of the mind that there’s a bright future. I’m just not sure that has sunk into the industry’s level of awareness these days.Guillermo: Yeah, I can agree with you in that sense. Mainly, I think we need to work on that awareness side. Because for example, if I go back to the other products we have in the company, you know, like the database, what the database team has done—and I’m not a database guy—and it’s like, “Guys, even being an infrastructure guy, customers doesn’t care about infrastructure. They just want to run their service, that it doesn’t fail, you don’t have a disruption; let me evolve my business.” But even though they came with this converged database, I was really impressed that you can do everything in a single-engine rather than having multiple database implemented. Now, you can use the MongoDB APIs.It’s like, this is the key of success. When you remove the learning curve and the frictions for people to use your services. I’m a [unintelligible 00:13:23] guy and I always say, “Guys, click, click, click. In three clicks, I should have my service up and running.” I think that the world is moving so fast and we have so much information today, that’s just 24 hours a day that I have to grab the right information. I don’t have time to go and start learning something from scratch and taking a course of six months because results needs to be done in the next few weeks.Corey: One thing that I think that really reinforces this is—so as I mentioned before, I have a free tier account with you folks, have for years, whenever I log into the thing, I’m presented with the default dashboard view, which recommends a bunch of quickstarts. And none of the quickstarts that you folks are recommending to me involve step one, migrate your legacy data center or mainframe into the cloud. It’s all stuff like using analytics to predict things with AI services, it’s about observability, it’s about governance of deploy a landing zone as you build these things out. Here’s how to do a low-code app using Apex—which is awesome, let’s be clear here—and even then launching resources is all about things that you would tend to expect of launch database, create a stack, spin up some VMs, et cetera. And that’s about as far as it goes toward a legacy way of thinking.It is very clear that there is a story here, but it seems that all the cloud providers these days are chasing the migration story. But I have to say that with a few notable exceptions, the way that those companies move to cloud, it always starts off by looking like an extension of their data center. Which is fine. In that phase, they are improving their data center environment at the expense of being particularly cloudy, but I don’t think that is necessarily an adoption model that puts any of these platforms—Oracle Cloud included—in their best light.Guillermo: Yeah, well, people was laughing to us, when we released Layer 2 in the network in the cloud. They were like, “Guys, you’re taking the legacy to the cloud. It’s like, you’re lifting the shit and putting the shit up there.” Is like, “Guys, there are customers that cannot refactor and do anything there. They need to still run Layer 2 there. Why not giving people options?”That’s my question is, like, there’s no right answers to the cloud. You just need to ensure that you have the right options for people that they can choose and build their strategy around that.Corey: This has been a global problem where so many of these services get built and launched from all of the vendors that it becomes very unclear as a customer, is this thing for me or not? And honestly, sometimes one of the best ways to figure that out is to all right, what does it cost because that, it turns out, is going to tell me an awful lot. When it comes to the price tag of millions of dollars a year, this is probably not for my tiny startup. Whereas when it comes to a, oh, it’s in the always free tier or it winds up costing pennies per hour, okay, this is absolutely something I want to wind up exploring and seeing what happens. And it becomes a really polished experience across the board.I also will say this is your generation two cloud—Gen 2, not to be confused with Gentoo, the Linux distribution for people with way more time on their hands than they have sense—and what I find interesting about it is, unlike a lot of the—please don’t take this the wrong way—late-comers to cloud compared to the last 15 years of experience of Amazon being out in front of everyone, you didn’t just look at what other providers have done and implement the exact same models, the exact same approaches to things. You’ve clearly gone in your own direction and that’s leading to some really interesting places.Guillermo: Yeah, I think that doing what others are doing, you just follow the chain, no? That will never position you as a top number one out there. Being number one so many years in the cloud space as other cloud providers, sometimes you lose the perception of how to treat and speak to customers you know? It’s like, “I’m the number one. Who cares if this guy is coming with me or not?” I think that there’s more on the empathy side on how we treat customers and how we try to work and solve.For example, in the startup team, we find a lot of people that hasn’t have infrastructure teams. We put for free our architects that will give you your GitHub or your GitLab account and we’ll build the Terraform modules and give that for you. It’s like now you can reuse it, spin up, modify whatever you want. Trying to make life easier for people so they can adopt and leverage their business in the cloud side, you know?Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don’t leave managing your database to your cloud vendor because they’re too busy launching another half-dozen managed databases to focus on any one of them that they didn’t build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: There’s so much that we folks get right. Honestly, one of the best things that recommends this is the always free tier does exactly what it says on the tin. Yeah, sure. I don’t get to use every edge case service that you’ve built across the board, but I’ve also had this thing since 2019, and never had to pay a penny for any of it, whereas recently—as we’re recording this, it was a week or two ago—that I saw someone wondering what happened to their AWS account because over the past week, suddenly they went from not using SageMaker to being charged $270,000 on SageMaker. And it’s… yeah, that’s not the kind of thing that is going to endear the platform to frickin’ anyone.And I can’t believe I’m saying this, but the thing says Oracle on the front of it and I’m recommending it because it doesn’t wind up surprising you with a bill later. It feels like I’ve woken up in bizarro world. But it’s great.Guillermo: Yep. I think that’s one of the clever things we’ve done on that side. We’ve built a very robust platform, really cool services. But it’s key on how people can start learning and testing the flavors of your cloud. But not only what you have in the fleet here, you have also the Ampere instances.We’re moving into a more sustainable world, and I think that having, like, the ARM architectures in the cloud and providing that on the free space of people can just go and develop on top, I think that was one of the great things we’ve done in the last year-and-a-half, something like that. Definitely a full fan of a free tier.Corey: You also, working over in the Developer Evangelist slash advocacy side of the world—devrelopers, as I tend to call it much to the irritation of basically everyone who works in developer relations—one of the things that I think is a challenge for you is that when I wind up trying to do something ridiculous—I don’t know maybe it’s a URL shortener; maybe it is build a small app that does something that’s fairly generic—with a lot of the other platforms. There’s a universe of blog posts out there, “Here’s how I did it on this platform,” and then it’s more or less you go to GitHub—or gif-UB, and I have mispronounced that too—and click the button and I wind up getting a deploy, whereas in things that are rapidly emerging with the Oracle Cloud space, it feels like, on some level, I wind up getting to be a bit of a trailblazer and figure some of these things out myself. That is diminishing. I’m starting to see more and more content around this stuff. I have to assume that is at least partially due to your organization’s work.Guillermo: Oh, yeah, but things have changed. For example, we used to have our GitHub repository just as a software release, and we push to have that as a content management, you know, it’s like, I always say that give—let people steal the code. You just put the example that will come with other ideas, other extensions, plug-in connectors, but you need to have something where you can start. So, we created this DevRel Quickstart that now is managed by the new DevRel organization where we try to put those examples. So, you just can go and put it.I’ve been working with the community on building, like, a content aggregator of how people is using our technology. We used to have ocigeek.com, that was a website with more than 1000 blog and, like, 500 visits a day looking after what other people were doing, but unfortunately, we had to, because of… the amount of X reasons we have to pull it off.But we want to come with something like that. I think that information should be available. I don’t want people to think when it comes to my cloud is like, “Oh, how you use this product?” It’s like no, guys how I can build with Angular, React the content management system? You will do it in my cloud because that example I’m doing, but I want you to learn the basics and the context of running Python and doing other things there rather than go into oh, no, this is something specific to me. No, no, that will never work.Corey: That was the big problem I found with doing a lot of the serverless stuff in years past where my first Lambda application took me two weeks to build because I’m terrible at programming. And now it takes me ten minutes to build because I’m terrible at programming and don’t know what tests are. But the problem I ran into for that first one was, what is the integration format? What is the event structure? How do I wind up accessing that?What is the thing that I’m integrating with expecting because, “Mmm, that’s not it; try again,” is a terrible error message. And so, much of it felt like it was the undifferentiated gluing things together. The only way to make that stuff work is good documentation and numerous examples that come at the problem from a bunch of different ways. And increasingly, Oracle’s documentation is great.Guillermo: Yeah, well, in my view, for example, you have the Three-Tier Oracle. We should have a catalog of 100 things that you can do in the free tier, even though when I propose some of the articles, I was even talking about VMware, and people was like, “[unintelligible 00:22:34], you cannot deploy VMware.” It’s like, “Yeah, but I can connect my [crosstalk 00:22:39]—”Corey: Well, not with that attitude.Guillermo: Yeah. And I was like, “Yeah, but I can connect to the cloud and just use it as a backup place where I can put my image and my stuff. Now, you’re connecting to things: VMware with free tier.” Stuff like that. There are multiple things that you can do.And just having three blocks is things that you can do in the free tier, then having developer architectures. Show me how you can deploy an architecture directly from the command line, how I can run my DevOps service without going to the console, just purely using SDKs and stuff like that. And give me the option of how people is working and expanding that content and things there. If you put those three blocks together, I think you’re done on how people can adopt and leverage your cloud. It’s like, I want to learn; I don’t want to know the basics of I don’t know, it’s—I’m not a database guy, so I don’t understand those things and I don’t want to go into details.I just they just need a database to store my profiles and my stuff so I can pick that and do computer vision. How I can pick and say, “Hey, I’m speaking with Corey Quinn and I have a drone flying here, he recommends your face and give me your background from all the different profiles.” That’s the kind of solutions I want to build. But I don’t want to be an expert on those areas.Corey: Because with all the pictures of me with my mouth open, you wouldn’t be able to under—it would make no sense of me until I make that pose. There’s method to—Guillermo: [laugh].Corey: —my insane madness over here.Guillermo: [laugh] [unintelligible 00:23:58].Corey: Yeah. But yeah, there’s a lot of value as you move up the stack on these things. There’s also something to be said, as well, for a direction that you folks have been moving in recently, that I—let me be fair here—I think it’s clown shoes because I tend to think in terms of software because I have more or less the hardware destruction bunny level of aura when it comes to being near expensive things. And I look around the world and I don’t have a whole lot of problems that I can legally solve with an army of robots.But there are customers who very much do. And that’s why we see sort of the twin linking of things like IoT services and 5G, which when I first started seeing cloud providers talking about this, I thought was Looney Tunes. And you folks are getting into it too, so, “Oh, great. The hype wound up affecting you too.” And the thing that changed my mind was not anything cloud providers have to say—because let’s be clear, everyone has an agenda they’re trying to push for—but who doesn’t have an agenda is the customers talking about these things and the neat things that they’re able to achieve with it, at which point I stopped making fun, I shut up and listen in the hopes that I might learn something. How have you seen that whole 5G slash IoT slash internet of Nonsense space evolving?Guillermo: That’s the future. That’s what we’re going to see in the next five years. I run some innovation sessions with a lot of customers and one of the main components I speak about is this area. With 5G, the number of IoT devices will exponentially grow. That means that you’re going to have more data points, more data volume out there.How can you provide the real value, how you can classify, index, and provide the right information in just 24 hours, that’s what people is looking. Things needs to be instant. If you say to the kids today, they cannot watch a football match, 90 minutes. If you don’t get the answer in ten, they move to the next thing. That’s how this society is moving [unintelligible 00:25:50].Having all these solutions from a data perspective, and I think that Oracle has a great advantage in that space because we’ve been doing that for 43 years, right? It’s like, how we do the abstraction? How I can pick all that information and provide added value? We build the robot as a service. I can configure it from my browser, any robot anywhere in the world.And I can do it in Python, Java. I can [unintelligible 00:26:14] applications. Two weeks ago, we were testing on connecting IoT devices and flashing the firmware. And it was working. And this is something that we didn’t do it alone. We did it with a startup.The guys came and had a sandbox already there, is like, “let’s enable this on [unintelligible 00:26:28]. Let’s start working together.” Now, I can go to my customers and provide them a solution that is like, hey, let’s connect Boston Dynamics, or [unintelligible 00:26:37] Robotics. Let’s start doing those things and take the benefits of using Oracle’s AI and ML services. Pick that, let’s do computer vision, natural language processing.Now, you’re connecting what I say, an end-to-end solution that provides real value for customers. Connected cars, we turn our car into a wallet. I can go and pay on the petrol station without leaving my car. If I’m taking the kids to takeaway, I can just pay these kind of things is like, “Whoa, this is really cool.” But what if I [laugh] get that information for your insurance company.Next year, Corey, you will pay double because you’re a crazy driver. And we know how you drive in the car because we have all that information in place. That’s how the things will roll out in the next five to ten years. And [unintelligible 00:27:24] healthcare. We build something for emergencies that if you have a car crash, they have the guys that go and attend can have your blood type and some information about your car, where to cut the chassis and stuff when you get prisoner inside.And I got people saying, “Oh gee, GDPR because we are in Europe.” It’s like, “Guys, if I’m going to die, I don’t care if they have my information.” That’s the point where people really need to balance the whole thing, right? Obviously, we protect the information and the whole thing, but in those situations is like hey, there’s so many things we can do. There are countless opportunities out there.Corey: The way that I square that circle personally has always been it’s about informed consent, when if people are given a choice, then an awful lot of those objections that people have seemed to melt away. Provided, of course, that is an actual choice and it’s not one of those, “Well, you can either choose to”—quote-unquote—“Choose to do this, or you can pay $9,000 a month extra.” Which is, that’s not really a choice. But as long as there’s a reasonable way to get informed consent, I think that people don’t particularly mind, I think it’s when they wind up feeling that they have been spied upon without their knowledge, that’s when everything tends to blow up. It turns out, if you tell people in advance what you’re going to do with their information, they’re a lot less upset. And I don’t mean burying it deep and the terms and conditions.Guillermo: And that’s a good example. We run a demo with one of our customers showing them how dangerous the public information you have out there. You usually sign and click and give rights to everybody. We found in Stack Overflow, there was a user that you just have the username there, nothing else. And we build a platform with six terabytes of information grabbing from Stack Overflow, LinkedIn, Twitter, and many other social media channels, and we show how we identify that this guy was living in Bangalore in India and was working for a specific company out there.So, people was like, “Damn, just having that name, you end up knowing that?” It’s like there’s so much information out there of value. And we’ve seen other companies doing that illegally in other places, you know, Cambridge Analytics and things like that. But that’s the risk of giving your information for free out there.Corey: It’s always a matter of trade-offs. There is no one-size-fits-all solution and honestly, if there were it feels like we wouldn’t have cloud providers; we would just have the turnkey solution that gives the same thing that everyone needs and calls it good. I dream of such a day, but it turns out that customers are different, people are different, and there’s no escaping that.Guillermo: [laugh]. Well, you mentioned dreamer; I dream direct routing between satellites, and look where I am; I’m just in the cloud, one step lower. [laugh].Corey: You know, bit by bit, we’re going to get there one way or another, for an altitude perspective. I really want to thank you for taking so much time to speak with me today. If people want to learn more, where’s the right place to find you?Guillermo: Well, I have the @IaaSgeek Twitter account, and you can find me on LinkedIn gruizesteban there. Just people wants to talk about anything there, I’m open to any kind of conversation. Just feel free to reach out. And it was a pleasure finally meeting you, in person. Not—well in person; through a camera, at least being in the show with you.Corey: Other than on the other side of a Twitter feed. No, I hear you.Guillermo: [laugh].Corey: We will, of course, put links to all of that in the [show notes 00:30:43]. Thank you so much for your time. I really do appreciate it.Guillermo: Thanks very much. So, you soon.Corey: Guillermo Ruiz, Director of OCI Developer Evangelism. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice along with an insulting comment, to which I will respond with a surprise $270,000 bill.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About swyxswyx has worked on React and serverless JavaScript at Two Sigma, Netlify and AWS, and now serves as Head of Developer Experience at Airbyte. He has started and run communities for hundreds of thousands of developers, like Svelte Society, /r/reactjs, and the React TypeScript Cheatsheet. His nontechnical writing was recently published in the Coding Career Handbook for Junior to Senior developers.Links Referenced: “Learning Gears” blog post: https://www.swyx.io/learning-gears The Coding Career Handbook: https://learninpublic.org Personal Website: https://swyx.io Twitter: https://twitter.com/swyx TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don’t leave managing your database to your cloud vendor because they’re too busy launching another half-dozen managed databases to focus on any one of them that they didn’t build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: Let’s face it, on-call firefighting at 2am is stressful! So there’s good news and there’s bad news. The bad news is that you probably can’t prevent incidents from happening, but the good news is that incident.io makes incidents less stressful and a lot more valuable. incident.io is a Slack-native incident management platform that allows you to automate incident processes, focus on fixing the issues and learn from incident insights to improve site reliability and fix your vulnerabilities. Try incident.io, recover faster and sleep more.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. Some folks are really easy to introduce when I have them on the show because, “My name is, insert name here. I built thing X, and my job is Y at company Z.” Then we have people like today’s guest.swyx is currently—and recently—the head of developer experience at Airbyte, but he’s also been so much more than that in so many different capacities that you’re very difficult to describe. First off, thank you for joining me. And secondly, what’s the deal with you?swyx: [laugh]. I have professional ADD, just like you. Thanks for having me, Corey. I’m a—Corey: It works out.swyx: a big fan. Longtime listener, first time caller. Love saying that. [laugh].Corey: You have done a lot of stuff. You have a business and finance background, which… okay, guilty; it’s probably why I feel some sense of affinity for a lot of your work. And then you went into some interesting directions. You were working on React and serverless YahvehScript—which is, of course, how I insist on pronouncing it—at Two Sigma, Netlify, AWS—a subject near and dear to my heart—and most recently temporal.io.And now you’re at Airbyte. So, you’ve been focusing on a lot of, I won’t say the same things, but your area of emphasis has definitely consistently rhymed with itself. What is it that drives you?swyx: So, I have been recently asking myself a lot of this question because I had to interview to get my new role. And when you have multiple offers—because the job market is very hot for DevRel managers—you have to really think about it. And so, what I like to say is: number one, working with great people; number two, working on great products; number three, making a lot of money.Corey: There’s entire school of thought that, “Oh, that’s gauche. You shouldn’t mention trying to make money.” Like, “Why do you want to work here because I want to make money.” It’s always true—swyx: [crosstalk 00:03:46]—Corey: —and for some reason, we’re supposed to pretend otherwise. I have a lot of respect for people who can cut to the chase on that. It’s always been something that has driven me nuts about the advice that we give a new folks to the industry and peop—and even students figuring out their career path of, “Oh, do something you love and the money will follow.” Well, that’s not necessarily true. There are ways to pivot something you’d love into something lucrative and there are ways to wind up more or less borderline starving to death. And again, I’m not saying money is everything, but for a number of us, it’s hard to get to where we want to be without it.swyx: Yeah, yeah. I think I’ve been cast with the kind of judgmental label of being very financially motivated—that’s what people have called me—for simply talking about it. And I’m like, “No. You know, it’s number three on my priority list.” Like, I will leave positions where I have a lot of money on the table because I don’t enjoy the people or the products, but having it up there and talking openly about it somehow makes you [laugh] makes you sort of greedy or something. And I don’t think that’s right. I tried to set an example for the people that I talk to or people who follow me.Corey: One of the things I’ve always appreciated about, I guess, your online presence, which has remained remarkably consistent as you’ve been working through a bunch of different, I guess, stages of life and your career, is you have always talked in significant depth about an area of tech that I am relatively… well, relatively crap at, let’s be perfectly honest. And that is the wide world of most things front-end. Every time I see a take about someone saying, “Oh, front-end is junior or front-end is somehow less than,” I’d like to know what the hell it is they know because every time I try and work with it, I wind up more confused than I was when I started. And what I really appreciate is that you have always normalized the fact that this stuff is hard. As of the time that we’re recording this a day or so ago, you had a fantastic tweet thread about a friend of yours spun up a Create React App and imported the library to fetch from an endpoint and immediately got stuck. And then you pasted this ridiculous error message.He’s a senior staff engineer, ex-Google, ex-Twitter; he can solve complex distributed systems problems and unable to fetch from a REST endpoint without JavaScript specialist help. And I talk about this a lot in other contexts, where the reason I care so much about developer experience is that a bad developer experience does not lead people to the conclusion of, “Oh, this is a bad interface.” It leads people to the conclusion, “Oh, I’m bad at this and I didn’t realize it.” No. I still fall into that trap myself.I was under the impression that there was just this magic stuff that JS people know. And your tweet did so much to help normalize from my perspective, the fact that no, no, this is very challenging. I recently went on a Go exploration. Now, I’m starting to get into JavaScript slash TypeScript, which I think are the same thing but I’m not entirely certain of that. Like, oh, well, one of them is statically typed, or strongly typed. It’s like, “Well, I have a loud mechanical keyboard. Everything I do is typing strongly, so what’s your point?”And even then we’re talking past each other in these things. I don’t understand a lot of the ecosystem that you live your career in, but I have always had a tremendous and abiding respect for your ability to make it accessible, understandable, and I guess for lack of a better term, to send the elevator back down.swyx: Oh, I definitely think about that strongly, especially that last bit. I think it’s a form of personal growth. So, I think a lot of people, when they talk about this sending the elevator back down, they do it as a form of charity, like I’m giving back to the community. But honestly, you actually learn a lot by trying to explain it to others because that’s the only way that you truly know if you’ve learned something. And if you ever get anything wrong, you’ll—people will never let you forget it because it is the internet and people will crawl over broken glass to remind you that you’re wrong.And once you’ve got it wrong, you will—you know, you’ve been so embarrassed that you’ll never forget it. So, I think it’s just a really good way to learn in public. And that’s kind of the motto that I’m kind of known for. Yeah, we can take the direction anywhere you want to go in JavaScript land. Happy to talk about it all day. [laugh].Corey: Well, I want to start by something you just said where you’re doing the learning in public thing. And something I’ve noticed is that there are really two positions you can take—in the general sense—when you set out to make a bit of a reputation for yourself in a particular technical space. You can either do the, “I’m a beginner here, same as the rest of you, and I’m learning in public,” or you can position yourself as something of an expert. And there are drawbacks and advantages to both. I think that if you don’t look as wildly over-represented as I do, both of them are more fraught in different ways, where it’s, “Oh, you’re learning in public. Ah, look at the new person, she’s dumb.”Or if you’re presenting yourself as an expert, you get nibbled to death by ducks on a lot of the deep technical nuances and well, actually’ed to death. And my position has always been and this is going to be a radical concept for some folks, is that I’m genuinely honest. I tend to learn in public about the things that I don’t know, but the things that I am something of a subject matter expert in—like, I don’t know, cloud billing—I don’t think that false modesty necessarily serves me particularly well. It’s yeah, I know exactly what I’m talking about here. Pretending otherwise it’s just being disingenuous.swyx: I try to think of it as having different gears of learning in public. So, I’ve called this “Learning Gears” in a previous blog post of mine, where you try to fit your mode of learning to the terrain that you’re on, your domain expertise, and you should never over-represent the amount that you know because I think people are very rightly upset when there are a lot of people—let’s say on Twitter, or YouTube, or Udemy even—who present themselves as experts who are actually—they just read the docs the previous night. So, you should try not to over-represent your expertise.But at the same time, don’t let your imposter syndrome stop you from sharing what you are currently learning and taking corrections when you’re wrong. And I think that’s the tricky balance to get which is constantly trying to put yourself out there while accepting that you might be wrong and not getting offended when or personally attacked when someone corrects you, inevitably. And sometimes people will—especially if you have a lot of followers, people will try to say—you know, someone of your following—you know, it’s—I kind of call this follower shaming, like, you should act, uh—invulnerable, or run every tweet through committee before you tweet after a certain sort of following size. So, I try to not do that and try to balance responsibility with authenticity.Corey: I think that there’s something incredibly important about that, where there’s this idea that you either become invulnerable and get defensive and you yell at people, and down that path lies disaster because, believe it or not, we all get it wrong from time to time, and doubling down and doubling down and doubling down again, suddenly, you’re on an island all by yourself and no one respectable is going to be able to get there to help you. And the other side of it is going too far in the other direction, where you implicitly take any form of criticism whatsoever as being de facto correct. And I think that both paths don’t lead to super great places. I think it’s a matter of finding our own voices and doing a little bit of work as far as the validity of accepting a given piece of feedback goes. But other than that, I’m a big fan of being able to just more or less be as authentic as possible.And I get that I live in a very privileged position where I have paths open to me that are not open to most folks. But in many respects so to you are one of the—easily—first five people I would think of if someone said, “Hey if I need to learn JavaScript for someone, who should I talk to first?” You’re on that list. And you’ve done a lot of things in this area, but you’ve never—you alluded to it a few minutes ago, but I’m going to call it out a little more pointedly—without naming names, let’s be clear—and that you’re never presented as a grifter, which is sort of the best way I can think of it of, “Well, I just learned this new technology stack yesterday and now I’m writing a book that I’m going to sell to people on how to be an expert at this thing.” And I want to be clear, this is very distinct from gatekeeping because I think that, “Oh, well, you have to be at least this much of an expert—” No, but I think that holding yourself out as I’m going to write a book on how to be proud of how to become a software engineer.Okay, you were a software engineer for six months, and more to the point, knowing how to do a thing and knowing how to teach a thing are orthogonal skill sets, and I think that is not well understood. If I ever write a book or put something—or some sort of info product out there, I’m going to have to be very careful not to fall into that trap because I don’t want to pretend to be an expert in things that I’m not. I barely think I’m an expert in things that I provable am.swyx: there are many ways to answer that. So, I have been accused a couple of times of that. And it’s never fun, but also, if you defend yourself well, you can actually turn a critic into a fan, which I love doing.Corey: Mm-hm.swyx: [laugh].Corey: Oh yes.swyx: what I fall back to, so I have a side interest in philosophy, based on one of my high school teachers giving us, like, a lecture in philosophy. I love him, he changed my life. [Lino Barnard 00:13:20], in case—in the off chance that he’s listening. So, there’s a theory of knowledge of, like, how do you know what you know, right? And if you can base your knowledge on truth—facts and not opinions, then people are arguing with the facts and not the opinions.And so, getting as close to ground truth as possible and having certainty in your collection of facts, I think is the basis of not arguing based on identity of, like, “Okay, I have ten years experience; you have two years experience. I am more correct than you in every single opinion.” That’s also not, like, the best way to engage in the battlefield of ideas. It’s more about, do you have the right amount of evidence to support the conclusions that you’re trying to make? And oftentimes, I think, you know, that is the basis, if you don’t have that ability.Another thing that I’ve also done is to collect the opinions of others who have more expertise and present them and curate them in a way that I think adds value without taking away from the individual original sources. So, I think there’s a very academic way [laugh] you can kind of approach this, but that defends your intellectual integrity while helping you learn faster than the typical learning rate. Which is kind of something I do think about a lot, which is, you know, why do we judge people by the number of years experience? It’s because that’s usually the only metric that we have available that is quantifiable. Everything else is kind of fuzzy.But I definitely think that, you know, better algorithms for learning let you progress much faster than the median rate, and I think people who apply themselves can really get up there in terms of the speed of learning with that. So, I spend a lot of time thinking about this stuff. [laugh].Corey: It's a hard thing to solve for. There’s no way around it. It’s, what is it that people should be focusing on? How should they be internalizing these things? I think a lot of it starts to with an awareness, even if not in public, just to yourself of, “I would like advice on some random topic.” Do you really? Are you actually looking for advice or are you looking—swyx: right.Corey: For validation? Because those are not the same thing, and you are likely to respond very differently when you receive advice, depending on which side of that you’re coming from.swyx: Yeah. And so, one way to do that is to lay out both sides, to actually demonstrate what you’re split on, and ask for feedback on specific tiebreakers that would help your decision swing one way or another. Yeah, I mean, there are definitely people who ask questions that are just engagement bait or just looking for validation. And while you can’t really fix that, I think it’s futile to try to change others’ behavior online. You just have to be the best version of yourself you can be. [laugh].Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: So, you wrote a book that is available at learninpublic.org, called The Coding Career Handbook. And to be clear, I have not read this myself because at this point, if I start reading a book like that, and you know, the employees that I have see me reading a book like that, they’re going to have some serious questions about where this company is going to be going soon. But scrolling through the site and the social proof, the testimonials from various people who have read it, more or less read like a who’s-who of people that I respect, who have been on this show themselves.Emma Bostian is fantastic at explaining a lot of these things. Forrest Brazeal is consistently a source to me of professional envy. I wish I had half his musical talent; my God. And your going down—it explains, more or less, the things that a lot of folks people are all expected to know but no one teaches them about every career stage, ranging from newcomer to the industry to senior. And there’s a lot that—there’s a lot of gatekeeping around this and I don’t even know that it’s intentional, but it has to do with the idea that people assume that folks, quote-unquote, “Just know” the answer to some things.Oh, people should just know how to handle a technical interview, despite the fact that the skill set is completely orthogonal to the day-to-day work you’ll be doing. People should just know how to handle a performance review, or should just know how to negotiate for a raise, or should just know how to figure out is this technology that I’m working on no longer the direction the industry is going in, and eventually I’m going to wind up, more or less, waiting for the phone to ring because there’s only three companies in the world left who use it. Like, how do you keep—how do you pay attention to what’s going on around you? And it’s the missing manual that I really wish that people would have pointed out to me back when I was getting started. Would have made life a lot easier.swyx: Oh, wow. That’s high praise. I actually didn’t know we’re going to be talking about the book that much. What I will say is—Corey: That’s the problem with doing too much. You never know what people have found out about you and what they’re going to say when they drag you on to a podcast.swyx: got you, got you. Okay. I know, I know, I know where this is going. Okay. So, one thing that I really definitely believe is that—and this happened to me in my first job as well, which is most people get the mentors that they’re assigned at work, and sometimes you have a bad roll the dice. [laugh].And you’re supposed to pick up all the stuff they don’t teach you in school at work or among your friend group, and sometimes you just don’t have the right network at work or among your friend group to tell you the right things to help you progress your career. And I think a lot of this advice is written down in maybe some Hacker News posts, some Reddit posts, some Twitter posts, and there’s not really a place you to send people to point to, that consolidates that advice, particularly focused at the junior to senior stage, which is the stage that I went through before writing the book. And so, I think that basically what I was going for is targeting the biggest gap that I saw, which is, there a lot of interview prep type books like Crack the Coding Career, which is kind of—Crack the Coding Interview, which is kind of the book title that I was going after. But once you got the job, no one really tells you what to do after you got that first job. And how do you level up to the senior that everyone wants to hire, right? There’s—Corey: “Well, I’ve mastered cracking the coding interview. Now, I’m really trying to wrap my head around the problem of cracking the showing up at work on time in the morning.” Like, the baseline stuff. And I had so many challenges with that early in my career. Not specifically punctuality, but just the baseline expectation that it’s just assumed that by the time you’re in the workplace earning a certain amount of money, it’s just assumed that you have—because in any other field, you would—you have several years of experience in the workplace and know how these things should play out.No, the reason that I’m sometimes considered useful as far as giving great advice on career advancement and the rest is not because I’m some wizard from the future, it’s because I screwed it all up myself and got censured and fired and rejected for all of it. And it’s, yeah, I’m not smart enough to learn from other people’s mistakes; I got to make them myself. So, there’s something to be said for turning your own missteps into guidance so that the next person coming up has an easier time than you did. And that is a theme that, from what I have seen, runs through basically everything that you do.swyx: I tried to do a lot of research, for sure. And so, one way to—you know, I—hopefully, I try not to make mistakes that others have learned, have made, so I tried to pick from, I think I include 1500 quotes and sources and blog posts and tweets to build up that level of expertise all in one place. So hopefully, it gives people something to bootstrap your experience off of. So, you’re obviously going to make some mistakes on your own, but at least you have the ability to learn from others, and I think this is my—you know, I’m very proud of the work that I did. And I think people have really appreciated it.Because it’s a very long book, and nobody reads books these days, so what am I doing [laugh] writing a book? I think it’s only the people that really need this kind of advice, that they find themselves not having the right mentorship that reach out to me. And, you know, it’s good enough to support a steady stream of sales. But more importantly, like, you know, I am able to mentor them at various levels from read my book, to read my free tweets, to read the free chapters, or join the pay community where we have weekly sessions going through every chapter and I give feedback on what people are doing. Sometimes I’ve helped people negotiate their jobs and get that bump up to senior staff—senior engineer, and I think more than doubled their salary, which was very personal proud moment for me.But yeah, anyway, I think basically, it’s kind of like a third place between the family and work that you could go to the talk about career stuff. And I feel like, you know, maybe people are not that open on Twitter, but maybe they can be open in a small community like ours.Corey: There’s a lot to be said for a sense of professional safety and personal safety around being—having those communities. I mean, mine, when I was coming up was the freenode IRC network. And that was great; it’s pseudo-anonymous, but again, I was Corey and network staff at the time, which was odd, but it was great to be able to reach out and figure out am I thinking about this the wrong way, just getting guidance. And sure, there are some channels that basically thrived on insulting people. I admittedly was really into that back in the early-two-thousand-nothings.And, like, it was always fun to go to the Debian channel. It’s like, “Yeah, can you explain to me how to do this or should I just go screw myself in advance?” Yeah, it’s always the second one. Like, community is a hard thing to get right and it took me a while to realize this isn’t the energy I want in the world. I like being able to help people come up and learn different things.I’m curious, given your focus on learning in public and effectively teaching folks as well as becoming a better engineer yourself along the way, you’ve been focusing for a while now on management. Tell me more about that.swyx: I wouldn’t say it’s been, actually, a while. Started dabbling in it with the Temporal job, and then now fully in it with Airbyte.Corey: You have to know, it has been pandemic time; it has stood still. Anything is—swyx: exactly.Corey: —a while it given that these are the interminable—this is the decade of Zoom meetings.swyx: [laugh]. I’ll say I have about a year-and-a-half of it. And I’m interested in it partially because I’ve really been enjoying the mentoring side with the coding career community. And also, I think, some of the more effective parts of what I do have to be achieved in the planning stages with getting the right resources rather than doing the individual contributor work. And so, I’m interested in that.I’m very wary of the fact that I don’t love meetings myself. Meetings are a means to an end for me and meetings are most of the job in management time. So, I think for what’s important to me there, it is that we get stuff done. And we do whatever it takes to own the outcomes that we want to achieve and try to manage people’s—try to not screw up people’s careers along the way. [laugh]. Better put, I want people to be proud of what they get done with me by the time they’re done with me. [laugh].Corey: So, I know you’ve talked to me about this very briefly, but I don’t know that as of the time of this recording, you’ve made any significant public statements about it. You are now over at Airbytes, which I confess is a company I had not heard of before. What do y’all do over there?swyx: [laugh]. “What is it we do here?” So Airbyte—Corey: Exactly. Consultants want to know.swyx: Airbyte’s a data integration company, which means different things based on your background. So, a lot of the data engineering patterns in, sort of, the modern data stack is extracting from multiple sources and loading everything into a data warehouse like a Snowflake or a Redshift, and then performing analysis with tools like dbt or business intelligence tools out there. We like to use MetaBase, but there’s a whole there’s a whole bunch of these stacks and they’re all sort of advancing at different rates of progress. And what Airbyte would really like to own is the data integration part, the part where you load a bunch of sources, every data source in the world.What really drew me to this was two things. One, I really liked the vision of data freedom, which is, you have—you know, as—when you run a company, like, a typical company, I think at Temporal, we had, like, 100, different, like, you know, small little SaaS vendors, all of them vying to be the sources of truth for their thing, or a system of record for the thing. Like, you know, Salesforce wants to be a source of truth for customers, and Google Analytics want to be source of truth for website traffic, and so on and so forth. Like, and it’s really hard to do analysis across all of them unless you dump all of them in one place.So one, is the mission of data freedom really resonates with me. Like, your data should be put in put somewhere where you can actually make something out of it, and step one is getting it into a format in a place that is amenable for analysis. And data warehouse pattern has really taken hold of the data engineering discipline. And I find, I think that’s a multi-decade trend that I can really get behind. That’s the first thing.Corey: I will say that historically, I’m bad at data. All jokes about using DNS as a database aside, one of the reasons behind that is when you work on stateless things like web servers and you blow trunks and one of them, oops. We all laugh, we take an outage, so maybe we’re not laughing that hard, but we can reprovision web servers and things are mostly fine. With data and that going away, there are serious problems that could theoretically pose existential risk to the business. Now, I was a sysadmin and a, at least mediocre one, which means that after the first time I lost data, I was diligent about doing backups.Even now, the data work that we do have deep analysis on our customers’ AWS bills, which doesn’t sound like a big data problem, but I assure you it is, becomes something where, “Okay, step one. We don’t operate on it in place.” We copy it into our own secured environment and then we begin the manipulations. We also have backups installed on these things so that in the event that I accidentally the data, it doesn’t wind up causing horrifying problems for our customers. And lastly, I wind up also—this is going to surprise people—I might have securing the access to that data by not permitting writes.Turns out it’s really hard—though apparently not impossible—to delete data with read-only calls.swyx: [crosstalk 00:28:12].Corey: It tends to be something of just building guardrails against myself. But the data structures, the understanding the analysis of certain things, I would have gotten into Go way sooner than I did if the introduction to Go tutorial on how to use it wasn’t just a bunch of math problems talking about this is how you do it. And great, but here in the year of our lord 2022, I mostly want a programming language to smack a couple of JSON objects together and ideally come out with something resembling an answer. I’m not doing a whole lot of, you know, calculating prime numbers in the course of my week. And that is something that took a while for me to realize that, no, no, it’s just another example of not being a great way of explaining something that otherwise could be incredibly accessible to folks who have real problems like this.I think the entire field right now of machine learning and the big data side of the universe struggles with this. It’s, “Oh, yeah. If you have all your data, that’s going to absolutely change the world for you.” “Cool. Can you explain how?” “No. Not effectively anyway.” Like, “Well, thanks for wasting everyone’s time. It’s appreciated.”swyx: Yeah, startup is sitting on a mountain of data that they don’t use and I think everyone kind of feels guilty about it because everyone who is, like, a speaker, they’re always talking about, like, “Oh, we used our data to inform this presidential campaign and look at how amazing we are.” And then you listen to the podcasts where the data scientists, you know, talk amongst themselves and they’re like, “Yeah, it’s bullshit.” Like, [laugh], “We’re making it up as we go along, just like everyone else.” But, you know, I definitely think, like, some of the better engineering practices are arising under this. And it’s professionalizing just like front-end professionalized maybe ten years ago, DevOps professionalized also, roughly in that timeframe, I think data is emerging as a field that is just a standalone discipline with its own tooling and potentially a lot of money running through it, especially if you look at the Snowflake ecosystem.So, that’s why I’m interested in it. You know, I will say there’s also—I talked to you about the sort of API replication use case, but also there’s database replication, which is kind of like the big use case, which, for example, if you have a transactional sort of SQL database and you want to replicate that to an analytical database for queries, that’s a very common one. So, I think basically data mobility from place to place, reshaping it and transferring it in as flexible manner as possible, I think, is the mission, and I think there’s a lot of tooling that starts from there and builds up with it. So, Airbyte integrates pretty well with Airflow, Dexter, and all the other orchestration tools, and then, you know, you can use dbt, and everything else in that data stack to run with it. So, I just really liked that composition of tools because basically when I was a hedge fund analyst, we were doing the ETL job without knowing the name for it or having any tooling for it.I just ran a Python script manually on a cron job and whenever it failed, I would have to get up in the middle of night to go kick it again. It’s, [laugh] it was that bad in 2014, ’15. So, I really feel the pain. And, you know, the more data that we have to play around with, the more analysis we can do.Corey: I’m looking forward to seeing what becomes of this field as folks like you get further and further into it. And by, “Well, what do you mean, folks like me?” Well, I’m glad you asked, or we’re about to as I put words in your mouth. I will tell you. People who have a demonstrated ability not just to understand the technology—which is hard—but then have this almost unicorn gift of being able to articulate and explain it to folks who do not have that level of technical depth in a way that is both accessible and inviting. And that is no small thing.If you were to ask me to draw a big circle around all the stuff that you’ve done in your career and define it, that’s how I would do it. You are a storyteller who is conversant with the relevant elements of the story in a first-person perspective. Which is probably a really wordy way to put it. We should get a storyteller to workshop that, but you see the point.swyx: I try to call it, like, accessibly smart. So, it’s a balance that you want to make, where you don’t want to talk down to your audience because I think there are a lot of educators out there who very much stay at the basics and never leave that. You want to be slightly aspirational and slightly—like, push people to the bounds of their knowledge, but then not to go too far and be inaccessible. And that’s my sort of polite way of saying that I dumb things down as service. [laugh].Corey: But I like that approach. The term dumbing it down is never a phrase to use, as it turns out, when you’re explaining it to someone. It’s like, “Let me dumb that down for you.” It’s like, yeah, I always find the best way to teach someone is to first reach them and get their attention. I use humor, but instead we’re going to just insult them. That’ll get their attention all right.swyx: No. Yeah. It does offend some people who insist on precision and jargon. And I’m quite against that, but it’s a constant fight because obviously there is a place at time for jargon.Corey: “Can you explain it to me using completely different words?” If the answer is, “No,” the question then is, “Do you actually understand it or are you just repeating it by rote?”swyx: right.Corey: There’s—people learn in different ways and reaching them is important. [sigh].swyx: Exactly.Corey: Yeah. I really want to thank you for being so generous with your time. If people want to learn more about all the various things you’re up to, where’s the best place to find you?swyx: Sure, they can find me at my website swyx.io, or I’m mostly on Twitter at @swyx.Corey: And we will include links to both of those in the [show notes 00:33:37]. Thank you so much for your time. I really appreciate it.swyx: Thanks so much for having me, Corey. It was a blast.Corey: swyx, head of developer experience at Airbyte, and oh, so much more. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice or if it’s on the YouTubes thumbs up and subscribe, whereas if you’ve hated this podcast, same thing, five-star review wherever you want, hit the buttons on the YouTubes, but also leaving insulting comment that is hawking your book: Why this Episode was Terrible that you’re now selling as a legitimate subject matter expert in this space.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
Comments (1)

Felipe Alvarez

it seems the volume changes from high to low every few seconds. please fix?

Jun 10th
Reply
Download from Google Play
Download from App Store