Ep. #19, Have We Forgotten How To Code?

Paul Biggar: There's people who write the tools and there's people who use the tools.

Edith Harbaugh: Why don't you just grind your own wheat and make your own bread from scratch and eat it at lunch also?

Paul: This is why there's always haters on Hacker News.

Software is a very fashion-oriented world.

Edith: Well, I think it goes back to the medieval times.

Paul: There's wildly varying positions on what this means for open source.

Edith: Okay, what are our points of failure? Let's not make that a point of failure again.

Paul: Each of these incidents brings the Node ecosystem quite a bit forward.

Hi, I'm Paul Biggar, founder of CircleCI.

Edith: I'm Edith Harbaugh, CEO and co-founder at LaunchDarkly.

Paul: And you're listening to To Be Continuous, a podcast about continuous delivery and software development.

Edith: You can get in touch with us anytime at our Twitter handle, @ContinuousCast.

Paul: The show is brought to you by Heavybit. To learn more, visit heavybit.com. And while you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.

The Left-pad Incident

Edith: Hey Paul, so somebody unpublished a module called left-pad and the internet basically exploded. And now it's been a couple weeks since then, so there's a lot of different topics to cover in terms of what this means for open source, whether it means that people have forgotten how to program.

Paul: So the LaunchDarkly, sorry, the left-pad... I'm obviously looking at my own T-shirt.

Edith: You're looking at your LaunchDarkly T-shirts?

Paul: The left-pad incident was pretty interesting, and one of the things that we did when coming up to this is just looking at all of the Hacker News discussions and comments on it. There's wildly varying positions on what this means for open source, what it means for whether people can program, what it means about the Node ecosystem. So to kick this off, why don't you give us a recap of the entire incident?

Edith: Yeah. So in short, a programmer had a module called kik, K-I-K, which is also a very popular messaging app. I mean, even I have heard of it, and I'm not a teenager. The Kik lawyers contacted the dude, said, "Please take down your module, change it."

And the programmer said, "No, I basically don't believe in copyright laws." So then the Kik people contacted npm. The npm people took down the module called kik.

And the guy, in, I'd say, kind of a fit of pique, said, "Okay, you took down my one module. I'm gonna take down all my modules." Basically, I'm taking my marbles and going home.

Paul: Okay.

Edith: So he took down all of his modules, one of which turned out to be an incredibly popular module called left-pad, which said basically what the name implied. It inserted left pads. And without this, an npm, the quote I read was, "Builds all over the world started to fail very quickly."

Paul: So as I understand it, left-pad was used by another module, which was used by another module, which was used by something core, or like this incredibly popular module.

Edith: Yeah, it was basically turtles on top of turtles on top of turtles, and at the bottom turtle was this guy who...

Paul: Took his marbles and went home.

Edith: Yeah, or yanked the bottom Jenga, basically.

Paul: Gotcha. And so all the builds started to fail. People's deploys started to fail. And people got pissed.

Edith: People got pissed because they didn't even realize that they had this dependency. So what happened was npm basically republished it. They said that even though you're the original publisher...

Paul: So there was a lot of outcry on the internet, and then they republished it.

Edith: npm just basically said, "You don't have the rights to unpublish this."

Paul: Gotcha. But they had an unpublish button, as I understand it. So he did something that was built into npm to remove a module.

Edith: Yeah, and since then they've tightened their rules a lot. They've basically said that this should be something that's used extremely rarely, if at all, and not something that you could just do capriciously.

Paul: So one thing that's interesting about the npm ecosystem, and the Node ecosystem in general, is they've had a bunch of things like this. Things that are security related or that are drama related or... And people have repeatedly come out and say, "Oh, Node.js is a tarpit or ghetto," or whatever it is that people use to describe poor ecosystems, that Node programmers don't know how to code, and so on and so on.

And what it looks to me like is each of these incidents brings the Node ecosystem quite a bit forward, to the point that, I mean, Node is relatively new. It hasn't had fifteen years before prime time in the way that Ruby or Python did, or twenty years before it got big.

And it's a real kind of bottom-up thing. You know, someone built it and there was some stewards, and then someone else took over, and then npm took over, and it's this sort of weird language from an ecosystem perspective, and it didn't have the same advantages that, like, Java had, or that C or C++ had in order to get to maturity.

And so each of these incidents propels it closer to maturity.

Edith: Yeah. I compare it somewhat to when electricity first started coming out. And then you were just working through all the issues that could happen. Like, for example, when a power line went down or transformer blew.

You work through, okay, what are our points of failure? Let's not make that a point of failure again.

Paul: Mm-hmm. So npm changed their policies on this basis, right?

Edith: Yep. So now you can't have an individual contributor being able to unpublish even if they originally published. Now you have to contact them, and it's much more stringent.

Paul: Gotcha. Their response, despite people being... well, people were annoyed at their response.

Edith: It seems perfectly rational to me.

Paul: Oh, I think it seems perfectly rational, but this is the thing about the internet. There's always some set of people who are annoyed at everything you do.

Edith: Well, you know, the internet is full of billions of people. I mean, there's gotta be some sort of standard deviation, or otherwise it would be a very boring internet.

Paul: Right, right. And this is why there's always haters on Hacker News. No matter what you do, if you do it one way, there'll be one group of haters. If you do it another way, there'll be a different group of haters.

Who Owns a Namespace?

And so with left-pad, there was people who were annoyed that npm retroactively decided that they had ownership over their namespace, which previously they had sort of gifted to people in the community, the first people to take the namespace.

Edith: Oh, you mean the original thing about Kik?

Paul: So there's that, but there's also the left-pad. They took away the sort of autonomy of the left-pad author.

Edith: I read a good article that basically says part of open source is, if it truly is open source, do you maintain the rights to it, or is it a community project?

Paul: The traditional way of deciding that sort of ownership was the fork.

Edith: Yeah.

Paul: That you could go and you could fork, and when you forked, you weren't necessarily entitled to the same name, you weren't entitled to the same trademarks. When Debian forked Firefox, they called it Iceweasel. When Jenkins was forked, or Hudson was forked, it became Jenkins. Even Emacs has had multiple forks, and GCC got forks, and they all got new names.

But what was the fork here? It wasn't even a fork. It was a republish of the same thing, the same software, in a namespace that the original author didn't control.

So typically you'd own the URL, you know, if you have a software library, you'd own the URL. And they did own, in some sense, the URL within npm, or the name within npm space, and npm, the sort of owners of that namespace, overwrote it.

Edith: And I think they did what was necessary for good order.

Paul: Sure, I completely agree. I think that was a good decision.

Edith: I mean, to take this to a logical conclusion, what if you had a module named left-pad and you decided that, like, as a prank, you were gonna rewrite it to be right-pad? Like, sure, is that within your rights? Kind of, but it's a real jerk thing to do that you probably want to prevent.

Paul: This is kind of one of the dangers of the whole ecosystem around centralized package managers, of sort of trusted package managers, where there's no actual trust involved. So what I mean by that is what is published does not have any gatekeepers.

Edith: Yeah, exactly.

Paul: And it's great. It allows an ecosystem to grow much faster.

But it has safety concerns because people generally don't pin their packages to an exact version.

Edith: Well, you're basically pinning it to floating value.

Paul: You're pinning it to a floating value. And that floating value could change. You know, there's no guarantee of what a minor number means, what a major number means, of what a patch number means, of what a patch even means, right?

There's some sort of, like, general community guidelines in semantic versioning. But there's no enforcement of that.

Edith: Yeah, and I mean, this basically goes back to the whole cathedral versus bazaar thing.

Paul: Right, right. It's very bazaar.

Edith: Yeah. It's, well, bizarre and bazaar.

Paul: And intentionally so. It's the thing that allows a community to grow. And is there a point at which open source communities need to, like, batten down the hatches to prevent that sort of behavior? Or to prevent that sort of risk?

Edith: Yeah, and it's really interesting. I think this goes back to something we talked about with Nadia and Sean a couple episodes ago, about when you buy from a corporation, you have some sort of guarantee of standards that they publish somewhere that, you know, "This is our guarantees," versus with open source, you don't normally have those same sort of guarantees.

Paul: Even when you do, it tends to be...

Edith: There's no enforcement mechanism.

Paul: Right. So there's no enforcement mechanism in the bazaar.

Edith: Well, there's no, like... So with a company, if a company has said, "Well, okay, we're gonna release twice a year and it's gonna have this level of quality," and they break that, you can walk with your feet or your money.

Paul: Right, right, right.

Edith: If an open source project stops or does something that you don't like, there's very little enforcement.

Paul: And theoretically you could walk, you know, you could fork the thing, but when you end up in an ecosystem that has massive network effects in the way that npm does, you can't walk with it. Because the deps are all transitive. The left-pad was, you know, a turtle way at the bottom of the stack.

Edith: It was almost like a turtle hatchling.

Paul: So to fork left-pad, you would have to fork the entire npm ecosystem.

Edith: Yeah, and it's not even... the fork is not the issue. The issue is that people were depending on it being in a certain place in a certain way. It wasn't that it wasn't forked, it was just like this entire structure at the bottom was gone.

Paul: I guess what I'm saying is that if you get pissed off with a certain set of developers or a certain set of practices within the npm community, because they're so interlinked, you don't have the traditional open source option of forking.

Edith: Absolutely.

Paul: You could rename it, you could pick a different library, but because of the way... and I'm not sure if they still do this, because I read something about it, but npm, it used to be that the version of the software that your library took was the version that library got.

Do you know what I mean by that? Okay. So let's say I've... there's some software, let's call it left-pad.

Edith: All right. Let's call it right-pad.

Paul: Right-pad. So right-pad version one is being used by library A, and right-pad version two is being used by library B. In the Java ecosystem and many, many other ecosystems, only one version of right-pad will be used.

Edith: Yeah.

Paul: And I believe in npm, both versions of right-pad will be used. So library A will use version one and library B will use version two. Now, that's definitely the way that it used to be. I'm not sure if they've actually changed that.

Edith: Yeah, I don't know, but that sounds very messy very quickly.

Paul: So both options are terrible.

Edith: Yeah.

Paul: And there isn't a right way to do it. So really, what it depends on... So the npm way of doing it, every library gets its own version of the dependencies, is fantastic, because it lets that library do what it actually intended to do under the covers.

If, however, you start, you know, having an object that gets sent around, that two modules are, you know, let's say it receives a, I think a HTTP packet or something like that would be a good example. Let's say you take a HTTP packet from one module, you send it to another module, and they have different understandings of the shape of that object or what functions you can run on that object, then that's messy.

Edith: Yeah.

Paul: But on the other hand, if you override the version of right-pad that some other library is using, you end up with all sorts of problems. And this is a major problem in the sort of the Maven ecosystem.

At CircleCI we used Clojure, and Clojure is built on top of the Maven ecosystem, and we would have bugs where the library that we used... One library that we used had a dependency, and that was in the package file before another thing. And so that's the version of the package that we used.

So we would have to pick... and to resolve that, we have to pick the library that works. And often there isn't a library that works for both, or there isn't a sort of sub-dependency that works for both. So there's a clusterfuck no matter which way you do it.

Edith: Yeah, I mean, we run into the same issues with our SDKs, because we have dependencies on other libraries that sometimes conflict.

Paul: Right. Disaster.

Have We Forgotten How to Program?

Edith: On the other hand, I mean, the bigger point, I think, was... So there was an interesting backlash, I think, that everybody kind of projected what they wanted on the npm thing. People said, "Oh, people don't know how to program anymore. You know, why do you even use something called left-pad? You know, why don't you just grind your own wheat and make your own bread from scratch and eat it at lunch also?"

Paul: I have no sympathy at all for this view of the world.

Edith: That you should make everything from scratch?

Paul: Oh, we've forgotten how to program because we left-pad.

Edith: Yeah, the fact that we have modules available that make it easier and easier and easier to program is a good thing.

Paul: I think it's kinda different to that. I think it's more of a curmudgeon view of the world. It's a get-off-my-lawn.

Edith: Wait, that you're the curmudgeon?

Paul: No, the people who complain, "Have we forgotten how to program anymore?" I think it's a No True Scotsman kind of argument.

Real programmers know how to write their own left-pad. Real programmers write in statically typed languages.

Real programmers write in C. Real programmers, and I'm going back in time here, real programmers write in assembly. Real programmers hand code their... tweak the needles on the...

Edith: They do their own garbage collection.

Paul: Oh, garbage collection. What is this? Garbage collection is for amateur coders, right? You could just as easily have written, "Have we forgotten how to manually allocate memory?"

Edith: Or you might as well say, like, nobody knows how to make their own transistors anymore. I mean, like, the entire computer revolution has been that you no longer have to assemble a computer by hand.

Paul: Right. And each layer of the ecosystem allows a vastly larger number of people to take part in it.

So I think the thing that we're seeing now, and this has been happening for five or ten years, but what's getting really, really big now is people who only know how to front-end program. People who only know JavaScript and HTML and CSS, and they go to boot camps and they come out, they maybe have some, like, vague Rails understanding. And they call themselves coders, and we, the real programmers who know how to write a quicksort and who know how to write a left-pad, we don't welcome them to our community.

And frankly, I mean, I have felt this in the past. I've since changed my mind. But it's a very, very common perspective.

Edith: Well, I think it goes back to the medieval times.

Paul: Mm-hmm. Sure, no, I totally see it.

Edith: The entire reason why there were guilds and apprenticeships was because people didn't want their knowledge to be cheap.

Paul: Right.

Edith: Like, so it used to be, so the reason why being a doctor takes seven years is because they wanted it to be something that only a few could do. And that they could then charge a lot of money.

Paul: Right. You can protect your industry as a... to protect your own value to society.

Edith: Yeah.

Paul: Or just your earnings even.

Edith: Your earnings. So if somebody could go to boot camp and learn for six months and become an adequate coder in front-end, suddenly it's very hard to justify charging, you know, or getting paid, like, $180,000.

Paul: Right, right, right. The more programmers there are in the ecosystem, the more people available to hire, the less valued your skills become.

Edith: Yeah, and I'm certainly gonna argue both sides. I think a talented programmer does have a deeper knowledge of the bubble sort, the quicksort. But I think there is a place in the world, and a welcoming place, to be honest, for people to do lighter coding, and that those people with more time on the job can become more sophisticated.

Toolmakers and Tool Users

Paul: One of the ways that I learned to look at programmers is that there's people who write the tools and there's people who use the tools.

Edith: Yeah, exactly.

Paul: Right. And to say... I think a lot of people would think, "Oh, you know, only real programmers write the tools." And I was susceptible to this because I worked in compilers, and we wrote the tools that the toolmakers wrote.

Edith: Did you have a factory that made mini factories?

Paul: I wish. So I think what we see with people who are kind of at the end of the chain, the people who are harnessing the libraries and the npm ecosystem and the browser and that sort of thing to build really fast, really, like, innovative things, that frankly the toolmakers would never have thought of.

And so I don't think that looking at the world who, you know, toolmakers are the real programmers and the other guys are kind of, they use the tools. I think it's not a very good way to look at the world. I think, you know, everyone's focusing on where their real value is.

Edith: Yeah. I think a lot of the wave of innovation that we've seen in the last 10 years has been because programming is a lot easier.

Paul: Right, exactly.

Edith: Like, the fact that there's, you know, Uber was because you didn't really need a lot of fancy programming when they first started out. I mean, now they have incredibly complex programs. But just to get a basic...

Paul: We're seeing this happening in the AI revolution that's happening at the moment. Facebook launched its platform today, and I'm not sure what the platform does exactly, but, you know, now to build bots is infinitely cheaper than it was, like, a year ago. And building AI and connecting people to these things is so, so much easier than it was a year ago, and a million times easier than it was, like, ten years ago.

Edith: Yeah. I mean, I remember, like, it used to be, to even get, like, a basic web app up, you had to have knowledge all the way up and down a stack.

Paul: Right.

Edith: And now with AWS and Heroku and DigitalOcean, like, you could get stuff up just much quicker. And then you could focus on what's the true business value of what you're building, instead of just, it's so hard just to get words on a page.

Paul: I've been talking, I'm not sure if I've talked about it on the podcast, but certainly this is an idea that I've been running through with a while. In the future, people are gonna build with, like, very, very simple understanding of, or possibly no understanding of, the tools beneath them.

You're gonna be able to build something that's like the level of complexity of, like, a Tinder or a Snapchat with just one product manager who's not really a coder in the way that, you know, maybe now, or maybe five years ago that we thought of a coder. But they're going to be building applications with barely any coding knowledge at all.

Edith: And I think that's great.

Paul: Oh yeah, completely agree.

Edith: I compare it to the Industrial Revolution, where it used to be that just getting power was a task in and of itself. Like, the reason why there were mills on streams was they had to locate them there so they could grind the wheat. Suddenly when there was electricity, you could focus on, "Hey, not everything has to be around a river. Let's build factories everywhere. Let's make cool stuff."

And that was the... it wasn't just about generating power, it was, what can we build now that we have power?

Paul: Right. And as a necessary part of this, you end up with something like left-pad. Left-pad is a little bit trivial in that sense, but I think that one of the most entertaining parts of this thing is that left-pad had a quadratic bug in it. So accidentally the left-pad had some form of quadratic behavior in it.

And no doubt it would have got fixed in a later version, and everyone got automatically upgraded, and so on.

But the idea that you're coding your own version of left-pad means that you're probably going to have quadratic behavior in it, and that's going to be used to exploit your system at some later date in the future.

Edith: Yeah, I mean, I totally agree with your metaphor that there'll be less and less coding and more and more value.

Paul: Right.

Edith: I mean, I do think, though, to argue the other side, I do think, though, that I'm thinking more about what you said about the people who build the tools.

Paul: Okay.

Edith: I do think they need to have a different skill set then. I mean, I go back to the...

Paul: Yeah, I mean, you have a different set of users, you have a different set of responsibilities, and they do different things and have different properties.

Edith: Yeah, it reminds me very much of... so I love Richard Feynman. I read all his books when I was growing up. And he talked about how he'd learned so much from assembling and reassembling radios.

Paul: Okay.

Edith: Which just wasn't something that was possible when I was a kid, because we'd gone to solid-state electronics.

Paul: Oh, okay. Right, right.

Edith: And I think we're starting to see a similar transition in coding.

Paul: That people can't take apart their things?

Edith: I think in twenty years people will be amazed at how much access we had to different...

Paul: Different components, like how open the ecosystems were, that sort of thing.

Edith: Yeah, it's just that things that we take for granted now will become completely opaque.

Open Ecosystems and the Best Product

Paul: So people are lamenting this and have been lamenting this for a long, long time. So one of the first comments I saw on Hacker News on the Facebook Messenger platform thing today is, you know, "Whatever happened to the open ecosystems? We had IRC. Do we really need this thing?"

And I agree to a certain degree that the open ecosystems are...

Edith: You agree to a degree?

Paul: I agree that open ecosystems have some value. What I think that people often don't get, especially in the open source ecosystems, is that usually the best product wins. And the openness, basically no one cares.

And especially now that these tools go from early adopter to, like, mass market in such a short period of time. Like, IRC is a fucking wasteland. All the tools around AIM and Pidgin and that sort of thing around the IM ecosystem are terrible.

XMPP, the protocol, doesn't allow you to do anything remotely like what Facebook platform or what WeChat, which is what Facebook platform is going to become. And as a result, they lose. Right? I mean, primarily they don't have nine hundred million people on them.

Edith: Yeah, you know, when you were talking, it reminded me very much of Steve Jobs and the Apple computers. 'Cause originally the computers were in the province of the hobbyists who would literally, like, build them from hand from parts, like Radio Shack.

Paul: Yep.

Edith: And then he said, "Okay, now it's this closed box that you can't open."

Paul: Yeah.

Edith: And everybody's like, "You're not gonna sell any," and he sold gajillions.

Paul: So he eventually sold gajillions, right? And if you're in San Francisco, you'll see that there's only MacBooks. Like, if you see someone using a Windows computer...

Edith: It's funny because I'm looking around the podcast room and there's six separate Apple products for three people.

Paul: Right. And there's actually five computers.

Edith: I blame our producer Ted for the proliferation.

Paul: So everyone in this ecosystem is familiar with Linux. At least all the engineers, let's say, within this ecosystem have used Linux, use Linux on the server. Maybe they use Linux as a kid, or, you know, when they were growing up, or at least in some way younger, on, you know, a Dell, or...

Edith: I remember when Linux was new.

Paul: Well. We'll just move on from that. What I'm saying, though, is it's not like we were unfamiliar with Linux. I mean, I use Linux for many years and I eventually got sick of it.

Like, Apple products were a much, much better product. And so I couldn't dig into the, you know, deep, deep down into... well, actually I didn't do that very much on Linux anyway.

Edith: Yeah.

Paul: I could, theoretically, if I learned all the stuff and all the vast, vast competing bullshit that goes on in the Linux development community, in the Linux desktop community, but frankly, I couldn't use Wi-Fi. And so I switched.

Engineers Who Design for Themselves

Edith: Yeah, I mean, it's an eternal debate of product manager versus engineers, except for if the product manager is Stockholm Syndrome.

Paul: So this goes back to our discussion from a few weeks ago about how open source needs product managers.

Edith: Well, no. I mean, I remember, this is when I worked for a content management system, and I would go on customer visits with our architect. And the architect had designed a system for architects.

Paul: Of course.

Edith: You know, like, where you could tweedle everything, you know, disassemble everything on the fly, had complete control. We go out to these newspapers or hotels, "I just want to put a picture of a horse on a page. Like, I do not want to twizzle all this stuff."

And it was amazing because the architect and I would go to the same meeting. We go back and he's like, "Did you see how excited they were about my configuration?" I'm like, "Marriott screamed at you."

Paul: Well, so actual Stockholm syndrome.

Edith: No, I was the product manager at the time, and I was just like, "No, they do not want your configuration. They want less configuration. They are tired of configuring."

Paul: So there are definitely engineers who can only design for themselves, can only kind of empathize with their own usage.

Edith: Yeah, it goes back to... I mean, have you read The Inmates Are Running Their Asylum?

Paul: I haven't.

Edith: It's a really good book. It's about, as a tin says, you know, engineers designing for... They think everybody wants all this control, and people just want to turn on the tap and have water come out.

Paul: And so going back to the left-pad debate, you know, a lot of the people who are screaming about the, you know, "Have we forgotten how to code?" are basically applying their worldview to a whole bunch of people who have a different worldview.

Edith: Yeah. I mean, to continue on the electricity example, like, no, I don't wanna have to do my own electricity. I just wanna flip the switch and have it come on.

Paul: Right, right. Oh, but I want to be able to dig into my house, and I wanna be able to change the voltage because I might have a thing that, something, something, something.

Edith: I might have brought back a hairdryer from Australia and urgently need to use it.

Paul: And I need one in every room, so every room has to have a one-ten thing and a two-twenty thing.

Edith: And that house is gonna burn to the ground.

Paul: Right. Yeah. And exactly, the system is built around having people's houses not burned down to the ground. And this is kind of, again, back to left-pad. The npm people have vested interest in not letting people's houses burn to the ground.

Edith: Oh yeah. I think it's like with anything, you don't realize it's an issue until it's an issue.

Paul: Right, right, right. What did they do to fix that? What was their...

Edith: Well, like I said, they changed their policy. They said you can no longer unpublish.

Paul: Okay.

Edith: It's for rare emergencies, and so basically a "contact us" instead of a button.

Paul: So they stood by the Kik naming decision.

Edith: Yep.

Paul: So they stood by the ability to remove the kik package. Good thing the kik package wasn't at the bottom of that turtle thing.

Edith: You know, that might have been an interesting thing. I mean, what if it turned out that the kik one was... that was actually...

Paul: Left-pad was kik.

Edith: It's interesting. I actually think the Kik people were in the right. I think it was something that was very easy to confuse. Kik is a well-known brand.

Paul: Right. So why was it called kik?

Edith: The guy said he thought... the original, the non-Kik guy, the programmer just said he liked the name.

Paul: Oh yeah, yeah. I mean, that happens all the time. You got a three-letter thing that sounds a bit like a common word in English. That's gonna happen.

Edith: And he also said that he didn't believe in copyright law.

Paul: Well, I mean, this seems like a very expected result of when you don't believe in a thing that all the rest of society believes in. Eventually society's gonna step in and just sort of move you aside.

Edith: Yeah, I thought the way you handled the Uber logo was very gracious.

Paul: Well, I mean, we don't own the idea of sort of two concentric circles.

Edith: What?

Paul: I know, right? Now, theirs was a square and a circle, so you might argue that it wasn't even like our logo, but, yes, certainly when I look at it, I see the circle logo and it reminds me every time.

Edith: Well, does it make you feel good that you can go all over the world now and see CircleCI's logo?

Paul: No, no, because it's not actual CircleCI's logo. It's Uber's logo. At least until they have their next top-down redesign.

Left-pad as a Microservice

Edith: Yeah. So I think the overall thing is I think the system is still working out what it means to really truly have microservices. I think that this is a good trend.

Paul: Sorry, microservices? How does this tie to microservices?

Edith: Well, I mean, the left-pad was basically a microservice.

Paul: Well, the left-pad was a library. Now someone made left-pad as a service.

Edith: LPaaS?

Paul: Yeah, and LPaaS is now a category in the way that normal PaaSes are. Someone took left-pad to put it behind a REST API and published it. Humorously, not for actual money, I think. I'm sure you could pay them.

Edith: Yeah, people pay for feature flags.

Paul: So I wonder what is their monetization then? Is there a management service behind left-pad that you can control, like, how much padding your developers can have? You can have an audit trail of how often people pad their strings, and you might want to make sure that your developers aren't padding strings too often.

Edith: I feel like you're teasing me, but I'm not completely sure.

Paul: No, no, no. The general way that people turn something like this into a service is they provide a management layer behind it. I mean, feature flags is the same thing. Your product is the management layer of the feature flags, not the idea that you can write an if statement in code.

Edith: Yeah, I mean, that's certainly why people are using us.

Paul: Right.

Edith: A feature flag is easy, management is hard.

Paul: It's the same with CircleCI. People don't buy it in order to run a test somewhere, because they have plenty of machines that they could run tests on. They buy the management service, the unifying place where everyone is doing this thing, or everyone on their team is doing this thing.

So back to the microservice idea. I think one of the difficult things that people have to deal with is where do you make the boundaries of your microservice. And I mean, if you're being strict on the use of the word micro, left-pad would very easily be, like, the level of micro that people are talking about.

I think that's a little small, personally. I kinda think of it as, like, we have our billing service. And then once billing gets small enough that we need to abstract it into, like, the PayPal service and the Stripe service and the invoicing service, then that's as a result of how large our architecture or our customer base is.

Edith: I don't know. I think it's an entirely appropriate size. I mean, it reminds me of, like, so I get asked a lot, what's the appropriate level of feature flag? And my basic answer is, like, wherever you want. Like, you could do it with literally with every check-in.

Paul: Wait, every check? Oh, you could have a different flag for every... Do people do that?

Edith: Sometimes.

Paul: That's interesting. Is that how people do their continuous deployment in a LaunchDarkly world?

Edith: And then we have other people who do it at a much, much higher level, just for, like, a major infrastructure project.

Paul: Gotcha. Oh, that's interesting. So you would have your... as you deploy a new web server, it would check the feature flag for what version that is?

Is that kind of the level that we're talking about? That's really cool.

Edith: Yeah. But I mean, I think of feature flagging as a technique, and I think people sometimes expect us to be a lot more prescriptive than we are.

Paul: Okay.

Edith: Like, in terms of, oh, how much... like what you said about microservices, how much should I have to feature flag? And I think I give what comes across as a bit wishy-washy, which is, like, however much you want.

Paul: Right, right, right, right.

Edith: But I think people think they have to... At one end, there's the people who have wholly bought into trunk-based development and they feature flag everything.

Paul: Oh, okay. Okay. I guess once you can feature flag everything, you buy into trunk-based, as opposed to, like, branch-based.

Edith: Yeah. So if you're full in on the... if you're full trunk.

Paul: That actually makes a lot of sense. I mean, if you start from the start as your branch is your feature flag, you probably get very, very good at making sure that you don't break shit as a result of it. And you also get very good at the idea that everything goes out with a feature flag.

One of the problems that we had with feature flags initially, when we started using it, was that people would build a new feature and it would start by, like, eviscerating the old feature. And then they would build a new feature in its place and it would look perfect. And then someone would say, "Oh, we should put a feature flag on this," and it's like, "Well, that would take me another two days. I'd have to go back, redo all this work, put it over there, put it behind a feature flag," and yeah.

I see that it got very difficult to build a competency of, we're building this other feature on the side and then we're gonna switch to it by a feature flag.

Edith: Yeah, I mean, the way I think of it is, you're building a new railroad track.

Paul: Right.

Edith: And so the first way is basically, there's this hilarious cartoon I've seen of, like, somebody trying to fix a railroad track while the train is moving. And, you know, you're like... and the feature flag way is more like, "Hey, let's build another track."

Paul: Yeah, yeah. And then we're ready to switch to it.

Edith: Yeah. So that reminds me of microservices. I think microservices is kind of a made-up term. So I think everybody's still kinda trying to figure out what does that actually mean. How micro and mini are these, and macro?

Paul: I mean, as far as I can tell, microservices are just service-oriented architecture. And they're really... anyone who's using microservices is largely building SOAs, and microservices is the common term that people use to describe it, and no one's really that concerned with making them micro.

Edith: Yeah. Why do you think SOA fell out of so much favor?

Paul: Because it got tied to things like SOAP, and, like, software is a very fashion-oriented world.

Edith: It's funny that way.

Paul: Yeah, it really is. I mean, maybe everything else is.

Edith: No, just terms, like, terms are popular. Like, CORBA used to be the thing.

Paul: Right, right, right.

Edith: And now people are like, "CORBA?"

Paul: Right. And CORBA is the kind of thing that you associate with service-oriented architectures.

Edith: Oh no, CORBA was before it.

Paul: I kinda put them in a similar box.

Edith: The box of old things.

Paul: Yeah, the box of old things that I touched once or twice, and I'm happy that I'm never near them again.

Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit and hosted by me, Paul Biggar of CircleCI, and Edith Harbaugh of LaunchDarkly.

Edith: To learn more about Heavybit, visit heavybit.com. While you're there, check out their library, home to great educational talks from other developer company founders and industry leaders.