Kin Lane asks Where Do Developers Get Idea That APIs Should Never Go Away?,

Why do developers feel so burned when a popular API like Netflix, ESPN, or Google Translate is deprecated? The main reason is lack of communication. Developers need a long runway that an API is going to be deprecated.

This isn’t why.

Developers hate it when APIs go away because it means they have to rebuild something they already built once, and because the discontinuance will cause apparent bugs in their product for their end users.

For reasons including process and toolset churn and environmental changes, developers often find themselves writing the same functionality over and over. Things that tend to encourage rewriting something already written, rather than allowing the developer to write something new, will cause frustration. It’s hard to get ahead on a treadmill.

Things the developer already sold will break, and the end users are uninterested in the decisions of some third party to deprecate an API.

The analogy with physical products

Kin again:

I cannot think of a single business, where you select a supplier of products or services you depend on for your business, without any chance of that company going bankrupt, shutting down, being sold, or possibly just discontinuing providing a particular products or services you depend on.

I’ve never sourced parts from vendors for a physical product (that shipped), but I can certainly imagine some irritation at a supplier which chose to discontinue a line of parts without some lead time. But what would we use that lead time for? In the case of a physical product, we’d look for replacements, think about redesigning our product to use other parts, and, crucially, we’d buy a stockpile of these parts in whatever amount we felt appropriate. Modulo difficulties like decay or cash on hand, in this situation we as the manufacturer can build in our own buffer for when we have to do something drastic like redesign our product. When we use a third-party API like Netflix or ESPN, there is usually no way to stockpile API requests for when we need them later.

We could cache, but there are a bunch of other circumstances that prevent us from caching as much as we’d like: the client might not have a lot of storage, terms of service might prevent us, the request and/or response might be time-sensitive, etc. Nevertheless, I think caching is a good fit for a lot of API uses, and API client developers don’t use it enough.

However, there’s a separate problem: programs which shipped weeks, months, or years ago will now break as the API they depend on goes away. Even if the product is caching, it’s unlikely to need only the same data over and over (else we would have just built that data into the product, no?). In the best case, the product functionality will degrade over a period of days or weeks, and in the more common worst case, the product turns into a brick or a useless waste of space on the devices of people who may well have paid good money for it.

It does happen in the physical product world, of course, that some products depend on third-party services or products after shipping. My appliances don’t work correctly without constant supplies of electricity and/or water. The furnace needs filters at regular intervals, and certainly didn’t come with a lifetime supply of them. My car needs oil and gasoline. There’s an interesting difference in the situation between an app that uses Google Translate and all these physical product scenarios, though. In the physical product world, the application user is expected to arrange supply of the third-party parts. In some cases, this is easy because there are multiple providers, and in some cases it’s easy because the product is essentially the same no matter what single supplier you have (water, for example). In the world of apps, this isn’t as easy, since APIs aren’t often compatible with each other, and even if they were, this kind of solution would need some place for the user to enter the API endpoint. Most API clients probably do their work without the end user ever being aware of the source API at all.

Sometimes there is a very specialized third-party product required for the end use product. This often results in the very specialized product being made available directly from the manufacturer of the end use product. I have a Nissan with a CVT, which requires a particular sort of fluid for the transmission, not widely available. Nissan sells this fluid itself, in effect caching the supply from the manufacturer and insulating me from any period where the manufacturer might discontinue the product, requiring Nissan to find another supplier. It might be that Nissan developed and manufactures this fluid in-house, but the point is that I don’t have to care, because my interface to it is through Nissan: they are hosting the supply needed by the product they sold me.

When a widely used third-party product is phased out, we often get a lead time greater than the mean lifetime of the products that use the third-party product! For example, automobile engines used leaded gasoline (at least in the US) to prevent knock, and when that was discontinued, automobile manufacturers and users were given about twenty years, or over twice the typical lifetime of a car in the 1970s.

Do we need to provide every API for twenty years? No. But we do need to provide an API (which is in use) for enough time that end users will usually not notice it has gone away, which means at least the typical lifetime of the product you encouraged use by. In general, people use application and software-based devices for times a lot shorter than the lifetime of a car (even a 1970s car), due to upgrade pressure from other sources. For a lot of applications, two to five years is probably enough time to provide an API after deprecation announcements.

Committing up front to supporting a (probably free) API for many years after you’ve decided to discontinue it might seem like quite a burden, and might even discourage an API provider from exposing it at all, so here’s some alternatives.

Deprecate before launch

Announce the deprecation information in the launch documentation. If, when it launches in 2015, the initial documentation for AwesomeAPI v1.0.0 says that v2.x.x will be preferred past May 2017, and that v1.0.0 will be discontinued in June 2019, client developers will feel secure in using your API if and only if they ought to feel secure for their use case. As Kin points out,

[client API developers] should carefully research your vendors, make sensible choices about why you use one over another

Providing full information about when an API will be deprecated before anyone is using it facilitates that research.

Use flexible architectures

Often we want to deprecate an API not because we’re not providing the information any more, but because there’s a newer API which has advantages for us or the clients, or both. The best deprecation is no deprecation at all, so try to design your API such that it can be extended and changed. Hypermedia can assist with this.

Encourage caching

Try to design your app to separate cachable information from time or location sensitive information, so that app functionality isn’t impacted where it need not be.

Use your terms of service to promote caching as much as possible.

Enforce lifetimes

Providing one to two lifetimes of product use before discontinuance of an API is only possible if the lifetime in question starts about the same time you announce deprecation. New cars sold in the US after the mid 1970s couldn’t use leaded gasoline, which meant that the deprecation clock had a definite start. If cars that needed lead had still been sold, as in the UK, the applicable lifetime would have started later, requiring some sort of shim replacement.

One way to avoid this is to have a term of service that prohibits developer distribution or sales of products using the API at some point after deprecation is announced. In this way, you can ensure that client developers are thinking about providing a new version of their product or some other workaround early. Or, if not, at least you have done what you could.

Client developers

Client developers should mirror many of the suggestions above. If it’s convenient to cache long term on the client, do so. Ideally, cache as much as possible for the expected lifetime of the app or device. Prefer using providers who support hypermedia APIs if you have a choice so that providers can make minor changes without disruption.

Ideally, if your app or device depends strongly on some non-unique API and the terms of service allow it, host an API that uses the source API in its backend. Controlling both sides of an API is the best way to ensure longevity, and if you’re caching aggressively on your server, it can be possible to switch to another backend provider without any change to your app or device at all.

Take away

So, learning some lessons from physical manufacturers, let’s sum up:

As an API provider:

• be clear about lifetime of the API
• ideally give notice of deprecation early enough for two product lifetimes (US leaded gasoline strategy), but one product lifetime is probably good enough
• announce limited lifetimes for your API at launch to assist client developers in planning
• allow caching when upstream licensing permits and it makes sense
• use API architecture that can be changed without breaking conforming clients (hypermedia, for example)
• consider terms of service requiring downstream deprecation

As an API client:

• cache on the client whenever convenient and permitted
• prefer hypermedia APIs which can (hypothetically) change without breaking your app
• host your own server and cache more aggressively on it if possible and permitted (Nissan’s CVT fluid strategy)

Speaking about fifteenth century Europe, Dan Jones:

In this age, men fight. That’s what they do.

I was listening to Dan Jones on the Diane Rehm Show the other day, and he made this comment in the middle of explaining that the society of the Wars of the Roses was much more militaristic than today’s western society, something she found surprising. The link above has a transcript; the exchange starts at 11:26:27, but the whole show is quite interesting, I thought.

In any case, there’s a widespread misconception (on all sides of the political arena) that modern times is particularly violent, when exactly the opposite is the case. Violent crime has been falling since the early 1990s in the US, and there’s a longer term trend toward massively lower violence rates in the West. So, when I read Kyle Bennett’s modest proposal, I wondered if strictly enforced rules against murder aren’t part of why we have this long term trend.

Perhaps Kyle is too optimistic about the number of people who, like him, have never thought about homicide. The homicidal ideation on Wikipedia suggests right up front that the number is closer to 1 in 2 than 1 in a million, and that’s for the lower bound of those who are willing to admit that to a researcher.

I have a lot of trouble getting started some days.

Part of this is the paradox of choice regarding all the stuff I need to do, but part of it is that it doesn’t matter much, for a goal the completion of which is weeks or months away, whether I get started this morning at 8am or 8:15am. Unfortunately, this doesn’t stop applying just because it’s 8:15am. This happens even with things I really, really want to do and am excited to do, except there it takes a slightly different form, which is that I feel as though I’m putting off getting to do the fun thing (without making it impossible to do the fun thing, as in putting off going to the movies, which can be put off until the theater isn’t open). That is, it feels oddly virtuous to avoid doing the fun, productive thing that I would like to do.

I’ve read about and tried various things to combat this.

• pomodoro timers
• limited work hours
• filtered or throttled internet

These are things that were unworkable or didn’t seem to help much. Throttled internet isn’t especially compatible with my usual methods (I’m googling around for quick answers to questions a lot of my day), and limited work hours just meant that I felt less guilty when I knocked off for the day, assuming I’d get more done tomorrow. The 25 minute timer is a great idea if you’re not having much trouble getting started, or if length of work is the primary problem. Of those, the timer plus limited work hours together came closest to helping, since not starting the timer at time X would mean at least one fewer block of work that day, which is an effect that I can wrap my subconcious around immediately. The small size (so it’s easier to get started) is in direct conflict, though, with the number of units being small enough to have a major effect on my perception of how much is lost for the day if I don’t start right away.

I found two things that really help.

• other-directed scheduling
• close deadlines

The thing these have in common is that they require other people to assist. Someone else has to be available to have a check-in at the start of the workday, or someone has to be prepared to be satisfied or disappointed depending on the status of a deadline.

Neither of these were a huge problem when I was going to work where other people were depending on my meeting deadlines (which were always only a few days away at worst), and expecting me to show up for scrums. Working alone has not provided that external motivation.

Hence this idea: scrums as a service.

One of the features of Agile development is the daily “scrum”, where everyone on the team meets, or calls in, and gives a brief progress report and sets expectations for the next work period. The purpose of all this is to give everyone on the team an idea of where the project is, and to provide an opportunity to resolve problems that require collaboration between parts of the team who don’t normally interact, and so on. However, for the purposes of this post, the part that seemed most helpful was the schedule and inherent deadline. Setting a time to meet means that at that time the people involved are working – are in a working mindset, ready to talk about what was accomplished since the last scrum, and what they hope to accomplish before the next. Doing this at the beginning of the workday seems most helpful to me for getting started, and doing it at the end seems most helpful for having an impending deadline. When I’ve worked in a shop with scrum, only a single daily scrum was usually used (separate meetings, of course, were often scheduled at the scrum), but I wonder if, for procrastination assistance, a start-of-day and end-of-day meeting might be helpful.

Bookending the workday with these meetings could have other salutary effects, such as helping set boundaries between work and home, something that those of us who work out of the home often have issues with already. In an ordinary daily cycle, there would be a push to get as much done as possible between start and finish, and a clear delineation of the workday’s end.

I imagine signing up for such a service and setting my start and end meeting times and methods (Skype, Google Hangout, Vidyo, whatever), and then calling into a start meeting, where the “scrum master” gets started, calling on those present to describe what they hope to do today (or did since the last meeting; perhaps there would be separate groups for those who do one meeting a day and those who do two), managing the discussion to prevent monopolization and to bring things to an close 10 or 15 minutes in.

Some differences from scrums I’ve participated in would be natural; after all, since this is a service, people wouldn’t necessarily have any idea what others were talking about, at least at first. Also, this would probably be less useful for those in a stealthy startup or whose company wouldn’t allow them to discuss what they did even in bite-sized form. Perhaps for such folks, the “scrum” could be just the scrum master and the worker, somewhat more expensively. Also, this is a motivational tool rather than a development methodology, so burndown charts and other agile paraphernalia would be missing, as I envision it.

Someone focused on one or two adjacent timezones might have two runs of 15 minute meetings, 7am to 9:30am at the half hours, and 2:30pm to 5pm similarly. At a maximum of 10 people each, that’s 120 person-meetings per day (can I use “person-meetings” as a unit of measure?). Assuming a cost of $75/person-meeting/month, so a person like me who would prefer a start and stop meeting every day would pay $150/mo, this would top out at $108K/yr before accounting, overhead, and taxes. Maybe a little too low to start as a scrum master in DC or SF, but possibly competitive in much of the rest of the US.

Does this service exist already? Let me know @randallsquared or randall@randallsquared.com.

I found Dave Cheney’s post about options in function signatures interesting.

I have, of course, been guilty of adding a new parameter for each option until the signature is ridiculously unwieldy. Moving on to his example of a config struct, this seems like a very common solution to this problem in Javascript (this was advocated by folks I’ve worked with in the past), but it’s always struck me as inherently opaque. I wanted to know how to get a NewWidget, and now I have to read all about these Widget options to make sure everything is how I think it is! Also, there’s the zero value issue Dave mentions, but that’s a problem specific to structs (and Go), since this pattern in Javascript can differentiate between 0 or null and undefined. Anyway, I agree that it’s probably a better solution than ever-longer parameter lists.

The initial example of functional options left me quite unconvinced. I will include this one.

func NewServer(addr string, options ...func(*Server)) (*Server, error)

func main() {
    srv, _ := NewServer("localhost") // defaults

    timeout := func(srv *Server) {
        srv.timeout = 60 * time.Second
    }

    tls := func(srv *Server) {
        config := loadTLSConfig()
        srv.listener = tls.NewListener(srv.listener, &config)
    }

    // listen securely with a 60 second timeout
    srv2, _ := NewServer("localhost", timeout, tls)
}

At this point, I was thinking “Why does the user of NewServer need to know so much about the internal structure of a new srv?” It seemed like a major step backwards. But then it became apparent that that wasn’t what was intended. Instead, the idea was to expose config functions from the configured package to be used in whatever way is appropriate (I’ll come back to that).

config := loadTLSConfig()
srv, _ := serverPackage.NewServer("localhost", serverPackage.setTimeoutSeconds(60), serverPackage.useTLS, serverPackage.useTLSConfig(config))

This isn’t the code he used, but it preserves the important details from his penultimate example, I believe. A minor nit, here, is that I’d prefer everything that is similar look similar, so passing a function that just sets a flag should look the same as passing one to which you have to provide an argument. But here’s the thing that I don’t get about using functional parameters: if you’re going to expose a bunch of configuration functions anyway, why bother passing them around? Instead we might have a bunch of methods on the result:

config := loadTLSConfig()
srv, _ := serverPackage.NewServer("localhost")
srv.setTimeoutSeconds(60)
srv.useTLS()
srv.useTLSConfig(config)

The only reason I can see is to centralize error handling, which (A) only matters for Go, and (B) it removes useful choices from the user of the package. “Okay, so the cert we were given is invalid, but let’s start up the non-TLS part of the site anyway and complain”. This is a decision that has to be made at useTLSConfig-time, and shouldn’t be buried.

I don’t think that we can settle on a single pattern for configuration. Sure, having separate constructors for all possible permutations is silly. Sure, there are issues with the difference between default and zero values (though passing maps in Javascript doesn’t have this problem, as mentioned above). But there really are major differences between a server that uses TLS and one that doesn’t; if there’s a single option that implies a whole different set of configuration parameters, it’s reasonable to create a different constructor for it, I would think.

So we have separate entry points or constructors, structs, parameters, functional options, and methods.

When we’re designing the API for our package, we have to consider all of these, and use the appropriate ones for those options that are best suited for them.

• constructors for distinct things (in usage, if not type)
• structs when you need a bag of scalar options
• parameters to the constructor for required options
• methods for non-required options where failure to apply the option can result in decisionmaking
• functional options for non-required options that cannot fail to be applied or where failure means the whole constructor must fail

A friend quoted this observation about walking behavior from reddit to me:

Walking towards somebody. The person who is walking the fastest is usually the person that makes room for the person walking slower. If you don’t feel like moving to the side when you’re walking towards somebody - slow down below their pace. Works all of the time, most of the time.

I would say that the person walking faster is responsible for not causing a collision. This could be read as the same thing, but isn’t quite: the faster person might slip through a quickly closing gap to exit the collision space, or might just walk around the slower person rather than “moving to the side” so that the slower person can pass. It applies more widely than only walking toward a facing person, too, since “don’t cause a collision” applies even if you are both walking the same way.

Also, while I understand the impulse behind slowing to avoid the responsibility of deciding where to go, in situations where one’s attention is not on the encounter, this can lead to awkwardness. I have been in multiple situations where I found myself trying to figure out why I was standing in front of someone who was looking at me expectantly or even already starting a conversation, because they thought I had stopped to talk to them, when I was just trying to be unobtrusive and slip by at a slower speed.

I thought about using “Radial Land Gnarl All Nerd” as the title of this blog, since anagramming is all the rage these days, but confusion is never fun. Well, almost never.

Well, sometimes.

Anyway, I’m back. I don’t recall if I ever actually blogged on randallsquared.com; I think it was always more of a businessy site than my personal site. The business “RandallSquared” has been defunct since at least 2008, so it’s probably safe to switch it up, now.

I’m going to make a concerted effort to move a lot of stuff here that would otherwise be comments on Reddit, Hacker News, Facebook, or wherever.

Starting a blog on randallsquared.com is something I’ve been “meaning to do” for many years, but there have been reasons why it seemed like a good idea to put it off. I didn’t like most of the blogging engines available (like WordPress, etc). I didn’t want to point randallsquared.com at something like Blogger or Blogspot. Generator tools seemed clunky and I didn’t want to bother with the languages they tended to be written in (Perl and Ruby were the major ones). Writing my own blogging engine was attractive, but I didn’t have time to do even the minimal feature set I would have preferred.

Later, creating a site became more and more of a potential chore, because the server randallsquared.com was on was old and busted (I was using CPanel on CentOS 5.x to manage several domains for myself and friends), and it became more and more difficult to install newer software on it, culminating in a situation where CPanel couldn’t update itself (for the third or fourth time in the last five years) without manual help.

I had a number of miscellaneous issues installing node.js and npm on the old server, and basically the only thing that was easy to keep up to date on it was PHP. PHP is not the worst thing in the world (well, some differ), and is probably the language I have the most experience with, but I would have preferred something like Go. Due to the CPanel issues, I decided to start moving things over to a new server rather than continue messing with the old and busted one, and to push me into it, I went ahead and started up a new server before I left on a trip, to create an artificial sense of urgency about the switch when I got back. That worked okay, and I moved over the randallsquared.com domain, and went looking for a blogging engine. Et voila! There was a nice generator tool written in Go, and it seemed well thought out. So, that’s what I’m using.

If you start watching now (Nov 4, 2014), you’ll probably see some churn, as I move things around and learn more about hugo, the blogging engine I’m using. When I add things I wrote before the blog start time, I might backdate them, but if that seems confusing, I’ll probably just mention the approximate date of creation in the post. There’s that confusion issue, again.