Successfully evolve APIs by monitoring consumer usage

Successfully evolve APIs by monitoring consumer usage

Written by:

Martin Boerman

If you expose functionality through web APIs, having a poor understanding of your users’ needs can drain all the success out of the services you provide. It can slow adoption, push existing clients to leave your platform, damage trust from the users you’re trying to win over, or simply cause conflict about responsibilities between development teams.

Believe me, I’ve been in the middle of such a dispute.

Thankfully, there’s a way to avoid never-ending discussions and declining user satisfaction around APIs. And that’s by getting to know your API consumers, thereby ensuring that you’ll actually serve them with the right thing in the right way – as opposed to offering services you think are beneficial to them.

Why build an API in the first place?

The notion of an API (Application Programmable Interface) is commonly used to refer to basically any kind of predesigned means of communication between pieces of software. To clarify, when I’m talking about APIs here, I’m referring to web services, exposing functionality to other applications over a network (often through HTTP).

There are plenty of reasons to create web APIs, like “I’ve read that RESTfulness is the next big thing!” But that’s not a sound reason. The driver should always be in line with the business goals that an organization is trying to achieve. In my work as a software quality consultant, I talk to many different organizations every day that create and buy software. So I’ve seen all kinds of different reasons for commissioning web APIs. Here are a few good ones:

  • Driving a software monolith towards a modular IT architecture
  • Increasing the user base of a product
  • Creating synergy with partnering organizations
  • Collecting valuable customer data to use or sell
  • Enabling a mobile strategy by developing multi-platform applications

APIs can either be used for applications within an organization, or to link with applications of other parties. In terms of software quality, having software products communicate through clearly-defined interfaces is generally accepted as a good way to stay flexible and adapt to changing demands.

Nobody’s responsibility (a real-world example)

At the beginning of this post, I mentioned a dispute (or let’s say, a heated discussion) about responsibilities between development teams. This happened at a company that develops software related to on-demand video streaming. One team maintained a backend for the hosting and streaming services; the other team developed mobile applications that made heavy use of that backend.

Over the course of several years, both teams had tried to work together on a solution to compete with the ordinary television experience. A modern platform spanning on-demand content, digital guides, seamless integration of smart TVs with devices like Chromecasts or Apple TVs and mobile phones, you name it. As you can imagine, getting there required integration of existing backend systems and modern frontend applications running on a variety of devices, such as tablets, laptops, and phones. Aligning these two worlds proved to be a significant challenge.

The issues were centered around the API between the frontend applications and the (older) backend realm. Problems were caused by both teams operating in too much isolation and as such, over time, developing incompatible views on the best means of communication between their software products. The frontend developers (“These modern apps need the APIs to meet our demands!”) got into recurring arguments with the backend team (“It’s way too hard for us to change this legacy stuff, so you solve the problem in your modern app!”). The effects of the misalignment ranged from an unnecessarily high amount of calls required for simple transactions… to high data and memory consumption… and even to concerns about reduced battery life of mobile applications. Arguments took place around many other topics, too, but it all boiled down to the older backend APIs simply not being a good fit with today’s frontend systems.

These discussions proved to be unwanted disturbance in what could have been a happy collaboration. Had the backend team followed the frontend team’s needs more closely as they evolved over the years, they would have been able to adapt iteratively. What was surprising to me was that these teams had diverged so much when they were situated in the same building, part of the same company. So, can you imagine how easy it would be to misalign with API consumers when they aren’t sitting just steps away down the hallway from you? And in fact, this organization was planning to expand the exposure of functionality over web APIs to external partners – and maybe even go public with them.

How do you maintain consumer satisfaction when you expand your offerings to consumers you don’t know? And don’t even know what drives them?

Keep your customer base front and center

Consumers can range from just one internal team to a number of teams in partnering businesses or even hundreds or thousands of developers in public communities. This influences the ease at which consumers can be engaged.

A 2018 report by Forrester Research on API management solutions, Now Tech: API Management Solutions, states that providers should treat developers as customers, especially when it comes to public APIs. The report stresses the need for “great API documentation, rich developer experience, and superior tools … to engage with developers individually and as groups.” This “user relationship management,” as they call it, is vital when treating developers as customers. Even though it might be more trivial in some contexts to maintain communication lines with API consumers (e.g. two teams in the same organization), the general approach to tracking user demands should be the same, regardless.

So, how do you do it? The key here is monitoring usage. By collecting usage information, analyzing it, and using it to improve the interface, an API can be kept in accordance with consumers’ needs. This is much like collecting usage information and user feedback from regular end users of a software product. The developers of consuming systems are the end users. An API is almost like a regular user interface, except it’s not graphical and the users are known to be developers.

Draw invaluable insights from actual usage

User feedback is important at every step along an API’s lifecycle – starting even before the actual design phase when determining the basic reasons and purposes for creating it in the first place. Incremental software development is common practice, so APIs are expected to evolve while they’re live. Collecting usage data starts right after go-live and should never end, as this is key to staying relevant in a world that will undoubtedly change.

By monitoring the usage patterns and behavior of an API in production, useful data can be gathered on how services are used by consuming IT ecosystems. Insights might lead to reshaping existing endpoints, or making them more flexible and relevant to use cases that might not even have been known during the initial design. In addition to functional improvements, non-functional aspects such as data optimization and performance characteristics can also be addressed by looking at how API clients behave. I’ve seen cases in which improvements had beneficial side effects, like reducing development efforts, which is directly linked to the software’s cost of maintenance later on.

Since nobody’s ambition should be to reinvent the wheel, selecting the right tool for the right context and goals can save a lot of effort. API management tools nowadays offer a range of functionality, from basic publishing features to user relationship management, monetization features, support for API ecosystems, and governance. If collecting usage data is the intent, make sure to select the right tool. In the selection process, be careful not to consider only the goal of collecting usage data, but other aspects as well (e.g. security, rate limiting, reliability). At the same time, remember that tools have their limits, too. API management tooling can go a long way in helping collect usage patterns, but API providers should extend their efforts beyond the features of such tools.

Engage directly with developers and product owners of consuming applications. Have them provide their feedback through discussions boards, surveys, or by simply talking to them. Remember, they’re customers – and as with any product, success is driven by their happiness.

Bottom line: Usage monitoring is your key to success

If there’s misalignment between software products communicating over the network, it’s quite often expressed in a suboptimal API. This is something that can gradually worsen over time. As an API provider, collecting data regarding usage patterns is key in staying aligned with consuming applications. Tooling can provide support here, but increase your chances at success by reaching out in different ways as well. It’s not only about usage data in production, but about collecting user feedback as well.

Treat API consumers as customers. Doing so may lead to increased adoption, strategic success, better performance and even lower maintenance costs.