Common integration issues #1: breaking API

This is a new blog series about common integration issues. Market feedback of our growing client base is inspiring this series, as are some online fora and websites we’ve come across, where people vent about integration experiences :).

In this blog series, we will relay scenarios that crop up in the space of integrations, and share some tips on how to prevent these from happening, in our experience. We hope that by sharing our knowledge and lessons learned, that the space of integrations can be a happier place for everyone.

Some of our recent client intake conversations have been revolving around API updates and how painful and time consuming it can be, if these break. We will firstly explain some of the underlying concepts, causes and the impact of these issues. After this, we will share our view on recommendations and best practices to deal with breaking APIs.

In a nutshell: it is not about the API breaking, it is about how you respond.

What is an API?

An API, or Automatic Programming Interface, allows an application to exchange data with the outside world, including other applications. This is much needed in our modern cloud based environment, where the number of applications is increasing significantly over time, and so is data volume.

APIs are all around us, and you are using them every day - maybe without realising. Each time you interact with apps on your phone, and when you decide to share something from say Linked-In to WhatsApp: API. Paying with PayPal: API. Logging into other platforms with Google or Facebook: API. We could go on for a while.

For Harmonizer, and many other integration technology/service providers, APIs are key. They provide the mechanism through which the integration can access and distribute the relevant data.

What does it mean if an API breaks?

An API “breaking” is obviously not a physical process. From the point of view of the owner of the API, it is not necessarily negative either. It just means that something in the code base of the API has changed. Perhaps as a flow-on effect of a change to the system it belongs to, or it could be that bugs were fixed, or new bugs have arisen due to updates. Alternatively, maybe data fields were added or modified in the API or other enhanced functionality was added to it.

A plethora of reasons for why APIs can break, but usually leading to the outcome of ‘garbage in, garbage out’ for the integration - because the API does not work as anticipated when the integration technology tries to extract information from it, or tries to write to an application.

Why is this happening more and more?

In recent years, there has been mass adoption of flexibility and agility in dealing with applications. These approaches have great benefits in terms of delivering more functionality in a shorter time, with a higher quality. Think SCRUM, and other agile development methods. We see a trend in that the adoption and improvement of these methods will only increase further. This in turn will increase flexibility and agility of software development more.

In the past, companies had all their software on premise and would run with software for 10 years, or maybe even longer. This is changing too. The cloud and other developments that offer flexibility and agility in procuring and implementing software allow companies to opt for best of breed applications, that change with the times at a much quicker rate, and in line with their own growth journey. Their integrations, therefore, need to change with their changing software landscape.

The upshot of the above, in plain english, is that APIs will change at an increasing speed. And ‘break’, insofar as integrations are concerned. To the point that in our longer term staff planning, we are taking into account we will need more support staff working on this in a year’s time than we have now.

For you, this means you need to find a way to go with the flow of integration requirements in your organisation. As this problem will only grow and you don't have any say in the development roadmap a company takes - unless you are one of their key, or very close clients.

How do you fix this?

You have a few options.

• You can keep amending your custom code or use any integration platform you may have whenever someone in the business complains, and build new code or workflows for each new piece of software your company adopts, if integrations are not available out of the box. You will need to get more manpower onto this, given the growing complexities in the landscape.
• You can leverage an outsourced IT provider (consultants or a worry free managed service like ours) to do this for you. Depending on the outsourced provider, you might need to budget more for this in the future. We for one, never charge for changes to APIs.
• However you approach integrations, you can't prevent broken APIs. It is a fact of life, much like taxes and cyber attacks. In line with the cyber-rhetoric though, it is all about the response rather than the issue occurring in and of itself.

Our baseline advice sounds simple but the realities can be complex. You should aim to detect a broken API early, as well as following it up quickly before it leads to an actual operational problem. Breaking this down, these are our recommendations and best practices for the team or provider responsible for your integrations, to do this well:

• Deploy a monitoring layer over your solution that continuously tests the execution of the code as well as whether data flowing through meets expectations. Our proprietary platform Blackhawk does this for us. Enabling us to instantly spot breaking APIs, before it affects our customers, and deploy fixes in one go to all affected customers.
• If you don’t have the option to deploy a monitoring layer, find another way to detect changes in API timely, and make sure it doesn't cost significant issues in your production environment.
• In addition, if you can, test all steps of data transformations that are part of the integration separately. This will assist in finding the issue with the integration quicker, allowing for a quicker response by saving time for the analysis of the problem as well as deployment of the solution.
• In case you develop your own code for integrations, aim to do this as modular as you can. In addition to saving you troubleshooting time in case an integration breaks, and allowing for quicker fixes in line with the above, this will also allow you to implement any augmentation in functionality in a quick and seamless way.
• If you outsource your integrations, see if you can agree a service level agreement (SLA) with your provider. That way, you have some assurances on when integration should come back online in case of any issues. Harmonizer offers each client an SLA, no matter how big or small.
• Should API developers be good enough to let you know they are making changes (sometimes they do, and sometimes they don't), it is recommended to test the APIs through for every release.
• For this reason, it pays to be closely aligned with your most important SaaS applications so you get a message up ahead and you have the time to focus on said message. Ideally you spend time with all of them - however we realise not everyone does this full-time like us and you have to prioritise your workload.
• At the very least, subscribe to any news feeds, offered by your SaaS providers, regarding deprecated features. This should give you some time / a horizon to work towards if the worst of all API changes happen: sunsetted API features. Usually SaaS providers give you lead-time when they deprecate or sunset features or entire APIs.

Conclusion

In case this all sounds very complex - we are here to help. Whether that looks like a quick phone call to explain any of the above or answer another question (all free of charge), or whether you are interested in having a conversation about using our worry free, managed service.

At Harmonizer, our main purpose is to make integrations work in the best possible way and by helping others on the journey, we get to have interesting conversations, meet interesting people and we always learn something along the way.

Photo by Jackson Simmer on Unsplash