Common integration issues #3: Where is your data?

This is Big.

In reality, once we started thinking about this topic, which frequently comes up in our line of business, we felt it was much bigger than just an ‘integrations issue’. There is a lot to unpack. It is a hidden problem as well, in that many organisations have no idea that they may have this issue.

This is due to the complexity of modern technology, in combination with an ever increasing and more complex compliance burden around data. In addition, professionals usually have an in-depth understanding of either technology or compliance but the combination is pretty rare.

Let us explain.

Background

Initially, this topic ended up on our blog list due to ad-hoc commentary from clients, prospects and partners regarding some lower-cost integration competitors. What was holding them back from utilising these lower cost services, was the fear of a ‘data black box’. In other words, the uncertainty of where their data would end up physically, and be stored along the way. Or what would happen to their data, should they decide to stop using the integrations provider. Would they get it back? Would it go somewhere else as well? Or had it already been dispersed?

In extreme cases, the prospect or partner in question had abandoned the integration project altogether, out of fear of ‘what might happen’, often fueled by a lack of understanding of cloud/modern computing. This is, obviously, a shame. It keeps them trapped in a less than optimal IT environment, where manual work is common, and wastes time that could otherwise add so much value.

The broader data risk

As you can imagine there are all sorts of potential compliance issues and traps associated with third party data black boxes, integrations or in general, for any other functionality.

Most countries have adopted some form of data privacy legislation which mandates you to look after your customers’ data properly. In some countries like the U.S., there are known additional data risks with agencies like the NSA having unrivaled powers in terms of any data stored in their jurisdiction.

All of this makes it less than desirable for a company to use a third party data service that isn’t fully understood, with due mitigation of risks in line with risk appetite.

What we would like to add to this, but hear less frequently if we speak with our clients and partners, is the risk of custom code - again for integrations or any other purpose. There are many great software developers out there, who develop outstanding custom code. So this is in no way a blanket statement that custom code for integrations or anything else is necessarily a bad thing.

However, unless there is a framework driving the code development that takes into consideration all the compliance requirements, and so does the implementation and continued monitoring of the code, one just cannot be certain of associated data risks and appropriate mitigation thereof.

Modern technology

These considerations we have heard from some of the people we work with, who understand that integrations require data to pass through somewhere in order to end up in a destination system. Over the last few years we have also learned that some people assume the integration works like magic. In other words, the integration makes data appear in the other application, without anything happening between the source and destination applications. Which once you bring it up in any amount of detail, obviously can’t be true, and everyone gets that as well. The problem occurs if this kind of conversation isn't had in the first place.

Even though it isn't necessary for our clients to understand the ins and outs of integration technology, it is important to know that in order for an integration to work, activity is happening outside of the applications involved and the data is passing through somehow. Because otherwise, the associated risks with data whereabouts cannot properly be considered by the organisation.

Cloud computing takes this consideration to another level, due to its nature: it enables a lot of efficiency and flexibility in launching applications and integrations as well as switching between them, however the downside is the visibility on all the touchpoints of your data decreases even further. Unless you make a conscious and continuous effort to keep track of all your dataflows, companies involved and their processes/locations for handling your data.

This is where we feel the crux of the issue is: in our experience, virtually no-one does that.

Triggers

We then considered at what point in time our prospects, clients and partners discuss data flows, whereabouts and processes with us. It usually comes up in meetings when we discuss the specifications of integration requests for new clients. At that point there is a clear trigger for almost any client or partner to consider the flow, and the data touchpoints and any additional storage that could happen along the way. In a nutshell: what our way of working looks like.

But then just thinking through to what usually happens after that, and once the deal is sealed - not very much. Once the integration is up and running, a client might request another integration or changes to an existing integration. Usually, the topic of where data goes or changes to data processing does not come up again. Nor does it on a yearly or otherwise periodic basis, in line with refreshing risk registers, or any other associated data compliance process.

We do our best to be good citizens and to send our clients updated data processing agreements where we can. However, we also see that some (not all!) software developers are a bit lazy, and not everyone follows this best practice. And finally, in line with the introduction to this blog, we do see a trend in that this (third party) data management is a very complicated piece for most, and the burden of complying with most modern requirements as stipulated in laws and regulations is onerous to say the least and probably just way too complicated.

A growing issue

So what started as a relatively simple blog idea around the data logistics of integrations, quickly morphed into trying to define and write about a pretty wicked issue.

In line with some of the above statements, it is also a growing issue. It is unlikely we will go back in terms of the advancements in the cloud, and likely we will advance further. Even though there are still quite a few organisations that are yet to embrace the cloud to the fullest extent, the expectation is that this will happen over the coming decade or so.

Therefore, unless there will be a massive education drive for all (data) risk & compliance professionals involved with decision making around risk mitigation, this issue will mushroom quite significantly. It might take a few publicised mishaps to coral public attention and due regulatory scrutiny to the issue, however this is probably a matter of time.

After that, we can see increased compliance standards develop in line with common awareness around these issues. Not necessarily making resolving data touchpoint issues any easier, however at least it would be a commonly accepted challenge, which is the first step to any good resolution.

Back to integrations

If you came across this blog post because you have questions or doubts about what your current integration/s are doing with your data, all of the above is a long-winded way of saying that we want to be part of the solution at Harmonizer.

We can give you a complete map of everything that happens to your data, and where. In addition, we promise to keep our and your understanding of data flows current, even after you become our client and specifications change. We have plans on our roadmap to enhance our functionality so you can export this with a single click from our web-application, and have an overview of security, any information you might need for audit purposes, as well as a functional description of what happens to your data.

Should this blog have assisted in educating you on the topic of data whereabouts and risks associated with modern technology more broadly, and integrations specifically, we are thrilled as well. We are keen to discuss any remaining questions you might have.

Photo by Denniz Futalan