Integration best practices - thoughts on ESB and SOA

When talking about integration, there is an infinite amount of tips that claim to lead to the holy grail of integration. Most tips are related to specific technologies and based on the latest and greatest, but in these series I will take an approach which will not be dependent on any specific technology.

I am a big fan of new technologies and frameworks and I am often one of the first to try something that's just been introduced. Trying the newest technologies and creating things that were previously thought to be impossible, is a lot of fun to me. This is all good in theory, but in practice, keeping up with the newest technologies and still delivering value when creating software, can require a lot of effort. There are many topics to cover, I will start with two which are very common.

Two of the most discussed topics in the field of integration are 1- service based architectures, like SOA and microservices and 2- integration helper software, like the Enterprise Service Bus (ESB) and integration frameworks (e.g. Camel). Whether you need an ESB in microservices is another discussion altogether, but I like to handle things as practically as possible and use whatever fits my needs, like in this case an ESB.

Thoughts on service based architectures and ESB's differ greatly, so some guidelines on which are not related to a specific technology and which can help creating lasting value could come in handy. Here are some of my thoughts on the subject: All tips and guidelines provided here are based on personal experience and opinion, feel free to agree (or disagree). When you select an ESB or integration framework, it's probably because you need it. Soon enough, the Bus becomes the godfather of your integration and everything needs its permission going forward. It takes some courage to speak up to It, but do this sooner rather than later. Otherwise you might realize that adding a new feature to your internal coffee counter application interferes with your invoicing process...

So the first tip that comes to mind:

Not everything needs to go over the Bus (or at least not over a single Bus). It's generally a bad idea to use a single Bus as the central hub for everything that looks like integration, since this could lead to it being a single point of failure and/or a maintenance nightmare.

It is perfectly acceptable to use a dedicated connection to an application that has a single purpose and is solely being used by a single application, but what about maintaining this connection?

Let's assume in this case an HTTP connection is used and you are worried about maintaining a point to point connection... there are simple solutions for location transparency. It can be achieved by using DNS with a load balancer/proxy as an abstraction. So you get all the benefits and no extra overhead is introduced.

What about monitoring and controlling these 'loose' elements? Is it really that interesting to know what messages are passing through the ESB? Sure you need to know if something goes wrong in the ESB, but there is an application that initiated the request (this is likely not the ESB). Logging this first action is more important than knowing if it arrived at the ESB, and knowing the message arrived on the receiving end is more important than knowing it left the ESB. The ESB can still be used as a log collector though, or as a log mediator to a log collector.

Building on this, there is the option to split the bus into multiple, useful and dedicated components. This results in providing the integration environment with the right tools for the right job. However, defining the granularity of these components can be a difficult job. So, the key is to start with larger components and see if they can be split up into smaller parts, keeping transaction issues and interservice communication in mind.

Dedicated components

Apart from using dedicated components, the general idea should be to keep the Bus lean and mean: use it for exchanging data, transforming data and routing data. Do not use it for central web services, workflow engines, rules engines and automated registries. You don't need them on the Bus. If you feel the need for any of these functionalities, just create a separate service.

Wrapping up

These tips might look like common practice, but in most environments, they aren't. Not so long ago I did some consultancy for a government corporation. They wanted to transform their application environment to full SOA to solve the problem they were having with a large application. This company had never done anything to support this architecture other than the fact that they had more than one application. Their most important application was a large monolith application which had a large amount of business logic, with a lot of different change cycles. Different change cycles can become an issue if they occur frequently and thus interfere with each other. They had tried implementing SOA themselves but it did not work as expected when their central Bus was creating more trouble than value. They used their Bus to handle everything that was not present in the applications to support SOA. This seemed to work ok in the beginning but got them into all sorts of trouble later on. The different change cycles in the monolith were directly reflected in the central Bus. So, they now had the same issues as before but now with a central Bus as a bonus.

What seems like a good idea in the beginning, might not work anymore after a few months of development. And sure, we work agile and can handle change with some (or a lot of) extra effort, but preventing this could save you a few (or a lot of) sprints in the end.

My next blog will focus on integration best practices again and I will try to propose some more tips/guidelines... So stay tuned!