By Evan Rosenfield
Have you ever wondered whether it would be possible to bind together a particular collection of separate software products into a unified, consolidated application with all of the combined functionality of the constituent components? If so, this new series of articles is being written just for you. We’re going to explore, in plain language, some of the arcane mysteries of system integration, so that you can make better-informed technology decisions and evaluate the potential vertical solutions that might fall well within your immediate reach.
The first lesson: the anatomy of… lasagna
It’s actually one of my favorite foods. When I think of lasagna, I tend to imagine its various layers: the layer of sauce, the layer of cheese, and the layer of noodles. Pile those layers on top of one another, and the result is both functional and delicious.
Believe it or not, contemporary applications are structured in a manner similar to lasagna. That is to say, in a metaphorical sense, that a contemporary application likewise has clearly defined layers (which are only slightly less tasty than the lasagna layers). These layers are sometimes called application “tiers” (and you might sometimes hear the term “n-tier architecture,” which simply invokes the idea that an application will have some number of separate layers).
The layers (or tiers) of a contemporary data-driven application would normally include the following:
- A Presentational Layer – These are the screens and graphic layouts that are displayed for users. This layer is also called the “The Display Layer”, “The User Interface,” the “UI/UX Layer,” and the “Front End.” The purpose of the Presentational Layer is to accept input (clicks, keystrokes) from the user, and return output (everything the viewers are shown in response to their activities).
- A Business Logic Layer – The Business Logic Layer is traditionally where most of a software product’s intelligence resides. This is usually the layer where the complex thinking and processing happens in an application. This layer typically “listens” to the Presentational Layer. It analyzes input and events, and then responds by executing and regulating whatever procedures are appropriate.
- A Data Layer – The essential function of the Data Layer is the control of data access. Whenever the Business Logic Layer needs to handle or access stored information, it will work through this layer. You could think of this layer as something like an information concierge, a gatekeeper, and a butler, all rolled into one.
- A Data Source – This is essentially the information storage bucket of an application. It is the place where the precious data lives for the long term. It often takes the form of a database, but some applications still use other (often proprietary) storage formats.
Above and beyond these four basic layers, there is one more important component of an application that should be taken into consideration (maybe think of this component as the optional garnish for the lasagna). Specifically, there is something called a “Service” that often plays an integral role in integration. The Service is like a very special secondary, non-graphic interface that can be operated by other applications. It’s something like a remote control that one program can use to give instructions to another. The positioning of the Service can vary, so we’ll just say that it sits to the side, next to the Business Logic Layer. In reality, more complex applications can sometimes include multiple services.
Incidentally, the use of the term “layer” probably warrants some further consideration at this juncture. In truth, in this context, the concept of a “layer” really has nothing to do with any sort of physical spatial arrangement. The so-called layers do not actually reside atop one another. It’s really a bit of a misnomer, but one that is applied for good reason.
To be exact, we invoke the term “layer” because it seems to capture some important ideas. Specifically, the concept of a layer seems to suggest a degree of separation between respective components. Moreover, insofar as the notion of a “stack” is closely related, the concept of layers helps us to imagine a pattern of organization wherein some components will touch one another, whereas others won’t touch.
At the end of the day, the separation implied by the notion of a layered stack is what should really be understood. The Presentation Layer might touch the Business Logic Layer, but probably would not touch the Data Layer or the Data Source. The Business Logic Layer might touch the Presentation Layer, the Service, and/or the Data Layer, but probably not the Data Source. The Data Source might be touched by the Data Layer, but probably would not be touched by the Presentation Layer.
The second lesson: dealing with multiple lasagnas
The only thing more delicious than a single lasagna is bunch of them, each with its own flavor. But laying out a table of many lasagnas can prove challenging, especially when you want to make sure that the taste of one lasagna will complement the next. The trick is learning how to effectively combine the flavors of the respective constituent layers from each finished dish.
And at the end of the day, making the various layers work together is really what software integration is all about. While the basic operation of integration – simply moving data between systems – is easy enough for just about anyone to conceive, the reality is that the actual approach to integration is slightly more complex. But a good way to think about the exercise is in terms of each respective application’s layers.
Middleware is the software that makes it all happen. This software usually works behind the scenes to pull, push, transform, repurpose, and generally shuttle data back and forth between programs. You can think of middleware as the glue that ultimately laces the separate software systems together. Or alternatively, you could maybe think of middlewear as the exquisite silver serving tray that brings the whole thing together, so that the presentation of the lasagnas is more enticing, and so that the “users” of the lasagnas will find the lasagnas that much more appetizing and delightful to consume.
Metaphors aside, one of the trickiest aspects of middleware is that is normally invisible to the user. It doesn’t have any screens, and no one ever actually sees it while it is operating. In fact, it technically doesn’t even have a visual appearance. It is just a program that runs in the computer’s memory. This is probably why so many people have a hard time understanding the term.
The third lesson: the scope of possibilities
If you’ve read this far, and you understand that integration has something to do with making the layers of various applications work together, then you might already be guessing at the various potential methods that could be used to achieve integration. You might be imagining all manner of possibilities (ex. Sending information from the Presentation Layer to the Data Source, or from one Data Layer to another.)
While this is getting close to the right idea, it is very important to understand not all approaches to integration are created equal. Some approaches are more common, and some are less common. Some are much easier to implement, and some are much harder. Some are considered preferable (“best practices”), whereas others are considered naughty. Some are more secure, and others are (much) less secure. Some result in a better user experience, and some in a worse experience. Some would result in instantaneous/real time integration, whereas others would result in delayed integration. Some would prove easier to update and maintain when software changes, and some would become very challenging to update over time.
So at the end of the day, while there might actually be numerous “wild west” approaches to integration, only a few are genuinely advisable. Specifically, as long as the applications targeted for integration are using one or both of the following elements, then any developer worth his/her salt should be able to write some responsible code to enable the associated data to move around:
- Databases: A database is a very common type of Data Source. Some of the most common databases include Microsoft SQL Server, Oracle, MySQL, SQLLite, IBM DB2, and PostGreSQL. Other databases also exist. It is fairly easy for a developer to write code that will make a connection to a database, extract data, transform it, and move it elsewhere.
- Web Services: As explained earlier, Services give programs (as opposed to users) the ability to interact. A Web Service is simply a Service that communicates through the internet. Many contemporary applications publish Web Services. These Services can be used to efficiently send, retrieve, and update information remotely.
The final magic trick (often without even looking at the software)
When a developer is asked about a potential integration project, the developer will usually follow-up with some additional questions. During the ensuing conversation, the developer might show some apparent signs of consternation. The developer might hem and haw, scratch his chin, and then finally, with a distant yet contemplative look in his eye, proclaim, “yeah, sure… it can be done.”
Non-initiated users often wonder how the developer could ever confidently assert — without looking at the actual software — that integration is assuredly possible. But after reading this article, you should have an inkling of how the magic trick really works.
At a basic level, you could just ask, “Do the involved systems either store information in databases, or publish web services?” If so – and you’ll be pleased to know that this is likely the case with 90%+ of contemporary software – then integration is likely possible. (Note: this isn’t to say that other sorts of integration are not plausible, but this one question will very quickly ascertain if the some of the easiest and most common methods can be used).
Then it just boils down to a question of time and resources.