IT Transformation with SOA

Ten years ago the world was abuzz with worry, holding its collective breath while waiting to see if the turn of the millennium with the much touted Y2K bug would bring an end to civilization.  (Fools, I say, we all know the end of civilization will occur in 2012!)

The so-called “Millennium Bug” wasn’t actually a bug but rather a side effect of legacy applications written during a time when the cost of storing one byte was about ten million times higher than it is today. Storing only the last two digits of the year did save tons of money for those using super-expensive disk storage. The problem was that legacy applications over-extended their welcome as they are prone to do. Avoiding a negative impact to the information systems with the coming of the year 2000 became a critical imperative[1]. 

With a total worldwide estimated cost of $200B (about one fifth the estimated cost of the Iraq war), the industry faced the challenge driven by the hype associated with the problem. In the end, the only firework displays came from the beautiful millennial ceremonies in Sydney harbor and the Eiffel Tower. January 1, 2000 came and went with barely a hiccup.

Soon a narrative of how we had all been duped by the IT people with all their “Y2K fear mongering” began to take shape. During a CNN interview, a US Senator who was opposing a new budget request stated, “Let’s not have another Y2K project where we spent so much money and then . . . nothing happened!” I was thunderstruck. Leave it to a politician. His line of reasoning was akin to this, “I paid all that life insurance money to cover me this year and nothing happened! What a bad investment that was!”

All this got me thinking about the fuzzy nature of success. For the most part, failure is easy to recognize (although good spin doctors can be masters at hiding them!), but success is not.  Even endeavors that were clearly successful in terms of having accomplished their stated goals (“send a man to the moon and bring him back safely”) have had their detractors. “The project was over-budget”.  “Astronauts died in the process”. “The whole mission was a fool-errand”. “That money could have been used to solve world-hunger”, et cetera, et cetera. 

Typically, a project that succeeds in each and every one of its dimensions is either a chimera or something that was not ambitious enough to begin with.  Every successful project will always have areas were things do not come up perfectly. This is especially true with complex transformation projects!

If the objective of a project is to deliver something tangible, you can at least make a compelling argument for its success. However, projects intended to avoid risks will always be more difficult to defend.

As far as that US senator was concerned, the Y2K intervention had been a failure because he was looking only at the cost involved in the effort. He was unable to understand that “nothing should happen” was actually the criteria for success. The hundreds of billions of dollars spent to avert the Y2K bug was money that had been well spent precisely because “nothing happened”; plus there’s anecdotal evidence that the Y2K remediation efforts also drove beneficial application makeovers. The Y2K catastrophe was avoided, but sadly, I still hear many who should know better (CFOs. CEOs, TV pundits) making comments similar to the senator’s.

I have witnessed many projects deprived of recognition simply because they provided a quiet success. No drama. It is a mark of our times that projects that first fail and then eventually get fixed (usually at extra expense and time) are the ones that tend to get credited and celebrated the most (Hubble Space telescope anyone?). However, projects that deliver their promise right off the bat; especially when the promise is one of risk aversion, receive little or no recognition.  I have witnessed projects that delivered and even exceeded their key goals, but because they failed to meet one hundred percent on original expectations (and I believe no serious project ever does!), they ended up being perceived as failures. This is particularly true for large projects were the magnifying lens of politically-driven critiques from those keen on shooting down a successful effort or casting a bad light on other’s accomplishments happily point out missed milestones or highlight wrinkles in the delivery.

Still, it’s good to remember that lack of recognition is one of the main reasons seasoned professionals search for greener pastures. Success should always be assessed fairly and realistically and then be celebrated. Every successful milestone and accomplishment should be recognized and used as a foundation for the next step up the ladder.  And for every successful step we should also give thanks.

Happy Thanksgiving!

Data is what we put into the system and information is what we expect to get out of it (actually, there’s an epistemological argument that what we really crave is knowledge. For now, however, I’ll use the term ‘information’ to refer to the system output). Data is the dough; Information the cake. When we seek information, we want it to taste good, to be accurate, relevant, current, and understandable. Data is another matter. Data must be acquired and stored in whatever is best from a utilitarian perspective. Data can be anything. This explains why two digits were used to store the date years in the pre-millennium system, leading to the big Y2K brouhaha (more on this later).  Also, data is not always flat and homogeneous. It can have a hierarchical structure and come from multiple sources. In fact, data is whatever we choose to call the source of our information.

Google has reputedly hundreds of thousands of servers with Petabytes of data (1 Petabyte = 1,024 Terabytes), which you and I can access in a manner of milliseconds by typing free context searches. For many, a response from Google represents information, but to others this output is data to be used in the cooking of new information. As a matter of fact, one of the most exciting areas of research today is the emergence of Collective Intelligence via the mining of free text information on the web. Or consider the very promising WolframAlpha knowledge engine effort (wolframalpha.com) which very ambitiously taps a variety of databases to provide consolidated knowledge to users. There are still other mechanisms to provide information that rely on the human element as a source of data. Sites such as Mahalo.com or Chacha.com actually use carbon-based intelligent life forms to respond to questions.

Data can be stored in people’s neurons, spreadsheets, 3 x 5 index cards, papyrus scrolls, punched cards, magnetic media, optical disk or futuristic quantum storage. The point is that the user doesn’t care how the data is stored or how it is structured. In the end, Schemas, SQL, Rows, Columns, Indexes, Tables, are the ways we IT people store and manage data for our own convenience. But as long as the user can access data in a reliable, prompt, and comprehensive fashion, she could care less whether the data comes from a super-sophisticated object oriented data base or from a tattered printed copy of the World Almanac.

How should data be accessed then? I don’t recommend handling data in an explicit manner the way RDBMs vendors tell you to handle it. Data is at the core of the enterprise, but it does not have to be a “visible” core. You don’t visualize data with SQL. Instead, I suggest that you handle all access to data in an abstract way. You visualize data with services and this brings up the need via a Data Sentinel Layer. This layer should be, you guessed it, an SOA enabled component providing data accesses and maintenance services.

To put it simply, the Data Sentinel is the gatekeeper and abstraction layer for data. Nothing goes into the data storages without the Sentinel first passing it through; nothing gets out without the Sentinel allowing it. Furthermore, the Sentinel allows decoupling of how the data is ultimately stored from the way the data is perceived to be stored. Depending upon your needs, you may choose consolidated data storages or, alternatively, you may choose to follow a federated approach to heterogeneous data. It doesn’t matter. The Data Sentinel is responsible for presenting a common SOA façade to the outside world. 

Clearly, a key tenet should be to not allow willy-nilly access to data by bypassing the Sentinel. You should not allow applications or services (whether atomic or composite) to fire their own SQL statements against a data base. If you want to maintain the integrity of your SOA design, make sure to access data via the data abstraction services provided by the Sentinel services only.

Then again, this being a world filled with frailty, there are three exceptions where you will have to allow SOA entities to bypass the abstraction layer provided by the Sentinel. Every castle has secret passageways. I will cover the situations where exceptions may apply later: Security/Monitoring, Batch/Reporting, and the Data Joiner Pattern.

Obviously, data abstraction requires attention to performance, data persistence, and data integrity aspects. Thankfully, there are off-the-shelf tools to help facilitate this abstraction and the implementation of a Sentinel layer, such as Object-Relational mapping, automated data replication, and data caching products (e.g. Hibernate). Whether you choose to use an off-the-shelf tool or to write your own will depend upon your needs, but the use of those tools is not always sufficient to implement a proper Sentinel.  Object-Relational mapping or use of Stored Procedures, for example, are means to more easily map data access into SOA-like services, but you still need to ensure that the interfaces comply with the SOA interface criteria covered earlier. In the end, the use of a Data Sentinel Layer is a case of applying abstraction techniques to deal with the challenges of an SOA-based system, but one that also demands engineering work in order to deploy the Sentinel services in front of the Data Bases/Sources. There are additional techniques and considerations that also apply, and these will be discussed later on.

A number of new needs have emerged with the advent of SOA. First of all, there was no standard way for an application to construct and deliver a service call.  Secondly, there was no standard way to ensure the service would be delivered.  Thirdly, it was not clear how this SOA environment could be managed and operated effectively. Fourthly . . .  well, you get the idea; the list goes on and on.

SOA demands the existence of an enabling infrastructure layer known as middleware. Middleware provides all necessary services, independent of the underlying technical infrastructure.  To satisfy this need, vendors began to define SOA architectures around a relatively abstract concept: the Enterprise Service Bus, or ESB.   Now, there has never been disagreement about the need to have a foundational layer to support common SOA functions—an enterprise bus of sorts. The problem is that each vendor took it upon himself to define the specific capabilities and mechanisms of their proprietary ESB, oftentimes by repackaging preexisting products and rebranding them to better fit their sales strategies.

As a result, depending on the vendor, the concept of Enterprise Service Bus encompasses an amalgamation of integration and transformation technologies that enable the cooperative work of any number of environments: Service Location, Service Invocation, Service Routing, Security, Mapping, Asynchronous and Event Driven Messaging, Service Orchestration, Testing Tools, Pattern Libraries, Monitoring and Management, etc. Unfortunately, when viewed as an all-or-nothing proposition, ESB’s broad and fuzzy scope tends to make vendor offerings somewhat complex and potentially expensive.

The term ESB is now so generic and undefined that you should be careful not to get entrapped into buying a cornucopia of vendor products that are not going to be needed for your specific SOA environment.  ESBs resemble more a Swiss army knife, with its many accessories, of which only a few will ever be used. Don’t be deceived; vendors will naturally try to sell you the complete superhighway, including rest stops, gas stations and the paint for the road signs, when all you really need is a quaint country road. You can be choosy and build your base SOA foundation gradually.  Because of this, I am willfully avoiding use of the term “Enterprise Service Bus”, preferring instead to use the more neutral term, “SOA Fabric.”

Of all the bells and whistles provided by ESB vendors (data transformation, dynamic service location, etc.), the one key function the SOA Fabric should deliver is ensuring that the services and service delivery mechanisms are abstracted from the SOA clients.

A salient feature that vendors tell us ESBs are good for is their ability to integrate heterogeneous environments. However, if you think about it, since you are going through the process of transforming the technology in your company (the topic of this writings after all!), you should really strive to introduce a standard protocol and eliminate as many of legacy protocols as you can.

Ironically, a holistic transformation program should have the goal of deploying the most homogeneous SOA environment possible; thus obviating the need for most of the much touted ESB’s transformation and mapping functions. In a new system, SOA can be based upon canonical formats and common protocols; thus minimizing the need for data and service format conversion. This goal is most feasible when applied to the message flows occurring in you internal ecosystem.

Now, you may still need some of those conversion functions for several other reasons, migration and integration with external systems being the most obvious cases. If the migration will be gradual, and therefore requires the interplay of new services with legacy services, go ahead and enable some of the protocol conversion features provided by ESBs. The question would then be how important this feature is to you, and whether you wouldn’t be better off following a non-ESB integration mechanism in the interim.  At least, knowing you will be using this particular ESB function only for migration purposes, you can try to negotiate a more generous license with the vendor.

There are cases whereby, while striving for a homogeneous SOA environment, you may well conclude that your end state architecture must integrate a number of systems under a  federated view. Your end state architecture in this case will be a mix of hybrid technologies servicing autonomous problem domains. Under this scenario, it would be best to reframe the definition of the problem at hand from one of creating an SOA environment to one of applying Enterprise Application Integration (EAI) mechanisms. If your end state revolves more around integration EAI, it would be better suited to performing boundary-level mapping and transformation work.  In this case, go and shop for a great EAI solution; not for an ESB.

If the vendor gives you the option of acquiring specific subsets of their ESB offering (at a reduced price) then that’s something worth considering. At the very least, you will need to provide support for service deployment, routing, monitoring, and management, even if you won’t require many of the other functions in the ESB package. Just remember to focus in deploying the fabric that properly matches your SOA objectives and not the one that matches your vendor’s sales quota.

A quick word regarding Open Source ESBs. . . There are many, but the same caveats I’ve used for vendor-based ESB’s apply. Open Source ESBs are not yet as mature, and the quality of functions they provide varies significantly according to the component. Focus on using only those components you can be sure will work in a reliable and stable manner or those which are not critical to the system. Remember you are putting in place components that will become part of the core fabric. Ask yourself, does it make sense in order to save a few dollars to use a relatively unsupported ESB component for a critical role (Service Invocation or Messaging, come to mind), versus using a more stable vendor solution?

In the end, if you are planning to use the protocol conversion features packaged in a vendor-provided or open source ESBs, I suggest you use them in a discrete, case-by-case basis, and not as an inherent component of your SOA fabric. This way, even as you face having to solve integration problems associated with the lack of standards, at least you won’t be forced into drinking the Kool-Aid associated with a particular vendor’s view of ESB!

Back in the XIX century (that’s the 19^th century for all you X-geners!), there was a composer who didn’t know how to play the piano. In fact, nor did he know how to play the violin, the flute, the trombone, or any other instrument for that matter. Yet, the man managed to compose symphonies that to this day are considered musical masterpieces. The composer’s name was Louis Hector Berlioz, and he achieved this feat by directing the orchestra through each step of his arrangement and composition. His most recognized work is called “Symphonie Fantastique” and, according to Wikipedia, the symphony is scored for an orchestra consisting of 2 flutes(2nd doubling piccolo), 2 oboes (2nd doubling English horn), 2 clarinets (1st doubling E-flat clarinet), 4 bassoons, 4 horns, 2 trumpets, 2 cornets, 3 trombones, 2 ophicleides (what the heck is an ophecleide? A forerunner of the euphonium, I found out. What the heck is a euphonium? Well, check it out in Wiki!), 2 pairs of timpani, snare drum, cymbals, bass drum, bells in C and G, 2 harps, and strings.

By now, you probably get the idea. Mr. Berlioz fully exemplifies the ultimate back-end composite services element: The Orchestrator. Berlioz composed some pretty cool stuff by knowing a) what he wanted to express, b) what specific set of instruments should be used at a particular point in time, and c) how to communicate the notes of his composition to the orchestra.

Every SOA-based system needs its Berliozes.

There are several dimensions involved in defining the role of an orchestrator for SOA. First, as discussed earlier, most orchestrator roles will be provided within the context of an application; not as a part of a service. That is, the orchestration is what defines an application and makes one application different from another. The orchestration is the brain of the application, and it is the entity that decides the manner and SOA services calling flow.

In some instances, you might even be able to reuse orchestration patterns and apply them across multiple applications. Better still, you can build orchestration patterns by utilizing the emerging Business Process Modeling technologies (BPM). BPM simplifies the work of creating orchestration logic by providing a visual and modular way of assembling orchestration flows. A small commentary of mine is this: BPM is not SOA, but BPM requires SOA to work properly.

An apropos question is to ask how much orchestration should be automated in the SOA system as opposed to letting the user manually orchestrate his or her own interactions. To answer this question it is best to remember the complexity rule I stated earlier:  the simpler the user interaction; the more complex the system, and vice-versa. 

Then again, there are limits to the complexity of an orchestration. A full-fledged Artificial Intelligence system could become the ultimate orchestration engine but, unfortunately, such a machine remains in the realm of science fiction.  Cost-Benefit compromises must be made.

Say we have a travel oriented system and need to find the coolest vacation spots for the month of September. Should we let the user manually orchestrate the various steps needed to reach a conclusion? Each step would indirectly generate the appropriate service calls for searching destinations, filtering unwanted responses, obtaining additional descriptions, getting prices, initiating the booking, and so forth. Or we could consider developing a sophisticated orchestration function that’s able to take care of those details and do the hard work on behalf of the prospective traveler. But should we do it?

The answer lies in the size of “the market” for a particular need. Clearly, there is a need for a travel orchestration capability that can take care of all the details mentioned. After all, isn’t this why Travel Agencies emerged in the first place? If the orchestration is need by only a few users, then it is best not to spend money and effort attempting to automate something that is too unique. On the other hand, if the request becomes common, then it is preferable to create an automated orchestration function that organizes and integrates the use of SOA services.

The orchestrators design should always accommodate the transparency tenets in order to allow horizontal scalability. In other words, if you provide the orchestration via servers located in the system membrane, you will need to design the solution in such a way that you can always add more front end servers to accommodate increased workloads, without disrupting the orchestration processes in existing servers. Because orchestration usually requires the server to maintain some form of state, at least for the duration of a transaction, you will need to incorporate some form of session-stickiness in the orchestration logic. Later on, I will write more about why I recommend that this is the one and only area where a “session state” between the user and the orchestration should exist, even as I still advice to keep backend services discrete and sessionless.