Managing SOA complexity brings up the question of session state. By ’state’ I mean all the required information that must be maintained and stored across the series of interactions needed to complete a full business exchange. Maintaining the state of a service interaction implies remembering at what stage are the conversing partners and the working data in effect. It will often be at your discretion designing services to either depend more or less on the use of state information. At other times the problem at hand will force a specific avenue. In either case, you should remember this simple formula: State-Keeping = Complexity.
Maintaining state might be inescapable in automated orchestration logic, but it comes with a cost. State-Keeping constrains the options for maintaining high availability and indirectly may increase SOA’s fragility by making it more difficult to add redundant components to the environment. With redundant components you must ensure that messages flowing through the system maintain their state, regardless of the server resources used. Relying on session states, while also allowing flexible service flows, is hard to do. It’s done, yes, but the price you will have to pay is an increase complexity and performance penalties related to the need to propagate the state of a particular interaction across several nodes. Therefore, a key SOA tenet is that you should use sessionless flows whenever possible. In other words, every request should ideally be atomic and serviceable regardless of the occurrence of previous requests.
Do you want to know the name of an employee with a given social security number? No problem. As a part of the request pass the social security number, and receive the name. If you next want the employee’s address, you can pass either the social security number or the name as part of the request. While atomic, sessionless, requests such as these do impose a requirement that the client maintains the state of the interaction and holds the information elements related to the employee, this approach does simplify the design of systems using server clusters.
Still, while the preferred tenet is to avoid session keys. On occasion, it becomes impossible for the client to keep the state, forcing the server to assume this responsibility. In this case, the approach is to use a uniquely generated “session-id” whereby the server “remembers” the employee information (the state). You will have to ensure the session key and associated state data is accessible to all servers in a loosely-coupled cluster, making your system design more complicated.
For an example of keeping a session-based state, consider an air booking process where the client is reserving a pair of seats. The server will temporarily decrease the inventory for the flight. For the duration of the transaction the server will give a unique “reservation id” to the client so that any ongoing requests from the client can be associated with the holding of these seats. Clearly, such a process will need to include timeout logic to eventually release the two seats in the event the final booking does not take place before a predetermined amount of time.
This discussion leads to another tenet: maintaining state, either in the client or in the server, along the lines mentioned is ultimately acceptable. Keeping the state inside the intermediate nodes? Not so much. Why? An intermediate component should not have control in timing-out a resource that’s being held in the server. If it did, it would be disrupting the server’s ability to maintain integrity in its environment. Also, an intermediate component will not have full awareness of the business semantics of the service request/response. Relying on an intermediate component to preserve state is like expecting your mail carrier to remind you that your cable bill is due for payment on the 20th of each month. He might do it, yes, but the moment you forget to tip him during the holidays, he just might “forget”!
Ironically, many of today’s vendors offer solutions that encourage the processing of business logic in their intermediate infrastructure products, encouraging you to maintain state in these middleware components. They do so because enabling middleware is an area that does not require them to be aware of your applications, and thus is the easiest area for them to offer you a “value-add service” in a productized, commoditized fashion. You should resist the melodious chant of these mermaids and refrain from using their tempting extras services. If not, you may find yourself stuck with an inflexible design and with a dependency on specific vendor architecture to boot.
My advice is to avoid these vendor-enabled approaches. There is much that can get complicated with the maintenance of state, especially when the business process requires transactional integrity, referential integrity, and security (and most business processes do). The moment you give up this tenet and maintain session state inside the SOA middleware as opposed to the extreme end represented by the Client and the Server, you will be ensuring years of added complexity in the evolution of your SOA system.
The complexity of a project is an intrinsic variable ultimately determined by the business processes. Keeping in mind the key dictum that a technology solution can never be of lesser complexity than the underlying problem it solves, you should first ensure that prior to automation, the business processes are as optimized as possible[1]. Many technology projects end up being unyieldingly complex because they try to tackle what is in essence a mash up of chaotic business rules.
Having said this, there is no reason why the technology solution should be more complicated than necessary. For instance, poor technical decisions or the use of inappropriate software tools can add unnecessary complexity to a project.
Recognizing that a project has an intrinsic complexity framed by the business requirements, your challenge is to efficiently tame this complexity within the realm of the technology and budget available to you. Here we have the “Law of Preservation of Complexity”: N = S + U; where N is the total complexity of the solution (as determined by the complexity of the business problem), S is the technology system complexity, and U is the complexity that the user faces when interacting with the system. Given that the intrinsic complexity of a project is “N”, it is your decision how to apportion the complexity between the system and the user.
Thinking about it, the simpler the user interface, the more complex the technology backend that supports that iteration is bound to be. I’m fairly certain that the underlying software supporting the Apple iPhone is quite complex, given that it is a device universally recognized as providing an extremely-intuitive interface.Alternatively, simple technology implementations tend to deliver solutions that are more manual in nature and hence more complex for the user. A complex solution that delivers an easy-to-use implementation is bound to cost more to develop, but its operational costs, due to reduced support and training costs will be lower. A simple solution that places the burden of complexity on the user can typically be developed more rapidly and at lesser expense, but its operational cost will be higher.
Taken to the extreme, the simplest automation solution is one in which automation ceases to exist and all business processes are manual. The most complex automation solution will probably involve the use of a sophisticated, artificial-intelligence system that is able to take on all automation burdens[2]. This capability, however, has yet to be invented.
In the end, it behooves you as technology leader to ensure that no unneeded technical complexity is introduced into your projects (watch out for that young bright, software maverick who pushes for that “cool”, dynamic rules engine to do the payroll!). Always keep in mind that the single most important independent variable in apportioning complexity is the understanding of the key business priorities and the true capabilities of technology.
The trick is to strike the proper balance in the deliverable. A good technical delivery will endeavor to place the bulk of complexity inside the solution black-box, making the interaction with the software simpler and more user friendly, but not at the cost of creating an un-implementable solution.
Many people confuse complexity with over-automation. If the complexity of the technology solution does not ultimately simplify the interaction with the user then you are not improving things. A personal peeve of mine is the recent “popularity” of so-called voice-response systems. If they were used only for simple voice driven selections, then there would be no problem. The issue arises when these interactive-voice-response (IVR) applications are forced to carry more complex interactions. The fact is that technology is not yet at the point where it can support these interactions in a seamless manner. In the end, the only purpose these “voice-recognition” systems accomplish is to weed out customers’ access to the support staff, at the price of lowered service levels, and also heightened customer dissatisfaction. The manner in which companies use these systems is a typical example of adding complexity for the wrong reason.
[1] In fact, one of the intrinsic benefits of IT Transformation is that, just like the act of smiling can make people feel happier via backward neural influence, this transformation can also serve as a trigger for much-needed business transformation.
[2] As I write this, I’ve come across the announcement that IBM is starting a project funded by DARPA to build a new type of computer that simulates the way the brain operates.A multi-disciplinary working team that includes Neuroscientists and other disciplines will work on this project. I doubt this type of approach will work (men finally invented airplanes when it stopped trying to emulate the way birds fly), but who knows?
The Strategy and Planning for the IT Transformation effort must emanate from the definition of scope and the understanding of the requirements that will be targeted by the initiative.Many compromises must be made in the process, and these compromises become both cause and effect of the ultimate strategic direction.
The compromises deal with the interplay of these parameters for each requirement:
§Quality: The solidity of the deliverable
§Cost: The amount of money available for delivery
§Schedule: The expected delivery timelines
§Scope: The functionality that is being delivered
These variables are known as QCSS attributes, taken from the initials of each. The business requirements can only reasonably specify three of these factors. The fourth factor will be a result of the project’s dynamics. For example, if the business wants a project done within certain cost restraints, under a specific timeline, and with a specific scope, the quality of the project will be in question.
In this diagram, Functionality is represented by the surface area of the square. The Cost used to meet the Functionality is shown in light grey, and the Schedule (timeline) is in dark grey. Quality is the percentage of the Functionality surface that is being covered by the Cost and Schedule squares.The diagram shows that if the Functionality scope is increased (larger surface), you will either have to increase the Cost, the Schedule, or both to match the new requirements; if not, Quality will suffer.
This “law” applies to all projects and to any combination of attributes. Furthermore, imagine the QCSS variables as knobs on a control panel, each influencing the other[1]. The more money available for a given project, the more potential resources that can be committed; thereby enabling faster delivery and/or more extended functionality. With less funding, chances are the scope will need to be reduced if timelines are to be maintained.
Managing the project budget and the timelines is a straightforward project management function. Project Management disciplines to effectively do this have been developed and can be applied in a systemic way. But truth be said, the most complicated challenge during this stage is to properly manage the amount of requirements you are likely to receive from the business. Let’s face it, this is an IT Transformation project that, most likely, has been sold as the mother-of-all-solutions and that will, no doubt, have the business salivating like a kid in a candy store with all the potential for functions that they have for so many years been deprived of.
However, the fact is that the requirements values tend to follow Pareto’s 80-20 rule. That is, typically 20% of the requirements represent 80% of the solution value, and the other 80% the remaining 20%. Now please don’t go to the business and tell then you will cover 20% of their requirements. Not a good idea. First, the Pareto principle is just that—a principle; not a law. It might be that in your case 40% of your requirements represent 70% of the value. Secondly, that little extra business value that can be attained by covering a few additional requirements might mean the difference between having a highly competitive solution or an also-ran.
What you first need to do is identify the 20% to 40% of the requirements that everyone agrees are a must, and then proceed to carefully analyze and prioritize the remaining requirements in light of their diminishing returns. My own philosophy is to do as much as possible with the money and time allocated (in essence, establish first the cost and schedule surfaces) and then draw a line in terms of what requirements are to be met during that first phase. You can place the unfulfilled requirements in a subsequent phase that might or might not be funded on its merits. If you can do so without eliciting the wrath of the business team, then you have succeeded!
As you can see, managing the overall scope is an activity best handled by the project leader and not to be delegated to line project managers. It requires finesse and political skills in dealing with business customers.
A second dimension that is often left unexamined during the requirements phase is the need to assess the project complexity and define the strategy of how this complexity will be managed. By this, I mean the determination of the strategy balancing the target automation complexity versus the reliance on manual processes. This is the subject of my next blog.
This blog covers the practical techniques, trials and tribulations associated with the transformation of IT systems from legacy technologies to systems using SOA and modern open systems. I also include the occasional interlude with rants about technology in general.
About Me
Name: Israel del Rio
Location: Atlanta, GA, United States
Israel has been recognized by Computerworld as one of their Premiere 100 IT honorees. Israel is a business and technology leader who has contributed the technology vision as key strategist and designer behind the enterprise technology roadmaps of large hospitality and travel companies. Israel has also have developed and deployed various mission-critical systems and in the process, he's been instrumental in creating and building effective and skilled development organizations.
