IT Transformation with SOA

The Architecture Scope

Because architectures should strive for flexibility, there is always a danger that, if they are not properly scoped, you may end up with a pile of “architecture” documents that no one will care to read, let alone implement. When defining the architecture it is essential for you and your team to define it with the right scope. If you are to keep it real, boiling the ocean is not an option. (“Let’s boil the ocean to get rid of those German U2s, the implementation is up to you.”)

For example, when defining the scope, ask yourself if you are trying to address only the customer-facing automation side of your business, or if you will also include the backend support systems [1]. If the latter, chances are you will need to consider integration to the ERP vendor. In any case, you’ll need to define whether to take an end-to-end customer lifecycle perspective or to focus on specific core business processes only.

Should you support international environments? If your company is a conglomerate, will you cover multiple business units or just one particular division? Are you even empowered to define the backend architecture?

Answering these questions requires a candid, non-delusional understanding of whether you have the kind of organizational governance to define far-reaching enterprise architectures or are limited to specific areas under a narrower span of control. That subsidiary headquartered in Paris might not appreciate your trying to dictate their technology!

Understanding your own limitations is essential. Clearly, it would be nice if the scope of the architecture encompassed as much of the “problem-space” as possible, but the fact is that real-life problem-spaces are highly irregular.

Were you to depict a typical problem space using a shape, chances are you would end up with a splattered blob like the one that follows:

Non-withstanding your governance limitations, you could define the architecture with a scope so broad that it covers all possible points of the problem space—like throwing a stick of dynamite into lake to do some fishing. This ‘solve-world-hunger’ approach will specify a great deal of things that just don’t belong to your specific transformation initiative. While all-encompassing, this ambitious approach will most likely be very expensive and non implementable.

The next diagram depicts a more realistic approach. The architecture scope here represents a compromise between specifying the largest possible conjunction of common elements in the problem set while ignoring the more “far-out” cases. It is this conjunction that forms the core of enterprise architecture—the main area of focus for the strategic architecture group.

Areas of the problem space not covered by the core architecture can always be handled as exceptions on a case-by-case basis. Who cares if content integration with the three person art-department happens with an 8GB USB stick? And yes, you have to accept the branch office in Namibia entering their one daily purchase order on a typewriter and then faxing the batched weekly orders on a Friday night. When the day comes that Internet connections become cost-effective there, you can then bring them into the fold of your architecture.

Such pragmatic exceptions avoid the need to over-architect data bases or data exchange formats and do help keep the architecture simpler. Of course, it is best to minimize these special cases as much as possible. Were you to set the scope in a too-narrow a fashion then most everything will becomes an exception! In addition to assuring yourself hours of the extra work needed by this fly-swatting approach to solving issues, you will also be costing the company more than necessary.

Even though the exceptions should be in the periphery of interest for your architecture team, you should at least be aware of their existence. Better yet, the architecture group should establish preferred guidelines on how best to admit exceptions so that the possibility of their future alignment with the core architecture is not compromised. For instance, establishing the use of a standardized Comma-Separated-Value (CSV) file for “non-traditional” data exchanges (including the art department’s USB stick), or defining the product codes to be typed in the manual purchase orders from Namibia, would at least ensure that exceptions can be better handled in the future.

It’s also worth remembering that the fluid nature of a business requires the ability to append new problem space areas to the original space (imagine when a business merger or company reorganization occurs). Typically, these “out-of-space” problems will have to be first handled on a reactive basis following the rule that new features are always needed “by yesterday”. However, you should have you team work on integrating these to the mainstream architecture as soon as practical. When this occurs, you will need to start the process of redefining the architecture scope all anew.

But what does the term “Scope” means in practical terms? What’ is it precisely that your architecture team should be defining as being either in or outside of scope?

The architecture scope should define the layers of technology to use (off-the-shelf or homegrown). Also, the scope should detail the applicability of the architecture as it concerns your organization’s structure, both horizontally (how many departmental functions and geographies it spans), and vertically (what level of detail will the architecture attain? Will you stop at the regional level or define the departmental elements?). In regard to timelines, your scope should go all the way to the long-term—there is no such a thing as a short-term architecture.

Perhaps you recall that back in the late eighties there was an Open Systems Interconnect standard (OSI) being pushed by the International Standards Organization. The specification reflected a design-by-committee approach that failed to take hold when faced with more effective de-facto standards such as TCP/IP, but one of its most enduring legacies was the definition of the seven-layer interoperability stack. This stack divided the various system elements in accordance to the type of service provided by each layer.

This framework can be useful in showing how the IT industry has been traversing the OSI layers toward commoditization. In the early days of IT, much blood was shed in battles centered on which technologies were to be chosen for the networking layers. Architecture scopes had to deal, for example, with whether to use SNA or DECnet, or even whether to implement something proprietary (it happened; one of my deliverables in a project long ago was for a proprietary communications protocol). Today it would be incomprehensible to debate against the use of TCP/IP as the protocol of choice, unless you are in a research organization testing the next big thing. The industry has converged to standards that moved the battleground upwards in the stack; toward OSI’s highest layers.

The diagram below shows the OSI layers. Shown in the horizontal axis are the trends toward standardization. In consequence, these days the focus is on actual application and service specifications; not on the underlying networking layers that, for all practical purposes are now completely commoditized.

For all the criticisms against it (the OSI specification truly became a boiling-the-ocean example[2]), it at least served as a unifying schema. The problem is that the OSI Reference Model is now outliving its usefulness. Despite the industry trend toward standardization and commoditization, an area of “infrastructure” that remains yet to be standardized is the Service Distribution Fabric[3]. SOA is today in a similar state of maturity as the networking world back in the eighties. With the so-called Enterprise Service Bus (ESB) we have vendors resisting standardization measures in order to sell their own proprietary solutions and get competitive advantage. Just as happened with TCP/IP, the world awaits a vendor independent SOA standards specification.

More work is needed to define the detailed structure standards for the Presentation and Application layers. This burden has been taken up by a variety of organizations such as the W3C, the OMG (Object Management Group), the Java Community Process, IEEE, many others, but we are not yet at a point where we can say that application and presentation layer standards have been universally adopted (Web browsers are still not guaranteed to display content the same way), but I have no doubt the industry will continue to converge towards more standards for SOA and the upper application development layers.

While that happens, your role will be to apply you know-how to best adapt from the range of available technologies and standards the solutions that best meet your specific needs. Defining the scope of the architecture, even when “only” focusing on the highest layers of the OSI stack, remains a challenging and fundamental proposition.

Life is still fun!

Labels: ESB, IT, Networks, organization, OSI, protocols, Scope, SOA, Standards, Strategy, Tactis, Technology, Transformation

posted by IT Transformation with SOA @ 11:29 AM

The Architecture Process

The progression of the architecture definition process begins at stratospheric heights, moving all the way down to the ground in what is hopefully a smooth-landing implementation rather than an uncontrolled crash. All architecture definition processes should follow a top-down progression. As discussed earlier, this progression is grouped into levels: Level I representing the most abstract steps, and Level III dealing with the detailed design and engineering.

The objective of the Level I process is to develop a conceptual architecture model that properly mirrors the business process it is meant to solve. In order to be meaningful, this model must offer an end-to-end view of the defined scope; so the first Level I task is to define the scope of the architecture.

At this point in the process it is fine if the model is suitably high-level, otherwise you could trade-off coverage for specificity (to abuse the cliché: you could fail to see the forest for the trees). At this early stage you should resist the impulse to over-define the solution. I say this fully aware that a common criticism of architecture is that, due to its abstract nature, it is can become too ethereal and impossible to implement. The alternative, however, is to fail to develop the most flexible solution in an attempt to tackle detailed implementation concerns such as performance or operability. Remember, God was allegedly able to create the universe in six days and then be so satisfied with the results that he could rest on the seventh day (I still contend He missed out on quality assurance). You on the other hand, have to take the more humble view that, in all likelihood, the solution you define will not be perfect initially. Your focus should be one of flexibility. Architectures defined under the pressure of parochial concerns often end up being the opposite: stilted and monolithic.

A composer represents a symphony in a musical score and a building architect conveys her vision through the use of a mock-up. Likewise, a good IT architecture model should be fashioned in a manner that business people can relate to. As a high level construct, the architecture model does not need to depict all the elements needed for a proper technology solution, but it should serve as the basis from which the detailed technology components emanate. The visual representation of the architecture model is known as the Architecture Reference Framework. Creation of a powerful framework as a communications tool is a key ingredient to the success of the architectural process.

Last in Level I list, you will need to provide the bridge between the abstractness of the architecture model and the reality of the technology design. This bridge is provided by the definition of tenets and standards.

For example, the United States of America was formed as a federated republic with three independent powers to ensure checks-and-balances. This arrangement represents the architecture model created by America’s founding fathers. The Constitution describes the standards that define the workings of this model, and the Bill of Rights represents the tenets.

Like the Constitution, which can be amended according to Article V, the standards can be further refined into a more detailed form as the architecture development steps are followed. Standards should not be seen as an inflexible mantra never to be deviated from, but they do provide a foundation against which to measure technology decisions.

Now, like the Constitution, which seems like a nuisance at times to politicians, architectures are sometimes seen as an expensive burden in terms of performance and practicality. The reason for this is that oftentimes, people attempt to reify[1] the architecture as opposed to instantiating it[2]. Attempts to reify the architecture by trying to implement architectures in a too-literal a fashion will usually result in contrived, inflexible systems. Instead, the architecture model, tenets, and standards should be seen as a kind of Rosetta stone for the purpose of translating the more ethereal architecture concepts into more concrete, practical implementations.

I present the view that architecture processes should be driven from a Top-Down perspective. There is, amazingly, a school of thought out there that proposes the idea that a successful implementation can serve as a baseline pattern for a new architecture. This bottom-up view is most commonly espoused by some extreme proponents of Agile methodologies who have forgotten that methodology’s forte is in applying quick development and implementation processes; not in the creation of comprehensive, repeatable frameworks. A detailed implementation, no matter how well executed, cannot constitute a true architecture because it is an implementation; one that intrinsically lacks an appropriate level of abstraction. In the end, “architecture solutions” derived by trying to generalize a successful implementation end up being so narrowly defined that the best one can hope for is to use them as the proverbial hammer, turning everything into a nail.

Moving downwards from the Space Shuttle-height Level I, the Commercial Jet Level II architecture deals with the current and target architecture models that map to specific business functions.

A drum-roll here!

We have finally arrived at the technology level that ultimately corresponds to the domain of Service Oriented Architecture! In Level II we deal with the various methodologies and processes needed to define the services’ ontology and the enterprise design for the entire SOA fabric. Later on, I will expand on this level (after all, we are dealing with IT Transformation with SOA!).

There’s no arguing the fact that the architecture should ultimately deal with practical details, it’s just that the mapping to reality is more of an engineering exercise; not an architectural one. Call it semantics, but Level III is actually a stage dealing with the engineering and detail designs in which the architecture principles are correlated with specific technology elements and pragmatic tenets; not to wash-out the principles, but to apply them. Practical trade-offs can be made here to ensure the system will perform under real-life situations. Engineering in Level III is when and where you can and should address all questions of performance, practical scalability, and operability; not before. Hopefully, your great work at defining flexible architecture principles during the Level I and II steps will make this engineering effort easier.

Engineering in this sense is the art of translating the architecture patterns into real world solutions that take into account well-known practical implementation constraints. As such, the definition of the Level III architecture is more about the practice of engineering .

Remember this key dictum:

Architect for Flexibility, Engineer for Performance.

The engineering and implementation processes are responsible for instantiating the abstract architecture into a concrete form. Level I and Level II Architecture may have defined the need for a glass ceiling in the structure, but the engineering process will decide the thickness and brand of that glass ceiling. The strategic engineering role will seek further reuse of common solutions (e.g. use same glass manufacturer for all glass ceilings), without infringing on the overall architecture guidelines. Architecture is all about doing your best; engineering is all about doing what’s best.

Next week, we'll go on how to define the Architecture Scope. The very first Level I activity.

Labels: Architecture, IT, Levels, Methodology, Model, Scope, Services, SOA, Strategy, Technology

posted by IT Transformation with SOA @ 4:05 PM

The Architecture as Foundation for the Technology Solution

While your executives should be able to understand the “What” and the “When” as defined by the technology strategy, it is in the “How” that the magic occurs. So, “How” do we do things?

Defining the “How” is an art; not a science—an art expressed around the creation of the architecture. Still, you should always ensure that, if the architecture is to meet its goals, you keep an eye on the way the architecture supports the strategy. Think of how a building is constructed . . . If the business plan is “Promote Religion”, your strategy could be “Build a Temple”; if your business is to “Enable Commerce”, your strategy might be “Build a Mall”. The architecture should match the strategy just as the strategy should match the business. How you architect a Catholic Church would be different from how you architect a Synagogue. There are in fact a set of patterns and principles that drive the design styles for each of these buildings.

Still, the fundamental question from the technology perspective remains: How do you design the temple or church to build? “How” do you build these structures?

What’s the best architecture?

Well, 99% of the time the answer to that any given question (architecture or otherwise) is that “It depends”. Neither will you arrive at the correct architecture by blindly following a set methodology. If it were that easy to dissect and replicate an expert’s judgment, most “thinking” jobs would have been taken over by computers long ago (do you remember all the hype about expert systems supposedly being able to capture and replicate domain-specific expertise?) No, defining and executing the “How” is what justifies your paycheck. Creating the architecture requires inner judgment, a judgment that’s the inherent product of many years of experience combined with a very high dose of common-sense.

This is how the federal government defines architecture.[1]

An enterprise architecture is a strategic information asset base which defines the business, the information necessary to operate the business, the technologies necessary to support the business operations, and the transitional processes necessary for implementing new technologies in response to the changing needs of business. Stated differently, the enterprise architecture is a strategic asset repository, which consists of models that define the current and target architecture environments, and the transitional processes for evolving from the current to the target.

A bit of a mouthful, but what did you expect? Remember, this is a definition from the government. So, to put it more simply, a modern architecture must meet the following conditions:

1. Provide a unified, end-to-end systems approach to technical architecture and design. In other words, it should provide a Unified Architecture Reference Framework. This framework should converge, consolidate, and functionally align the various technology elements in a coherent manner with the business.

2. Define the architecture patterns the company will live and die by. At its apex, the architecture must follow what I call a broad theme. Here I suggest that if the architecture is to have the required flexibility to properly meet the business objectives, the broad theme ought to be the use of the service oriented paradigm (SOA).

3. Incorporate principle-based evolution to move your company from its current state toward a target state. This includes a roadmap of development initiatives that incorporate the adopted strategic tenets and standards.

As shown in the diagram above, the process of architecture is ultimately a process of abstraction. Unfortunately this is often confused with the idea that architectures are just a bunch of un-implementable constructs.

It is your mission to ensure that the architecture is viewed as a stepping stone to a process that will ultimately lead to concrete solutions. This process should have a top-down approach that progressively moves the architecture from a high level definition down through levels of finer detail:

· Level I, the highest conceptual level, contains standards, current architecture, architecture segments, sub architecture models, target architecture, strategic direction, transitional processes, and external architecture drivers.

· Level II breaks down the current and target architecture models to cover specific business functions and technology.

· Level III further breaks down the technology architecture into data, systems, and infrastructure.

Next week I will cover the specifics for each level in this process.

Enjoy!

Labels: Architecture, Definition, Foundation, IT, Levels, Methodology, Process, SOA, Solution, Technology

posted by IT Transformation with SOA @ 10:56 AM

Creating the Technology Strategy

So far so good, the business strategy is understood, the requirements are known with a reasonable degree of certainty and, by now, you have framed the relevant scope and priorities of your company (are you in a business that requires a view of technology as a competitive weapon or as a utility?). To boot, you have also apportioned how you will deal with complexity. It is finally time to develop the overall technology strategy that most clearly matches the technology plan of your business.

At its simplest, the technology strategy should provide a summary, a digest of the technical “Whats” and “Whens”, that translates in technical terms the key business needs. These are really the technical strategy answers to the business requirements, defined in terms that business folks can understand. Here are some examples:

What: “We will replace the entire system with new technology components via gradual investments.”

When: “Over a two-year period, starting with a Phase 1 deliverable that will support new customer analysis tools.”

What: “We will buy a new ERP from vendor XYZ and outsource all new development to India.”

When: “We will hold off until next year. In the meantime we will research outsourcing partners and develop an RFP.”

What: “We will continue to use the operating system ‘as is’, but we will use new technologies to develop any new business systems.”

When: “We will do so on an on-going basis as new business systems become approved.”

If you’ve ever watched a group of children playing a casual game of soccer, you may have been amused at the way they wildly chase after the ball as a pack (believe me, this happens whenever children are let loose with a soccer ball!) There is no structure to the way they play and, as a result, their efforts are usually wasted as they comically interfere with each other. The children know the “What”: to play soccer; they also know the “When”: right-now, but what they lack is knowledge of the “How”.

Just as the “What” and the “When” should have emerged from the definition of the agreed transformation previously discussed, the “How” is the essence, the secret-sauce, of the technology strategy. The “Whats”, the “Whens”, and the comprehensive explanation of the “Hows”, represent the detailed strategy that is to serve as the blueprint for the multi-year IT transformation plan. This technology strategy blueprint is depicted in the diagram below: the core technical solution, including the overall technical approach, the high level architecture, and the standards and tenets you will adopt.

The technical strategy is not meant to be a photograph—something static—but rather an evolving movie. To become a living, breathing, strategy, it must be continuously cross-checked with the changing business requirements. If adjustments are needed, you should ensure the solution continues to conform to the business requirements, and that any resulting changes to the architecture and standards only occurs on an as-needed basis and not as a result of whims or sudden changes in direction driven by dictates of fashion or politics. You will need to implement a governance to oversee the evolution of the strategy, as well as to manage the change control processes in order to avoid the dangers that so-often plague many large projects: scope diffusion, runaway requirements, or irrelevance of the solution.

One of the key roles of the strategic governance is to ensure the strategy is communicated and understood at all levels of the organization. The technical staff, from the most junior programmer to the most senior manager, should be intimately familiar with all the elements of the technology strategy.

While your communication to the technical staff will most likely emphasize the “How” of the strategy, the message to less-technical constituencies, such as the executives, will have to be framed in a more easy-to-understand ‘elevator-chat’ form dealing with the “Whys” and the “Whats” of the strategy.

In this elevator talk you can still speak about the core technical solution and the general principles regarding the transformation: Will you develop the solution internally? Will you rely entirely on the services of a third party vendor? Will you use a hybrid of internally developed modules, with modules available externally as a service? (Software-as-a-Service: SaaS). If you are going to build the solution internally, what role will the IT department play? Will you hire new developers, or integrate third party augmentation services? Will you bring in temporary contractors for software development or will you off-shore development? If so, which components will you off-shore? Will you outsource the whole damn thing? What are the timeframes and general costs? What are the benefits? When will the benefits be realized? Which vendors do you consider strategic?

Clearly, this (possibly long—it’s a tall building) elevator talk will not easily accommodate the explanation of the detailed architecture or standards, but you should at least be able to cover the general concepts, such as whether you plan to use open source software or whether the solution will be accessible via the web.

Now, why are there no longer elevator operators? Remember the days when operating an elevator was considered to be such a specialized role that it required a, usually bored-looking, attendant to press those buttons as a service to you?

Come to think about it, you may want to mention to the CFO that you will be using something called Service oriented Architecture. . .

Labels: Architecture, IT, outsourcing, SaaS, SOA, Software Development, Strategy, Technology, Transformation

posted by IT Transformation with SOA @ 8:58 AM

Taming Complexity

The complexity of a project is an intrinsic variable ultimately determined by the business processes. Keeping in mind the key dictum that a technology solution can never be of lesser complexity than the underlying problem it solves, you should first ensure that prior to automation, the business processes are as optimized as possible[1]. Many technology projects end up being unyieldingly complex because they try to tackle what is in essence a mash up of chaotic business rules.

Having said this, there is no reason why the technology solution should be more complicated than necessary. For instance, poor technical decisions or the use of inappropriate software tools can add unnecessary complexity to a project.

Recognizing that a project has an intrinsic complexity framed by the business requirements, your challenge is to efficiently tame this complexity within the realm of the technology and budget available to you. Here we have the “Law of Preservation of Complexity”: N = S + U; where N is the total complexity of the solution (as determined by the complexity of the business problem), S is the technology system complexity, and U is the complexity that the user faces when interacting with the system. Given that the intrinsic complexity of a project is “N”, it is your decision how to apportion the complexity between the system and the user.

Thinking about it, the simpler the user interface, the more complex the technology backend that supports that iteration is bound to be. I’m fairly certain that the underlying software supporting the Apple iPhone is quite complex, given that it is a device universally recognized as providing an extremely-intuitive interface. Alternatively, simple technology implementations tend to deliver solutions that are more manual in nature and hence more complex for the user. A complex solution that delivers an easy-to-use implementation is bound to cost more to develop, but its operational costs, due to reduced support and training costs will be lower. A simple solution that places the burden of complexity on the user can typically be developed more rapidly and at lesser expense, but its operational cost will be higher.

Taken to the extreme, the simplest automation solution is one in which automation ceases to exist and all business processes are manual. The most complex automation solution will probably involve the use of a sophisticated, artificial-intelligence system that is able to take on all automation burdens[2]. This capability, however, has yet to be invented.

In the end, it behooves you as technology leader to ensure that no unneeded technical complexity is introduced into your projects (watch out for that young bright, software maverick who pushes for that “cool”, dynamic rules engine to do the payroll!). Always keep in mind that the single most important independent variable in apportioning complexity is the understanding of the key business priorities and the true capabilities of technology.

The trick is to strike the proper balance in the deliverable. A good technical delivery will endeavor to place the bulk of complexity inside the solution black-box, making the interaction with the software simpler and more user friendly, but not at the cost of creating an un-implementable solution.

Many people confuse complexity with over-automation. If the complexity of the technology solution does not ultimately simplify the interaction with the user then you are not improving things. A personal peeve of mine is the recent “popularity” of so-called voice-response systems. If they were used only for simple voice driven selections, then there would be no problem. The issue arises when these interactive-voice-response (IVR) applications are forced to carry more complex interactions. The fact is that technology is not yet at the point where it can support these interactions in a seamless manner. In the end, the only purpose these “voice-recognition” systems accomplish is to weed out customers’ access to the support staff, at the price of lowered service levels, and also heightened customer dissatisfaction. The manner in which companies use these systems is a typical example of adding complexity for the wrong reason.

Labels: Business Rules, Complexity, Interfaces, IT Transformation Defined, IVR, Project, voice recognition

posted by IT Transformation with SOA @ 11:41 AM