This being my 50th blog, it represents a good vantage point to take stock of the road traversed and the reason why we are in this journey in the first place. I started my blog describing the promise of technology and the importance of technology transformation to accomplish that promise. I moved on to a discussion on how to make the business case to start the technology transformation ball rolling. I then proceeded to cover more technical matters, such as the characteristics of Service Oriented Architecture and the various classifications of services, and then delving even deeper into the detailed considerations for SOA design and management.
In fact, we have gone so deep that I am reminded of this “gedanken” (mental experiment):
Assume there is a tunnel so deep that it reaches the center of the earth. In fact, imagine digging this tunnel until it reaches the surface on the opposite side of the Earth (the antipode).Now, let’s have a brave athlete jump into the tunnel. What would happen?
Removing all other physical considerations such a as air resistance, and temperature and pressure, as the athlete reaches the center of the Earth, she should begin to feel less and less gravity. In the center of the Earth she should be completely weightless. The force of gravity is zero down there. The reason being that gravity is caused by the Earth’s mass. At the center, the gravitational pull is offset by the Earth’s surrounding mass.
So far, so good, but now the athlete has inertia and will continue to “fall upwards” towards the surface on the other side of the Earth! As the athlete falls upward, the gravitational pull will increase (more and more mass from the Earth will be behind her), slowing her down until the “upward fall” is halted just as she reaches the surface at the antipode. At this point, our athlete will begin to fall once again toward the center of the planet until she returns to the entrance of our tunnel . . .only to fall again. In this hypothetical, frictionless environment, our athlete would act like a perpetual Yo-Yo, repeatedly falling and re-falling back to the surface.
So, imagine that this SOA blog is a bit like this athlete. It feels like we have reached the center and that it’s time to now “fall” upwards towards the surface.Next I will be covering detailed engineering considerations (remember, we are still near the SOA core!), followed by less technical discussions. These items will be related to program execution governance, project management, and organizational and people matters. That is, we will return from the detailed to the general.
Still, given that we are still knee deep in the details, it is also good to remind ourselves why we are on this journey. In the end, this is not about SOA or even Technology, but about what we can do with SOA and with Technology. Yes, there is the technologist viewpoint regarding the power of SOA. While you can certainly run non-SOA system in a Cloud Computing environment, without SOA it is almost impossible to truly leverage the power of Cloud computing on behalf of an enterprise-wide system. Then again, the labor involved in creating SOA systems has an objective beyond Cloud Computing or using Software-as-a-Service. The most exciting goals are all about shaping the future of technology. That is, our ability to make technology so flexible that it eventually becomes hidden.
Arthur C. Clarke’s famed third law states that any sufficiently advanced technology is indistinguishable from magic.I would add a fourth law: The best indication that a technology has matured is that it has become invisible.
Think of electricity, the water supply, or even the internal workings of an automobile. In all these cases, we operate these technologies almost obliviously in a Switch On/Switch Off basis.
For the most part, technologies follow a well-defined life-cycle that takes them from inception in a lab all the way to invisibility. The time spent within a cycle is technology-dependent, but the average time to maturity can span decades.
Many futurists believe that one of the main evolutionary aspects of computing in the future is to have it also become invisible—embedded in the fabric of the thing we call “reality”. Instead of screens, keyboards and mouses, users will interface with computers in a seamless manner.
The ultimate interface achievement will be to hide the fact that a user is accessing, or even programming, a computer. This later attribute is often confused with the famed Turing Test of Artificial Intelligence (AI).However, the Turing Test establishes that Artificial Intelligence will only be achieved when a computer is able to hide the fact that it is a computer when communicating with a human in a broad domain. AI has been long in coming, and many believe it to be still a century away; others that it is around the corner. But AI requires common-sense and pattern recognition capabilities if it is to work, and progress has been fairly slow on these fronts. I tend to agree that AI as originally envisioned will take a long time to be achieved.However once it happens, AI will not appear as an overnight invention; instead, we will continue to see improvements in computer systems that gradually appear to make them smarter and smarter.
Think about your car’s navigation system which already appears quite smart and of the novel capabilities of your digital camera, such as face recognition. Pseudo-AI behavior in narrow knowledge domains is arriving thanks to the growing computer power made possible by Moore’s law. Consider that in the beginning it was assumed that a chess program capable of beating a chess grandmaster would require a full-fledged AI system. However, this feat has been achieved thanks to the use of the brute-force represented by massive parallel processors and the ingenuity of sophisticated heuristics; not by the invention of a human mind emulator. In May, 1997 an IBM computer nicknamed Deep Blue beat World Chess champion Garry Kasparov much to the chagrin of the Grand Master who found it difficult to accept he had been beaten by a computer! To all intents and purposes, playing against a chess computer does convey the eerie feeling of competing against an “intelligent” device. The machine behaves like AI, but it is actually based on the narrow domain of chess-playing, making the computer an “idiot-savant” of sorts.
As discussed earlier, most transformative technologies are the result of synergistic combinations of various evolutionary advances.To the degree that we see continued advances in user interface paradigms as represented by gestures ala iPhone or voice recognition, combined with improved algorithms and availability of ultra-fast communication bandwidths, we will see a wealth of interesting applications; many of them with true transformative effects. For example, enhanced user interfaces in the future, combined with more advanced artificial intelligence heuristics and the merging social networking paradigms, can deliver a suite of Virtual Sidekick capabilities:
·Attaining complete knowledge of your preferences. In fact, complete knowledge of you as a person.
·Exercise controlled empowerment to take independent action.
·Have immediate access to all sources of information available electronically. The ability to alert you to those specific developments that interest you, such as breaking news or TV specials.
·Adopt different service personalities based on context.
·Monitor actions performed on your behalf in a non-obtrusive manner.Certain events will automatically initiate pre-approved actions. For example, a calendar event schedule change will automatically trigger an action from your Virtual Sidekick to initiate a flight change.
This type of automated avatar will spawn new industries just as the Internet has spawned the multi-billion dollar Google. The Virtual Sidekick is but one example of the kind of thinking that should be propelling your R&D efforts. There are others. For example, it’s logical to imagine a future in which web access devices will have become so small and non-intrusive that they can be implanted into our bodies. In a world permeated with wireless access to the Web (the” Infosphere”, I discussed earlier), imagine a scenario where you can search and access the Internet by simply thinking about it; where you can “Skype” your wife and talk to herusing your own embedded phone. You won’t even need to speak to communicate. A microprocessor embedded in your brain will convert your brain waves into speech.Think of this scenario as technology-enabled telepathy!These and other interesting possibilities can be extrapolated from the intriguing technology forecasts by author, Ray Kurzweil, in his book “The Singularity is Near: When Humans Transcend Biology”.
There can be no doubt that the transformative effects of such future inventions will generate heated debates about the ethics and dangers associated with their use, but that’s a subject matter for a future blog.
Sooner or later it happens to most of us. We grow up and no longer can continue to live in the cocooned environment created by our parents—the comfort and coziness of our youth is gone (except if as a result of the Grand Recession you are obliged to return to your parents home and are forced to experience the George Constanza-like awkwardness of adulthood, but I digress). Either way, we have to enter the real world, a world where people speak the language of credits and debits and where behaviors are no longer governed by Ms. Manner’s etiquette or Mom’s nagging but rather by a set of complex social rules that help us interface with the world. The way we engage with the world, the set of rules we follow, the processes and mechanisms we use to interact with others, the whole cultural context of how to say “please”, or “keep the change”, are equivalent to a boundary layer between us and the rest of humanity.
Having created a suitable SOA system (either homogenous or federated via Enterprise Application Integration tools), we need to enclose it in its own protective cocoon, lest the reckless world outside trample with its internal fabric. The trick is to prevent what is not wanted from getting in while allowing what is wanted to access the system. Here, biology provides us with an ideal model in the workings of the living cell. Just as the membrane of a healthy cell acts as a barrier against harmful agents and judiciously allows the exchange of the enzymes needed to make the cell work in concert with the rest of the organism, we must maintain an SOA membrane that allows the necessary information exchange to take place while keeping the bad guys out of the system.
In IT terms, the membrane is known as the DMZ (Demilitarized Zone). Frankly, I never cared for this term. A DMZ is a buffer zone designed to keep warring factions apart—a zone void of hostilities. The term is deceiving because, in reality, the DMZ is the area where all access battles are fought. Also, the layer’s role is not to keep warring factions apart but to allow the controlled exchange of participating partners. With the emergence of virtualization approaches such as Cloud Computing, we should take the perspective that the membrane is the region where safe trade occurs. In this area the presentation logic is placed alongside a number of public applications. This is the layer that deals with the Business-to-Consumer (B2C) and the Business-to-Business (B2B) interactions. In this layer you also must perform data transformations for data exchange with external entities.
In engineering terms the membrane consists of an arrangement of technologies carrying the interaction with the external world in each layer of the computing stack, from the security guard manning the entrance to the Data Center to the application displaying the sign-on screens. In the networking layer you have the protocol convertors, VPN gateways, IP routers and load balancers. Further up in the stack, the membrane includes firewalls with the appropriate system-level access passwords and permissions; including specific field-level encryption. Even higher up, the membrane contains the needed data mapping and conversion services. Moving on to the application space the membrane includes spam filters and user-level authentication mechanisms.
Rather than give a subliminal message, let me state it as loudly and plainly as a used car commercial before a Memorial-day sale: it’s preferable to create the membrane with off-the-shelf technologies rather than to try to develop your own. The capabilities and features needed for this layer are usually standard to the industry, and thus it makes sense to use vendor solutions. In fact, a trend is to have many of the functions needed by the membrane be handled by special-purpose hardware appliances.
Alternatively, if you plan to outsource operations, then let the hosting provider worry about the make-up of the membrane. Still, you have to define the required levels of service and make certain the monitoring tools and processes exist to ensure these levels. Either way, the membrane is a component that’s rapidly becoming commoditized. A good thing too, for this is not where you ought to seek competitive IT differentiation (that is, unless you are one of those hosting providers!).
To sum up, the membrane is not the area to invest in internal development. The challenge is to create and view the membrane as an integrated component that can be managed and monitored in a holistic manner even if it consists of an assemblage of products and technologies. If you are creating a membrane you should focus on sound engineering work and vendor component integration; not software development.
Ultimately, a well-designed membrane should be trustworthy enough to allow some relaxation of the security levels inside the system. Also, a well-designed membrane should be flexible enough to allow support for a variety of access devices. Once you take care of your system’s membrane you can then focus on what happens inside, where the real work takes place, with the Orchestrators.
If SOA is so “old” why do we still have all this excitement about service oriented architecture applied to IT? Isn’t SOA an obvious choice anyway? In fact, the history of computer science has always been about movement from the very complex to the obvious. While in the pioneering days of computing it took a John von Neumann to work out programming concepts, these days even an Alfred E. Neuman could easily write a decent program.The reason the “obvious” solution was not used in earlier decades is because “obvious” solutions require more complex technologies (recall my earlier discussion on the preservation of complexity). Think of the electronic spreadsheet which was invented in the early 80’s (Visicalc). In hindsight this invention is something obvious. Why then wasn’t the electronic spreadsheet invented even earlier?If we take a look at the character oriented computer screens prevalent in the decade of the 70’s, it’s clear that a spreadsheet model would not have been adequate for the teletype-like devices of that era. It took cheaper bit-mapped displays for the electronic spreadsheets idea to become viable.
What’s obvious these days is that implementing systems with SOA can lead to better solutions as long as the inherent issues of SOA are tamed and appropriate service interfaces and service management governances are established.In this context, a service-oriented view provides many advantages:
1.Allows a directmapping to a business perspective. SOA allows the implementation of solutions that can directly mirror the business processes.In the past, system designers had to translate business processes into computer driven structures. System implementations based on awkward mappings of business requirements did work. However, given that the most natural way to describe business processes and organizational flows is through a consumer view and given the fact that older computers couldn’t cope with these views, the resulting IT systems almost always ended up being difficult to use.
In fact, SOA’s facilitation of a direct mapping with the business is key to the emergence of higher-layer business tools such as Business Process Modeling (BPM). BPM represents an even higher level of abstraction of automation which can be still be used to dynamically generate software solutions.The capability to give the definition of business processes to actual business users and then have these definitions used to generate actual applications is only feasible when these processes can call predefined services.
Without SOA there is no BPM.
1.
2.Enables Reuse of well-defined logic blocks. This is a Lego approach. Just as a building contractor assembles the skills needed to build a house, services can, in turn, call services. A general-purpose service can be re-used by several applications.
Admittedly, reuse is not a new concept. It has been the holy-grail of computer science for many years. For example, in the early days code reuse was sought via definition of macros or subroutines—chunks of source code that could be embedded into the application.
While this approach had the advantage of reusing source code providing generic functions (“Convert Data”, “Hash a Table”, etc.) it did have its issues. First of all, programmers would sometimes tweak the library code to better match their requirements; making the code non-reusable. Also, as new programming languages emerged, these libraries became outdated and could no longer be used.This is not to say the use of macros is no longer valid. The use of macros or functions for repetitious snippets of code is still a recommended best practice, but only within the confines of a single application.
In time, other code reuse techniques emerged. The most important of these were the linkable libraries. Unlike macros, the programmer could use the library without having to know the source code; thus giving some degree of protection against improper changes. However, as with macros, these static libraries become embedded in the executable code, using more memory and ultimately forcing the updating of programs whenever the libraries changed.
Enter the Dynamic Libraries. The familiar Windows DLL (Dynamic Link Library) is basically a library that becomes dynamically linked to a program, pretty much on demand. However, dynamic libraries require the use of a specific running environment (i.e. MS/Windows) making them viable only when executed in the same system as the calling application.
Dynamic Libraries represented a good step forward, and indeed they have been extremely popular as commercially available add-ons for development tools such as Visual Basic and more broadly under MS/Windows frameworks. Still, what if you wanted to use a dynamic library from a different platform? What if you could place dynamic libraries into any system and be able to call them from any platform?
Enter the concept of Services!
Just as with DLLs, you can acquire and use external tools and services, but more fundamentally, because of the loose coupling, you can run your service in a system completely different from your own. You can call a Linux service from a Windows application, or call a service located somewhere in a cloud.
SOA is all about transparency . . .
3.SOA is the foundation for transparency. If you call a help desk these days, chances are that the person at the other end of the line is based in a foreign country. Thanks to the lower communication costs and the benefits provided by educational standards and globalization, companies benefit by sourcing the services wherever they are the most cost effective. Likewise, you can run services anywhere and have them accessed by any authorized user. SOA allows you to place a function where it makes the most sense. But, since what makes sense today may not make sense tomorrow, SOA is also about allowing change with a minimum of effort. We can finally decouple the way we logically partition functions from the way we deploy physical computer systems.Service Oriented Architectures should provide as many of the following transparency tenets as possible:
·Access Transparency. Provide the ability to access the system from different devices and mechanisms.
·Failure Transparency. Designing the systems with automatic service failure fallback, without affecting the application.
·Location Transparency. The ability to deploy the system in any location.
·Migration Transparency. Allow minimum or no impact to the existing system when upgrading service implementations.
·Persistence Transparency. If the desired service has not been used, you can load it automatically. If it has been used before you can reuse the code already resident in memory.
·Relocation Transparency. The system should allow you to movie a service from machine A to machine B without impacting clients.
·Replication Transparency. The system should be able to provide the same service from different locations. This supports failure transparency and it can also be used to increase performance via horizontal scalability.
·Technology Transparency.As long as you get the service to do what the service is meant to do, the client shouldn’t care about the technology used to implement the service.
These transparency attributes facilitate legacy integration with new technologies. Since the implementation of the service is hidden from the service consumer, the service-oriented approach enables the integration of older legacy software with emergent software. Once older applications are properly encapsulated under the guise of services, it is also possible to gradually transition a system by re-implementing services one step at a time. This removes the necessity to incur in risky “big-bang” system migrations. Also, these transparency attributes are what makes emerging technologies such as Cloud Computing possible. Without transparency there are no clouds!
4.Simplifies software development by decoupling business processes, decisions, and data. The agility gained from SOA comes from the inherent simplification of the software design. It becomes easier to assign development of different modules to different groups and isolates the way the program accesses the data. More importantly, because SOA can mirror the business processes, you can create organizational structures in IT that truly mirror the business structures.
The diagram above shows this concept pictorially. It’s far easier to decompose and assign each of the various business processes inside the box to the right, using external services, than having to adopt responsibility for the way data is accessed or manipulated.
Now that I have sung the praises of SOA, I want to bring us down to reality just a bit. There are certainly many traps and difficulties in using SOA, and it makes sense to be aware of them so that you can use the well-travelled roads of successful past experiences as much as possible. This means that designing SOA is all about figuring out the concept of patterns and the way services fit into these patterns.
This blog covers the practical techniques, trials and tribulations associated with the transformation of IT systems from legacy technologies to systems using SOA and modern open systems. I also include the occasional interlude with rants about technology in general.
About Me
Name: Israel del Rio
Location: Atlanta, GA, United States
Israel has been recognized by Computerworld as one of their Premiere 100 IT honorees. Israel is a business and technology leader who has contributed the technology vision as key strategist and designer behind the enterprise technology roadmaps of large hospitality and travel companies. Israel has also have developed and deployed various mission-critical systems and in the process, he's been instrumental in creating and building effective and skilled development organizations.
