Linthicum on Mashup Governance...over the top?

I usually agree with David Linthicum of ZapThink when he writes about SOA. He understands the problems faced by organizations moving to SOA-based applications development. He gets that organizations need to put governance in place to avoid SOA turning into merely A Bunch of Services. (ABOS) I'm a fan.

But I think his recent article on mashup governance in the SOA Institute newsletter was off the mark. Not completely, but I think he's making the mistake that many of us make in thinking that mashups need the same level of governance, the same rigor, as composite applications developed within IT.

Here's my take: Not only don't they need the same level of governance, they won't get it.

So let's take a look at what Linthicum has to say. He's right when he says that mashups blur the line between the web-at-large and the behind-the-firewall enterprise. In his words, "...We could find that we soon live in a world where it's difficult to determine where the enterprise stops and the web begins. Now that's scary and exciting at the same time."

I completely agree.

Next he talks about how organizations need to prepare for mashups. Linthicum says that organizations need to design their SOA with mashups in mind, and on this point we also agree. Although SOA has come to be synonymous with the WS-* standards, it need not be. In fact, early SOAs were completely custom, and even today there are a number of competing SOA standards. Yes SOAs need to be designed around standards, but not just any standards. They need standards that make the services and widgets usable by many different SOA consumers.

Let's consider an example. I happen to know of a company that created SOAP-based services to expose some of their products capabilities. However, these services were built in the image of their old API. When it came time to use these services in a BPEL orchestration, never mind within a mashup, it was extremely difficult. These services weren't built with their end use in mind. More services by techies and for techies.

When it comes time to adopt your SOA standards, keep in mind that you will have many different SOA consumers. Not just by coders in VS .NET or Eclipse, but portals, BPM, and even mashups. And when it comes time to implement your SOA, remember that you may have multiple consumers about which you know little to nothing. So architect for flexibility, scalability and security.

Linthicum goes on to discuss mashup governance in some detail, and I'm afraid here is where he and I disagree. Readers will recall I've written about mashup governance before. I believe mashup governance is necessary. Mashers need version control. Mashers need some form of feedback loop to manage defects and enhancements. Mashers need to practice safe scraping and need to be aware of the provenance of the data they use in their mashups.

However, Linthicum believes mashups should be governed with the same rigor as Big Applications developed by R&D.

"So, what do you need to do to prepare for mashups? It’s a matter of addressing the following areas: Requirements, Design, Governance, Security, Deployment, and Testing. In essence, these are core architectural activities that are required to get you to the Promised Land of mashups on top of your existing activities when you create a SOA."

Nice in theory, but impractical in practice. Mashups are to be built by subject-matter experts, not IT professionals. Think Ultra-Agile rather than Waterfall when putting together a framework for mashup governance.

Requirements? How about "build it," "try it," and, "change it." In Big Applications this may be a license for creep. The on-ramp to disaster. But for mashups, inherently smaller and more agile applications, formal Requirements Management is nothing but unnecessary overhead.

Design? Those of us with Big Applications pedigrees, myself included, will cringe, but a formal design process for mashups seems like overkill. If it works, it's probably good enough. If it stops working, then fix it. Will some mashups be inefficient? Perhaps some mashups will be slow. So what? If the mashup is going to be mission critical, if it will service a large number of transactions, then it will likely be developed by IT, not as a mashup. If it's being assembled by a subject-matter expert, it's likely that inefficiencies won't be a problem. When you bet, figure pot odds.

Governance I've already talked about, as well as security. While Linthicum and I may disagree about governance, we completely agree about security. If an organization is going to over-govern any aspect of mashups, my choice would be security. Unlike SOA's notorious performance issues, security breaches have the potential to materially harm organizations. Jail time? Wall Street disfavor? Stockholder uproar? All these have resulted from IT security problems. 'Nuf said.

Deployment is an interesting issue because it should be easy, but can be difficult. After all, what's the harm in allowing a masher to put a mashup into production? A mashup can't hurt the operational environment, right?

Clearly some governance is necessary. I suggest IT provide mashers with a secure and stable deployment platform, distinct from any mission-critical applications, and then leave mashup deployment to the mashers. IT should not attempt to require the same deployment gates for mashups that they do for SAP upgrades, changes to the financial system, or updates to eCommerce. Once the secure and stable environment is in place, leave mashers to mash.

Testing, testing, testing. I think testing falls into the same category as requirements. Should mashers test all possible configurations, scenarios and error conditions? Sure. Will they? Hardly. Putting gates in place to force mashers to test will only put mashers in an awkward position. Should they ignore the testing gates, or falsify the documentation? Darwin and sheer embarrassment will take care of quality control in the long run. That may not be optimal, but quality control will be more effective if it is instituted as a bottom-up rather than a top-down requirement.

My take-away is that mashers need to develop their own governance practices. Trying to force-feed Big Application lessons to mashers will only result in resentment and eye-rolling. Just as with Agile teams, let mashers define their own rules, and let them modify these rules as they learn, discover and create.

ZapThink Predictions for 2008

Continuing in my series on the 2008 mashup predictions made by others, I'd like to talk about a recent ZapThink article by Jason Bloomberg. He makes several predictions about the future of SOA in 2008, to include the sale of ZapThink itself. I won't comment on the acquisition discussion except to say that it seems a bit odd for ZapThink to tack up a For Sale sign like that. I can only guess they are already in the final stages of negotiation and want to boost their price with a little self-promotion, or perhaps want to get an idea of how their current clients will react to the sale.

Whatever the reason, good luck with it.

In addition to ZapThink’s future, Bloomberg makes some straightforward predictions about SOA and mashups. To summarize: The economic downturn in 2008 may cause some businesses to slow down or cancel their SOA initiatives, especially if they are struggling with ways to define the ROI. This will be good for businesses that have already turned the corner on SOA, but bad for the businesses that cancel.
"...For those organizations, we do predict cancellations of SOA projects and deferments of SOA spending, to their detriment. Because after all, their competition might very well be leveraging SOA to be more successful during a downturn, leaving the unsuccessful companies out in the cold."

I agree with both points...to a point. In difficult economic times, businesses will tend to cut back on strategic spending, especially if the ROI is a bit ephemeral, as it can be in Big SOA initiatives. So yes, I suspect we will se a number of SOA initiatives delayed or cancelled.

However, I think this only applies to Big SOA, not to Guerrilla SOA. Guerrilla SOA projects will likely continue forward because their ROI is tied to the success of a specific business initiative, not to some hard-to-define measure of enterprise IT effectiveness. Organizations with Guerrilla SOA projects in the works will still be able to leverage their investment, "...leaving the unsuccessful companies out in the cold." See my previous post on the subject, as well as predictions made by Hinchcliffe and Chappell.

Later in his post, Bloomberg defines a mashup as a "...governed, managed composition of loosely coupled Services within the context of a rich, collaborative, Internet-based environment." Except for the fact that he left out human interaction, that's a pretty good definition. He goes on to say that "...enterprise mashups are what organizations actually do with SOA."

Spot on.

Organizations leverage their SOA investments in many ways. Old-style programmers can boot up Eclipse or VS .NET and use java, C# or other languages to build applications. When used this way the services available in the context of a SOA are really just another API. I'm sure to get hate mail for this, but for an organization trying to maximize their SOA investment, relying on old-style programmers won’t do the trick.

What about portals? Once the portal framework is in place, end-users can customize at will, without depending on expensive IT resources. Making use of a SOA infrastructure will help increase an organization’s SOA ROI. Unfortunately, portals don’t scale as well. Most people want one portal, or if necessary, two or three, but only under duress. See Oracle’s comments on portals. Portal proliferation is a problem that needs to be solved, so trying to increase your SOA investment's ROI by increasing the number of portals is a losing bet.

Another way to leverage SOA investments is through BPM initiatives. BPM and SOA have been joined at the hip for several years. Before SOA, BPM tools could define a process, but executing the process was something else. With integrated SOA BPM end-user could do their jobs while actually in the BPM tool. No more application-hopping. Additionally, with SOA the BPM vendors had a workable standard for integrations. Not every install had to have a unique set of adaptors built specifically to integrate the BPM tool with back-end systems. No more introspecting DLLs to figure out how to talk to home-grown applications. Give the BPM system a WSDL and it could move the world.

As with portals, BPM solutions don’t scale. They are big systems, mostly behind the firewall, built for process gurus to build big supply-chain-type solutions. BPM practitioners need to have training, they need Big IT support and they need to spend a lot of money to install and maintain BPM systems. So while BPM does help businesses get a bit closer to the sort of ROI the EA team promised as a result of Big SOA, it isn't enough.

Now let's talk about mashups. Specifically, business mashups. Mashups are easy to assemble. They are easy to deploy. They scale down beautifully, but scale to the enterprise as well. With new mashup platforms coming on the market, anyone with enough tech savvy to build Excel macros or administer a wiki can build a mashup. Better yet, mashups can take advantage not only of the SOA, but of the WOA as well, opening up the entire world to mashers. Best of all, business mashups put the aggregated data and display elements into the context of an actual business activity. You have the data you need, when you need it, and can take action on the data without having to jump from application to application.

So when Bloomberg says that enterprise mashups will become the killer use-case for SOA in 2008, I want to agree, I really do. But I don’t. First, I think business mashups are actually the killer app, not enterprise mashups. This isn't just a war over verbiage. Enterprise mashups don't include human workflow, but human workflow is a vital part of most applications. Maybe 2008 will be the year that the term 'enterprise mashup' starts to include human processes. If so, then I'll stop harping on business mashups. Until then I'll continue to make the distinction. Second, I think 2008 will be the year mashups get mindshare outside of IT. However, the business won't actually start cranking out mashups until 2009.

I hope I'm wrong and Bloomberg is right.

Good luck, ZapThink, on your pending sale. And thanks, Bloomberg, for your 2008 predictions

Bloomberg says mashups need governance as well as an easy development environment

Jason Bloomberg, a ZapThink analyst, is planning to hold a session at SOAWorld 2008, where he will discuss the need for mashup governance when developing Service-oriented business applications. (SOBAs) I won't be at his session, or even at the conference, but I read what looks like his abstract in the press.

Bloomberg's conclusion: mashups need governance, and here's why.

Way back in the day, when PCs first made their way into the business, we developed many and varried applications. These apps were complete one-offs and at some point they became too expensive for the business to maintain. The business dropped these apps right back in ITs lap. IT proceeded to toss many of them, replace others and rewrite still others. The apps didn't have legs for long-term usefulness. Their data sources were siloed, their interfaces inconsistent and their development environments primitive.

Now we have a new wave of business-built applications, or mashup-based SOBAs in Bloomberg's terms. If we don't want to repeat our past mistakes, we need to make sure this new crop of applications is architected for the enterprise. They will need to be loosely coupled to back-end systems, employ rich interfaces akin to those now common in the consumer world, and have a development environment that will make it easy for the original business mashers to maintain them as business requirements evolve.

To this Bloomberg adds,

"...Clearly, no business would risk allowing any of its employees to assemble and reassemble business processes willy, nilly, with no controls in place to ensure that the resulting SOBAs followed corporate policies."

The problem with this statement is that employees are assembling and reassembling SOBAs willy-nilly. The lack of controls to make sure these applications follow corporate policies is one of the very things that make fast innovation possible. The ungoverned free-for-all of mashup construction is part of the mashup DNA.

So mashers are mashing, not just in the wilds of the uncontrolled web, but also behind the firewall. We don't want to stop them, and we can't trust them to self-govern. Not because they are evil, lazy or stupid, but because they should be busy innovating, not fighting a version control system. Bloomberg seems to agree, and he goes on to say that when governance is put in place the mashup environment starts looking less like a Web 2.0 environment and more like a standard development environment.

Not necessarily. The trick is to integrate light-handed governance with a mashup construction environment that looks and feels like a MS Office application, or better yet, iTunes. Certainly not like Popfly or Eclipse. But we also want mashup-based SOBAs to have legs for the long term. So we need security. We need version control. We need change management. What we don't need is to discourage our would-be mashers by making governance obstructive. Governance should be a silent side-effect of building a mashup. A Side-effect ignored right up to the point where somebody accidentally honks up one of the production mashups.

Is this a mere dream?

Yes and no. There are some mashup construction tools on the market that incorporate version control. There are some that include security. There are some that include change management. Unfortunately, there are none yet that include all the old application development disciplines. So yes, it is a dream for the moment.

However, the fact that some mashup enablement vendors are already thinking about governance tells me that the tools will evolve, and likely very quickly. So maybe this isn't a dream after all. Maybe it's just a product roadmap issue. Unlike our first go-round with business-built applications, we have some experience with what does and doesn't work. This time around IT should be on hand to provide a safety net rather than a brick wall. This time things just might be different.

If we've learned from the past we will create an environment where business-built applications will remain with the mashers, to wax and wane and ebb and flow right along with the business.