Gartner says that initial SOA adoption rates are slowing. I think they are wrong.

I just read a Government Technology article stating that Gartner says there is a dramatic decrease of organizations planning first-time SOA projects. Here's the money quote.

Since the beginning of 2008, there has been a dramatic fall in the number of organizations that are planning to adopt SOA for the first time. In 2008, this was cut by more than one-half, down to 25 percent from 53 percent in 2007, while the number of organizations with no plans to adopt SOA more than doubled from 6 percent in 2007 to 16 percent in 2008.

Unfortunately, I no longer have direct access to the Gartner research (One of the things about leaving my former employer that I will miss the most.) so I have to comment only on the secondary source, the above mentioned article. That's too bad because I can't look at the methodology used to gather the research. Knowing Gartner from the past, however, I can make some guesses.

I'm guessing they talked to IT departments and asked, “So, what are your spending plans for SOA?” and probably, “Have you already adopted SOA within your organization?” Those aren't bad questions, but I think we could all have predicted the results. Given today's economic uncertainties, Big IT isn't going to spend lots of money on big IT projects that don't have an established track record for success. Unfortunately, the success of SOA in the enterprise, at least Big SOA, is not a given.

But these results don't even come close to telling the full story. I personally have worked on three projects over the past few months that were absolutely based on a services architecture. However, if you were to ask the IT departments of these organizations about the projects, they would not have tagged them as SOA.

Why not?

Because these weren't 'SOA projects,' they were business initiatives whose solutions happened to make use of SOA. In all three cases we didn't buy expensive middleware to run the software. We didn't embark on an orgy of service writing to SOA-enable myriad legacy systems. In two of the three cases it's probable that I was the only person who knew the underlying architecture of the solution was services oriented. These were true Guerrilla SOA projects.

So while I completely believe that Big IT departments are slowing down in their implementation of Big SOA projects, I don't believe for a minute that these same organizations aren't expanding their use of SOA. It's just that IT doesn't know. And what they don't know, they can't blab to Gartner.

So take what Gartner says with a grain of salt. Sure some of the Big IT SOA projects may be on hold, but don't assume that Big IT owns all the action. SOA is happening all over, without IT knowing anything about it.

Hello again, everyone. (Or all two of you)

Loyal readers may remember that I stopped writing this blog when my boss instituted a policy requiring all blogs be approved by PR before publication. Since I wouldn't submit my blog to PR, I stopped writing it, at least under my own name.

I'm no longer with my prior company, so it is time to start up this blog again. I've got a backlog of issues to write about. So many, in fact, that you can think of this as my Q4 2008 editorial calendar.

• Finally, after all my pleading, all my whining and all my cajolery, I get to test drive JackBe Presto. We've gone a round or two in the past, with JackBe refusing me access, and me swearing that I would give them a fair review. You can read my requests here, here and here. I'd like to say that my razor sharp reasoning finaly convinced JackBe that free and open access to their technology is the best marketing strategy, but it wouldn't be true. It's just that I now no longer work for a company JackBe perceives as a competitor.
  
  So far it has been a bit of a trial. Since I had to give back my corporate computer, I'm starting from scratch with a laptop having only, god help me, Vista, installed. Before I can install the Presto suite I have to install the JDK and Eclipse, which reminded me of all the myriad libraries and jar files I already had installed on my old machine. I suppose I'll figure out what's missing when I write code and don't have the appropriate jars...
  
  So I haven't quite gotten Presto installed yet, but hopefully some time today I'll be able to dig in.



• The financial services industry has been an early adopter of business-driven initiatives using guerrilla SOA and mashups. Needless to say, this market has other things on its mind just now. The question we are all asking is, "What's in store for us given the meltdown?" I've been building mashups for a handful of global banks for the past nine months, and I've got some opinions about what we can expect for the next nine months.
  


• Interneer looks like another promising application builder tool, along the same lines as Coghead and ActiveGrid (WaveMaker). Once I finish with Presto, I'll take Interneer through its paces and let you know how it stacks up against its rivals.



• Despite our best efforts, the hype surrounding SOA is dimming. This is actually a good thing, not a bad thing. SOA is no longer newsworthy because it is passe. Mainstream. Old school. Dull. As SOA moves to the mainstream, raging debates about SOAP versus REST, stateless web services, and whether Event-Driven Architecture (EDA) is an extension of SOA, seem to be naval gazing. I'll give my take on the state of SOA, where it is, and where I think it will be moving in the next year.



• The industry is lousy (Nasty word. Read the etymology here.) with WS* standards. Along with the product reviews, I'll start to explain some of these specifications and how they fit in to the evolving SOA 'standards.'

That should take me through Q4 2008, along with random rants from the field. Feel free to write with suggestions for topics, or just to say, "Hello."

It's good to be back.

Farewell Business Mashups Blog...I don't want a PR review

To all three of my readers out there, I'm sad to say that this will be my last post for this blog.

We now have a policy stating that all external communications have to be run through our PR department. Since I'm not comfortable having my blog reviewed, I'm going to stop this one.

No need to weep in despair, however. I will be blogging under another name and another account, so keep a lookout.

Sincerest Regards,

Kelly A. Shaw

IDC says HR to help Web 2.0 adoption. Um...what?

I just read an article by INQUIRER.net writer Lawrence Casiraya that had me raising my eyebrows. He interviewed IDC's Shalini Verma about the concept of "unified communications" and how Web 2.0 technologies such as social networking can help distributed organizations collaborate more effectively.

Here's the money quote.

So how can businesses begin to embrace Web 2.0? The task may fall into the hands of the human resource department.

What was Verma thinking? HR, together with IT, have been the main forces preventing adoption of Web 2.0 technologies in the workplace. According to Verma, HR should conduct a survey to find out what technologies are already being used and then make a plan for company-wide adoption of the most useful tools.

Well, I doubt that will work. Pretend we are listening to a conversation between two 'screenagers' when they get a survey from HR...

Nancy Networker: Did you see this survey from HR? They're trying to find out what sort of web apps we use. Why do they want to know?

Wally Widgetuser: (Smirking knowingly to Nancy) The survey says it's because they want to help us collaborate more effectively across the company.

Nancy and Wally have a good, cynical laugh.

Nancy: Well, I'm not telling them anything. The last thing I need is for them to shut down Facebook. It's the only way I keep in touch with the guys in Thailand.

Wally: You got that right. If I tell them I'm building mashups to keep track of my customer information, they'd freak.

Nancy: I'm just going to say I use Google Docs. I can live without them if HR and IT shut them down.

...

OK, the situation may not be that bad, but I'd wager that, given HR's history, there won't be a lot of trust between HR and the employees.

What's the answer?

I think HR could take the lead, but they'll have to do some groundwork first to establish some trust and credibility with the rest of the organization. Executive sponsorship wouldn't hurt either. In my own case, I spent well over a year blogging under a different name, knowing I'd be shut down if HR found out. I came out of the closet when Serena's executive leadership started promoting the use of social networking.

In addition to getting very visible executive support, HR could start publishing internal case studies about how people in the company use Web 2.0 to promote collaboration. Maybe someone in HR uses pbwiki to collaborate on ideas for the company picnic. Maybe someone in PR uses Second Life to conduct press conferences. (OK, that one's off the wall.) Perhaps someone in sales uses a mashup to get information before a customer call. If HR spends some time up front promoting the use of these collaborative tools, then maybe they will be trusted.

Maybe.

But we all remember when HR shut down MySpace, told us we couldn't blog, and forbade us from giving recommendations on LinkedIn. They've got a ways to go before we trust them with the dark secret that we use OpenKapow to scrape competitive information, or that we swap musical stations with clients on Pandora.

News from the mashup trenches

I’ve neglected this blog for a while, and now it’s time to get back to it. I didn’t think changing jobs would have much of an effect on my blogging output. Wrong again, Shaw. These days I have to do my reading and blogging mostly after hours, and that does change the dynamic. What takes priority, doing laundry, weeding the garden, helping with homework, balancing the checkbook, playing fetch with the dog, splitting wood, or...writing this blog? I'm afraid lately it's been those other things, and I'm sorry.

On the bright side, I’ve been in the trenches more than I have been in the past couple of years. It’s one thing to read analyst reports, discuss findings with customers and partners, write academic papers and pontificate from on high. It’s another to jump in and actually develop mashups. It is definitely harder to view the entire industry from a trench. Especially in the context of a fixed price contract. On the other hand, I’ve become reacquainted with the reason I went into technology in the first place: I enjoy it. I’ve written more applications (Don't be fooled. Mashups are applications.) in the past couple of months than in the past two years combined.

In a departure from my standard mashup posts, I’d like to share some things I’ve learned about mashups in the past few weeks rather than commenting on mashup news or reviewing some new mashup tool. Some of my experiences have reinforced what I thought before I embarked on this change in job title. Some have made me change my thinking, and in ways that might not make my current employer happy. Sorry in advance, Serena Software, if what I have to say doesn't agree with our marketing message.

First, I’d like to talk about the idea that mashups will allow the business to deploy their own applications without having to bother IT. I never thought this would work. After all, even applications in the tail will need access to back-end systems. Even applications in the tail will need version control of some sort. Even applications in the tail will need governance, even if that governance is light-handed compared to governance used to manage more strategic applications built by IT itself. After my time in the trenches, I’m even more convinced that mashup development has, at the very least, to be a loose partnership between the business and IT.

Let me give you an example.

I built a series of mashups as a prototype for a hardware technology company who wanted to get better control of their rebate program. It was a very intense three-day engagement where we spent the entire day working with the customer to understand their business problems, and then I got to spend the nights building mashup prototypes. (I got about four hours of sleep during those entire three days. I don’t recommend it steady-state, but it was nice to know I could get back to my developer roots if I had to do so. I even had my first fully caffeinated non-diet pop in years. I drew the line at pizza, however.)

The meetings with the client were sponsored by IT, and were run by the business. They were sponsored by IT because the business didn’t know mashups from shinola, but IT did. The business didn’t have a clue that mashups could help them, but IT did. The business was frustrated at IT’s repeated attempts to solve their growing rebate program problem, and IT responded by searching for alternatives to their same old development paradigm.

IT and business worked together to help us define the requirements. IT and business together helped identify what infrastructure was already in place to support the effort. IT and business worked together to help define what parts and pieces of the proposed new ‘system’ could be managed by end users and what parts and pieces would be governed by IT. In other words, there was deep collaboration between IT and the business. And while there were definitely self-preservation undertones in IT's motives, there was also realization on both sides that the old way of doing app dev didn’t work. Everyone was on-board to try something new.

If business had tried to run this meeting without IT, mashups wouldn't even have been on the table. It would have been strictly a shadow IT project that may or may not have taken advantage of recent Web 2.0 trends. And it would have been killed as soon as it became apparent that the project required access to CRM and financial systems. If IT had tried to run the meeting without the business, then we would likely have had a great architecture, and a deep understanding of how the pieces and parts were going to work together, but we never would have understood the deep and abiding frustration felt by the business, nor the financial imperatives that required something be done about the problem soon.

Both had to work together for success. Perhaps this isn't true for all mashup projects, but I suspect it will be true for many of them, especially any that require integration with the existing IT infrastructure.

Here’s something else I learned in the past few weeks. We’ve been saying for a while that business users would start writing their own applications because they can no longer wait for IT to do things for them. The idea is that account reps, shipping clerks, marketing program specialists, etc. would write their own mashups, and we will give them the tools to do so quickly and easily. Now that I’ve been out in the world a bit and have seen what is really going on, here’s what I think: Bullshit.

Even the young 'Net Gen' men and women on the business side don’t want to write their own apps. They may be forced to do so, but they aren’t doing it by choice. That’s because these younger workers want to make progress in their chosen career. Account reps of any age want to spend their time selling, not writing apps. Business analysts want to spend their time figuring out how to beat the competition, not writing apps. Marketing program specialists want to put programs in the field, not spend time writing apps.

So while these end users may be writing apps, they are doing so reluctantly because it takes time away from their chosen careers. Mashers are the exception rather than the rule.

What I do see is a rise in shadow IT. Certainly the business can’t wait for IT to get to their applications in the tail. But the business isn’t building these apps itself. they are hiring outside help. Unlike the wholesale shadow IT trend in the 80’s however, these new shadow IT projects have to operate below the radar. They have to be inexpensive enough to be funded within a departmental budget and not cause an IT governance blip. They also have to show fast ROI so that, when the project finally gets outed, the business can justify what it’s done in a ‘ask forgiveness not permission’ model.

(Note: web widgets are the exception. I'll write more about them in a future post.)

Here’s something that may surprise you. These shadow projects may be instigated by IT as well as by the business. IT itself is getting frustrated with its own inability to react quickly to requests for applications in the tail. In fact, I worked on a project a couple of weeks ago where I designed a guerrilla SOA infrastructure for an IT department that was trying to get an internal project into production before Big IT saw what was happening. The irony of IT setting up shadow IT to subvert Big IT made the job all the more enjoyable.

The take-away for this post is that IT and business have to work together for successful mashup projects, just as they do for any development effort. (Let's all hold hands and sing Kumbaya.) However, when business does take the lead, it is likely to be in the form of a low-cost shadow IT project rather than in the form of an internal masher. I know that isn't doctrine. Perhaps my opinion will change in the future. But for now, I calls it the way I sees it.

Mashups in the financial sector aren't just for the back office

I spent last week in NYC talking about mashups to a number of customers in the financial sector. I love going to NY, and last week the weather was beautiful and all the designer dogs were out in force in Central Park. The outlook from our financial institution clients wasn’t quite so perfect, however.

Here’s the message I heard over and over again: Mashups in the financial industry were only good for the back-office, not the front-line. So while we can help make their order-to-cash process mean and lean, we can’t help them bring innovative products to their customers.

I understand the reasons. Financial institutions have to be conservative. When bankers and investment institutions stray from the straight and narrow, somebody will likely be in front of Congress right before they go to jail. S&L bailout anyone? Would you like to invest in some junk bonds? Let's depend on Enron for our retirement portfolio. Oh yes, let’s not forget subprime mortgages.

So while I understand their reluctance to adopt mashups on the front-end of their business, I think it is a mistake. I wouldn’t expose the banking systems until we get better mashup security. But financial institutions have a lot of other offerings that aren’t tied directly to their transactional back-end systems.

Why should they bother?

Financial institutions have to walk a fine line. They are in a constant struggle to balance the need for governance, the heavy load of compliance, and a cutthroat competitive landscape. And the financial sector depends heavily on technology to be competitive. And not necessarily technology within a traditional IT organization.

According to a Booz Allen Hamilton study, for every dollar spent on ‘real’ IT, most industries also spend 78 cents on ‘shadow IT.’ That is, IT funded directly by, and implemented within the business. In the financial sector I’d be willing to bet the ratio is much higher. One bank employee I talked to said that embedding IT within the business is a necessary practice just to stay competitive. When one bank innovates, the others have to be right behind. That means tight coupling between the technologists and the business so new and innovative offerings can be out the door fast.

This sounds like a perfect job for mashups.

I’m not an expert on the institutional side, but I do have a number of personal and small business accounts with a couple of handfuls of banks and investment firms. As a consumer of financial services, I’ve got a number of ideas for how they could use mashups without compromising their core banking systems.

How about a money management mashup? Most banks have money management information, but wouldn’t it be a good idea to mash information from multiple sites, mashing book sales from Amazon? Then not only could the company provide good value to their customers, they might also be able to turn their website into a profit center.

Ditto for investment information. I have accounts with several investment firms, yet when I want to do any investment research, I have to search Yahoo! finance to get the financials and Google for any relevant news. I’d use a mashup that pulled that information together into a single page.

How about a mashup that pulls together many investment strategies? Again mashing up books from Amazon and information from some of the leading personal finance strategists. How about a mashup that lets me compare and contrast a company’s performance against some of its nearest competitors? Then mash in some Google Docs to let me save my analysis so I can retrieve it later.

And on the other side of the equation, banks and investment firms should turn some of their free web content into widgets. For example, Fidelity has a DJIA chart on their home page. If this was a widget, and they modified it to show provenance, Fidelity would get free advertising whenever someone added the widget to a mashup.

I’m not buying that mashups aren’t a good fit for the financial services industry. Most of the innovation, at least on the consumer side, isn’t in the back-end transactional systems. It’s out front, providing services, information and advice to customers.

Again, a perfect job for mashups.

Smoking pot and stealing music. Some things never change.

OK, I admit I wrote that title to see if I could trick some people into reading this post. But really, I will actually compare the two. My motivation is a recent article by Linda Tucci, a writer for SearchCIO.com. It made me smile because it was about how millennials don't respect organizational, hierarchical or other boundaries. These millennials are going to cause security headaches because they don't respect IT policies and procedures either.

This is a hot news flash?

In her defense, Tucci was simply reporting on the results of a Symantec survey, first blogged by Symantec employee Samir Kapuria. But those of us who have either been interacting with these younger workers, or have children of that age who are about to enter the workforce, already know we've got an IT compliance disaster waiting to happen. I know that my own daughters have absolutely no respect for IP rights. In their minds, anything on the public web is and ought to be theirs for the taking. Lectures about the morality of downloading music and video fall on deaf ears. As do discussions about network security and malware.

These conversations reminded me of discussions I had with my parents about pot smoking when I was a teenager. My parents lectured me on the evils of marijuana, but in my peer culture at the time, nearly everyone smoked it. In fact, the University of Michigan and Michigan State University had parties every spring, called the Hash Bash , to protest pot laws. While I never had the guts to light up on the steps of the capital and get carted away in nonviolent protest, I wasn't above cutting class (I was in High School at the time) and joining in the party.

Bear with me. This isn't just a stroll down memory lane. It really is about mashups.

In my view at the time, and the view of many in my generation, pot was not only a civil right, it was symbol. Sure, flaunting the anti-pot laws was fun. But it was also morally defensible to break the laws in protest of unnecessarily restrictive rules and regulations. I believed my parent's views were not only behind the times, not just old fashioned. They were wrong, and nothing they said changed my mind.

That's the attitude I see in my children. Talking to them about network security, IP rights, privacy, and even footnoting, is like talking to a brick wall. For them, free access and use of all information is not only a civil right. Breaking IP and security rules is a form of political protest against unnecessary and restrictive rules and regulations. Here's the money quote from the article.

When asked whether they feel entitled to use whatever application or device or technology they would like, regardless of source or corporate IT policies, 69% of millennials said yes, compared with 31% of other workers. Indeed, 75% of millennials have downloaded software on their work computer for personal use, vs. 25% of other workers -- even though 85% of the organizations surveyed indicate their policies restrict that practice. Millennials also regularly store their corporate data on personal devices: 39% on personal computers, 38% on personal USB devices, 20% on personal hard drives and 16% on personal smartphones.

CIOs should be very afraid of these survey results. Especially since the same survey showed that IT and other corporate leaders believe they have good rules in place, and that everyone understands and mostly obeys them. Those who don't comply get fired.

Most of the Millennials I know aren't afraid of losing their job. They aren't going to get intimidated by getting yelled at by the boss. Organizations who try to restrict the use of personal devices, who prohibit social networking and other Web 2.0 applications, who try to legislate the use of web content, are either going to be mired in lawsuits, or are going to find that they can't hire innovative and out-of-the-box thinkers.

What's the alternative? I'd like to fall back on the agreement I've now forged with my children. I've worked for companies that blocked sites, monitored email, recorded web access and filtered out 'bad' words in IM. I didn't care for it, and I wasn't going to turn around and do the same thing in my own home. Nor could I simply ignore the problem. While I know pirating is illegal, I also believe it is wrong.

We finally came to a compromise that we worked out together. They don't completely like it, still believing I'm backwards-thinking. I don't completely like it, believing they will have ample opportunity to break the law. But because it is a negotiated agreement rather than a dictated policy, I have some hope of success.

• They are now free to download anything that is really free, not pirated free. MySpace is full of 'really free' music and video, and a lot of it is quite good.
• They can keep their MySpace accounts, but they must allow me access to their profiles. (Neither of them like Facebook. Probably because I use it.)
• They have an iTunes budget. It isn't large, but it is enough to buy a few songs now and then.
• They won't download software without my approval. I can only deny the download if the software is harboring malware, if it's content is objectionable or if it will cost too much.
• They agree not to store any pirated content on their computer.
• I've asked them not to 'borrow' pirated content from their friends. I've told them I'll throw away any media that I believe has pirated content.

So far it's either working or they are very good at making it appear to work. I won't take bets.

I think IT has to do something similar. In old paternalistic, hierarchical organizations it might be considered a sign of weakness to negotiate policy with subordinates. Our millennials are going to change that mindset. Corporate leaders will need to work with their employees rather than dictate to them, or they will face not being able to recruit or retain the quality of worker they need. So instead of a restrictive IT policy based on sanctions and Big Brother thinking, we'll probably end up with something similar to the agreement I have with my kids.

With respect to mashups, I think we'll also end up with something similar.

• If you mash content from the web, note the source.
• If you mash content from behind the firewall, make sure the content isn't sensitive.
• If you are mashing services from the web, make sure they don't have viruses, understand the costs, and try to use reputable sources.
• If you are mashing services from behind the firewall, make sure the services don't expose sensitive information.

Are these guidelines bulletproof? Of course not. There isn't an IT policy today that's bulletproof. What these guidelines do is help the masher understand what the issues are and why he/she should be concerned. These guidelines treat the masher like an adult, not like a naughty child or convicted felon that must be monitored.

Some may think this is mere kowtowing to these new bad boys entering the workforce. Further proof that the world is going to Hell in a hand basket. Me? I can't wait until these younger workers roll in and shake everyone up. Will we have chaos? Will there be security problems? Are there going to be mistakes, upheavals and disasters?

Most certainly. But there will also be progress.

(Note to horrified readers: I stopped smoking pot in High School. I didn't then, and still don't, think there is anything wrong with it. I just needed to get my act together academically. After HS I always ended up in jobs that required a security clearance. And now it just doesn't interest me.)

Can I take back what I said about BPM and mashups?

Back what seams a very long time ago, but was actually only October last year, I wrote a post suggesting that BPM was another form of business mashup. Like-minded blogger Sandy Kemsley agreed, and bemoaned the lack of mashup understanding in the BPM community.

I've kept an eye on the BPM community looking for activity around mashups. I've seen a few comments around the edges, but nothing I would call a trend.

I was confused. Presentation mashups and BPM may have little in common, although I would suggest that Tibco, with their focus on RIA composite applications, have been playing in the presentation mashup space for a while. (Others will likely disagree, and we can have a discussion.) However, once you get 'out of the map' and start considering mashups from a business or enterprise angle, the overlap becomes pronounced.

(Note: my good friend Summer Ficarrotta coined the term 'out of the map' months ago to help people understand that Google Maps mashups at the glass weren't the only mashups on the block.)

I think I fell into the trap of thinking that because two things look the same and act the same, they should be the same. (Do you remember Papa Bear in The Big Honey Hunt?) I have a recent article by TechTarget writer Rich Seeley to thank for getting my head out of my trap. Through his insights about BPM and SOA roles and responsibilities, I now understand just how different BPM and mashups are.

In his article about the business and IT roles within SOA and BPM, he lists eight different roles involved in creating a BPM/SOA application, four each in business and IT.

Business Roles

• Business Leader: Responsible for overall business performance, compliance and governance.
• Business Professional: Manages business performance and decides on strategic and tactical needs for a specific area of responsibility.
• Business Analyst: Interprets business professional and business leader requests and documents them into process models.
• Process Analyst: Specialized business analyst who concentrates on the simulation and analysis of processes in their business environments and their interactions.

IT Roles

• IT Leader: A Business Leader responsible for delivering technology solutions that enable the business.
• IT Analyst: Interprets business analyst inputs/requirements in the context of IT capabilities, works with team on IT-based business process improvement.
• IT Architect: Defines basic operational imperatives in the provisioning of IT services with a focus on resiliency, reuse and adaptability.
• IT Developer: Follows IT architectural principles to create "building blocks" for the construction of applications.

Whew! Imagine putting a mashup together where you needed four different roles to put an idea together before tossing it over the wall to IT where four more roles did the implementation. I'm not saying this is too much for BPM. High-value and highly-complex systems need governance and discipline during their ideation and construction, regardless of whether they are implemented as a custom application built from scratch by App Dev, or as a business process built atop a BPMS.

What I'm saying is this is too much for mashups.

The premise of Serena's paper on the long tail of applications development is that there are many applications that IT never implements because they aren't high value or complex enough to merit IT involvement. This is 'The Long Tail' of Applications Development. At the time we wrote the paper I thought that BPM could be one of the answers. After all, BPM was all about empowering the business to define and build applications.

And that's where I made my bloomer. Mashups need to be easy to build, easy to deploy and easy to maintain. Mashups need some governance, as I've written about here, here, and here. (Yes, this is a subject I care about.) Just not as much as an expensive and complex App Dev initiative. They also need some lifecycle management. Again just not as much as a typical App Dev initiative.

Now I'll go on record as saying they don't need as much governance and lifecycle management as Big BPM either.

Applications Development has a long tail, a tail that can be serviced, in part, through the use of mashups. Contrary to what I've said before, BPM also has a long tail. A tail that can be serviced, in part, through the use of mashups.

What I won't say any more is that BPM is a business mashup platform. It may look the same. It may act the same, but it isn't the same at all.

Dapper has a lot of promise, but boy can it be annoying!

I haven't done a review in a while, so I thought I'd get back into it, starting with Dapper. I came across them in a Hinchcliffe blog entry about the most promising mashup tools. He had Dapper on the list, along with other more well-known tools such as JackBe, who still won't let me test their product.

Dapper allows developers to pull content from websites and expose them using various APIs. There is nothing new about this. Nearly all the products I've reviewed have this capability. Dapper differentiates itself with the number of different APIs it supports, which I'll get to later. The list is very impressive, but doesn't include SOAP.

Too bad about SOAP, but I understand. SOAP is overkill for screen scraped content. You don't need transactional integrity or security (not that SOAP has that problem licked) if you're just pulling read-only content from a page. Still, it does mean I can't use the content in something like a BPEL orchestration. Clearly that use isn't something Dapper has in mind. Then again, neither did Intel MashMaker or Kapow.

For consistency with my other reviews, I attempted to create a feed from the news page on the Serena website. No luck. Dapper couldn't load the page. So next I went to Digg to add recent news items. While Dapper loaded the page, it scrambled the page elements. I couldn't pull the top news stories from digg/science or digg/technology.

Next I went to my own blog to see if I could pull my content into a feed. I know, everyone can already get a blog as a feed, but this was an experiment. The instructions for Dapper say its selection algorithm will work better with multiple similar pages, so I added the links for my most recent three posts and went to the next step, selecting the contents to scrape.

Unlike other screen scraping technologies I've played with, Dapper has some smarts built in. Their algorithm supposedly helps mashers select the right content to pull into the API without having to mess around with Xpath. Well, certainly there is an algorithm in place, but I found it much more of an annoyance than a help. I couldn't get selections to work correctly, and when I tried to de-select manually, I got a page script error and the interface stopped in its tracks. I couldn't interact with the application at all, and had to reload the entire page with a new URL.

I went back to the start and tried it again, and got similar results, except that I didn't even try manually de-selecting page elements. Instead I wondered if I should not give Dapper multiple pages to work with. I selected 'Back' in the interface to return to the page where I selected my inputs, intending to remove all but the latest blog entry. And guess what?

Right! I got an error on the page again. The 'Back' link didn't work either. At least this time the interface didn't freeze up.

After a while I defined something close to the selections I wanted. (I never did get the exact content.) And now for the reason Dapper is different. The reason I kept playing with Dapper despite its many flaws: I could expose the content as POX, RSS, Filtered RSS, HTML, a Google gadget, a Netvibes module, a PageFlake, a Google map, an image loop, an iCalendar, Atom, CSV, JSON, XSL, YAML or even as an email. True, a lot of these formats don't make sense for blog content, but it's nice to have the option.

I especially liked the preview that let me take a look at the content before finalizing my output format choice. That was sweet.

Bottom line. I wouldn't use Dapper today for production mashups. It just isn't ready. However, when Dapper fixes their algorithm so it isn't annoying, when they do some serious debugging, when they fix their performance issues and when they otherwise clean up their usability, it will be one killer application for creating mashable content.

The reports of IT's death are greatly exagerated. Or are they?

It’s been a while since I wrote in this blog. I’ve had a lot of things happen, so I hope you will all forgive me. First, Marketing decided they couldn’t afford a pet nerd any more, so I had to find another job. I can’t blame them. They’ve let me run wild for a year, reading and writing and playing with software. How many people get to do their dream job for even a year?

Serena’s professional services department has taken me on to do for a fee what I used to do for free: give advice to customers about best practices with respect to ALM, SOA Governance, Agile development, Web 2.0 and mashups. I’ll still have the chance to read and write, and I’ll get involved with customers earlier in our relationship. In all, a very fair trade.

I also took a vacation before hitting the ground in services. I’ve been a skier for 20 years or so, but decided to put them aside for the week and learn to board. Why? I had become complacent and was no longer progressing or pushing myself. I wanted something new, and decided to try the board. Having heard from other skiers that it’s very hard to switch, especially at my age, I deliberately left my skis at home.

A week later I had a bruised tailbone, a wrenched shoulder, a twisted knee, possibly a broken thumb and a couple of very sore wrists. I also can ride. Not well. Not nearly as well as I skied. But even though I’m back on the greens, I’ve progressed.

I’m back now and I’d like to talk about a post by ZD Net blogger Michael Krigsman. In his recent post he talks about the very real possibility that IT will become an endangered species, going the way of the steno pool, vast accounting departments keeping the books with adding machines, and company cars. The article was interesting, citing the now dusty ‘IT as commodity’ argument as well as a number of others. Not quite as interesting as the article are the comments. Except for a very few insightful posts, they fall into three categories.

• IT sucks and deserves to die.
• Tech illiterate users suck and will beg IT for mercy when their systems crash.
• IT isn’t going away, it’s just evolving into a new life form.

They remind me of what poses as political debate on slashdot. “Democrats suck.” “No! Republicans suck!” “No! Democrats suck.” And so on. I keep reading because once in a while I come across some creative invective that makes it all worthwhile. In case you were wondering, there wasn't any noteworthy invective in the comments.

When I first read the article I believed Krigsman was being deliberately provocative. I thought, "He’s picking a fight. I know it, he knows it and I bet everyone else knows it too." Once I started considering his points, however, I wasn't so sure. In fact, I think all three conclusions posted by Krigsman's readers have some truth. In some cases IT does deserve to die, in some cases users will still rely heavily on IT, and in some cases IT will survive, just not in it's current form.

Krigsman bases his predictions on seven trends, but I think there are really only three: IT is a commodity, IT isn’t in tune with the business, and the world is changing around IT.

IT is a commodity. Krigsman cites three different examples of IT as a commodity. First, IT has become defensive rather than offensive as applications such as email become as necessary to everyone as electricity. This is the argument Carr made in his post dot-com article about why IT doesn’t matter. Second, applications that used to be differentiators, such as CRM, are now so standard they can be offered as a service, completely bypassing the need for IT support. Salesforce.com is the poster child for SaaS, but many others exist, and even venerable enterprise app vendors such as SAP are planning to offer their software through subscription services. Third, software vendors are volume pricing their offerings, pushing IT organizations to make single-vendor deals. In other words, vendors such as Microsoft and IBM are pricing their offerings as commodities.

But is it a valid argument that IT as a commodity means IT is doomed to extinction? Most IT professionals would say, “No.” Older, established capabilities such as file servers, email, and enterprise apps may move outside of the firewall, but there will always be the need for newer and better capabilities that haven’t yet been commoditized.

I’m not sure either position is correct. If all a company needs is vanilla IT, then it doesn’t make sense to maintain an internal IT organization. If all you need is standard power, would you bother generating your own, or would you buy it from the grid? In some cases, however, innovative IT is a strategic weapon, not merely a defensive necessity. Wal-Mart, Google, eBay and Amazon have shown that IT can be such a weapon. Businesses that differentiate based on innovative customer interaction, a hyper-lean supply chain or platform as a service, are going to need their own IT.

The bottom line is that commoditized IT will put some, but not all, internal IT departments out of business.

IT is out of touch with the business. Krigsman discusses two manifestations of this root cause: IT leadership is alienated from corporate senior management, and corporate senior management doesn’t understand how IT can be a strategic asset.

Both of these points have the ring of truth. Where IT is a provider of services rather than a business partner, IT is nothing but a defensive necessity and the CIO merely a cost cutter, probably reporting to the CFO. Under those circumstances, why would senior leadership pay attention? Would the CEO reserve a place on the board for the person responsible for ordering the most cost effective paper clips? A corollary would be that senior leadership also won’t understand how IT can be a strategic asset rather than a cost center.

But does this signal the end of IT? Again, not necessarily. Let’s consider Wal-Mart versus K-Mart. K-Mart didn’t care about IT and didn’t consider IT as a strategic asset. Wal-Mart did, and put in place a supply chain system that drove K-Mart into bankruptcy. K-Mart's IT definitely took a hit then, although I bet is has rebounded since, and not merely as a provider of email. In the Darwinian business world, organizations win when they make the best use of their strategic assets. They lose when they don’t.

So the bottom line is similar. Where the IT/business divide is strong, it will put some internal IT organizations out of business.

The world is changing, lessening the need for IT. We won’t need IT because we can get everything we need from enterprise-ready consumer-inspired apps. These apps are getting to market faster and better fueled by VC investments.

This is a hotly debated topic, very much tied up with Enterprise 2.0 and mashup debates. The business wishes that it were true, IT knows it isn’t. The reality is somewhere in between. I already use Facebook and LinkedIn as much as corporate email for business correspondence. I don’t need an intranet to share content. We happen to have a wiki set up for internal use, not supported by IT by the way, but if it wasn’t there I’d use pbwiki. I can share presentations on Facebook, documents on Google and important links on del.icio.us. Mashup tools such as Serena Business Mashups (OK, you knew I had to work self-promotion in someshere.) will soon be good enough to enable end users to construct their own non-critical applications.

But these apps, because they are available for everyone to use, won't be strategic differentiators. At least not for long. And as for building my own applications, I may be able to use mashup tools, but I need a SOA infrastructure in place to provide the services that I'll mash. There is a place for IT, even in companies that rely heavily on readily available consumer-oriented applications. Just not the same IT.

And the bottom line is the same. Some internal IT will go under, but not all.

What does it all mean? Statistics tells us that the probability of combined events may approach zero even if the individual probabilities of the events are nowhere near zero. So while none of these three trends will take IT down, their combined influences will take a toll.

I’m bullish about IT professionals, but bearish about the current structure of internal IT organizations. Can we assume that IT professionals will continue to concentrate on keeping the LAN operational, providing desktop support, email and file servers? I wouldn't make that assumption. Will internal app dev departments continue to build applications that take years to create and months to modify? I'm not sure how much longer the business will be willing to fund such an organization.

Will there be a place for IT professionals who keep their skills fresh, engage with the business and provide input on how IT can become a strategic weapon? That's a bet I'd make. But it requires that we IT workers change the way we do business. We've got to learn new skills, keep fresh, bounce around new ideas even if someone tells us why we can't do it that way.

It’s time to leave the skis at home, endure the bumps and bruises and the ignomy of being back on the greens, and move forward.

IT run-around crowd can't bypass the need for 'real' web services

I just finished reading a post by Patty Seybold in her Outside Innovation blog. In this post she bemoans the looming mashup hype, but notes that it's about time mashups enter the mainstream.

Too true.

Seybold goes on to say that mashup vendors are trying to convince potential customers that they don't need IT. In Seybold's words, these vendors are saying,

You don't have to wait for your IT department to wrap your back-end application functionality into real Web Services, you can "cheat" by extracting data in the form of reports and turning them into RSS feeds, which you can mash up with graphical user interfaces and tools.

She is implying, although she doesn't come out and say it outright, that this view of mashups is flawed. In this I completely agree.

Consider the sort of 'application' you can create with this approach. You can pull together data, you can bring in interesting visual elements, you can consolidate information, and you can look at it in a unified GUI. But you can't do anything. The data and visual elements aren't in the context of a business activity.

Clearly these mashups aren't going to be the killer apps Hinchcliffe notes are necessary for mashups to become a valuable enterprise tool.

Sorry, IT run-around crowd (Seybold's term) but any mashup vendor saying you can bypass IT and get what you need through RSS feeds, screen scraping and widgets isn't considering that you will need access to back-end enterprise services to get the most out of your mashup investment.

Let's consider an example. Let's say your sales manager wants to keep track of important news cross-referenced to either existing accounts, or opportunities in the pipeline. This could be put together by mashing up CRM data, available through reports and screen scraping, with RSS news feeds, all without the help of IT. That would give the manager some great data, but he can't do anything with it unless he hops out of the mashup and into the CRM system.

The real killer application would allow the sales manager to action the data within the mashup. So, for example, if the manager sees that a company in his pipeline just hired a new COO, the manager could push a button to create a TODO item in the CRM system requesting the account manager arrange an introductory call with the new executive. That sort of interaction isn't enabled by RSS feeds. It is enabled by services connected to back-end systems controlled and managed by IT.

What mashup platforms will do is reduce your dependence on IT to construct new applications. IT can take on the role of a trusted partner rather than gatekeeper, but you still need them. Any mashup vendor who says otherwise doesn't understand your business.

More 2008 predictions from the 'unnamed analyst'

Let me start out by congratulating Encanvas. They have joined JackBe in the illustrious group of mashup vendors who won’t let me review their mashup tools for this blog. Here’s what they had to say when I asked.

As Serena is a close competitor to Encanvas I’m not sure that would be appropriate but we have the greatest admiration for any organization that is encouraging agile computing and it’s great to have competition - so keep doing good things.

The funny thing is that as far as I can tell, Serena’s never been in a deal against Encanvas. Competition. It’s all in the mind.

And now for something completely different…

The 2008 prediction machines are still at it, although I assume now that we are into February we will see them less often. One leading analyst firm, whose name I can’t give you since they would require I run this post through their vendor relations department, has made Web 2.0 predictions for 2008. Hint: They aren’t Gartner. As in the past, I’ll just call them the Unknown Analyst. (UA)

Here’s their 2008 mashup predictions.

• Mashup vendors will start seeing a return on their investment in 2008, but at the expense of other Web 2.0 technologies.

Bloomberg at ZapThink already made a similar prediction. And just as I did before, I must disagree. 2009 will be the year of mashups rather than 2008. I hope to be wrong, but I don’t think I will be. Web 2.0 is just emerging into the enterprise, and mashups will be at the tail end of Web 2.0 spending. RSS, wikis, blogs and social networks are all going to be on the short list for 2008, with mashups entering the list in a big way in 2009.

• Enterprise mashups will come out of the closet, and the vendors who influence standards boards and best articulate mashup value will set market expectations.

My take is that standards boards aren’t going to make a big difference. If mashups were stressing technology to IT customers, then standards would be important. But we need to remember that mashup customers will likely come from the business side of the house. Business users don’t give a hoot about mashup standards for security, widgets, REST contracts, etc. These are vendor problems the vendors need to solve.

However, while I disagree with the notion that standards board leadership will make much of a difference in market leadership, I completely agree that articulating mashup value will make a difference. If I can convince you that mashups will help you leverage your SOA investments to build those applications now languishing at the bottom of IT’s priority list, then I will likely get your business. If I can't convince you, then I won't.

• Vendors from adjacent markets such as EAI and portals will enter the mashup fray in 2008. Pure-play Web 2.0 vendors will be the losers because big players like Microsoft, IBM, Oracle and HP will Web 2.0-enable their current offerings.

This is a very interesting prediction. In effect it says that stand-alone Web 2.0 technologies don’t matter as much as Web 2.0 technologies in the context of existing business problems. So will I need a social network solution in my company if my BI vendor puts one in the reporting portal? Do I need a stand-alone enterprise RSS solution if my existing reporting tools, Sharepoint and CRM vendors add the ability to export notifications as RSS or Atom feeds?

I think, for the most part, the answer is ‘no.’ Enterprises will be able to make do with Web 2.0 features added to existing vendor offereings. At least when combined with open source readers, wikis and blogs.

But mashups? I don’t think so.

Consider a scenario. Assume you want to pull your financial information together with lead data from CRM, marketing program plans from MRM and historical statistical information from BI. You also want to wrap a process around these data to calculate whether a program will generate the number of leads necessary, and alert Marketing to take action when it looks like a program will not meet expectations. If you were depending on one or the other of these tools to provide a mashup platform, you wouldn’t have a unified end-user experience, your process would be dictated by the host tool you happened to choose, and you would end up with the same point-to-point integration problems that have been causing headaches since the dawn of applications development.

In other words, you would have an integration, not a mashup.

So while I agree that some Web 2.0 technologies such as social networking, wikis and blogs may not be good long-term product strategies for independent pure-play vendors, I think mashups have a chance.

Right up to the point where they are purchased by one of those big guys, something not predicted by UA.

What would George Carlin say about the Ten Mistakes Companies Make When Implementing SOA?

In Paul Callahan’s recent eWeek article he lists ten mistakes companies make when they implement SOA Projects. It’s a pretty good list, and I’ll summarize it here, but I’d like to do some tweaking.

Do you remember George Carlin explaining how the Ten Commandments could be boiled down to two, with another added for good measure? If not go take a look. That’s what I think about this list. It isn’t bad, but it needs some tweaking.

Let’s take a quick look at the ten first, and then I’ll discuss how I would consolidate.

1. Taking a Shotgun Approach. That is, SOA-enable everything, regardless of whether there is a business need for the resulting services. LogicLibrary’s Brent Carlson likes to call this A Bunch of Services. (ABOS) and is one of the most common pitfalls of Big SOA.


2. Failing to Involve Business Analysts. Technologists tend to think of SOA as a technology initiative rather than a business initiative. When that happens, not only do the wrong things get built, but it is unlikely the organization will ever see the sort of ROI promised by these technologists.


3. Spending More Time on SOA Products Than SOA Planning. Given lack of business direction, IT will tend to think that SOA is something that you buy rather than something you do.


4. Tackling the Largest Projects First. Sure it is inviting to make a big splash with your SOA initiative. But isn’t it better to try to walk before you run?


5. Forgetting that SOA is a Business Problem. It’s not about the technology; it’s about the solution to a problem.


6. Treating Identity as an afterthought. Readers of this blog will know that this is one I take very seriously. Security and identity are the soft underbellies of SOA, and we need to build for security and identity up front.


7. Buying New Products When Existing Investments Suffice. This is a special case of thinking of SOA as a technology rather than a business solution. You don’t always need to buy new stuff, no matter how much you are told to do so by vendors.


8. Misunderstanding Company Key Players. We live in a world of silos, and if you want to play in a sandbox belonging to someone else, you’d better ask first. Don’t assume you will be allowed to SOA-enable legacy systems, deploy to servers or install clients. Find out first, ask, and then get to work.


9. Expecting the SOA Project to Spread Quickly. This isn’t just an issue for SOA, but for any enterprise project where the success or failure of a project is tied to ‘the enterprise’ rather than to a specific project.


10. Lacking Necessary Elements. Embarking on Big SOA without Big SOA resources.

Here’s how I would rework the list. Numbers 1, 2, 3, 5, 7, 9 and 10 are actually different manifestations of the same problem: Treating your SOA rollout as a technical rather than a business problem. If you focus on your business issue, you won’t have ABOS, you will include business in the discussions, you won’t try to ‘buy SOA’ rather than solve your problem, you won’t think it’s a necessity to buy new stuff, you will tie the success of your SOA rollout to a specific business problem and you won’t go down the Big SOA path if you won’t be able to carry through with it.

So now we have four remaining issues.

Numbers 4 and 8 belong together as well. Both are saying the same thing: Start small. Organizations that start small won’t try to tackle the big, high-profile problems first. They also won’t run into the problem of stepping on somebody’s toes. Or at least the likelihood will be much less.

That leaves number 6, treating identity as an afterthought. I’d like to expand this a bit and say that treating scalability, security and identity management as an afterthought is the real mistake.

I’d also like to add an additional mistake. Forgetting your consumers. Effective SOAs will have multiple consumers, and multiple types of consumers. Design your SOA so it can be used by composite applications developers, BPM process gurus, portal programmers and mashers. Then you will be able to get the ROI you need to make your SOA investment an unqualified success.

So like George, I’d like to restate the items in a positive light.

1. Always remember that you are solving a business problem.


2. Start small and scale up as you learn.


3. Design for scalability, security and identity management up-front.


4. Remember your consumers.

Of course, if I really wanted to be true to George I’d add a 5th to the effect that I should keep my &^%$# opinions about SOA to myself.

Linthicum on Mashup Governance...over the top?

I usually agree with David Linthicum of ZapThink when he writes about SOA. He understands the problems faced by organizations moving to SOA-based applications development. He gets that organizations need to put governance in place to avoid SOA turning into merely A Bunch of Services. (ABOS) I'm a fan.

But I think his recent article on mashup governance in the SOA Institute newsletter was off the mark. Not completely, but I think he's making the mistake that many of us make in thinking that mashups need the same level of governance, the same rigor, as composite applications developed within IT.

Here's my take: Not only don't they need the same level of governance, they won't get it.

So let's take a look at what Linthicum has to say. He's right when he says that mashups blur the line between the web-at-large and the behind-the-firewall enterprise. In his words, "...We could find that we soon live in a world where it's difficult to determine where the enterprise stops and the web begins. Now that's scary and exciting at the same time."

I completely agree.

Next he talks about how organizations need to prepare for mashups. Linthicum says that organizations need to design their SOA with mashups in mind, and on this point we also agree. Although SOA has come to be synonymous with the WS-* standards, it need not be. In fact, early SOAs were completely custom, and even today there are a number of competing SOA standards. Yes SOAs need to be designed around standards, but not just any standards. They need standards that make the services and widgets usable by many different SOA consumers.

Let's consider an example. I happen to know of a company that created SOAP-based services to expose some of their products capabilities. However, these services were built in the image of their old API. When it came time to use these services in a BPEL orchestration, never mind within a mashup, it was extremely difficult. These services weren't built with their end use in mind. More services by techies and for techies.

When it comes time to adopt your SOA standards, keep in mind that you will have many different SOA consumers. Not just by coders in VS .NET or Eclipse, but portals, BPM, and even mashups. And when it comes time to implement your SOA, remember that you may have multiple consumers about which you know little to nothing. So architect for flexibility, scalability and security.

Linthicum goes on to discuss mashup governance in some detail, and I'm afraid here is where he and I disagree. Readers will recall I've written about mashup governance before. I believe mashup governance is necessary. Mashers need version control. Mashers need some form of feedback loop to manage defects and enhancements. Mashers need to practice safe scraping and need to be aware of the provenance of the data they use in their mashups.

However, Linthicum believes mashups should be governed with the same rigor as Big Applications developed by R&D.

"So, what do you need to do to prepare for mashups? It’s a matter of addressing the following areas: Requirements, Design, Governance, Security, Deployment, and Testing. In essence, these are core architectural activities that are required to get you to the Promised Land of mashups on top of your existing activities when you create a SOA."

Nice in theory, but impractical in practice. Mashups are to be built by subject-matter experts, not IT professionals. Think Ultra-Agile rather than Waterfall when putting together a framework for mashup governance.

Requirements? How about "build it," "try it," and, "change it." In Big Applications this may be a license for creep. The on-ramp to disaster. But for mashups, inherently smaller and more agile applications, formal Requirements Management is nothing but unnecessary overhead.

Design? Those of us with Big Applications pedigrees, myself included, will cringe, but a formal design process for mashups seems like overkill. If it works, it's probably good enough. If it stops working, then fix it. Will some mashups be inefficient? Perhaps some mashups will be slow. So what? If the mashup is going to be mission critical, if it will service a large number of transactions, then it will likely be developed by IT, not as a mashup. If it's being assembled by a subject-matter expert, it's likely that inefficiencies won't be a problem. When you bet, figure pot odds.

Governance I've already talked about, as well as security. While Linthicum and I may disagree about governance, we completely agree about security. If an organization is going to over-govern any aspect of mashups, my choice would be security. Unlike SOA's notorious performance issues, security breaches have the potential to materially harm organizations. Jail time? Wall Street disfavor? Stockholder uproar? All these have resulted from IT security problems. 'Nuf said.

Deployment is an interesting issue because it should be easy, but can be difficult. After all, what's the harm in allowing a masher to put a mashup into production? A mashup can't hurt the operational environment, right?

Clearly some governance is necessary. I suggest IT provide mashers with a secure and stable deployment platform, distinct from any mission-critical applications, and then leave mashup deployment to the mashers. IT should not attempt to require the same deployment gates for mashups that they do for SAP upgrades, changes to the financial system, or updates to eCommerce. Once the secure and stable environment is in place, leave mashers to mash.

Testing, testing, testing. I think testing falls into the same category as requirements. Should mashers test all possible configurations, scenarios and error conditions? Sure. Will they? Hardly. Putting gates in place to force mashers to test will only put mashers in an awkward position. Should they ignore the testing gates, or falsify the documentation? Darwin and sheer embarrassment will take care of quality control in the long run. That may not be optimal, but quality control will be more effective if it is instituted as a bottom-up rather than a top-down requirement.

My take-away is that mashers need to develop their own governance practices. Trying to force-feed Big Application lessons to mashers will only result in resentment and eye-rolling. Just as with Agile teams, let mashers define their own rules, and let them modify these rules as they learn, discover and create.

ZapThink Predictions for 2008

Continuing in my series on the 2008 mashup predictions made by others, I'd like to talk about a recent ZapThink article by Jason Bloomberg. He makes several predictions about the future of SOA in 2008, to include the sale of ZapThink itself. I won't comment on the acquisition discussion except to say that it seems a bit odd for ZapThink to tack up a For Sale sign like that. I can only guess they are already in the final stages of negotiation and want to boost their price with a little self-promotion, or perhaps want to get an idea of how their current clients will react to the sale.

Whatever the reason, good luck with it.

In addition to ZapThink’s future, Bloomberg makes some straightforward predictions about SOA and mashups. To summarize: The economic downturn in 2008 may cause some businesses to slow down or cancel their SOA initiatives, especially if they are struggling with ways to define the ROI. This will be good for businesses that have already turned the corner on SOA, but bad for the businesses that cancel.
"...For those organizations, we do predict cancellations of SOA projects and deferments of SOA spending, to their detriment. Because after all, their competition might very well be leveraging SOA to be more successful during a downturn, leaving the unsuccessful companies out in the cold."

I agree with both points...to a point. In difficult economic times, businesses will tend to cut back on strategic spending, especially if the ROI is a bit ephemeral, as it can be in Big SOA initiatives. So yes, I suspect we will se a number of SOA initiatives delayed or cancelled.

However, I think this only applies to Big SOA, not to Guerrilla SOA. Guerrilla SOA projects will likely continue forward because their ROI is tied to the success of a specific business initiative, not to some hard-to-define measure of enterprise IT effectiveness. Organizations with Guerrilla SOA projects in the works will still be able to leverage their investment, "...leaving the unsuccessful companies out in the cold." See my previous post on the subject, as well as predictions made by Hinchcliffe and Chappell.

Later in his post, Bloomberg defines a mashup as a "...governed, managed composition of loosely coupled Services within the context of a rich, collaborative, Internet-based environment." Except for the fact that he left out human interaction, that's a pretty good definition. He goes on to say that "...enterprise mashups are what organizations actually do with SOA."

Spot on.

Organizations leverage their SOA investments in many ways. Old-style programmers can boot up Eclipse or VS .NET and use java, C# or other languages to build applications. When used this way the services available in the context of a SOA are really just another API. I'm sure to get hate mail for this, but for an organization trying to maximize their SOA investment, relying on old-style programmers won’t do the trick.

What about portals? Once the portal framework is in place, end-users can customize at will, without depending on expensive IT resources. Making use of a SOA infrastructure will help increase an organization’s SOA ROI. Unfortunately, portals don’t scale as well. Most people want one portal, or if necessary, two or three, but only under duress. See Oracle’s comments on portals. Portal proliferation is a problem that needs to be solved, so trying to increase your SOA investment's ROI by increasing the number of portals is a losing bet.

Another way to leverage SOA investments is through BPM initiatives. BPM and SOA have been joined at the hip for several years. Before SOA, BPM tools could define a process, but executing the process was something else. With integrated SOA BPM end-user could do their jobs while actually in the BPM tool. No more application-hopping. Additionally, with SOA the BPM vendors had a workable standard for integrations. Not every install had to have a unique set of adaptors built specifically to integrate the BPM tool with back-end systems. No more introspecting DLLs to figure out how to talk to home-grown applications. Give the BPM system a WSDL and it could move the world.

As with portals, BPM solutions don’t scale. They are big systems, mostly behind the firewall, built for process gurus to build big supply-chain-type solutions. BPM practitioners need to have training, they need Big IT support and they need to spend a lot of money to install and maintain BPM systems. So while BPM does help businesses get a bit closer to the sort of ROI the EA team promised as a result of Big SOA, it isn't enough.

Now let's talk about mashups. Specifically, business mashups. Mashups are easy to assemble. They are easy to deploy. They scale down beautifully, but scale to the enterprise as well. With new mashup platforms coming on the market, anyone with enough tech savvy to build Excel macros or administer a wiki can build a mashup. Better yet, mashups can take advantage not only of the SOA, but of the WOA as well, opening up the entire world to mashers. Best of all, business mashups put the aggregated data and display elements into the context of an actual business activity. You have the data you need, when you need it, and can take action on the data without having to jump from application to application.

So when Bloomberg says that enterprise mashups will become the killer use-case for SOA in 2008, I want to agree, I really do. But I don’t. First, I think business mashups are actually the killer app, not enterprise mashups. This isn't just a war over verbiage. Enterprise mashups don't include human workflow, but human workflow is a vital part of most applications. Maybe 2008 will be the year that the term 'enterprise mashup' starts to include human processes. If so, then I'll stop harping on business mashups. Until then I'll continue to make the distinction. Second, I think 2008 will be the year mashups get mindshare outside of IT. However, the business won't actually start cranking out mashups until 2009.

I hope I'm wrong and Bloomberg is right.

Good luck, ZapThink, on your pending sale. And thanks, Bloomberg, for your 2008 predictions

Here's a man excited about mashups

You can tell just by looking at this guy how excited he is about mashups. Energy and excitement just rolls off his picture in waves.

Really, Tim's a great guy, and spoke to India's Express Computer about business mashups. The gist is that organizations have too many portals and too many applications that suck up too much of IT's operating budget.

The benefits of aggregation into a portal are lost if end-users then have to remember URLs and passwords for multiple portals. Mashups in a portal context allows users to aggregate aggregations so they don't have to jump around from portal to portal.

The effects of too many applications are twofold. First, just as with portals, end-users have to jump from application to application to get anything done. The second is that there is no budget to work on new and innovative applications because IT's budget is already being spent maintaining existing apps. Mashups help with application aggregation, and because they can be developed without writing code, tech-savvy end users can build these apps themselves.

The article doesn't offer a lot of new and deeply technical information, but it does summarize nicely the view of mashups by vendors such as Serena Software and Oracle, as well as Gartner's Dave Gootzit. At the risk of being tagged as a member of the lazysphere, I'm just going to point you at the article rather than commenting on it.

Cheers.

Hinchcliffe and Chappell Predictions for 2008 Part 2 - Mashup Governance

Welcome to Part 2 of my series on 2008 mashup predictions made by Dave Chappell and Dion Hinchcliffe. In Part 1 I discussed how both are predicting the end of Big SOA. If you are interested, read the original posts and circle back to my comments. If you aren’t interested in the death of Big SOA, but are interested in mashup governance, read on.

Both Chappell and Hinchcliffe note that organizations will start pushing for mashup governance in 2008. We all cringe at the G-word, linking it with over-governance as mandated by legalistic process Nazis. Too much governance and you bring innovation to a standstill. Not enough and you have chaos. Light-handed governance can be a great help, insuring that security is in place, that there are recovery procedures, that everyone knows the lines of responsibility.

I’ve written before about how IT and business need to work together to make sure mashers have a secure and reliable mashup platform. Now let’s take a look at what Chappell and Hinchcliffe have to say along the same lines.

Chappell believes it's only a matter of time until a business-critical mashup fails. Some feed you didn’t even know you used will go away, some 3rd party service will change endpoints, somebody will change the mashup on Friday evening and then leave to hike the Inca Trail for two weeks. What happens? As Chappell says, “…many will be scrambling to figure out who is supposed to fix it, and who to point the finger at.”

He goes on to say that by the end of the year, driven by these mashup failures, mashers and IT alike will understand that mashup governance has to be put in place. Does it need to be as rigorous as the rules and processes surrounding making changes to the SAP ERP Financial system? Probably not. But neither can it be ignored completely.

Hinchcliffe makes similar predictions, but believes the demand for governance won’t happen because of a catastrophe, but because IT will start voicing loud concerns. "...it will likely be widespread grassroots behavior outside of IT that will drive the need for these solutions within IT departments." Security will become an issue as mashers pull data from many sources behind the firewall, from SaaS applications, and even from the web-at-large leveraging Web-oriented Architecture. (WOA) Mashers should ask questions about these data. Are the data reliable? Are the data available? Are the data to be used appropriate for the audience? After all, you don't want sensitive financial data leaked to the wild by a clueless masher.

While mashers should ask these quesions, it is likely that IT will be the one doing the asking. IT will make noises to govern access to mashable data within the enterprise. Additionally, IT will demand a SOA governance infrastructure that will provide security, policy enforcement and scalability across both on premises and SaaS mashup platforms and data sources.

Unfortunately, I’m just not sure whether any SOA governance solution can put the necessary controls in place without unduly interfering with masher innovation. Let’s take it as given that in a SOA world IT will be able to control access to services. Will these same controls work in a WOA world? I doubt it, at least not without cutting off access to uncontrolled services and data. Frankly, I doubt if the business would stand for these tight restrictions.

As I've asked before, what's to be done?

While IT may not be able to enforce governance outside the firewall and known SaaS applications, it can act as an advisor, explaining to mashers why reliable data sources are important. Why it might not be a good idea to use the Unreliable News RSS feed as input to a mashup for the CEO. Why scraping content from a random web site may land him in trouble. In the end the masher will mash, but IT can at least help the masher understand the risks.

Besides the SOA Governance infrastructure Hinchcliffe believes will be put in place by IT, I believe there are some actions that IT and the business can take together to implement safe mashing in the enterprise.

• Agree up-front who owns the mashup and who is responsible for maintenance, updates and enhancement requests. Who is backup in case the responsible masher goes on holiday? The lines of responsibility should be documented before you have to send a helicopter down to the Andes to airlift your masher back to NY so he can fix the mashup.


• Define the deployment and rollback plan and rules of engagement. What is IT's involvement in a mashup deployment? What is the masher's involvement? What is the rollback plan in case a deployment fails?


• Give the masher access to a version control system. Mashers will be generating code, XML, a model, or some other artifact that defines the mashup. That code, model, etc. has to be saved so when the production copy needs to be modified or recovered, the masher doesn’t have to start from scratch. Hint to mashers: a copy sitting on your machine isn’t reliable version control.


• Work with IT to make sure your mashup platform is scalable, reliable and secure. IT should put a registry and a repository in place so mashers know what services are available, the SLAs for the services, and any security, scalability and reliability concerns for their use.


• Require the masher document the data sources before allowing mashups into production. I know, documentation sucks, but so does getting pulled back from your vacation to explain why you've been getting market sizing information from astrology.com

These recommendations won't cover all the bases, but they will provide a framework for governance, even in a world where there is, in Hinchcliffe's words, "...still too much steering committee and not enough operating governance infrastructure." And most importantly, they will help mashers understand that when they mashup a business application, it isn't just a cool science project. It's serious business.

Hinchcliffe and Chappell Predictions for 2008 Part 1 - The Death of Big SOA

It's that time of year again. Everyone is writing their predictions for 2008. Some are bullish, some bearish, some worth reading, some not. Of the myriad predictions out there, I like the ones written by Dave Chappell and Dion Hinchcliffe the best. I almost completely agree with everything they have to say. However because I don't completely agree, it gives me an excuse to write.

I suggest you read both posts in full. They are definitely in the 'worth reading' category. but I want to comment on two points made by both gentlemen. First is about the demise of Big SOA. The second is about the need for mashup governance. Unfortunately, there is too much to write for a single post, so I'll break it up into a two-part series for your reading pleasure.

2008 – The Year Big SOA Died

Chappell and Hinchcliffe are as one in their predictions for the demise of Big SOA. “That’s nice,” you say, “but what is Big SOA and what does it have to do with business mashups?” Good questions, both, so let me address them one at a time.

I’m not sure we have a good definition of Big SOA. Like porn, however, we know it when we see it. Does the initiative span the entire organization? Does it incorporate a lot of planning? Does it have a multi-year rollout schedule independent of any mission critical project? Does the SOA plan stress measures of effort, such as the number of services to be built, rather than results?

Most tellingly, when you ask the project owner to describe the business problem they are solving with their SOA initiative, do they say something like, “SOA enables developers to build applications more quickly, reducing costs through service reuse.” Hint: this isn’t a description of a business problem; it is a description of a technology solution in search of a business problem.

Big SOA is all about the big picture, which can be a very good thing. It’s good to have an understanding of what the organization’s SOA will look like in five years. Of course, in six months the five year plan will be out of date, but that’s beside the point. Thinking about the big picture can help tactical implementations fit into an overall strategy of growth, security and scalability. Big SOA becomes a problem, however, when the objectives are technical rather than business focused, when the measures of success describe effort rather than results, and when the initiative itself is not tied to the success of specific business initiatives.

Can Big SOA be done right?

Certainly. Let’s consider an example. Let’s say your sales organization has decided that adding social networking to your eCommerce site, currently a Web 1.0 app, will increase sales by 15% in the first year. The project gets funded, and the eCommerce product team makes an architectural decision to build the features using SOA. They consult the Enterprise Architecture group to see how the eCommerce initiative should fit into the enterprise SOA master plan. EA and App Dev work together to modify the enterprise SOA master plan, and to plan for the eCommerce upgrade. At implementation time the eCommerce team leverages work done elsewhere in the organization by re-using and re-factoring services and infrastructure.

In other words, Big SOA is the keeper of SOA strategy and an enabler of enterprise reuse strategy. Little SOA implements the strategy on a tactical level and makes use of the shared assets built in other projects.

Now consider Big SOA run amok. This is a story told to me by someone I met at a conference last year where I was giving a talk on consumption-side SOA governance. During the Q&A period, one of the attendees said her SOA project didn’t include any consumption-side considerations. This puzzled me since the consuming applications are where all the ROI will be realized. She said her company’s SOA initiative included a plan to roll out 500 services in the first year, and then make those services available to App Dev, who would be responsible for using the SOA infrastructure in any new initiative. When I asked her “Why 500 services?” she said it was the number they decided was right for an organization of their size.

Run away, far away. That’s what I told her, at least.

Big SOA has been fueled by middleware vendors trying to sell bloated app server stacks, by SOA governance and management vendors and by analyst firms. This next statement will likely generate hate mail, but I believe their efforts were cheered on by Enterprise Architects searching for a way to become more operational and less purely advisory. As a result, businesses have spent a lot of money and significant effort to plan and implement Big SOA initiatives. Every application, every server, every database had to be pulled into the SOA fold. After huge outlays for consultants and software, and after spending lots of time and effort planning for the brave new world of SOA, these same organizations ended up with a five year rollout plan, a huge proposed budget, and no well-understood way to calculate their SOA's ROI.

That’s a problem. If I’m about to spend $30M developing an enterprise-wide SOA, I’m going to want to see a return sooner rather than later. It’s not just the amount of money, it’s the politics. IT doesn’t control its own budget anymore. Not since the dot-com bust. Something as strategic as Big SOA now requires business buy-in and business approval. When it comes time to approve the Big Budget that goes along with Big SOA, business will want to know how soon and how much ROI to expect. If IT can’t answer, then the business won’t fund.

Hence the demise of Big SOA in 2008.

So let’s move on to the next question. What does this have to do with business mashups? Business mashups need services. They need widgets. They need workflow. Mashups behind the firewall need SOA to get access to the myriad enterprise data currently locked in siloed applications. Mashups outside the firewall need WOA to pull data and display elements from the web at large. For business mashers to take on the task of drawing down the application development backlog, SOA is a necessary backdrop. So if Big SOA is doomed, and I completely agree with Hinchcliffe and Chappell that it is, what’s going to take its place?

Smaller, directed SOA initiatives, called Guerrilla SOA by ThoughtWorks practice lead Jim Webber, have a much better chance of succeeding. Rather than taking an enterprise-wide approach, Guerrilla SOA initiatives are funded by specific projects. Because they are part of a larger project, they don't have the visibility, nor the political liabilities that come with big ticket IT infrastructure projects. Also because they are funded by specific projects, it isn’t necessary to calculate the ROI for the SOA work itself. If the project is a success, that’s enough.

2005 was supposed to be the year of SOA and wasn’t. Then 2006 was supposed to be the year of SOA and wasn’t. Then 2007 was supposed to be the year of SOA and wasn’t. If we toss out Big SOA and go with guerrilla SOA, 2008 actually has the chance to be the year of SOA, only nobody will know it. The ROI won’t come from the SOA initiative itself, but rather from IT finally getting to those pesky integrations, and from subject matter experts in the business building mashups.

The demise of Big SOA may be just what’s needed to get SOA off the drawing board and into production. And that’s what mashers need to start cranking out those mashups.

Next post: Mashup governance. Will IT help or will IT hinder?