IT run-around crowd can't bypass the need for 'real' web services

I just finished reading a post by Patty Seybold in her Outside Innovation blog. In this post she bemoans the looming mashup hype, but notes that it's about time mashups enter the mainstream.

Too true.

Seybold goes on to say that mashup vendors are trying to convince potential customers that they don't need IT. In Seybold's words, these vendors are saying,

You don't have to wait for your IT department to wrap your back-end application functionality into real Web Services, you can "cheat" by extracting data in the form of reports and turning them into RSS feeds, which you can mash up with graphical user interfaces and tools.

She is implying, although she doesn't come out and say it outright, that this view of mashups is flawed. In this I completely agree.

Consider the sort of 'application' you can create with this approach. You can pull together data, you can bring in interesting visual elements, you can consolidate information, and you can look at it in a unified GUI. But you can't do anything. The data and visual elements aren't in the context of a business activity.

Clearly these mashups aren't going to be the killer apps Hinchcliffe notes are necessary for mashups to become a valuable enterprise tool.

Sorry, IT run-around crowd (Seybold's term) but any mashup vendor saying you can bypass IT and get what you need through RSS feeds, screen scraping and widgets isn't considering that you will need access to back-end enterprise services to get the most out of your mashup investment.

Let's consider an example. Let's say your sales manager wants to keep track of important news cross-referenced to either existing accounts, or opportunities in the pipeline. This could be put together by mashing up CRM data, available through reports and screen scraping, with RSS news feeds, all without the help of IT. That would give the manager some great data, but he can't do anything with it unless he hops out of the mashup and into the CRM system.

The real killer application would allow the sales manager to action the data within the mashup. So, for example, if the manager sees that a company in his pipeline just hired a new COO, the manager could push a button to create a TODO item in the CRM system requesting the account manager arrange an introductory call with the new executive. That sort of interaction isn't enabled by RSS feeds. It is enabled by services connected to back-end systems controlled and managed by IT.

What mashup platforms will do is reduce your dependence on IT to construct new applications. IT can take on the role of a trusted partner rather than gatekeeper, but you still need them. Any mashup vendor who says otherwise doesn't understand your business.

Hinchcliffe and Chappell Predictions for 2008 Part 1 - The Death of Big SOA

It's that time of year again. Everyone is writing their predictions for 2008. Some are bullish, some bearish, some worth reading, some not. Of the myriad predictions out there, I like the ones written by Dave Chappell and Dion Hinchcliffe the best. I almost completely agree with everything they have to say. However because I don't completely agree, it gives me an excuse to write.

I suggest you read both posts in full. They are definitely in the 'worth reading' category. but I want to comment on two points made by both gentlemen. First is about the demise of Big SOA. The second is about the need for mashup governance. Unfortunately, there is too much to write for a single post, so I'll break it up into a two-part series for your reading pleasure.

2008 – The Year Big SOA Died

Chappell and Hinchcliffe are as one in their predictions for the demise of Big SOA. “That’s nice,” you say, “but what is Big SOA and what does it have to do with business mashups?” Good questions, both, so let me address them one at a time.

I’m not sure we have a good definition of Big SOA. Like porn, however, we know it when we see it. Does the initiative span the entire organization? Does it incorporate a lot of planning? Does it have a multi-year rollout schedule independent of any mission critical project? Does the SOA plan stress measures of effort, such as the number of services to be built, rather than results?

Most tellingly, when you ask the project owner to describe the business problem they are solving with their SOA initiative, do they say something like, “SOA enables developers to build applications more quickly, reducing costs through service reuse.” Hint: this isn’t a description of a business problem; it is a description of a technology solution in search of a business problem.

Big SOA is all about the big picture, which can be a very good thing. It’s good to have an understanding of what the organization’s SOA will look like in five years. Of course, in six months the five year plan will be out of date, but that’s beside the point. Thinking about the big picture can help tactical implementations fit into an overall strategy of growth, security and scalability. Big SOA becomes a problem, however, when the objectives are technical rather than business focused, when the measures of success describe effort rather than results, and when the initiative itself is not tied to the success of specific business initiatives.

Can Big SOA be done right?

Certainly. Let’s consider an example. Let’s say your sales organization has decided that adding social networking to your eCommerce site, currently a Web 1.0 app, will increase sales by 15% in the first year. The project gets funded, and the eCommerce product team makes an architectural decision to build the features using SOA. They consult the Enterprise Architecture group to see how the eCommerce initiative should fit into the enterprise SOA master plan. EA and App Dev work together to modify the enterprise SOA master plan, and to plan for the eCommerce upgrade. At implementation time the eCommerce team leverages work done elsewhere in the organization by re-using and re-factoring services and infrastructure.

In other words, Big SOA is the keeper of SOA strategy and an enabler of enterprise reuse strategy. Little SOA implements the strategy on a tactical level and makes use of the shared assets built in other projects.

Now consider Big SOA run amok. This is a story told to me by someone I met at a conference last year where I was giving a talk on consumption-side SOA governance. During the Q&A period, one of the attendees said her SOA project didn’t include any consumption-side considerations. This puzzled me since the consuming applications are where all the ROI will be realized. She said her company’s SOA initiative included a plan to roll out 500 services in the first year, and then make those services available to App Dev, who would be responsible for using the SOA infrastructure in any new initiative. When I asked her “Why 500 services?” she said it was the number they decided was right for an organization of their size.

Run away, far away. That’s what I told her, at least.

Big SOA has been fueled by middleware vendors trying to sell bloated app server stacks, by SOA governance and management vendors and by analyst firms. This next statement will likely generate hate mail, but I believe their efforts were cheered on by Enterprise Architects searching for a way to become more operational and less purely advisory. As a result, businesses have spent a lot of money and significant effort to plan and implement Big SOA initiatives. Every application, every server, every database had to be pulled into the SOA fold. After huge outlays for consultants and software, and after spending lots of time and effort planning for the brave new world of SOA, these same organizations ended up with a five year rollout plan, a huge proposed budget, and no well-understood way to calculate their SOA's ROI.

That’s a problem. If I’m about to spend $30M developing an enterprise-wide SOA, I’m going to want to see a return sooner rather than later. It’s not just the amount of money, it’s the politics. IT doesn’t control its own budget anymore. Not since the dot-com bust. Something as strategic as Big SOA now requires business buy-in and business approval. When it comes time to approve the Big Budget that goes along with Big SOA, business will want to know how soon and how much ROI to expect. If IT can’t answer, then the business won’t fund.

Hence the demise of Big SOA in 2008.

So let’s move on to the next question. What does this have to do with business mashups? Business mashups need services. They need widgets. They need workflow. Mashups behind the firewall need SOA to get access to the myriad enterprise data currently locked in siloed applications. Mashups outside the firewall need WOA to pull data and display elements from the web at large. For business mashers to take on the task of drawing down the application development backlog, SOA is a necessary backdrop. So if Big SOA is doomed, and I completely agree with Hinchcliffe and Chappell that it is, what’s going to take its place?

Smaller, directed SOA initiatives, called Guerrilla SOA by ThoughtWorks practice lead Jim Webber, have a much better chance of succeeding. Rather than taking an enterprise-wide approach, Guerrilla SOA initiatives are funded by specific projects. Because they are part of a larger project, they don't have the visibility, nor the political liabilities that come with big ticket IT infrastructure projects. Also because they are funded by specific projects, it isn’t necessary to calculate the ROI for the SOA work itself. If the project is a success, that’s enough.

2005 was supposed to be the year of SOA and wasn’t. Then 2006 was supposed to be the year of SOA and wasn’t. Then 2007 was supposed to be the year of SOA and wasn’t. If we toss out Big SOA and go with guerrilla SOA, 2008 actually has the chance to be the year of SOA, only nobody will know it. The ROI won’t come from the SOA initiative itself, but rather from IT finally getting to those pesky integrations, and from subject matter experts in the business building mashups.

The demise of Big SOA may be just what’s needed to get SOA off the drawing board and into production. And that’s what mashers need to start cranking out those mashups.

Next post: Mashup governance. Will IT help or will IT hinder?

How close are we to overcoming the 10 challenges facing business mashups?

Once again I’m going to delay my QEDWiki review. Really, I’m going to get to it. Honestly. However, I decided that I needed to discuss overcoming the challenges presented by Dion Hinchcliffe in his post last week, The top 10 challenges facing enterprise mashups. Simply taking a futurist approach as I did in my last post didn't seem like it would be enough.

Hinchcliffe’s post generated a lot of discussions within both the mashup community, and within Serena Software specifically. A number of us debated his points, discussed whether we could help overcome the challenges and even used his post to guide discussions on features we plan to put in our future product releases. I guess this makes Hinchcliffe an honorary Serena Product Manager. Thanks! And thanks to my many colleagues at Serena Software whose ideas have been integrated into this post.

Hinchliffe’s ten challenges fall into three broad categories: business challenges, governance challenges and technical challenges. Rather than addressing each of his ten issues, I’ll address the categories.

Business challenges: Lack of business support for mashups and lack of killer mashup applications.

Remember when the web started to grow? At first it was full of sites with pretty pictures and cool graphics. Organizations created websites as experiments or as another avenue for advertising. It wasn’t until the web killer app came along, eCommerce if you were wondering, that we had the dot-com explosion. We can repeat this story with SaaS and Salesforce.com. When the business sees a killer app, the business wants the killer app. Once we find the Salesforce.com equivalent for mashups, we’ll have the business lining up to invest.

Why hasn’t this happened yet?

Because we’re too busy talking about how cool mashups are. While cool is cool, it isn’t a killer app until it solves a business problem. We can take maps, charts and videos, we can pull in data from multiple sources and we can mash them together at the glass into a visually exciting experience for the mashup user, but no matter how cool it is, it won’t be a killer app until it’s scalable and useful. The problem with at-the-glass mashups is they don’t put the mashup in the context of a business activity. Yes, it’s great that I can pull data from many sources, but if the data aren’t actionable, what’s the point? If I can’t reuse the business logic across the organization, then why invest?

Let’s use an example. Assume I run a fleet of ice cream trucks and I want to make the best use of the trucks. I could use a presentation or data mashup to help by pulling local event information from online community calendars, school activity calendars, business announcements and even law enforcement announcements. I could map these events on a Google Map along with information about the likely size and times of the events. Using this information I could develop a schedule to optimize the routes of my trucks.

That’s a nice way to use mashups, but it isn’t a killer app. It’s not even a business mashup. It’s a data mashup with some cool graphics. A killer app would take the information from the mashup and use it automatically to schedule trucks, drivers and inventory to make sure the right trucks were at the right locations with the right inventory at the right time. The killer app would keep updating event information. A killer app would know when trucks are due for maintenance and schedule the maintenance around heavy usage days based on the mashed-up information. Our truck scheduling application is a business mashup because it puts the mashed up information in the context of the larger business problem, namely, optimizing ice cream truck utilization. The data aren’t enough. The data must be actionable and solve an actual business problem.

Once we understand that a killer mashup app has to be in the context of a business activity, that the mashup data has to be actionable, and that the mashup itself must solve business problems, then we will start to see a lot more businesses take mashups seriously. Until then, well we can always console ourselves that we are cool.

Governance Challenges: An immature services landscape, confusion over management and support of end-user mashups, chaotic data quality and accuracy and version management.

I’ve written about this issue before, both in my futurist post about the sematic web, and earlier when discussing the role IT can play as a trusted advisor to the business with respect to business mashups. Some discussions bear repeating, however, so I’ll cover some of the same ground again.

Lack of mashable content and data quality are interrelated. Without supported services tied to systems of record, mashers will have a difficult time ensuring the quality of their data. Long-term I believe this is a problem for the semantic web. Short-term, however, vendors need to start getting serious about enabling access to products through web services. At Serena we’ve already started this process, and we will continue to add services for the foreseeable future. As mashups become more accepted in the business community instead of just an IT tool I expect we will see this trend emerge with other software vendors. Note to business mashers: If you want your vendors to provide web services, you’d better start demanding them.

Management and support of mashups will be problematic and will get worse as more mashups are developed by the business community rather than IT. When talking to IT professionals about mashups developed by the business, this issue is where IT has the most heartburn. As Hinchcliffe notes, once upon a time this same scenario played itself out with PCs, databases and spreadsheets. The business started something, building applications, that it couldn’t support long-term and IT was tasked with providing support for applications about which they knew very little. IT has a long memory. I doubt if they will be taken by surprise again.

Surprised or not, IT isn’t going to be able to stop business mashers from developing mashups. Not only does the business have too much at stake, but the new generation entering the workforce doesn’t have a lot of patience with corporate hierarchies. They’ve grown up with technology and won’t wait around for IT to build their applications. To stay relevant, IT needs to become the partner of business and provide a secure and scalable infrastructure in which the business can build mashups.

It is inevitable, however that the business will eventually need support for their mashups. We could see a move towards centralization once more, just as we did when the business handed back all those Access databases to IT. However, business has a memory just as long as IT, and they will remember that while centralization did bring order to the mish-mash of rogue applications, the cost was business agility and strict IT control. I suspect that many on the business side of the house will look for an alternative.

Enter a new breed of vendor whose business will be to support the business. Budget oversight being what it is, these new vendors will likely provide support as part of a subscription process within a SaaS model. These vendors will need to fly under the capital expense radar and simply be a line-item on a department’s monthly expenses, similar to a cell phone bill. That means many business mashups will be purchased as part of a subscription model with support being provided by these new vendors. That way the business can build their mashups, but can also have a number to call when they need help.

I agree with Hinchcliffe that mashup version management has to be part of any mashup tool vendor’s offering. Lucky for Serena we’ve already got mashup version control as part of our mashup tools.

There is another version control issue that needs to be confronted, however. Version control of the individual services has long been a problem within SOA implementations. It’s a dark not-so-secret that uncontrolled services can cause disaster in SOA-based applications. If the SOA implementation has a successful reuse policy, the problem is even worse since a single bad service can bring down any number of applications. And yet there is no way for the SOA client to know whether a service has changed. Here vendors and 3rd party web service vendors need to be held accountable by consumers. Until that time, version control will continue to be a challenge.

Technical Challenges: No construction standards, the splintering of widgets, deep support for security and identity, and low-level mashup support by major software firms.

I’m bullish about overcoming the technical cited by Hinchcliffe. If we can get the business to throw their weight behind mashups, the vendors will have tremendous pressure to start providing some solutions that will make it easy for the business to adopt the business mashup model.

However, I’d like to challenge Hinchcliffe’s assertion that we need a unified method for mashup construction. Ditto for widget technology. It would be great if all the tools had a consistent approch, but I’m not sure I’d classify it as a challenge for mashup adoption in the enterprise.

Business mashers will have domain knowledge and a level of technical competence consistent with building Excel spreadsheet macros. Given that business mashups need to mash data and visual elements in the context of a business activity, it’s clear that model-based construction is the solution with legs. Our business mashers won’t be writing JavaScript. They won’t be writing any sort of code, even if that code is disguised as an XML document. They will be dragging and dropping visual, data and process elements using a familiar office-like interface. If that’s the case, the end user won’t care what is happening under the hood. A consistent method of construction may be a challenge for the vendors, but not for the mashers.

I do agree with Hinchcliffe that support for mashups among infrastructure and application vendors will continue to be an issue for some time. However, we might be able to solve some of the problems in the short-term. For example, if we are to put mashed content in the context of a business activity, we must have some sort of event driven architecture, or at the very least, a simple eventing system. Every vendor has one. Even Serena has one. We use the eventing system within the open source Eclipse ALF project. Eventing systems require participating software to kick off some external communication when important things happen.

Let’s consider our ice cream truck example. Ideally, the mashup would need an event to kick-off rescheduling truck routes when a concert gets cancelled, a new truck is purchased or a driver quits. the ALF project has tried to make the eventing system generic by providing web services to raise events, but again, the web services have to be tied to custom actions within the ALF event management system. While the pattern is the same for other vendors’ eventing systems, the devil is in the details.

One way to overcome this is to use eventing systems that already exist. Email leaps to mind, as do Outlook meeting reminders. Many back-end systems already know how to send emails and already integrate with outlook. While it may not be the best of all possible worlds, it would certainly jump-start event-oriented business mashups if the onus was on the mashup tool vendors to integrate with these existing event channels.

As for other low-level support, once again Hinchcliffe has it right. We can solve some of the issues, but the bulk have to wait until software vendors feel the squeeze from customers demanding low level mashup support.

I’ve saved the hardest problem for last: security. If anything is going to kill SOA and the companion consumption-side technologies, it will be security and identity management. Consider web-based applications. We’ve been at those for over ten years, and we still don’t have security under control. With SOA the problem is even worse because there are myriad potential back-end systems engaged in every mashup, and to date the most common method of passing around credentials is either as a parameter to service calls, or in the service header. With RESTful services the problem is aggrevated since the WS_* standards generally don’t apply at all.

One promising solution is the open source Eclipse Higgins identity management project. Many vendors have already signed up to use Higgins, but again, until all vendors adopt the standard, we are going to have the potential for serious security breaches within mashups. Especially mashups at the glass.

My conclusion is that yes, we have some challenges, but in many cases these challenges are either already in the works to be solved, or there is at least a roadmap for solving them. The ones that aren’t going to be overcome in the short-term will be side-stepped. How? I don't know. I do know that once the business understands the potential of business mashups, nothing will get in the way of widespread adoption.

Can the semantic web help with the ten challenges facing enterprise mashups?

I'm going to delay my review of QEDWiki yet again to comment on Dion Hinchcliffe's post, The 10 top challenges facing enterprise mashups. Hinchcliffe's blogs about Web 2.0 have been very influential over the past few years, and this excellent posting is no exception.

Fair warning: I’m going to use his post as an excuse to go off on a futurist binge and talk about the semantic web. Don't worry, though. I'm going to talk about the 'real' semantic web, not the ivory tower version.

I won't reiterate Hinchcliffe’s points, you can, and should, read them for yourselves. However, I do want to talk further about two of his challenges that I think are related, and relate directly to the power of the emerging semantic web. His #2 challenge is an immature services landscape. There just aren't enough services out there to provide mashable content. His #6 challenge relates to data quality and accuracy. How do mashers know whether the data are accurate and up-to-date?

I see these issues as interrelated. The lack of 'supported' services is driving people to create services for themselves using various tools, HTML screen scraping being the one I've been working with lately. Before you dismiss screen scraping as a viable content creation strategy, note that the number of robots available from OpenKapow outstrips the number of services from StrikeIron and the number of APIs available from Programmable Web combined. Lack of services is causing people to turn to self-help methods to get mashable content directly from web pages. Yet we all know that web pages often have out-of-date data or even absolutely trash data.

Do you know about The Greys? The Greys are a crossbreed between human and an extraterrestrial reptilian species. By visiting this site I learned that there are over 70 distinct species of Greys. Wow! Good thing I have this website around to help me find such valuable information.

‘The Greys’ is an extreme example, but there are others that are less silly. If you were scraping content from the US Open site about who was in the women's final you would get one set of names for 2006 and another for 2007. Yet once the data is abstracted through a service call and incorporated into a mashup, it won't be obvious the 2006 data is out of date. Mashup user won't, and shouldn't, be able to tell from where the data came. Mashups are first and foremost about presenting a unified experience to the mashup user. Noting where data comes from makes the mashup less of a mashup and more like a plain old integration.

How can mashers solve this problem? One way is to create more supported services so mashers will depend less on tactics such as screen scraping to get their mashup content. I doubt this will work. By some estimates there are between 19 and 30 billion web pages today, and that doesn't even count dynamic pages such as search results from the Snap-on Tools site. We aren't going to create web services to expose reliable data for all of those pages. People who need mashable content are going to get it where they can, and that means web pages themselves.

Another way to help with the data reliability problem, and this is where I think the web is going, is to start leveraging the capabilities of the semantic web. I’m talking about the practical semantic web that is emerging from the likes of del.icio.us, Facebook and Amazon. I’m not talking about the ivory tower semantic web with volumes of ontologies, deductive rules and AI searches. Some call this emerging web “Web 3.0” and some say the ivory tower version of the semantic web is “Web 3.0.” Personally, I don’t care what we call it, but I’m excited about what it is, or rather, what it can become.

To backtrack, the semantic web is a way of structuring web content so that it can be consumed both by humans and by machines. Most web content today is only consumable by humans. (Irony. It's everywhere.) That’s why we get so many trash results even from the greatest search engines. In the ivory tower version, every web page has both semantic information (what the information on the page means) as well as content. The semantic information makes the page machine consumable. A phone number is a phone number is a phone number. Once a program knows the content is a phone number, it knows how to handle said content.

In theory, but not in reality, since there are many ways to tag and format a phone number.

In practical terms today, web content is being slowly categorized by various tag clouds such as del.icio.us, social network sites and blogging sites such as the one you’re visiting now.

Today these clouds are disaggregated without any sort of consistency. However, while it is unlikely we will get universal acceptance on what amounts to a tag dictionary, it is highly likely we can get universal acceptance of a small number of tags. This has already happened in specialty areas such as research libraries. Imagine a rating tag being adopted by all tag clouds so site visitors can rate the quality of a web page ala Digg, or an expiration date so mashers know when content is out of date, or even a copyright tag telling mashers the page is off limits for scraping. Not that mashers would pay attention.

Imagine a world where a masher pulling content from a web page through HTML harvesting of some sort could be given a rating of how good the data is likely to be. And even with disparate tag clouds, it would be possible for mashing tools to suggest alternative content pages. Imagine a world where the mashup itself could warn users if content quality degrades below some acceptable level.

Finally mashers and mashup users would be able to have some indication whether they are getting the latest scores, the most reliable news or the best information on extraterrestrial species.

OK, this is all for the future, but perhaps the not-to-distant future.

I’ll see what I can do to convince Serena Software to start thinking about the semantic web and how we can use it to help business mashers. Meanwhile, go give Hinchcliffe's blog post a thumb's up vote.