News from the mashup trenches

I’ve neglected this blog for a while, and now it’s time to get back to it. I didn’t think changing jobs would have much of an effect on my blogging output. Wrong again, Shaw. These days I have to do my reading and blogging mostly after hours, and that does change the dynamic. What takes priority, doing laundry, weeding the garden, helping with homework, balancing the checkbook, playing fetch with the dog, splitting wood, or...writing this blog? I'm afraid lately it's been those other things, and I'm sorry.

On the bright side, I’ve been in the trenches more than I have been in the past couple of years. It’s one thing to read analyst reports, discuss findings with customers and partners, write academic papers and pontificate from on high. It’s another to jump in and actually develop mashups. It is definitely harder to view the entire industry from a trench. Especially in the context of a fixed price contract. On the other hand, I’ve become reacquainted with the reason I went into technology in the first place: I enjoy it. I’ve written more applications (Don't be fooled. Mashups are applications.) in the past couple of months than in the past two years combined.

In a departure from my standard mashup posts, I’d like to share some things I’ve learned about mashups in the past few weeks rather than commenting on mashup news or reviewing some new mashup tool. Some of my experiences have reinforced what I thought before I embarked on this change in job title. Some have made me change my thinking, and in ways that might not make my current employer happy. Sorry in advance, Serena Software, if what I have to say doesn't agree with our marketing message.

First, I’d like to talk about the idea that mashups will allow the business to deploy their own applications without having to bother IT. I never thought this would work. After all, even applications in the tail will need access to back-end systems. Even applications in the tail will need version control of some sort. Even applications in the tail will need governance, even if that governance is light-handed compared to governance used to manage more strategic applications built by IT itself. After my time in the trenches, I’m even more convinced that mashup development has, at the very least, to be a loose partnership between the business and IT.

Let me give you an example.

I built a series of mashups as a prototype for a hardware technology company who wanted to get better control of their rebate program. It was a very intense three-day engagement where we spent the entire day working with the customer to understand their business problems, and then I got to spend the nights building mashup prototypes. (I got about four hours of sleep during those entire three days. I don’t recommend it steady-state, but it was nice to know I could get back to my developer roots if I had to do so. I even had my first fully caffeinated non-diet pop in years. I drew the line at pizza, however.)

The meetings with the client were sponsored by IT, and were run by the business. They were sponsored by IT because the business didn’t know mashups from shinola, but IT did. The business didn’t have a clue that mashups could help them, but IT did. The business was frustrated at IT’s repeated attempts to solve their growing rebate program problem, and IT responded by searching for alternatives to their same old development paradigm.

IT and business worked together to help us define the requirements. IT and business together helped identify what infrastructure was already in place to support the effort. IT and business worked together to help define what parts and pieces of the proposed new ‘system’ could be managed by end users and what parts and pieces would be governed by IT. In other words, there was deep collaboration between IT and the business. And while there were definitely self-preservation undertones in IT's motives, there was also realization on both sides that the old way of doing app dev didn’t work. Everyone was on-board to try something new.

If business had tried to run this meeting without IT, mashups wouldn't even have been on the table. It would have been strictly a shadow IT project that may or may not have taken advantage of recent Web 2.0 trends. And it would have been killed as soon as it became apparent that the project required access to CRM and financial systems. If IT had tried to run the meeting without the business, then we would likely have had a great architecture, and a deep understanding of how the pieces and parts were going to work together, but we never would have understood the deep and abiding frustration felt by the business, nor the financial imperatives that required something be done about the problem soon.

Both had to work together for success. Perhaps this isn't true for all mashup projects, but I suspect it will be true for many of them, especially any that require integration with the existing IT infrastructure.

Here’s something else I learned in the past few weeks. We’ve been saying for a while that business users would start writing their own applications because they can no longer wait for IT to do things for them. The idea is that account reps, shipping clerks, marketing program specialists, etc. would write their own mashups, and we will give them the tools to do so quickly and easily. Now that I’ve been out in the world a bit and have seen what is really going on, here’s what I think: Bullshit.

Even the young 'Net Gen' men and women on the business side don’t want to write their own apps. They may be forced to do so, but they aren’t doing it by choice. That’s because these younger workers want to make progress in their chosen career. Account reps of any age want to spend their time selling, not writing apps. Business analysts want to spend their time figuring out how to beat the competition, not writing apps. Marketing program specialists want to put programs in the field, not spend time writing apps.

So while these end users may be writing apps, they are doing so reluctantly because it takes time away from their chosen careers. Mashers are the exception rather than the rule.

What I do see is a rise in shadow IT. Certainly the business can’t wait for IT to get to their applications in the tail. But the business isn’t building these apps itself. they are hiring outside help. Unlike the wholesale shadow IT trend in the 80’s however, these new shadow IT projects have to operate below the radar. They have to be inexpensive enough to be funded within a departmental budget and not cause an IT governance blip. They also have to show fast ROI so that, when the project finally gets outed, the business can justify what it’s done in a ‘ask forgiveness not permission’ model.

(Note: web widgets are the exception. I'll write more about them in a future post.)

Here’s something that may surprise you. These shadow projects may be instigated by IT as well as by the business. IT itself is getting frustrated with its own inability to react quickly to requests for applications in the tail. In fact, I worked on a project a couple of weeks ago where I designed a guerrilla SOA infrastructure for an IT department that was trying to get an internal project into production before Big IT saw what was happening. The irony of IT setting up shadow IT to subvert Big IT made the job all the more enjoyable.

The take-away for this post is that IT and business have to work together for successful mashup projects, just as they do for any development effort. (Let's all hold hands and sing Kumbaya.) However, when business does take the lead, it is likely to be in the form of a low-cost shadow IT project rather than in the form of an internal masher. I know that isn't doctrine. Perhaps my opinion will change in the future. But for now, I calls it the way I sees it.

Who will be the business mashers?

One of the top two leading industry analyst firms has been talking for several months about how software applications are brittle, difficult to maintain and aren't architected to keep up with the fast-paced demands of today's business environment. That's not new news. It isn't even old news. What is news is that now we have analysts saying out loud what we've all known for quite a while: custom software development is broken.

I can't tell you which analyst firm has been talking about this issue because they will insist that I run this post through their vendor relations department. I don't really want to have my blog sanitized, so instead I'll just call them the Unknown Analyst. (UA)

UA has suggested some solutions to this problem. They predict more development teams will adopt agile software development practices as a way to make app dev more responsive. That will help, but agile methods will only take us so far. Even with agile development practices, multisourcing, offshoring and otherwise scaling up capacity, app dev resources are scarce and expensive. IT has been scaled to work on big-ticket and highly complex software projects, not the myriad interesting and potentially business transforming smaller applications that are backing up in app dev's queue.

Regardless of the development methodology employed by the app dev team and how many new developers they hire, IT simply isn't going to be able to satisfy the demands of the business for new and innovative applications. Especially when, as UA says, they need to change at the speed of business: constantly.

UA suggests that this gap will be filled by business analysts. The role of the business analyst will need to change to become as much someone who assembles applications as someone who simply defines them on behalf of the business. In our world this means the business analyst will turn into the business masher.

So where do business analysts come from, and will they have the necessary skills to become business mashers? What skills will the business masher need?

Good questions, both. UA says that business analysts are grown, usually from an IT staffer who is interested in business or more likely from a business expert or project manager of some sort who becomes interested in IT. Today, the most important skills business analyst posses have to do with effective communications. Can the BA communicate effectively both with the business and with IT? Does the BA know how to write, analyze a problem and mediate disputes between what can be two opposing forces?

Technical skills such as process modeling, usability engineering, and data modeling are less important. Likely because those skills can be learned, but knowing how to communicate effectively both to the business and IT isn't something that can be picked up out of an instruction manual.

But are the soft skills currently important to BAs going to be adequate when these BAs have to start turning out applications?

It's a bit of a rhetorical question because I think I have an answer already. No, these skills won't be enough. I've worked with a number of BPM tools in the past. I've also worked with a number of Web 2.0 application assembly platforms, as you will know if you've read my reviews of Intel's MashMaker, Kapow's RoboMaker, Alpha Works' QEDWiki, WaveMaker's ActiveGrid and even Serena Mashup Composer. These tools all either don't actually build applications, MashMaker, Mashup Editor, QEDWiki and RoboMaker fall into this category, or require some strong IT know-how when it comes to the deep-down nuts and bolts of web service calls.

Yes the tools will get better, but for now I can't agree with UA's assessment. Today's BAs aren't going to become tomorrow's business mashers. Most of them lack the necessary technical skills.

I want to digress a bit. A little over ten years ago there were a lot of complaints in IEEE circles because engineers would found companies only to have them run, owned and grown by MBAs. (Microsoft excepted, of course.) Many IT departments were run by MBAs working for the CFO rather than by technical people with knowledge of technical issues. We nerds put ourselves in technical silos, and then complained like crazy about the lack of technical know-how among our top executives.

IEEE put out a call and said, basically, "Quit complaining and go get MBAs." In other words, IEEE decided that if we nerds wanted leaders with strong technical knowledge, we'd have to grow these leaders from our own ranks. It was a great message and got a number of us to start worrying about things like entrepreneurialism, sales, marketing and finance in addition to whether Windows or Unix would end up ruling the world.

I think we have a similar chance to expand our horizons today. The BA role is changing. BAs today need soft skills, but the BAs of tomorrow will need more technical know-how. Most of these BAs come from the business side of the house and have learned about IT on the job. That means when these newly minted mashers start to create mashup-based business applications they will make mistakes. They won't necessarily think about version control. They won't know about consumer-side SOA Governance. They won't understand how to use XPath, call web services or in many cases how to set up a feedback system to manage changes to their applications.

Remember the last couple of times business built their own applications? Remember the Access databases from the 80s? Remember the web applications in the 90s? The business needed agility and they got it, but they also needed engineering discipline, and this they didn't get. In the end, when it became clear that these applications were business critical, they ended up back in IT's lap, and both IT and business lost. IT because they became villains. (Business: remember when we could make changes whenever we needed? Now everything takes forever.) Business because they lost their agility. (IT: those business guys don't understand that we have 400 things on our TODO list, and the capacity to do five of them.)

We don't want to repeat the mistakes of the past if we can help it. If we technical people want to keep control over our businesses, we have to learn about business. If we don't want to repeat the problems we had with business built applications in the 80s and 90s, we need to step into BA roles now so we can bring our hard-learned disciplines to bear on this new application construction paradigm.

Calling all IT application developers! Why not take the plunge and become a BA? Walk away from Eclipse and give up Java. (Although you will probably still need javascript.) Learn what makes your business tick. Read "Crossing the Chasm." Listen to a marketing podcast. Change those sneakers for loafers, throw a sports coat over that t-shirt and go apply for the BA job in sales your HR department just posted.

This time around, let's do it the right way.