Thursday, March 27, 2008

Dapper has a lot of promise, but boy can it be annoying!

I haven't done a review in a while, so I thought I'd get back into it, starting with Dapper. I came across them in a Hinchcliffe blog entry about the most promising mashup tools. He had Dapper on the list, along with other more well-known tools such as JackBe, who still won't let me test their product.

Dapper allows developers to pull content from websites and expose them using various APIs. There is nothing new about this. Nearly all the products I've reviewed have this capability. Dapper differentiates itself with the number of different APIs it supports, which I'll get to later. The list is very impressive, but doesn't include SOAP.

Too bad about SOAP, but I understand. SOAP is overkill for screen scraped content. You don't need transactional integrity or security (not that SOAP has that problem licked) if you're just pulling read-only content from a page. Still, it does mean I can't use the content in something like a BPEL orchestration. Clearly that use isn't something Dapper has in mind. Then again, neither did Intel MashMaker or Kapow.

For consistency with my other reviews, I attempted to create a feed from the news page on the Serena website. No luck. Dapper couldn't load the page. So next I went to Digg to add recent news items. While Dapper loaded the page, it scrambled the page elements. I couldn't pull the top news stories from digg/science or digg/technology.

Next I went to my own blog to see if I could pull my content into a feed. I know, everyone can already get a blog as a feed, but this was an experiment. The instructions for Dapper say its selection algorithm will work better with multiple similar pages, so I added the links for my most recent three posts and went to the next step, selecting the contents to scrape.

Unlike other screen scraping technologies I've played with, Dapper has some smarts built in. Their algorithm supposedly helps mashers select the right content to pull into the API without having to mess around with Xpath. Well, certainly there is an algorithm in place, but I found it much more of an annoyance than a help. I couldn't get selections to work correctly, and when I tried to de-select manually, I got a page script error and the interface stopped in its tracks. I couldn't interact with the application at all, and had to reload the entire page with a new URL.

I went back to the start and tried it again, and got similar results, except that I didn't even try manually de-selecting page elements. Instead I wondered if I should not give Dapper multiple pages to work with. I selected 'Back' in the interface to return to the page where I selected my inputs, intending to remove all but the latest blog entry. And guess what?

Right! I got an error on the page again. The 'Back' link didn't work either. At least this time the interface didn't freeze up.

After a while I defined something close to the selections I wanted. (I never did get the exact content.) And now for the reason Dapper is different. The reason I kept playing with Dapper despite its many flaws: I could expose the content as POX, RSS, Filtered RSS, HTML, a Google gadget, a Netvibes module, a PageFlake, a Google map, an image loop, an iCalendar, Atom, CSV, JSON, XSL, YAML or even as an email. True, a lot of these formats don't make sense for blog content, but it's nice to have the option.

I especially liked the preview that let me take a look at the content before finalizing my output format choice. That was sweet.

Bottom line. I wouldn't use Dapper today for production mashups. It just isn't ready. However, when Dapper fixes their algorithm so it isn't annoying, when they do some serious debugging, when they fix their performance issues and when they otherwise clean up their usability, it will be one killer application for creating mashable content.




Tuesday, March 25, 2008

The reports of IT's death are greatly exagerated. Or are they?

It’s been a while since I wrote in this blog. I’ve had a lot of things happen, so I hope you will all forgive me. First, Marketing decided they couldn’t afford a pet nerd any more, so I had to find another job. I can’t blame them. They’ve let me run wild for a year, reading and writing and playing with software. How many people get to do their dream job for even a year?

Serena’s professional services department has taken me on to do for a fee what I used to do for free: give advice to customers about best practices with respect to ALM, SOA Governance, Agile development, Web 2.0 and mashups. I’ll still have the chance to read and write, and I’ll get involved with customers earlier in our relationship. In all, a very fair trade.

I also took a vacation before hitting the ground in services. I’ve been a skier for 20 years or so, but decided to put them aside for the week and learn to board. Why? I had become complacent and was no longer progressing or pushing myself. I wanted something new, and decided to try the board. Having heard from other skiers that it’s very hard to switch, especially at my age, I deliberately left my skis at home.

A week later I had a bruised tailbone, a wrenched shoulder, a twisted knee, possibly a broken thumb and a couple of very sore wrists. I also can ride. Not well. Not nearly as well as I skied. But even though I’m back on the greens, I’ve progressed.

I’m back now and I’d like to talk about a post by ZD Net blogger Michael Krigsman. In his recent post he talks about the very real possibility that IT will become an endangered species, going the way of the steno pool, vast accounting departments keeping the books with adding machines, and company cars. The article was interesting, citing the now dusty ‘IT as commodity’ argument as well as a number of others. Not quite as interesting as the article are the comments. Except for a very few insightful posts, they fall into three categories.


  • IT sucks and deserves to die.
  • Tech illiterate users suck and will beg IT for mercy when their systems crash.
  • IT isn’t going away, it’s just evolving into a new life form.

They remind me of what poses as political debate on slashdot. “Democrats suck.” “No! Republicans suck!” “No! Democrats suck.” And so on. I keep reading because once in a while I come across some creative invective that makes it all worthwhile. In case you were wondering, there wasn't any noteworthy invective in the comments.


When I first read the article I believed Krigsman was being deliberately provocative. I thought, "He’s picking a fight. I know it, he knows it and I bet everyone else knows it too." Once I started considering his points, however, I wasn't so sure. In fact, I think all three conclusions posted by Krigsman's readers have some truth. In some cases IT does deserve to die, in some cases users will still rely heavily on IT, and in some cases IT will survive, just not in it's current form.

Krigsman bases his predictions on seven trends, but I think there are really only three: IT is a commodity, IT isn’t in tune with the business, and the world is changing around IT.

IT is a commodity. Krigsman cites three different examples of IT as a commodity. First, IT has become defensive rather than offensive as applications such as email become as necessary to everyone as electricity. This is the argument Carr made in his post dot-com article about why IT doesn’t matter. Second, applications that used to be differentiators, such as CRM, are now so standard they can be offered as a service, completely bypassing the need for IT support. Salesforce.com is the poster child for SaaS, but many others exist, and even venerable enterprise app vendors such as SAP are planning to offer their software through subscription services. Third, software vendors are volume pricing their offerings, pushing IT organizations to make single-vendor deals. In other words, vendors such as Microsoft and IBM are pricing their offerings as commodities.

But is it a valid argument that IT as a commodity means IT is doomed to extinction? Most IT professionals would say, “No.” Older, established capabilities such as file servers, email, and enterprise apps may move outside of the firewall, but there will always be the need for newer and better capabilities that haven’t yet been commoditized.

I’m not sure either position is correct. If all a company needs is vanilla IT, then it doesn’t make sense to maintain an internal IT organization. If all you need is standard power, would you bother generating your own, or would you buy it from the grid? In some cases, however, innovative IT is a strategic weapon, not merely a defensive necessity. Wal-Mart, Google, eBay and Amazon have shown that IT can be such a weapon. Businesses that differentiate based on innovative customer interaction, a hyper-lean supply chain or platform as a service, are going to need their own IT.

The bottom line is that commoditized IT will put some, but not all, internal IT departments out of business.

IT is out of touch with the business. Krigsman discusses two manifestations of this root cause: IT leadership is alienated from corporate senior management, and corporate senior management doesn’t understand how IT can be a strategic asset.

Both of these points have the ring of truth. Where IT is a provider of services rather than a business partner, IT is nothing but a defensive necessity and the CIO merely a cost cutter, probably reporting to the CFO. Under those circumstances, why would senior leadership pay attention? Would the CEO reserve a place on the board for the person responsible for ordering the most cost effective paper clips? A corollary would be that senior leadership also won’t understand how IT can be a strategic asset rather than a cost center.

But does this signal the end of IT? Again, not necessarily. Let’s consider Wal-Mart versus K-Mart. K-Mart didn’t care about IT and didn’t consider IT as a strategic asset. Wal-Mart did, and put in place a supply chain system that drove K-Mart into bankruptcy. K-Mart's IT definitely took a hit then, although I bet is has rebounded since, and not merely as a provider of email. In the Darwinian business world, organizations win when they make the best use of their strategic assets. They lose when they don’t.

So the bottom line is similar. Where the IT/business divide is strong, it will put some internal IT organizations out of business.

The world is changing, lessening the need for IT. We won’t need IT because we can get everything we need from enterprise-ready consumer-inspired apps. These apps are getting to market faster and better fueled by VC investments.

This is a hotly debated topic, very much tied up with Enterprise 2.0 and mashup debates. The business wishes that it were true, IT knows it isn’t. The reality is somewhere in between. I already use Facebook and LinkedIn as much as corporate email for business correspondence. I don’t need an intranet to share content. We happen to have a wiki set up for internal use, not supported by IT by the way, but if it wasn’t there I’d use pbwiki. I can share presentations on Facebook, documents on Google and important links on del.icio.us. Mashup tools such as Serena Business Mashups (OK, you knew I had to work self-promotion in someshere.) will soon be good enough to enable end users to construct their own non-critical applications.

But these apps, because they are available for everyone to use, won't be strategic differentiators. At least not for long. And as for building my own applications, I may be able to use mashup tools, but I need a SOA infrastructure in place to provide the services that I'll mash. There is a place for IT, even in companies that rely heavily on readily available consumer-oriented applications. Just not the same IT.


And the bottom line is the same. Some internal IT will go under, but not all.

What does it all mean? Statistics tells us that the probability of combined events may approach zero even if the individual probabilities of the events are nowhere near zero. So while none of these three trends will take IT down, their combined influences will take a toll.


I’m bullish about IT professionals, but bearish about the current structure of internal IT organizations. Can we assume that IT professionals will continue to concentrate on keeping the LAN operational, providing desktop support, email and file servers? I wouldn't make that assumption. Will internal app dev departments continue to build applications that take years to create and months to modify? I'm not sure how much longer the business will be willing to fund such an organization.


Will there be a place for IT professionals who keep their skills fresh, engage with the business and provide input on how IT can become a strategic weapon? That's a bet I'd make. But it requires that we IT workers change the way we do business. We've got to learn new skills, keep fresh, bounce around new ideas even if someone tells us why we can't do it that way.

It’s time to leave the skis at home, endure the bumps and bruises and the ignomy of being back on the greens, and move forward.