Automation Pitfalls Snarls an Automation Vendor

I like automation testing tools. I like the challenge, I like the hobbyist programmer it brings out in me, and I like to use them, but only when they are useful. Automated tests are not sentient. They are not capable of subjective evaluations, only binary ones. Not only that, they are only able to monitor what they are told to monitor. These facts are often overlooked in the name of faster, better, higher quality testing. See the advertising quote below:

Benefits include ease of training (any member of your team can become proficient in test automation within a few days), staffing flexibility, rapid generation of automated test suites, quick maintenance of test suites, automated generation of test case documentation, faster time to market, lower total cost of quality, greater test coverage, and higher software quality.

This is the advertising from a tool I ran across today called SmarteScript, by SmarteSoft. I saw their ad, and downloaded a demo. The first thing I did was attempt to run through the tutorial to become acquainted with how the tool operates. Like most commercial for-profit tools, they have a rudimentary demo site that this designed to show off their product's bells and whistles. The basic tutorial was to learn the web-based objects (textboxes, buttons, etc...). Then, with their nifty excel like grid, add in some values for appropriate objects. Pretty simple. I used their tool to learn a dropdown box on their demo site per their instructions. *Emphasis added because it is possible to learn a table cell around the dropdown, the dropdown arrow graphic, or the text only portion of the dropdown list without learning the dropdown list as required. They had specific instructions on how to do this as well as how to tell if you screwed up. However, when going through the tutorial, with IE8, I noticed that it appeared to only learn the dropdown as static text (ie a label). I tried and tried to get it to recognize the dropdown as a dropdown, but it would not. I tried playing the script back just to see if the tool would overcome its own learning. But alas, it failed. I sent my observation to the support team. To their credit, I received an email back the same day stating they were able to reproduce the problem and their development team would look into it further.

I find it ironic that a vendor, selling this as a way to achieve faster time to market, lower total cost of quality, greater test coverage, and higher software quality ended up releasing a tool that is the contrary of that. I would be remiss if I didn't point out that I doubt I can be proficient at using a tool in a few days when it doesn't work on their own demo site; so strike down another claim. Most importantly I think this points out that test automation is not a silver bullet...nor a cheap one.