Monday, July 6, 2009

Automation Pitfalls Snarls an Automation Vendor

I like automation testing tools. I like the challenge, I like the hobbyist programmer it brings out in me, and I like to use them, but only when they are useful. Automated tests are not sentient. They are not capable of subjective evaluations, only binary ones. Not only that, they are only able to monitor what they are told to monitor. These facts are often overlooked in the name of faster, better, higher quality testing. See the advertising quote below:

Benefits include ease of training (any member of your team can become proficient in test automation within a few days), staffing flexibility, rapid generation of automated test suites, quick maintenance of test suites, automated generation of test case documentation, faster time to market, lower total cost of quality, greater test coverage, and higher software quality.

This is the advertising from a tool I ran across today called SmarteScript, by SmarteSoft. I saw their ad, and downloaded a demo. The first thing I did was attempt to run through the tutorial to become acquainted with how the tool operates. Like most commercial for-profit tools, they have a rudimentary demo site that this designed to show off their product's bells and whistles. The basic tutorial was to learn the web-based objects (textboxes, buttons, etc...). Then, with their nifty excel like grid, add in some values for appropriate objects. Pretty simple. I used their tool to learn a dropdown box on their demo site per their instructions. *Emphasis added because it is possible to learn a table cell around the dropdown, the dropdown arrow graphic, or the text only portion of the dropdown list without learning the dropdown list as required. They had specific instructions on how to do this as well as how to tell if you screwed up. However, when going through the tutorial, with IE8, I noticed that it appeared to only learn the dropdown as static text (ie a label). I tried and tried to get it to recognize the dropdown as a dropdown, but it would not. I tried playing the script back just to see if the tool would overcome its own learning. But alas, it failed. I sent my observation to the support team. To their credit, I received an email back the same day stating they were able to reproduce the problem and their development team would look into it further.

I find it ironic that a vendor, selling this as a way to achieve faster time to market, lower total cost of quality, greater test coverage, and higher software quality ended up releasing a tool that is the contrary of that. I would be remiss if I didn't point out that I doubt I can be proficient at using a tool in a few days when it doesn't work on their own demo site; so strike down another claim. Most importantly I think this points out that test automation is not a silver bullet...nor a cheap one.

2 comments:

Art Trevethan said...

Joe,

Thank you for taking a little bit of time to look at SmarteScript. We understand your issue and are working to resolve it. As IE8 is brand new, and not in wide distribution to our core customers we are still pursuing validation and our own QA process in this browser. The business of test automation tools is a tricky one; we are expected to deliver solutions for every situation in the wild, and in most cases do. A drawback is that, in some cases, we are running slightly behind to address a brand new interface or technology. Please be assured, it is on our roadmap and is being addressed.

We also agree with your ascertain that automation is no silver bullet. There are things that should not be tested using automation as it reduces the benefit. We recently were published on SQE with an article about just this topic, when to consider automation. A PDF of the article is available from our site at http://www.smartesoft.com/casestudies/Avoiding%20Catastroph.pdf. Feel free to browse the rest of the site and blogs at http://www.smartesoft.com.

I would invite you to discuss your needs with us, perhaps even sitting through an interactive demonstration of SmarteScript on your software. Failing that, I would encourage you to re-attempt your test using IE7, I believe that your result will be favorable.


Thank You,

Art Trevethan
SmarteSoft

Joseph Ours said...

Thanks Art. I appreciate your response and I'm glad you agree automation isn't a silver bullet. I look forward to trying your tool in the future. Still the tool, by just its tutorial, belies the marketing points on the website.