My Lessons Learned from the STPCon 2012 Test Competition

I attended STPCon Fall 2012 in Miami, FL.  I was there both as a track speaker and a first time conference attendee.  One interesting aspects of the conference, there were others I’ll cover in another blog post, was the testing competition that was available.  Matt Huesser, a principal consultant at Excelon Development, arranged and helped judge the competition.  A blog of his observations can be found at . 

I participated in the competition and have my own thoughts on the competition.

The rules were fairly simple.  We had to work in teams of two to five.  We had 4 websites we could choose to test and we had a bug logging system to report our bugs.  We also had access to stand in product owners.  We had 2 hours to test, log bugs, as well as put together a test status report.

My first observation is that it was a competition but it wasn’t.  The activity was billed as a “There can be only one” style of competition.  However, and more importantly, it was about sharing more than competing.  There were competitive aspects to the activity, but the real value was in sharing approaches, insights, and techniques with testers we have never met before.  Not enough can be said about the value of peering.  Through this exercise, I was able to share a tool, qTrace by QASymphony - for capturing steps to recreate our defects during our exploratory testing sessions, as well as my approach to basic web site security testing.  Although we didn’t do pure peering, it is obvious how valuable the peering approach is.

Secondly, a simple planning discussion over testing approach and feedback during testing is immensely valuable as it not only spawns brainstorming, it helps reduce the occurrence of redundant testing.  Through this exercise, my cohort, Brian Gerhardt, and sat next to each other and showed each other the defects we found.  We also questioned each other on things we had not tried, but were in our realm of coverage.  For side by side pseudo peering, this approach worked quite well for us and led to several bugs that we may not have looked for otherwise.

Lastly, I reflected on the competition and there are several observations that I have made as well as one startling curiosity that I think is most important of all.  Every single team in the competition failed to do one single task that would have focused the effort, ensured we provided useful information, as well as removed any assumptions over what was important.  We failed to ask the stakeholder any of importance regarding what they wanted us to test.  We did not ask if certain functions were more important to others, we did not ask about expected process flows, we did not even ask what the business objective of the application was.  Suffice to say, we testers have a bad habit of just testing, without direction and often on assumption.  I will be posting a blog post more on this topic. 

What I did notice is that testers when put under pressure, such as a competition or being time-bound, will fall back on habits.  We will apply those oracles that have served us well in the past and work with heuristics that make sense to us.  Often times this will produce results that appears to be great, but in the end, they really lead to mediocre testing.  If we had taken the time to ask questions, to understand the application and the business behind it, our focus would have been sharper on areas of higher priority, AND, we would have had context for what we were doing and attempting to do.

I will keep this blog post short, the moral of the exercise is simply to ask questions, to seek understanding, to gain context before you begin.