My Lessons Learned from the STPCon 2012 Test Competition

I attended STPCon Fall 2012 in Miami, FL.  I was there both as a track speaker and a first time conference attendee.  One interesting aspects of the conference, there were others I’ll cover in another blog post, was the testing competition that was available.  Matt Huesser, a principal consultant at Excelon Development, arranged and helped judge the competition.  A blog of his observations can be found at . 

I participated in the competition and have my own thoughts on the competition.

The rules were fairly simple.  We had to work in teams of two to five.  We had 4 websites we could choose to test and we had a bug logging system to report our bugs.  We also had access to stand in product owners.  We had 2 hours to test, log bugs, as well as put together a test status report.

My first observation is that it was a competition but it wasn’t.  The activity was billed as a “There can be only one” style of competition.  However, and more importantly, it was about sharing more than competing.  There were competitive aspects to the activity, but the real value was in sharing approaches, insights, and techniques with testers we have never met before.  Not enough can be said about the value of peering.  Through this exercise, I was able to share a tool, qTrace by QASymphony - for capturing steps to recreate our defects during our exploratory testing sessions, as well as my approach to basic web site security testing.  Although we didn’t do pure peering, it is obvious how valuable the peering approach is.

Secondly, a simple planning discussion over testing approach and feedback during testing is immensely valuable as it not only spawns brainstorming, it helps reduce the occurrence of redundant testing.  Through this exercise, my cohort, Brian Gerhardt, and sat next to each other and showed each other the defects we found.  We also questioned each other on things we had not tried, but were in our realm of coverage.  For side by side pseudo peering, this approach worked quite well for us and led to several bugs that we may not have looked for otherwise.

Lastly, I reflected on the competition and there are several observations that I have made as well as one startling curiosity that I think is most important of all.  Every single team in the competition failed to do one single task that would have focused the effort, ensured we provided useful information, as well as removed any assumptions over what was important.  We failed to ask the stakeholder any of importance regarding what they wanted us to test.  We did not ask if certain functions were more important to others, we did not ask about expected process flows, we did not even ask what the business objective of the application was.  Suffice to say, we testers have a bad habit of just testing, without direction and often on assumption.  I will be posting a blog post more on this topic. 

What I did notice is that testers when put under pressure, such as a competition or being time-bound, will fall back on habits.  We will apply those oracles that have served us well in the past and work with heuristics that make sense to us.  Often times this will produce results that appears to be great, but in the end, they really lead to mediocre testing.  If we had taken the time to ask questions, to understand the application and the business behind it, our focus would have been sharper on areas of higher priority, AND, we would have had context for what we were doing and attempting to do.

I will keep this blog post short, the moral of the exercise is simply to ask questions, to seek understanding, to gain context before you begin.  

Two Testing Giants Part Ways

Wow. What can I say? Scott Barber started a firestorm. Two great minds, James Back and Cem Kaner, in the testing community are parting ways on an idea, context-driven testing (CDT), which they helped create and foster. The reaction on each other’s blog has been like two parents getting divorced with testers in the field expressing disbelief, pain, and sadness.

Part of what is driving this is a changing viewpoint for Cem. Not uncommon since 2 other founding members have already broken off with CDT. To compound things between Cem and James, there is personal animosity between them. While I will say that I have a great deal of respect for James and Cem and they both have great ideas; I, for one, do not really care about them parting ways. And I do not think you care about their parting ways either; because CDT principles are an undeniable truth, not dependent on any one person.

I do agree with Cem’s general statement that there is more than one CDT school – more than one camp each with their own values and ontology (to quote James). The problem is, I disagree with using the word “school”. While it may be semantics, my issue is that school conjures up images of an institution with a predefined curriculum of which you cannot graduate unless you pass the courses. Sounds like certification, with which I disagree. CDT is a paradigm. I don’t like the implied argument, raised by James, that the ability to NOT follow something makes it an approach versus a school; because the implication is, that you must always follow something and that something that must be your identity. This makes CDT sounds like dogma and religion. For the sake of this blog post, I will still reference “schools” as schools to maintain some discussion continuity.

To me, CDT is more fundamental to the testing community than I have found anyone to say. My belief is based on the premise CDT is not about a new way doing things so much as it is an acknowledgement of reality. The CDT principles are more akin to truths than principles. Even if you do not positively use the principles to gain synergies, it does not negate the principles; it does not render them false. CDT is ingrained in the fabric of testing, regardless of which “school” a tester is following. The principles are an acknowledgement of “what is” not of “what can be”. To see what I mean, just paraphrase a few enlightened guys, “I hold these truths to be self-evident…”  Even if someone followed the “Factory School” of testing, the CDT principles still hold true. For example, Factory testers believe testing measures progress. While that is an information point, it does not negate CDT principles such as projects unfold in unpredictable ways or that people are still the most important asset. The principles of CDT can be found in these realities. Therefore, CDT fundamentally permeates all “schools” of testing. Because of these reasons, I do not see any significance in this parting of ways. 


Ultimately, there is an underlying “thing” that needs to be acknowledged, and that is: the purpose of software testing and consequently, software testers. Software testing’s purpose is to identify data, consolidate it into useful information, and provide it to stakeholders so that informed decisions can be made. That purpose exists regardless of a “school”. When thinking about it, I see the “schools” are really aligned to tools and types of information for specific decisions. The schools are not aligned to the real purpose of testing. I believe this is one of the reasons there are so many issues with every “school’s” beliefs and approaches. 

• Our job, as testers, is to extract data, synthesize it into information so that someone, a stakeholder, can make a decision. To do this we need tools. Those tools depend on the several factors: 
• I can only use tools I recognize – it is difficult to use something as a tool if you cannot see it in front of you Everyone has different tools – our tools are experience and knowledge based, accelerated - at times - but never replaced by technology 
•  Not every tool we know of is at our disposal – there may be great tools out there, ones we know about, but we simply might not be able to afford or capable of easily learning them 
• Every tool has a function – no matter the tool, it has a purpose, a way of being used, and expectation of what using it will do 
• Every tool can be used for any job, with varying degrees of success – you can use a butter knife as a regular screwdriver - sometimes 
• Using tools, in both traditional and non-traditional ways, will create new tools for us – it is about learning. With all tools being knowledge based, any learning leads to new tools 

Our challenge is to realize, as testers, our job is to extract data about a creative process in order to synthesize useful, information in a way so that stakeholders can make informed decisions; to use the tools we have available to do the best job we can. By the way, that translation of data to useful information is and should be influenced by the creative process (development process), our tools, knowledge, experience, and our understandings. We are researchers, inspectors, philosophers, teachers, learners, synthesizers, but most of all, we are information brokers.