Recently I did a 99 second talk at TestBash in Brighton in which I compared golf to testing:
Now I know golf is a rather dull subject to many people who may have never played it so I’ll try not to bore you all too much. 😝
The thing with golf as anyone who has ever watched it on TV knows is that you only ever need a single golf club to play golf at the highest level… Rory McIlroy, Sergio Garcia, Seve Ballasteros, Tiger Woods… They all only ever used one club! Let me explain…
- Those guys tee off using a driver to hit the ball as far as possible. BOOM and down the fairway the ball flies.
- Then they use the driver again to punch the ball down the fairway or onto the green.
- And if they hit it into the rough or a green side bunker guess what club they teach for? That’s right, the driver!
- Then they’re a foot from the pin where just a nice little tap in will win the hole. What club do they use?
Except that actually nobody uses one club all the time because at the very best professional level in golf it would be inefficient (and for the vast majority of weekend warriors it makes the game completely inaccessible and not even remotely enjoyable.)
Every golf club has a purpose and an ideal use case; some clubs have multiple purposes or several use cases but no club fits all the purposes and all the use cases and yet I’ve met testers who apply that “one tool fits all” mentality to their testing daily!
I’ve met testers who use one set of browser developer tools and never try others (even berating others without trying them!) and I’ve met testers who have a certain choice of tracking their exploratory test coverage and they never look into other possibilities:
- Is a notepad doc sufficient, useful and logical for others to read?
- Would Rapid Reporter give the notes more structure?
- Is the time trade-off worth the benefit to YOUR context?
- Should you be tracking coverage in a spreadsheet because your client is a 3rd party who requires a visual representation of the testing done?
- Does your tool integrate directly with your issue tracker? Should it?
Do people in YOUR context regularly evaluate your testing tools and techniques to make suggestions on improvements or do you sit quietly and not question the status quo?
For a long time I myself had one single choice of Proxy software to analyse what’s being sent from and fired back to a test app. I knew what I could do with it so I never looked into others, what was the point when I knew my choice of app well?
Sometimes tools can be used for purposes other than the intended one, much like someone may choose to “bump and run” a golf ball from the fringe of the green with a club normally intended for long shots off the fairway – For that specific shot it’s a great option. For that specific test you want to do perhaps cURL + JQ is a superb option to pull in some JSON and reorder it for comparison but for the rest of your testing there may be little value in those tools.
As testers we should strive to read about tools, try tools out, make notes on how you might use those tools best in your daily testing work and then maybe leave the tool alone until the task at hand DEMANDS that tool!
Maybe that will put us in a far less biased position when it comes to using the right tool for the job and it will expose us to more tools, better tools and ultimately make us more efficient in our day to day work.
The driver is not the best club for all shots all the time.
Recently at an event I overheard a conversation between two testers in which one tester was describing some examples he’d apply to an input field.
As the conversation went on there were a lot of “tests” (which I’d call “checks”) I found to be basically pointless and it got me thinking about how as professional testers we automatically use our experience, knowledge and heuristics to decide on the best approach and best set of tests/checks to run – from applying every test known to man to no tests at all, with anything in between.
The tester in question had used an example of:
“So in the First Name field you might try a vowel, a consonant, two consonants, two vowels, a vowel and a consonant etc”
To quote the phenomenal Michael Bolton (@michaelbolton) – “In traditional parlance, domain testing involves identifying equivalence classes— sets of things that we expect the program to (mis)treat in the same way” and to my mind from a pure data perspective, whether a consonant or vowel is used to check an input field’s validation or submission makes no difference.
If I was looking at physical size, letter widths etc that’s a different story completely, as would checking Unicode characters which are most definitely not an equivalent to non-Unicode characters but to my mind those tests I overheard are simply a waste of time that could be better spent testing something that could reveal valuable information.
Sometimes testing isn’t just about what we DO test/check, it’s about what we DON’T as well…
When I arrive at work after my morning tube commute the first thing I do is use some antibacterial hand gel before I check my emails and tasks for the day. I’d rather kill any cold or virus germs asap but not enough to walk from the ground floor up to the 3rd floor.
After I’ve checked emails, tasks and anything else urgent I’ll head upstairs to the second floor to make a coffee and en route I’ll pop up one more floor into the bathroom and wash my hands so that they’re clean as well as germ free.
I’m a clean kind of chap; not OCD clean but perhaps cleaner than I need to be.
With my testing I like my environments to also be as clean as necessary for the system under test:
- Websites (including mobile sites) – No unnecessary programs running in the background and a cleared down browser with no addons enabled that may affect the site.
- Mobile apps – Factory reset device with no unnecessary apps installed or accounts that aren’t essential to the OS. No backgrounded apps or non-test-app push notifications enabled (except for deliberate testing)
The idea behind this is to reduce any false positives – Any issues I discover will almost certainly be a problem with the system under test and I won’t be incorrectly reporting issues that are centred around interaction with other software, much of which could be unique to my own preference of apps.
For instance if I was testing software on a system that had an IDE installed on it and I was unaware that the software had a reliance on certain runtimes that were previously installed and were not native to the operating system I’d completely miss the fact that the software may not even run for many users.
Similarly certain setups from previous testing sessions may not have been returned to their base state so I could end up spending unnecessary time troubleshooting proxy/DNS settings, environmental variables that should be changed etc.
I’ve spoken to quite a few people about their test system preference and generally there seems to be two camps:
- The Angelically Clean Crew – Their machine/device is a freshly imaged desktop machine or mobile device irrelevant of any previous testing. If for example the only other testing that had been done on that environment was website testing and there’d be an almost non-existent chance of any false positives it doesn’t matter – “Image all the things!”
- The Dirty Dozen – A more “real world” approach – Machines/devices are more often than not their development machines or personal mobiles and filled with other apps, accounts, push notifications for lots of apps enabled, backgrounded apps etc – “But real users don’t have perfect environments”
There doesn’t seem to be many people with a “good enough” approach when it comes to test environments for some reason. That may be a completely factual statement but also it may be that the people I’ve spoken to have tacit knowledge about what they would accept and it’s not entirely clean/dirty but a happy Goldilocks place.
I completely see the benefit to all levels of dirtiness in testing setups but what I struggle with is lack of time on our many “one hit” (waterfall) projects and prioritising those levels of dirtiness in terms of time spent setting up versus bugs encountered versus time spent investigating and ruling out other system interactions as the cause etc.
I think until we have the time to perform mirrored testing on both a clean and dirty device simultaneously across many projects to prove one way or other how much impact there is to time due to device cleaning versus false positives sticking to a an “everything immaculately clean” approach is the best option, even if it’s overkill.
What are your thoughts? How dirty do you like it? Do you have any real world experience or statistics of the value between clean/dirty?
This blog post I’m hoping will actually be a good discussion on various methods and techniques you guys may use to track your coverage when performing manual compatibility testing on a project.
In the interest of not ‘seeding’ any specific ideas or techniques I’m going to avoid discussing the current process we use at work in this blog post but I’ll happily share after you guys have talked in the comments below.
Let’s say a project lands on your lap and you’re told “here’s our website redesign. We need it testing. The baseline environment is Windows 10 on the latest Chrome release but we want it tested on a total of 20 mixed Windows, Android and iOS environments”
How do you track your coverage while manual compatibility testing those 20 environments?
On Wednesday of last week the Software Testing Clinic held yet another excellent event in London; this time at the Thoughtworks offices in the West End (*love* your London office space Thoughtworks!) and the topic this month was Exploratory Testing.
During the clinic there was a lot of discussion as you’d imagine from a room of extremely intelligent and passionate testers and on one subject I made a comment along the lines of:
“I think there’s a difference between an assumption and a bias – a bias is always negative and an assumption mainly positive”
At that point I was (correctly!) questioned on that assertion by my friend Tony Bruce on why I think one is positive and one negative.
*cue mind melt in front of the whole group*
I felt 100% that a bias is a bad thing that serves no purpose in the mind of an exploratory tester but I couldn’t articulate why. I also knew that generally speaking the thought of assumptions doesn’t give me anywhere near the type of negative reaction than the thought of a bias does, but again I couldn’t explain why.
So I did the best thing I could – I stopped to question my opinion and whether it was actually an assumption or bias about assumptions and biases. 🙂
Let me start my thought process with two definitions:
1[MASS NOUN] Inclination or prejudice for or against one person or group, especially in a way considered to be unfair:there was evidence of bias against black applicants
the bias towards younger people in recruitment
1A thing that is accepted as true or as certain to happen, without proof:they made certain assumptions about the market[WITH CLAUSE]:
we’re working on the assumption that the time of death was after midnight
The same as I thought; one is “considered to be unfair” which lends to the negative connotation in my mind. The other is neither negative, nor positive; it is just something that has not been proven one way or the other.
How does that apply to my testing? Well from my perspective an assumption is something active – it’s the deliberate initiation of a thought designed to bring that thought to the front of my mind while testing so that I can prove/disprove it (Tony Bruce defined this as a ‘test idea’ which is definitely another great description for it – thanks Tony!)
Some examples would be:
- I assume that the logo at the top of this page will have a hyperlink back to the homepage.
- I assume that the navigation items all have hyperlinks to their correct pages.
- I assume that if I do not enter the mandatory fields on this form that some validation will take place and I will be presented with a suitable error explaining that the form could not be submitted in its current state and that I need to make some specific corrections in order for the form data to be submitted correctly.
All of those assumptions are things I actively think about and set out to prove or disprove during my testing. They are active, mainly positive thoughts used during testing.
One of my amazing colleagues Deborah also added that for seasoned testers it’s more likely that experience and domain knowledge over the years leads to a filtering of assumptions too; any time an assumption has been made in the past and it was wrong it was probably moved it out of a tester’s internal “safe” assumptions list and into an internal “unsafe” assumptions list (or in my case it’s removed from being an assumption completely and treated instead as an “unknown”)
A bias on the other hand is something passive and that I’m often unaware of until I break my normal processes, sit down and ask myself whether there’s any bias being applied.
It’s something I likely do as a natural process without realising it, whether that process came from previous experience, heuristics or just the way I think normally. A bias for me is generally a negative thing as it can potentially lead me away from things to explore that may be important or ways of working that are more efficient.
Some examples would be:
- I have a bias to navigating websites with my mouse; I use the mouse wheel to scroll, I move the mouse cursor to elements and click links rather than using keyboard navigation to scroll, tabbing through elements and select them with Space/Return. Unless I actively think about that (and in the process turn that bias into an assumption that tabbing through links and selecting them will be as effective as clicking) I could miss the fact that the tab order is completely erratic because of that bias.
- When testing new websites for clients I have a bias toward believing what the browser Status Bar tells me a link is. I know there’s a possibility the link displayed in the Status Bar won’t necessarily be the *actual* link but as I see no reason for a project owner to deceive me I trust that the link shown is the link that will be opened. It may be that the client has decided to make links look ‘neat’ by adding a hover status to them and I wouldn’t initially know because of that bias.
- I have a bias toward using my Windows machine for my test tools. Unless there’s a specific tool I need that is better on Linux or Mac I use what I’ve used for years. There are likely lots of excellent tools on other environments that I don’t know of because those environments aren’t used enough by me. My efficiency is undoubtedly compromised occasionally because of that bias.
So there you have it; my (possibly over-thinking it!) definition of how I see Assumptions vs Biases in exploratory testing.
EDIT: In this post I’m talking about the bias itself, not the outcome of the bias (which could be positive, neutral, negative or anywhere in between) which is a separate topic I want to blog about soon. 🙂