Select Page
Tracking manual compatibility testing coverage – How do you do it?

Tracking manual compatibility testing coverage – How do you do it?

This blog post I’m hoping will actually be a good discussion on various methods and techniques you guys may use to track your coverage when performing manual compatibility testing on a project.

In the interest of not ‘seeding’ any specific ideas or techniques I’m going to avoid discussing the current process we use at work in this blog post but I’ll happily share after you guys have talked in the comments below.

Let’s say a project lands on your lap and you’re told “here’s our website redesign. We need it testing. The baseline environment is Windows 10 on the latest Chrome release but we want it tested on a total of 20 mixed Windows, Android and iOS environments”

How do you track your coverage while manual compatibility testing those 20 environments?

The “red rag” heuristic

The “red rag” heuristic

A few weeks ago there was a problem with our phone system in the office; whenever the main phone rang and was picked up from a different phone the main phone registered the call as “missed”

There was a discussion about how nobody was answering calls except for the office manager and various people were told off who had been in the office at the time even though everybody insisted that no calls had been missed.

A discussion took place the following morning where the following phrase came up:

“It’s a simple phone system so it can’t be broken; it has to be the testers not picking up incoming calls!”

As you’d  imagine this assertion immediately got my Testy Senses™ tingling so I set out to prove the opposite; that the phone system can indeed have a defect and quite possibly did (or our domain knowledge  of the phone system needs improving to understand why it’s behaving in a way that seems unnatural).

My internal response to the assertion was instant, powerful and absolutely 100% necessary for me to investigate immediately – this couldn’t wait and I would stay late that day if I needed to!

I time boxed 15 minutes to investigated the issue, started straight away and did indeed find a bug with the system as expected.

After the fact I realised that the intensity of the reaction I’d had (which made me want to investigate desperately) was caused by the assertion itself – it was a statement of believing absolute fact with no evidence to back it up and my experience as a veteran tester immediately disagreed with it being “absolute”.

And that was the moment my “red rag” heuristic was born:

“The red rag heuristic” – Having an almost uncontrollable urge to charge straight into a situation with no pre-planning, no thought for the consequences or what effect there will be on other things that require your time or skill.

Language interpretation

Language interpretation

Recently while walking to the shops with my wife and 22 month old son in his buggy we approached a pedestrian crossing we needed to cross at (no, not Abbey Road!).

As we approached the crossing my wife noticed that the green man was showing so we’d be able to cross straight away, however as we got to the lights the man turned red; we’d have to wait a little while.

 “Oh, it’s turned red” she said as we got to the kerb.

 “It’s OK” I replied.

 At which point she took a step toward the road with the buggy just as the cars started to pull away at the lights…

 Talk about heart attack moment!

 As my mouth opened to speak she stopped and said, “I thought you said it was clear?”

 “Nnnooooo!” I said – “it’s OK” as-in it’s OK that the light went red because we’re not in a hurry and we can wait, not “it’s OK to walk out in front of the moving cars!”

 The consequences of language or terminology assumptions can be horrific; thankfully in this case they weren’t and good old fashioned self-preservation won the day. 😱

In testing make sure that what you’re reading or hearing is absolutely accurate and unambiguous before you make any decisions that affect the project; don’t walk out in front of a car!

The “Dolphin” Heuristic

The “Dolphin” Heuristic

My sister LOVES dolphins! She has dolphin statues, dolphin artwork, dolphin toys and even a dolphin table! She’s absolutely obsessed with them!

Except that I recently found out she hates them… HATES them!

So why does she have them all over the house you may ask… Well I’ll tell you…

When she was very young, she received a stuffed dolphin as a present for her birthday.

At Christmas that year she received a pair of dolphin bookends.

Her next birthday she received more dolphin paraphernalia.

Christmas came and guess what she got?

I’d always assumed she loved dolphins by the sheer volume of dolphin-related stuff around her house – I mean why would you fill your home with something you don’t like?

Fast forward 30 years or so and here we are – A house filled with dolphin stuff and the occasional new addition at birthdays and Christmases.

Just a couple of weeks ago though she told me she’d never really liked dolphins and hasn’t a clue why people keep buying them for her. As time has gone on she’s been bought more and more to the point of hating them completely but because she didn’t want to offend anyone she just kept everything!

It was then that I realised she had inadvertently become part of a family and friend-wide unquestioned assumption and self-perpetuating system of “she has loads of dolphins so clearly she loves them; I’ll buy something dolphin-related for her”

In testing we make assumptions and sometimes we don’t clarify those assumptions as they seem absolutely 100% concrete and “can’t be any other way”. And despite an overwhelming amount of evidence to back up that assumption there’s still a distinct possibility that it’s incorrect.

Don’t assume anything (or if you do, make sure you set out to prove or disprove that assumption) because otherwise at some point in time when you least expect it, that unquestioned assumption will rear its ugly head and yell, “I HATE dolphins!”

Logical Reasoning

Logical Reasoning

Recently on Twitter, Ben Simo posted an interesting photo of a set of road markings at the entrance to a driveway/car park etc.

Despite being a joke or rhetorical question my tester brain engaged immediately with the question posed by Ben and with the photograph presented.

After a minute or so of Logical Reasoning I’d reached my conclusion and was about to post a reply when I realised that the logical reasoning process all testers go through is different so why not share my own process with people in a blog post?

So here you go; this is mine:

  • Assumption: The lettering has been printed in correct orientation for the intended audience.
  • There is a road running horizontally along the bottom of the image, based on the white “central divider” line in the road. I will name this “road”
  • There seems to be an entrance/exit to a driveway, road, car park etc based on the low kerb into the road, the parked white vehicle to the top-right of the image and the width of the entrance being wide enough for vehicles. I will name this “driveway”
  • The spacing of the lettering (or the angle the photo is taken from!) indicates that there are two ways for it to be read:
    • “Do Not Enter” and “Exit Only”
    • “Do Not Exit” and “Enter Only”
  • Both sets of sentences are in the same orientation, indicating that both are meant to be read together from the same viewpoint. If the sentences were meant to be read in isolation, one set of sentences would likely be rotated 180 degrees to be correctly oriented for each intended audience. This also suggests that the driveway is intended for traffic in one direction only.
  • If I was turning from the road into the driveway I’d see both sentences correctly whereas if I was pulling out of the driveway into the road the sentences would be upside-down and not easily readable to me, therefore if the sign has been printed correctly the intended audience is one who would be attempting to “enter” the driveway.
  • My conclusion is that the entrance/exit is meant to be used by people driving out of the driveway into the road. The sign is to tell people on the road not to enter the driveway.
  • To answer Ben’s question; from the perspective of the intended audience of the sign, this is an Exit.
Assumptions vs Biases

Assumptions vs Biases

Software Testing ClinicOn Wednesday of last week the Software Testing Clinic held yet another excellent event in London; this time at the Thoughtworks offices in the West End (*love* your London office space Thoughtworks!) and the topic this month was Exploratory Testing.

During the clinic there was a lot of discussion as you’d imagine from a room of extremely intelligent and passionate testers and on one subject I made a comment along the lines of:

“I think there’s a difference between an assumption and a bias – a bias is always negative and an assumption mainly positive”

At that point I was (correctly!) questioned on that assertion by my friend Tony Bruce on why I think one is positive and one negative.

*cue mind melt in front of the whole group*

I felt 100% that a bias is a bad thing that serves no purpose in the mind of an exploratory tester but I couldn’t articulate why. I also knew that generally speaking the thought of assumptions doesn’t give me anywhere near the type of negative reaction than the thought of a bias does, but again I couldn’t explain why.

So I did the best thing I could – I stopped to question my opinion and whether it was actually an assumption or bias about assumptions and biases. 🙂

Let me start my thought process with two definitions:

Pronunciation: /ˈbʌɪəs/
1[MASS NOUN] Inclination or prejudice for or against one person or group, especially in a way considered to be unfair:there was evidence of bias against black applicants

the bias towards younger people in recruitment

Pronunciation: /əˈsʌm(p)ʃ(ə)n/
1A thing that is accepted as true or as certain to happen, without proof:they made certain assumptions about the market[WITH CLAUSE]:

we’re working on the assumption that the time of death was after midnight

The same as I thought; one is “considered to be unfair” which lends to the negative connotation in my mind. The other is neither negative, nor positive; it is just something that has not been proven one way or the other.


How does that apply to my testing? Well from my perspective an assumption is something active – it’s the deliberate initiation of a thought designed to bring that thought to the front of my mind while testing so that I can prove/disprove it (Tony Bruce defined this as a ‘test idea’ which is definitely another great description for it – thanks Tony!)

Some examples would be:

  1. I assume that the logo at the top of this page will have a hyperlink back to the homepage.
  2. I assume that the navigation items all have hyperlinks to their correct pages.
  3. I assume that if I do not enter the mandatory fields on this form that some validation will take place and I will be presented with a suitable error explaining that the form could not be submitted in its current state and that I need to make some specific corrections in order for the form data to be submitted correctly.

All of those assumptions are things I actively think about and set out to prove or disprove during my testing. They are active, mainly positive thoughts used during testing.

One of my amazing colleagues Deborah also added that for seasoned testers it’s more likely that experience and domain knowledge over the years leads to a filtering of assumptions too; any time an assumption has been made in the past and it was wrong it was probably moved it out of a tester’s internal “safe” assumptions list and into an internal “unsafe” assumptions list (or in my case it’s removed from being an assumption completely and treated instead as an “unknown”)


BiasesA bias on the other hand is something passive and that I’m often unaware of until I break my normal processes, sit down and ask myself whether there’s any bias being applied.

It’s something I likely do as a natural process without realising it, whether that process came from previous experience, heuristics or just the way I think normally. A bias for me is generally a negative thing as it can potentially lead me away from things to explore that may be important or ways of working that are more efficient.

Some examples would be:

  1. I have a bias to navigating websites with my mouse; I use the mouse wheel to scroll, I move the mouse cursor to elements and click links rather than using keyboard navigation to scroll, tabbing through elements and select them with Space/Return. Unless I actively think about that (and in the process turn that bias into an assumption that tabbing through links and selecting them will be as effective as clicking) I could miss the fact that the tab order is completely erratic because of that bias.
  2. When testing new websites for clients I have a bias toward believing what the browser Status Bar tells me a link is. I know there’s a possibility the link displayed in the Status Bar won’t necessarily be the *actual* link but as I see no reason for a project owner to deceive me I trust that the link shown is the link that will be opened. It may be that the client has decided to make links look ‘neat’ by adding a hover status to them and I wouldn’t initially know because of that bias.
  3. I have a bias toward using my Windows machine for my test tools. Unless there’s a specific tool I need that is better on Linux or Mac I use what I’ve used for years. There are likely lots of excellent tools on other environments that I don’t know of because those environments aren’t used enough by me. My efficiency is undoubtedly compromised occasionally because of that bias.

So there you have it; my (possibly over-thinking it!) definition of how I see Assumptions vs Biases in exploratory testing.

EDIT: In this post I’m talking about the bias itself, not the outcome of the bias (which could be positive, neutral, negative or anywhere in between) which is a separate topic I want to blog about soon. 🙂

Questions? Comments?