Select Page
Over thinking it – I can therefore I will

Over thinking it – I can therefore I will

At TestBash Brighton 2016 one of the superb talks was on “Building the Right Thing” with Lisa Crispin and Emma Armstrong and the talk started with a very basic task – “make something that flies through the air”. My first thought on hearing this task was, “I’ll scrunch up the paper and throw it; that would be funny” quickly followed by “but I’m sure they don’t mean that and anyway I can make superb paper planes which would be more fun”

I made a plane ready to fly and then we were told that the requirement was one of Minimum Viable Product (I hate this phrase but that’s a separate blog post!) so my initial thought would have been spot on. Gutted.

I’d completely over thought the task at hand.Paper planes

This year at TestBash Brighton in their excellent mentoring workshop Shey Crompton and Nicola Sedgewick tasked us with something similar; the task was to build an actual “paper plane” this time though so I got to make a great paper plane and be on task for the challenge. Win win!

This time I knew that as it was a mentoring workshop there would likely be a challenge to teach and learn what we created so I decided to design my plane in as simple a way as I could think of; I based it on a simple dart shape rather than complex wing design with the idea being that if I threw it hard enough and it would fly a nice long way anyway but there would be a benefit of being able to easily teach the design. As predicted it flew across the room and happy was I!

We were then told to describe the process of building our planes to a partner (nice to know my intuition was right on that).

I was also confident that I could easily describe the process for my partner to replicate my plane…

Except that what is simple for me isn’t necessarily simple for others. I understand what my mind can process, understand and retain easily but I can’t know that of someone else, especially someone I’d never met until that moment. My partner did successfully follow my instructions although it was far from as easy as I’d envisioned it – I’d have been better off making an even more simple plane that was not as detailed with the trade off in flight capability being matched in reproduction capability.

(K)eep (I)t (S)imple (S)tupid

As testers we often can’t see the wood for the trees. We look at a system or a website or an app to be tested and immediately our mind fills with the potential things we can subject the system to in order to test it. Sometimes our mind is exactly on task and the intended tests are precisely what’s needed. Other times though we over-think it and end up wasting time in convoluted tests or bug investigations that could have been much simpler and allowed us to move on to other important tests.

We also try to make judgements and estimations of others’ capabilities whether it’s in our bug reports, test scripts (ugh!) clients demand or in meetings we have with our team.

Let’s look at some practical examples:

Example #1 – “I’m going to create a tool for test data creation”

You have a website to test where many accounts can all be in differing states. You decide you need to create 10 test accounts on the system. Rather than go through the monotony of creating them all individually by hand you decide to create a snippet of JavaScript to automate it for you.

The JavaScript takes you an hour to write, debug and run successfully. Job done!

Except that an account takes 2 minutes to create and the project only has 2 days of test time so in reality you would have used 2/3 the amount of time just doing it manually for the duration of the project.

D’oh!

Example #2 – “The main banner image is missing on one specific browser”

You’re testing a website and your baseline is Google Chrome. You’re happy with your coverage on the baseline and you move on to test on other browsers. Firefox passes with flying colours, Edge is happy and so is Internet Explorer 11. Safari has a couple of minor issues that you report and the next browser on your hit list is Internet Explorer 10.

You boot the clean image test machine, clear down cookies and cache and load the page but the main banner is missing.

A quick investigation verifies that the image is declared correctly in the code and it exists on the server when the URL is added straight into the navigation bar of the browser so it seems like a rendering issue.

Better bug it!

Investigate lessExcept that you reckon you can track down the exact cause of the problem and provide the developer with a load more info than a simple “Banner image on Internet Explorer 10 does not display” bug report.

You end up spending a further hour figuring out the full problem (the CMS uses a handler that’s declaring a different mime-type than the image actually had) and you report the full details. How great is that?

Wait… you used an entire test session to investigate that? So now there’s another charter than needs to be dropped or postponed?

Maybe just bugging it after 2 mins of investigation and letting the developers investigate the actual cause was a better plan even if you were providing far less information with that bug report.

Over thought it again (and ironically thought nothing about the other charters impacted by over-thinking this one thing)

So you see as testers it’s very easy for us to get carried away with our work, over-think the task at hand and produce less value to the long term project targets in favour of short term value to that single task.

Try to be mindful of Pareto’s Principle – you’ll get 80% of the output from 20% of the input and most of the time 80% is more than adequate. If you know you can investigate further perhaps add that offer into the bug and discuss it at the next scrum or with your project manager and allow them to decide.

Fore! How to score better with your testing approach

Fore! How to score better with your testing approach

Recently I did a 99 second talk at TestBash in Brighton in which I compared golf to testing:

https://dojo.ministryoftesting.com/lessons/99-second-talks-testbash-brighton-2017

Now I know golf is a rather dull subject to many people who may have never played it so I’ll try not to bore you all too much. 😝

The thing with golf as anyone who has ever watched it on TV knows is that you only ever need a single golf club to play golf at the highest level… Rory McIlroy, Sergio Garcia, Seve Ballasteros, Tiger Woods… They all only ever used one club! Let me explain…

  • Those guys tee off using a driver to hit the ball as far as possible. BOOM and down the fairway the ball flies.
  • Then they use the driver again to punch the ball down the fairway or onto the green.
  • And if they hit it into the rough or a green side bunker guess what club they teach for? That’s right, the driver!
  • Then they’re a foot from the pin where just a nice little tap in will win the hole. What club do they use?

Driver!

Except that actually nobody uses one club all the time because at the very best professional level in golf it would be inefficient (and for the vast majority of weekend warriors it makes the game completely inaccessible and not even remotely enjoyable.)

Every golf club has a purpose and an ideal use case; some clubs have multiple purposes or several use cases but no club fits all the purposes and all the use cases and yet I’ve met testers who apply that “one tool fits all” mentality to their testing daily!

I’ve met testers who use one set of browser developer tools and never try others (even berating others without trying them!) and I’ve met testers who have a certain choice of tracking their exploratory test coverage and they never look into other possibilities:

  • Is a notepad doc sufficient, useful and logical for others to read?
  • Would Rapid Reporter give the notes more structure?
  • Is the time trade-off worth the benefit to YOUR context?
  • Should you be tracking coverage in a spreadsheet because your client is a 3rd party who requires a visual representation of the testing done?
  • Does your tool integrate directly with your issue tracker? Should it?

Do people in YOUR context regularly evaluate your testing tools and techniques to make suggestions on improvements or do you sit quietly and not question the status quo?

For a long time I myself had one single choice of Proxy software to analyse what’s being sent from and fired back to a test app. I knew what I could do with it so I never looked into others, what was the point when I knew my choice of app well?

Sometimes tools can be used for purposes other than the intended one, much like someone may choose to “bump and run” a golf ball from the fringe of the green with a club normally intended for long shots off the fairway – For that specific shot it’s a great option. For that specific test you want to do perhaps cURL + JQ is a superb option to pull in some JSON and reorder it for comparison but for the rest of your testing there may be little value in those tools.

As testers we should strive to read about tools, try tools out, make notes on how you might use those tools best in your daily testing work and then maybe leave the tool alone until the task at hand DEMANDS that tool!

Maybe that will put us in a far less biased position when it comes to using the right tool for the job and it will expose us to more tools, better tools and ultimately make us more efficient in our day to day work.

The driver is not the best club for all shots all the time.

Accept misnomers – The bigger picture is helping, not correcting language

Accept misnomers – The bigger picture is helping, not correcting language

EDIT: This post was mid-draft when I was reminded by @NicolaSedgwick on Twitter of @PaulHolland_TWN‘s #TestBash #99SecondTalk about not attacking people with a different understanding of terminology than you. I’m pretty sure there are improvements yet to be made on it but for the sake of remaining relevant I’m publishing it now. 🙂

In life there are misnomers everywhere:

  • “Coffee beans” – they’re actually coffee “seeds”
  • “Dr Spock” (Star Trek) is actually Mr Spock
  • Panama hats were designed in Ecuador
  • “Personal PIN Number” – PIN is an abbreviation for “Personal Identification Number” so in effect you’re saying “Personal Personal Identification Number Number”
  • “Wherefore art thou Romeo,” is often understood to mean “where are you Romeo?” When in actual fact “wherefore” means “why” – Juliet is asking why he is a Romeo, not where he is.
  • “Peanuts are nuts” – They’re not, they’re legumes as they grow under the ground, not on trees.
  • “A tomato is a vegetable” – It’s a fruit.
  • Koala Bears are not bears.
  • “Slider” is sometimes used to reference a gallery/slideshow/carousel function
  • “Checkbox” is frequently used to reference a radio button. Checkboxes though are multiple-selection “AND/OR” elements whereas radio buttons are single choice buttons.
  • “Drop down” has been know to reference an expandable <div> because the panel drops-down to display content.
  • “Quality Assurance” is regularly used to reference “Testing”

And so on.

So let’s use an example misnomer and discuss the potential states of it.

We’re working on a website that is early in development and has had no technical team members on it. There is a header section <div> that expands when clicked to reveal additional content.

The project manager and project sponsors have always referred to the <div> as a “header dropdown” as from their perspective that’s what it is; the header “drops down” when clicked.

With misnomers there are several states of understanding:

  1. Misnomer is accepted as being factually correct and not questioned. This often occurs on projects where the beginning of the project starts with substantial planning from non-technical team members or those with less domain knowledge.
  2. Misnomer is known to be factually incorrect but the person understands the context and sees no reason to correct the misnomer or hold a discussion about it. Using our example misnomer, a developer may start working on the project and as there is no other way to implement the required functionality (it is not achievable via an actual “drop down” element) there really is only one possible thing it can be right now – an expandable <div>, so why waste everyone’s time holding a conversation about it?
  3. Misnomer is understood to be factually incorrect. The person feels it is necessary to correct the originator of the misnomer with their own understanding of the misnomer. OK so let’s say a tester joins the project and decides that in order to make sure everyone is using the correct terminology and avoid any ambiguity the current “header dropdown” misnomer should be corrected. This will ensure that any future functionality that may be added to the header is not confused with the misnomer, say for example a “language” dropdown is added. There is no room for discussion here; it must be corrected.
  4. Misnomer is known to be factually incorrect. A conversation takes place with the project facilitator to gauge any potential risk associated with the misnomer. An agreement is reached whereby the most efficient solution is implemented for the misnomer. Using our example let’s say the tester joins but instead of taking a “must have” approach they explain to the project facilitator that there are ambiguity risks with using the terminology but that those risks may be extremely low depending on the development plan for the project. For example if there will never be anything added to the header of the site that resembles a dropdown then referring to the expandable <div> as a “dropdown” will have little impact. The project manager explains that once the site development is complete it will be handed over to a 3rd party and there will be no maintenance period and no future development on the project for the development team. The project is also only a 3 month project and there are no team member changes planned that could cause confusion with terminology. The “header dropdown” misnomer is then deemed to be unimportant and everyone is happy to use the misnomer rather than set about re-educating the team.

Inadvertent use of misnomers allows quicker communication in groups where there is a joint understanding of the domain being discussed and the context of the misnomer itself by all parties but herein lies the risk of not correcting or discussing a misnomer; there is an inevitable assumption that everyone has similar domain knowledge and understands the terminology used equally, hence there is no need to discuss or correct it.

Using terminology or descriptions which are even slightly ambiguous to those with different domain knowledge can create issues though.

From my perspective what we need to strive for is the establishment of a common “language” on projects that allow the participants the ability to discuss things in the most efficient manner possible adding as little ambiguity into the discussion as possible, not to pick apart people’s differing descriptions of the same thing.

I test software. I perform quality analysis. If you say I am “QA’ing” your software and I understand the task to be “testing” the software then there’s little value in me trying to redefine your vocabulary or use of the English language. If anything it will make everything even less efficient as there will be a period of readjustment before the new vocabulary is “normal” in day to day work.

The approach I normally take is to reiterate the required task using the correct terminology, for example:

“We’ve added Search functionality to the header dropdown on the staging environment. Can you exploratory test it this morning please?”

“Oh cool, yep absolutely! Exploratory test the Search function in the expandable header <div> – got it. I’ll start right after this test session and feed back as soon as possible.”

Now having said all if the above I do think there’s a huge benefit in establishing a common vocabulary within an industry or community where our “domain” is the industry itself. But that’s a whole new blog post… 😉

The Perfect Storm

The Perfect Storm

Well what a perfect storm couple of months it’s been.

Apologies for having been off the radar with social media, Slack channels, London Tester Gatherings, Software Testing Clinic mentoring and of course blog posts; I’ve been noting down plenty of ideas but unfortunately been unable to blog to any degree of depth for a few reasons:

  • Cold
  • Cough
  • Flu (after the cold)
  • Chest infection
  • Christmas
  • Second born child arriving
  • Paternity
  • Pneumonia and pleurisy

The latter of those I’m still getting over and have literally been back at work for a single day.

I have to say a massive thank you to my employer Test Partners for their unwavering support and understanding and all my family and friends who have helped through those issues. You guys really are special people! 👍🏻

So my plan from here on is to try and fit in a little blogging time when I can. It will likely be sparse and sporadic for a few weeks/months based on the prognosis for pneumonia but let’s see how we get on; normally my body is pretty good at fixing itself.

My next blog post is a relatively simple one – “People in glass houses”

Watch this space!

Do you like it dirty?

Do you like it dirty?

When I arrive at work after my morning tube commute the first thing I do is use some antibacterial hand gel before I check my emails and tasks for the day. I’d rather kill any cold or virus germs asap but not enough to walk from the ground floor up to the 3rd floor.

After I’ve checked emails, tasks and anything else urgent I’ll head upstairs to the second floor to make a coffee and en route I’ll pop up one more floor into the bathroom and wash my hands so that they’re clean as well as germ free.

I’m a clean kind of chap; not OCD clean but perhaps cleaner than I need to be.

With my testing I like my environments to also be as clean as necessary for the system under test:

  • Websites (including mobile sites) – No unnecessary programs running in the background and a cleared down browser with no addons enabled that may affect the site.
  • Mobile apps – Factory reset device with no unnecessary apps installed or accounts that aren’t essential to the OS. No backgrounded apps or non-test-app push notifications enabled (except for deliberate testing)

The idea behind this is to reduce any false positives – Any issues I discover will almost certainly be a problem with the system under test and I won’t be incorrectly reporting issues that are centred around interaction with other software, much of which could be unique to my own preference of apps.

For instance if I was testing software on a system that had an IDE installed on it and I was unaware that the software had a reliance on certain runtimes that were previously installed and were not native to the operating system I’d completely miss the fact that the software may not even run for many users.

Similarly certain setups from previous testing sessions may not have been returned to their base state so I could end up spending unnecessary time troubleshooting proxy/DNS settings, environmental variables that should be changed etc.

I’ve spoken to quite a few people about their test system preference and generally there seems to be two camps:

  1. The Angelically Clean Crew – Their machine/device is a freshly imaged desktop machine or mobile device irrelevant of any previous testing. If for example the only other testing that had been done on that environment was website testing and there’d be an almost non-existent chance of any false positives it doesn’t matter – “Image all the things!”
  2. The Dirty Dozen – A more “real world” approach – Machines/devices are more often than not their development machines or personal mobiles and filled with other apps, accounts, push notifications for lots of apps enabled, backgrounded apps etc – “But real users don’t have perfect environments”

There doesn’t seem to be many people with a “good enough” approach when it comes to test environments for some reason. That may be a completely factual statement but also it may be that the people I’ve spoken to have tacit knowledge about what they would accept and it’s not entirely clean/dirty but a happy Goldilocks place.

I completely see the benefit to all levels of dirtiness in testing setups but what I struggle with is lack of time on our many “one hit” (waterfall) projects and prioritising those levels of dirtiness in terms of time spent setting up versus bugs encountered versus time spent investigating and ruling out other system interactions as the cause etc.

I think until we have the time to perform mirrored testing on both a clean and dirty device simultaneously across many projects to prove one way or other how much impact there is to time due to device cleaning versus false positives sticking to a an “everything immaculately clean” approach is the best option, even if it’s overkill.

What are your thoughts? How dirty do you like it? Do you have any real world experience or statistics of the value between clean/dirty?

Tracking manual compatibility testing coverage – How do you do it?

Tracking manual compatibility testing coverage – How do you do it?

This blog post I’m hoping will actually be a good discussion on various methods and techniques you guys may use to track your coverage when performing manual compatibility testing on a project.

In the interest of not ‘seeding’ any specific ideas or techniques I’m going to avoid discussing the current process we use at work in this blog post but I’ll happily share after you guys have talked in the comments below.

Let’s say a project lands on your lap and you’re told “here’s our website redesign. We need it testing. The baseline environment is Windows 10 on the latest Chrome release but we want it tested on a total of 20 mixed Windows, Android and iOS environments”

How do you track your coverage while manual compatibility testing those 20 environments?