Why automated testing is essential

But we do manual testing; isn’t that enough?

Here’s the classic software development workflow I see in this situation:

1. The client - a business function owner or product manager - creates a specification or a set of outcomes for the engineering team to deliver. A new feature for an existing product.
2. The engineering team creates some code that delivers this feature to the specification and does a demo.
3. The client - maybe a QA team - then runs a series of tests manually to verify that the new feature meets the acceptance criteria.
4. Any issues found are fixed by the engineering team and the feature is re-tested by the QA team.
5. The feature is shipped to the customer.

On the surface, this seems to make sense. It’s low-effort to implement, the engineering team can focus on “coding”, and the feature is thoroughly tested before any real customer sees it.

But this masks a fundamental misunderstanding about the way software engineering works. Any change could cause issues in any previously shipped feature too.

Analogy time

Let’s take an analogy from another type of engineering: building a train.

Imagine last week you implemented a new feature: a cellular radio in the cab of the train relays messages about signals from boxes on the side of the track, so the driver has information about any unexpected speed restrictions or closed sections of track ahead. You did all your manual, real-world testing of the feature and it all worked so you shipped it.

This week, the feature is to add a microwave oven to the train’s galley so that passengers can have hot food on their long journey. Your team dutifully implemented the microwave oven and it worked perfectly in testing, so you shipped it.

But - as anyone who has tried to use a mobile phone near a microwave will attest - there’s a hidden cost here. Whenever the microwave is running, the cellular signalling system goes down. This is a major safety risk for the train! But in shipping the microwave feature, you only tested the new feature and you didn’t realise it was breaking an existing feature.

Regression testing

Testing existing features to make sure they haven’t broken is called regression testing.

You can probably see right away why manual testing doesn’t cut it if you want to do regression testing. Every time you add a new feature, your “bill” for testing increases in size. Soon enough, your test script will be so long that it will be the bulk of your time spent.

This is the hidden cost of skipping test automation. Features ship at lightning pace in the beginning when there are only a handful of them, but as soon as your product gets even remotely complicated, you’re either running a huge test script on every release, or you’re having to spend most of your time going back and fixing bugs in old features every time a new feature is released to the customer.

If you automate this, you have a test that runs every time, so that particular feature cannot regress without someone being alerted. Having a feature covered by a test is sometimes called defending it (like in chess), or you might hear coverage when talking about whether a specific file or function or line of code is tested during the regular build.

Testing is the engineers’ job

I firmly believe this one. Writing automated tests should be the job of the engineers implementing the feature that is being tested.

I’m not saying fire your QA team - they’re an incredibly valuable skillset. QAs are the best people to define the tests - to identify “happy and unhappy paths” through a feature and demand that certain acceptance criteria are met.

But don’t invest your energy upskilling your QA team to write automated tests. This part is absolutely the responsibility of the engineers who are writing the feature. They understand the codebase the best and will be able to implement the automation the fastest. They’ll also be able to add unit tests which test low-level things like individual methods and algorithms. Unit tests are not there to find issues - they’re there to help you quickly fix them when the other tests find the issues, and they can also help with Writing the algortihm in the first place.

It’s important to add automation to your definition of done. Here are some lines I typically like to add:

• Features are only done when they have tests automating all acceptance criteria. Acceptance criteria that cannot be automated should be marked as such in code comments (many testing systems allow you to add these to the report so you remember to manually test them).
• Bugfixes are only done when they are defended by a test. This can be a unit test or something higher level, but it must fail when the fix is not included and pass when the fix is merged.

Test-driven development

Test-driven development (TDD) is the idea that your tests drive development - designing your tests before you write your code actually helps you write code better and faster.

(I’m not a TDD purist - I don’t really believe that you need to fully automate the tests before you start writing any code. But I do think there’s enormous value in scoping them out and writing a pseudocode version before you begin.)

Very often, when people are writing code, they’ll sketch out test cases without even thinking. They’ll run their function and say to themselves “if I put in LS28 5UJ I get out Amity Brew Co” and they’ll test it in a terminal or development environment.

Then that test case is lost. But, if they were also writing an automation at the same time, they could have incorporated that test case into the automation, and very quickly worked it up to a number of other test cases too.

I’ll never quite believe how so many engineers have this seemingly infinite capacity to throw away their working once they have something they think is complete! Test automation is a good way to ensure that all that pen-and-paper thinking you were doing is kept somewhere.

This even applies on a tiny timescale, like a weekend. So often people get to the end of Friday and because all their working is in their heads or in paper scribbles, they have to start again almost from scratch on Monday. What a waste!

This is a culture change

So when this is so self-evidently beneficial, why does it so often not happen in real teams?

It’s because it’s a culture change.

Adding automated testing to your development workflow will look slower in the beginning.

Also, if engineers haven’t got a lot of practice doing it, they can find it tedious or frustrating. (Honestly, once you get the hang of these frameworks they can be an absolute delight, and the most fun part of the job! Who wouldn’t want to see a little video generated of exactly where your feature broke??)

Your engineering team want you to be happy, so they’re going to take shortcuts to get things to demo quicker. If you don’t have a team culture that insists on automation, your team is going to feel like they look bad when they insist on doing this slow thing that you don’t even care about.

But, if your team culture doesn’t care about automation: don’t be surprised if - in 6 months’ time - most of your work is bugfixing!

(PS. I’ve got a way to get you out of that hole if you’re already there. Let’s talk?)