How Google Tests Software – my review



James Whittacker, Jason Arbon & Jeff Carollo

First published 2012

What’s a developer doing reading a book on testing? Isn’t that something for the Tester to read?

…was how I once thought. These days my mindset is along the lines that a good developer is more than just being able to write good code. It’s more about being part of a team that delivers software. We need to be involved with defining the story of what our customer wants, writing the code using TDD techniques and being able to work with a tester to verify that we are delivering a piece of quality to our customer.

This book is focussed on the roles of the people who make Google’s software. All roles require some developer nous. Each part of the book goes into some detail, explaining what that role covers within Google. This book assumes you are familiar with development and testing practices.

The four main roles are:

  • The Software Engineer (SWE): A developer who delivers features. Of course done using TDD and also ‘participates in Tests’.
  • The Software Engineer in Test (SET): A developer focused on testability, quality and risk. Works closely with the SWE to help create test code, such as mocks and stubs. The book goes on to say they even create Testing Frameworks. Their goal is to test the developer and ensure code is of a high standard and consistent with the rest of the codebase.
  • The Test Engineer (TE):  Focussed on testing the user. i.e. ensuring that what’s getting delivered is what the user asked for and is of a good level of quality. They write test automation scripts. They are product experts. They are Quality and risk analysers.
  • The Test Engineer Manager (TEM): Primarily responsible for coordinating TE and SET. A take away for me was that these people are really product experts on top of really knowing their people.

Crowd Sourcing

Google also has the clever use of getting the public to tests its software for them. They call it crowd sourcing, it’s a natural extension of dog fooding (which they also do). The idea is to get users to run through tours, execute, specific testing and perform exploratory testing. They even have incentives for  bug finders! The Test Engineer’s role as I understood it was to coordinate this, to try and get a good level of coverage and facilitate how bugs are fed back to the developers.

The Future…

The final chapter is disappointingly not about killer robots, self driving cars or transportation devices. Whittacker waxes lyrical about how he sees the testing landscape at Google evolving over the next few years. Like combining SET and SWE roles. This was a thought I also had when reading the book –  if you’re a strong SWE – surely a lot of what the SET does are tasks you would do yourself. Especially writing mocks and stubs – for me an important tool I use when writing my code with a TDD approach. They are also using automation tools like Selenium and encouragingly he sees that having SWE/SETs contributing to open source frameworks such as Selenium is much more beneficial as opposed to the cost of innovating and maintaining an in-house testing framework.

Test engineers are reducing all the time. One key reason here is feedback. Making use of continuous deployment mechanisms they can get software to the customer (live, crowd testers, dog fooders) fast. Bugs can be raised fast and developers can fix these bugs and get them back out to production, fast. I read from this that TE will become more involved with orchestrating crowd tests and writing automation scripts and less involved with manual testing.

Google – just another software department?

This was an enjoyable and easy read but for me didn’t rock my world so to speak. It’s nice to know that the guys at Google aren’t doing anything significantly different to what I’ve seen in other good IT departments in terms of their approach. There is a focus on testing the ACC (Attributes, Components, Capabilities) of its software as opposed to test plans. I think this could read as Specification by Example, as the focus is on what problem the software is solving without getting bogged down in how it’s doing it. If this interests you, should read the book.

Write Less, Test More (part 2)

In my previous post I blogged about Xunit Theories, which provided a neat way to pass multiple test scenarios into one test method. Why I’m liking Xunit these days is due to its extensibility. In particular Adam Ralph and friends have extended it and come up with Xbehave.Net.

A popular pattern when testing is Given (my input), When (I test this), Then (expect this outcome). See Cucumber scenarios for more info. One of the problems with my last post is that the [Fact] didn’t use this language. The test output didn’t use this language. Wouldn’t it be nice if it did?

If we wrote that test with xBehave it would. This post revisits the problem from the first post and shows how we could test it using xBehave.Net.

Problem Recap

I want a class that, when given a sentence, will return a collection of strings containing all words in lowercase. This collection should only have alpha characters in it – i.e. spaces and punctuation are to be ignored.

To write our test for the zero case, using xBehave it could look like:

public void FindWordsEmptyStringScenario(string input, IEnumerable output)
"Given an Empty String"
   .Given(() => input = string.Empty);

"When I find words in this string"
   .When(() => output = _wordFinder.FindAllWordsInLowerCase(input));

"Then I expect the answer to be an empty list"
   .Then(() => Assert.Empty(output));

Cucumber uses scenarios, so does our test. The test method, takes as its inputs an input and output variable. The method cleanly has three calls, each coming off a string describing the action:
Spelling it out, the Given() sets up our input variable.
When(), does the business of calling our method under test, returning the output, into our variable, called output.
Then(), asserts if the output is what we expected.

Talk about readable!

It gets better – run this through reSharper and our output looks like:


Again very readable I think.

As I’m sure you guessed, this framework supports something very similar to Xunit’s [InlineData]. It has examples.

Our test scenario, with all example inputs and outputs, based on first blog, looks like

[Example("", new string[0])]
[Example("One", new [] { "one" })]
[Example("One two", new[] { "one", "two" })]
[Example("One...:; - ,two", new[] { "one", "two" })]
public void FindAllWordsInLowerCaseScenario(string input, string[] expectedOutput, IEnumerable output)
    "Given the sentence '{0}'"
        .Given(() => { });

    "When I find words in this sentence"
        .When(() => output = _wordFinder.FindAllWordsInLowerCase(input));

    string.Format("Then I expect the answer to be [{0}]", string.Join(",", expectedOutput))
        .Then(() => Assert.Equal(expectedOutput, output));

What’s nice here is that if you remember the outputs from the InLine tests – it didn’t output the expected string arrays’ content. Here, as the method name that gets generated for the test runner output is simply a string we can manipulate the string to show the contents of the expected array.


Each example provided effectively creates a new scenario, which are grouped in the output by scenarioNumber.StepNumber.

For the output names to be neatly formatted here (without repetition of the input variables) we are setting the attribute [OmitArgumentsFromScenarioNames] on the class, which suppresses them.

Write Less, Test More (Part 1)

Just like my software development skills, my approach to writing tests over the years has changed, but it’s only recently I’ve started to use more than the basic Assert functionality of testing frameworks.

This post explains an approach I used to test a particular module that came about through TDD and how to test a number of scenarios, by writing just one test method.

The problem

I want a class that, when given a sentence, will return a collection of string containing all words. This collection should only have alpha characters in it – i.e. spaces and punctuation are to be ignored.

My tests

As I’m dealing with a collection I think it makes sense that my method should cope with an empty sentence, a sentence with one word and a sentence with n words. At least one of these n word sentences should be peppered with some non alpha characters.

A reasonable approach might be to setup my test class, then write my first test to test the ‘0’ case. This could look like:

public void GivenAnEmptyStringWhenFindAllWordsExpectAnEmptyList()
    var expectedResult = new string[0]; 

    var result = _wordFinder.FindAllWordsInLowerCase(string.Empty);

    Assert.Equal(expectedResult, result);

(Where _wordFinder is the class under test. This test asserts that an empty collection is returned from FindAllWordsInLowerCase method).

Run the test – it’ll fail. Go off create a WordFinder class, with the method FindAllWordsInLowerCase(string input), write the code to return an empty string list

Re-run the test and hopefully the test will pass.

Great. Next test is the scenario with one word. Forget DRY; copy and paste previous test, rename it, change the inputs and expected output and rerun.

public void GivenOneWordWhenFindAllWordsExpectOneItemInLowerCase()
    var expected = new List<string> { "one" };
    var result = _wordFinder.FindAllWordsInLowerCase(OneWord);

    Assert.Equal(expected, result);

Write some code to get this to pass, then, copy and paste test class again, alter inputs and expected outputs… you get the idea.

This will work and is an approach I’ve used for a while. The outcome is great – we’ll have a class with very high coverage that was developed just as our tests informed it. For me though, I prefer to be a bit more DRY than this.

Cue…. Xunit Theories

In a nutshell, an XUnit Theory, is a means to perform a data driven test. Data is provided in an [InlineData(…)] attribute. Each InlineData attribute applied to a Fact, will result in  a different test.

Define your test method and define your theory’s data and you can run multiple inputs and test the expected outcomes through a lot less methods – one in this case! Putting this into practice, our two previous tests now look like:

 [InlineData("", new string[0])]
 [InlineData("One", new[] { "one" })]        
  public void FindAllWordsInLowerCaseTheory(string input, IEnumerable<string> expectedOutput)
     var result = _wordFinder.FindAllWordsInLowerCase(input);

     Assert.Equal(expectedOutput, result);

And when run with ReSharper, produces an output that looks like:


Unfortunately it can’t output expectedOutputs cleanly when it’s a collection, but we have a test that’s DRY and easier to read from a non technical perspective. Potentially a non developer could even provide these inputs and outputs, lending itself to BDD very nicely.

You can also achieve a similar result with nUnit, using it’s [DataSource] attribute.

As we are using XUnit, I suspect it may be possible to modify the output behavior, so that it can display the expected collection of strings better. Not in scope for this post though.

So finally, when all our test scenarios are created our code looks like:

[InlineData("", new string[0])]
[InlineData("One", new[] { "one" })]
[InlineData("One, two", new[] { "one", "two" })]
[InlineData("ONE,4--;@@ ::tWo", new[] { "one", "two" })]
[InlineData("paulthecyclist tested; PaulTheCyclist failed @#$@! PaulTheCYclist Coded, paulthecyclist tested => PaultheCyclist passed!", new[] { "paulthecyclist", "tested", "paulthecyclist", "failed", "paulthecyclist", "coded", "paulthecyclist", "tested", "paulthecyclist", "passed" })]
public void FindAllWordsInLowerCaseTheory(string input, IEnumerable<string> expectedOutput)
    var result = _wordFinder.FindAllWordsInLowerCase(input);

    Assert.Equal(expectedOutput, result);

This produces five tests. I think it’s easier to read than having a test class with five separate methods and certainly less code to write/copy and paste.

There is an even more fun way to write your tests using XUnit though, which addresses the readability of the output along with other things, that I’ll cover in part 2.