How Google Tests Software – my review



James Whittacker, Jason Arbon & Jeff Carollo

First published 2012

What’s a developer doing reading a book on testing? Isn’t that something for the Tester to read?

…was how I once thought. These days my mindset is along the lines that a good developer is more than just being able to write good code. It’s more about being part of a team that delivers software. We need to be involved with defining the story of what our customer wants, writing the code using TDD techniques and being able to work with a tester to verify that we are delivering a piece of quality to our customer.

This book is focussed on the roles of the people who make Google’s software. All roles require some developer nous. Each part of the book goes into some detail, explaining what that role covers within Google. This book assumes you are familiar with development and testing practices.

The four main roles are:

  • The Software Engineer (SWE): A developer who delivers features. Of course done using TDD and also ‘participates in Tests’.
  • The Software Engineer in Test (SET): A developer focused on testability, quality and risk. Works closely with the SWE to help create test code, such as mocks and stubs. The book goes on to say they even create Testing Frameworks. Their goal is to test the developer and ensure code is of a high standard and consistent with the rest of the codebase.
  • The Test Engineer (TE):  Focussed on testing the user. i.e. ensuring that what’s getting delivered is what the user asked for and is of a good level of quality. They write test automation scripts. They are product experts. They are Quality and risk analysers.
  • The Test Engineer Manager (TEM): Primarily responsible for coordinating TE and SET. A take away for me was that these people are really product experts on top of really knowing their people.

Crowd Sourcing

Google also has the clever use of getting the public to tests its software for them. They call it crowd sourcing, it’s a natural extension of dog fooding (which they also do). The idea is to get users to run through tours, execute, specific testing and perform exploratory testing. They even have incentives for  bug finders! The Test Engineer’s role as I understood it was to coordinate this, to try and get a good level of coverage and facilitate how bugs are fed back to the developers.

The Future…

The final chapter is disappointingly not about killer robots, self driving cars or transportation devices. Whittacker waxes lyrical about how he sees the testing landscape at Google evolving over the next few years. Like combining SET and SWE roles. This was a thought I also had when reading the book –  if you’re a strong SWE – surely a lot of what the SET does are tasks you would do yourself. Especially writing mocks and stubs – for me an important tool I use when writing my code with a TDD approach. They are also using automation tools like Selenium and encouragingly he sees that having SWE/SETs contributing to open source frameworks such as Selenium is much more beneficial as opposed to the cost of innovating and maintaining an in-house testing framework.

Test engineers are reducing all the time. One key reason here is feedback. Making use of continuous deployment mechanisms they can get software to the customer (live, crowd testers, dog fooders) fast. Bugs can be raised fast and developers can fix these bugs and get them back out to production, fast. I read from this that TE will become more involved with orchestrating crowd tests and writing automation scripts and less involved with manual testing.

Google – just another software department?

This was an enjoyable and easy read but for me didn’t rock my world so to speak. It’s nice to know that the guys at Google aren’t doing anything significantly different to what I’ve seen in other good IT departments in terms of their approach. There is a focus on testing the ACC (Attributes, Components, Capabilities) of its software as opposed to test plans. I think this could read as Specification by Example, as the focus is on what problem the software is solving without getting bogged down in how it’s doing it. If this interests you, should read the book.

Dropdown list with MVC5.1 – now out of the box!

Last year I made this post showing a clean way to render a dropdown list in MVC.

Recently MS released MVC5.1 (prerelase) and excitingly it now has Enum support.

Jon Galloway has written an excellent post outlining this change. The .Net team have added a new HTMLHelper extension called EnumDropDownListFor that will render an Enum as a dropdown box (or radio button list).

How does this compare to the long winded solution I outlined last year, I hear no one asking?

To recap, we want to render a dropdown list that looks like:


with our view rendering our model with one line.


We no longer need the custom Htmlhelper Extension EnumDropDownListFor used in the previous solution, as that now ships with MVC5.1 (well, not the same implementation!). We do, however, still want to be culture aware and use a Resx file to contain the resources of the actual names to display on screen. I’d now recommend using display attributes in the Enum to identify the resource. This will leave our Enum and model looking like:

public enum AirlinePreference
    [Display(Name = "AirlinePreference_BritishAirways",
                ResourceType = typeof(Resources.Enums))]
    [Display(Name = "AirlinePreference_EasyJet", 
                ResourceType = typeof(Resources.Enums))]
    [Display(Name = "AirlinePreference_RyanAir",
                ResourceType = typeof(Resources.Enums))]
    [Display(Name = "AirlinePreference_Virgin",
                ResourceType = typeof(Resources.Enums))]

public class PersonViewModel 
    [Display(Name="First Name")]
    public string FirstName { get; set; }

    [Display(Name = "Last Name")]
    public string LastName { get; set; }

    [Display(Name = "Airline Preference")]
    public AirlinePreference? AirlinePreference { get; set; }

For reference our resx file looks like this


Compile it, run it, go to the Add Person page and you’ll see the dropdown list as before…

5.1 is still prerelease and at the time of writing @Html.EditorForModel will not render an Enum dropdown list. It currently needs a push in the form of an Enum template (as described in Jon’s blog) to know how to display an Enum.
With my lean hat on, for the purpose of this blog the following will suffice.

Create a view called Enum.cshtml and put it in the EditorTemplate folder


Add this one line

@Html.EnumDropDownListFor(x => x)

Now go the Add Person page and voila, a DropDownList is present.

Write Less, Test More (part 2)

In my previous post I blogged about Xunit Theories, which provided a neat way to pass multiple test scenarios into one test method. Why I’m liking Xunit these days is due to its extensibility. In particular Adam Ralph and friends have extended it and come up with Xbehave.Net.

A popular pattern when testing is Given (my input), When (I test this), Then (expect this outcome). See Cucumber scenarios for more info. One of the problems with my last post is that the [Fact] didn’t use this language. The test output didn’t use this language. Wouldn’t it be nice if it did?

If we wrote that test with xBehave it would. This post revisits the problem from the first post and shows how we could test it using xBehave.Net.

Problem Recap

I want a class that, when given a sentence, will return a collection of strings containing all words in lowercase. This collection should only have alpha characters in it – i.e. spaces and punctuation are to be ignored.

To write our test for the zero case, using xBehave it could look like:

public void FindWordsEmptyStringScenario(string input, IEnumerable output)
"Given an Empty String"
   .Given(() => input = string.Empty);

"When I find words in this string"
   .When(() => output = _wordFinder.FindAllWordsInLowerCase(input));

"Then I expect the answer to be an empty list"
   .Then(() => Assert.Empty(output));

Cucumber uses scenarios, so does our test. The test method, takes as its inputs an input and output variable. The method cleanly has three calls, each coming off a string describing the action:
Spelling it out, the Given() sets up our input variable.
When(), does the business of calling our method under test, returning the output, into our variable, called output.
Then(), asserts if the output is what we expected.

Talk about readable!

It gets better – run this through reSharper and our output looks like:


Again very readable I think.

As I’m sure you guessed, this framework supports something very similar to Xunit’s [InlineData]. It has examples.

Our test scenario, with all example inputs and outputs, based on first blog, looks like

[Example("", new string[0])]
[Example("One", new [] { "one" })]
[Example("One two", new[] { "one", "two" })]
[Example("One...:; - ,two", new[] { "one", "two" })]
public void FindAllWordsInLowerCaseScenario(string input, string[] expectedOutput, IEnumerable output)
    "Given the sentence '{0}'"
        .Given(() => { });

    "When I find words in this sentence"
        .When(() => output = _wordFinder.FindAllWordsInLowerCase(input));

    string.Format("Then I expect the answer to be [{0}]", string.Join(",", expectedOutput))
        .Then(() => Assert.Equal(expectedOutput, output));

What’s nice here is that if you remember the outputs from the InLine tests – it didn’t output the expected string arrays’ content. Here, as the method name that gets generated for the test runner output is simply a string we can manipulate the string to show the contents of the expected array.


Each example provided effectively creates a new scenario, which are grouped in the output by scenarioNumber.StepNumber.

For the output names to be neatly formatted here (without repetition of the input variables) we are setting the attribute [OmitArgumentsFromScenarioNames] on the class, which suppresses them.