Tweaking our Nancy unit tests to run in 5 minutes instead of 40

TL;DR version:

• We value fast, automated tests
• Our suite has grown to over 4300 tests
• Our suite already took about 20 minutes to run, on average, and recently adding only a handful of tests increased total time to over 40 minutes
• Turns out the useful Browsertesting component from the Nancy framework was expensive to create for each of our 1500 server side UI unit tests
• Changing our Browser and bootstrapper configuration to run once per suite rather than once per test, resulted in our total test time being reduced to just over 5 minutes!
• That saving is worth the trade-offs we had to make

We value fast, automated tests

At DrDoctor we value automated tests, unit tests, TDD, and so on. We want our tests to run quickly so we

• Don't lose focus while waiting for tests to complete
• Are not discouraged from running all or most of the tests all the time while we are coding
• Are able to churn out builds (which go through an automated continuous integration process) quickly for frequent deployment to UAT environments or production

Nancy framework has a handy Browser testing component

We use the light-weight Nancy framework for our web applications hosted via ASP.NET. It has a really nice Browser testing component. This helps test our views more effectively, where we can assert the generated HTML is as we expect.

Server side UI test execution time was slowing down dramatically

We have a growing number of tests:

• About 4300 over our entire system
• 1500 tests are server side UI tests
• Tests are split to run in parallel, such as integration tests, unit tests, spec flow tests, database test, and JavaScript tests
• The longest running group is the unit tests, which has 3500 tests in it, including all the server side UI ones, and it takes 20 minutes to run (not ideal, but it gets worse!)

Recently, I added about 75 tests which somehow doubled the test time execution to over 40 minutes. Something strange was going on and we wanted to revisit this again as we couldn't go on like this.

I noticed in some UI test suites, each test would take a second or so longer than the previous test. Clearly this wasn't going to scale. Adding only a small number of tests could increase the overall test time a LOT.

Solution: use OneTimeSetUp instead of SetUp for Nancy Browser bootstrapping in NUnit

We use NUnit as our unit testing framework of choice. It lets you write SetUp and TearDown methods that run before and after each test. A great way to reuse common code.

So in our SetUp we would configure our Browser component, including defining the Bootstrapper which may include a lot of dependencies, some of which might be mocks to reduce testing inter-dependencies.

However, creating a new Browser instance together with all the bootstrapper configurations for every test run is somewhat expensive. The Browser component may be sophisticated doing many things for you so you don't have to, but with 1500 UI tests using this, this is a lot of repeated expensive set up each time.

The solution? Rather than using NUnit SetUp for every test use OneTimeSetUp instead to set up the Browser only once per suite of tests.

From a crazy 40 minutes our test suite now runs in around 5 minutes!

For each test suite we can now see the first test may take a few seconds for starting up (due to the bootstrapper/Browser) but all the other tests in that suite take just milliseconds, as we would expect.

Trade-offs

We still haven't pinpointed the exact reason why tests would gradually take longer than the previous test in a given suite, but it does seem to be related to the bootstrapper/browser and we are still investigating further.

The other thing I am a bit uncomfortable about with this approach is the notion of an independent unit test. Ideally each unit test is completely independent of other tests, even in a suite. But setting up the Browser and bootstrapper once for a suite of tests means you now have to watch for unintended state being set (session or cookies for example) between tests.

One way we do that is using the Bootstrapper's pre-request hooks to hook into the request pipeline where you can clear any cookies or session items to have a clean state.

As an example here is a very cut down version of what one of our bootstrappers might be doing:

[OneTimeSetUp]
public virtual void OneTimeSetUp()
{
	_service = new Mock();
	_bus = new Mock();
	_logger = new Mock();

	_bootstrapper = new ConfigurableBootstrapper(cfg =>
	{
		cfg.ApplicationStartup((container, pipelines) =>
		{
			FormsAuthentication.Enable(pipelines, _formsAuthConfig);
			Csrf.Enable(pipelines);
			CookieBasedSessions.Enable(pipelines);
		});
		cfg.StatusCodeHandlers(typeof(MethodNotAllowedCodeHandler));
		cfg.ViewFactory();
		cfg.Module();
		cfg.Dependency(_service.Object);
		cfg.Dependency(_bus.Object);
		cfg.Dependency(_logger.Object);
		// and various other dependencies added here as needed
	});

	_requestHelper = new RequestHelper(_bootstrapper.BeforeRequest);

	_browser = new Browser(Bootstrapper);
}

Initially we struggled to get a satisfactory solution to clearing session per test as we were hooking into a pipeline to add/remove the items as needed. But as a test suite builds up, these pipeline handlers would execute in subsequent tests which was highly inefficient.

The solution was to create named pipelines which can then be removed per test in one place. This is where we created a little RequestHelperto help manage session and cookies per request and we ended up with something like this:

public class RequestHelper
{
	private readonly BeforePipeline _beforePipeline;
	private readonly List _namedItems = new List();

	public RequestHelper(BeforePipeline beforePipeline)
	{
		_beforePipeline = beforePipeline;

		PipelineSetUp();
	}

	private void PipelineSetUp()
	{
		_beforePipeline.AddItemToStartOfPipeline(context =>
		{
			context?.Request?.Cookies?.Clear()

			context?.Request?.Session.DeleteAll();

			return null;
		});
	}

	public RequestHelper SetSessionVariable(string key, object value)
	{
		_namedItems.Add(key);

		_beforePipeline.AddItemToEndOfPipeline(new PipelineItem<Func<NancyContext, Response>>(key, context =>
		{
			if (context.Request.Session == null)
			{
				context.Request.Session = new Session(new Dictionary<string, object>());
			}

			context.Request.Session[key] = value;

			return null;
		}));

		return this;
	}

	public void Reset()
	{
		foreach (var namedItem in _namedItems)
		{
			_beforePipeline.RemoveByName(namedItem);
		}
	}
}

This class helps to clear any response cookies and session. It sets up named pipelines whenever a test wants to set up a session variable. Named pipelines means they can later be removed (in a per-test TearDownfrom NUnit) from the Nancy Browser request pipeline so that the next test doesn't try to add it to the session again.

Then in the TearDownof an NUnit test we can reset this helper:

[TearDown]
public void TearDown()
{
	_requestHelper.Reset();

	// and any other tear down related things
}

Assuming the above methods are in your test class you can use the request helper instance to set up session variables for a given test. Example:

[Test]
public void PostToOurModuleShouldWork()
{
	_requestHelper.SetSessionVariable("Some session key", "Some value");

	var expectedOutcome = // some expected result;

	var response = _browser.Post("/some/route/", with =>
	{
		with.TestCsrfValues();
		with.FormValue("Name1", "value 1");
		with.FormValue("Name2", "value 2");
	});

	AssertActualOutcomeIsExpected(response, expectedOutcome);
}

The above test is a contrived example. You could imagine if the form values were parameterised using TestCase you could quickly include a variety of inputs for testing without repeating the above. Without a OneTimeSetup for the Browser and bootstrapper, the ease of adding additional tests cases would mean you quickly find this running slowly (depending on what your bootstrapper needs to do).

Multiply that with other tests in the suite that might do similar things, you can see why we quickly get 1500 tests!

Your mileage may vary; you may not need all this with sessions/cookies. Or you may not even need to change from SetUp to OneTimeSetUp. For us it has made a big difference though! We are more productive and can release builds even more frequently now!

Originally posted on Anup's blog