Browsertesting component from the Nancy framework was expensive to create for each of our 1500 server side UI unit testsAt DrDoctor we value automated tests, unit tests, TDD, and so on. We want our tests to run quickly so we
We use the light-weight Nancy framework for our web applications hosted via ASP.NET. It has a really nice Browser testing component. This helps test our views more effectively, where we can assert the generated HTML is as we expect.
We have a growing number of tests:
Recently, I added about 75 tests which somehow doubled the test time execution to over 40 minutes. Something strange was going on and we wanted to revisit this again as we couldn't go on like this.
I noticed in some UI test suites, each test would take a second or so longer than the previous test. Clearly this wasn't going to scale. Adding only a small number of tests could increase the overall test time a LOT.
OneTimeSetUp instead of SetUp for Nancy Browser bootstrapping in NUnitWe use NUnit as our unit testing framework of choice. It lets you write SetUp and TearDown methods that run before and after each test. A great way to reuse common code.
So in our SetUp we would configure our Browser component, including defining the Bootstrapper which may include a lot of dependencies, some of which might be mocks to reduce testing inter-dependencies.
However, creating a new Browser instance together with all the bootstrapper configurations for every test run is somewhat expensive. The Browser component may be sophisticated doing many things for you so you don't have to, but with 1500 UI tests using this, this is a lot of repeated expensive set up each time.
The solution? Rather than using NUnit SetUp for every test use OneTimeSetUp instead to set up the Browser only once per suite of tests.
From a crazy 40 minutes our test suite now runs in around 5 minutes!
For each test suite we can now see the first test may take a few seconds for starting up (due to the bootstrapper/Browser) but all the other tests in that suite take just milliseconds, as we would expect.
We still haven't pinpointed the exact reason why tests would gradually take longer than the previous test in a given suite, but it does seem to be related to the bootstrapper/browser and we are still investigating further.
The other thing I am a bit uncomfortable about with this approach is the notion of an independent unit test. Ideally each unit test is completely independent of other tests, even in a suite. But setting up the Browser and bootstrapper once for a suite of tests means you now have to watch for unintended state being set (session or cookies for example) between tests.
One way we do that is using the Bootstrapper's pre-request hooks to hook into the request pipeline where you can clear any cookies or session items to have a clean state.
As an example here is a very cut down version of what one of our bootstrappers might be doing:
[OneTimeSetUp]
public virtual void OneTimeSetUp()
{
_service = new Mock();
_bus = new Mock();
_logger = new Mock();
_bootstrapper = new ConfigurableBootstrapper(cfg =>
{
cfg.ApplicationStartup((container, pipelines) =>
{
FormsAuthentication.Enable(pipelines, _formsAuthConfig);
Csrf.Enable(pipelines);
CookieBasedSessions.Enable(pipelines);
});
cfg.StatusCodeHandlers(typeof(MethodNotAllowedCodeHandler));
cfg.ViewFactory();
cfg.Module();
cfg.Dependency(_service.Object);
cfg.Dependency(_bus.Object);
cfg.Dependency(_logger.Object);
// and various other dependencies added here as needed
});
_requestHelper = new RequestHelper(_bootstrapper.BeforeRequest);
_browser = new Browser(Bootstrapper);
}
Initially we struggled to get a satisfactory solution to clearing session per test as we were hooking into a pipeline to add/remove the items as needed. But as a test suite builds up, these pipeline handlers would execute in subsequent tests which was highly inefficient.
The solution was to create named pipelines which can then be removed per test in one place. This is where we created a little RequestHelperto help manage session and cookies per request and we ended up with something like this:
public class RequestHelper
{
private readonly BeforePipeline _beforePipeline;
private readonly List _namedItems = new List();
public RequestHelper(BeforePipeline beforePipeline)
{
_beforePipeline = beforePipeline;
PipelineSetUp();
}
private void PipelineSetUp()
{
_beforePipeline.AddItemToStartOfPipeline(context =>
{
context?.Request?.Cookies?.Clear()
context?.Request?.Session.DeleteAll();
return null;
});
}
public RequestHelper SetSessionVariable(string key, object value)
{
_namedItems.Add(key);
_beforePipeline.AddItemToEndOfPipeline(new PipelineItem<Func<NancyContext, Response>>(key, context =>
{
if (context.Request.Session == null)
{
context.Request.Session = new Session(new Dictionary<string, object>());
}
context.Request.Session[key] = value;
return null;
}));
return this;
}
public void Reset()
{
foreach (var namedItem in _namedItems)
{
_beforePipeline.RemoveByName(namedItem);
}
}
}
This class helps to clear any response cookies and session. It sets up named pipelines whenever a test wants to set up a session variable. Named pipelines means they can later be removed (in a per-test TearDownfrom NUnit) from the Nancy Browser request pipeline so that the next test doesn't try to add it to the session again.
Then in the TearDownof an NUnit test we can reset this helper:
[TearDown]
public void TearDown()
{
_requestHelper.Reset();
// and any other tear down related things
}
Assuming the above methods are in your test class you can use the request helper instance to set up session variables for a given test. Example:
[Test]
public void PostToOurModuleShouldWork()
{
_requestHelper.SetSessionVariable("Some session key", "Some value");
var expectedOutcome = // some expected result;
var response = _browser.Post("/some/route/", with =>
{
with.TestCsrfValues();
with.FormValue("Name1", "value 1");
with.FormValue("Name2", "value 2");
});
AssertActualOutcomeIsExpected(response, expectedOutcome);
}
The above test is a contrived example. You could imagine if the form values were parameterised using TestCase you could quickly include a variety of inputs for testing without repeating the above. Without a OneTimeSetup for the Browser and bootstrapper, the ease of adding additional tests cases would mean you quickly find this running slowly (depending on what your bootstrapper needs to do).
Multiply that with other tests in the suite that might do similar things, you can see why we quickly get 1500 tests!
Your mileage may vary; you may not need all this with sessions/cookies. Or you may not even need to change from SetUp to OneTimeSetUp. For us it has made a big difference though! We are more productive and can release builds even more frequently now!
Originally posted on Anup's blog