​Testing Kinja with Selenium - the big picture - part 1

As our development team and our platform grows, it's impossible for us developers to keep track of all the platform features and the use cases of the different user groups. It's inevitable to have a safety net before releasing new features, so we can make sure, we don't break existing features, and the new ones are working as expected. Our first step towards this safety net is automated browser testing, and in the following series of posts we would like to share the process of how we are incorporating the Selenium browser automation framework in our development process. This post covers the why, and the basics of the "how".

First of all, let's see our expectations:

  • Write tests once, and target a wide range of browsers, operating systems, devices. Fortunately Selenium does that.
  • In the beginning writing and maintaining tests seems like an extra effort, so we want to put the least possible burden on devs, meaning using a familiar language for the tests. Our sites are javascript-heavy, and our frontend build is done via Grunt, so having tests written in javascript was an obvious choice.
  • Speed and scalability. Let's admit it, firing up test machines, launching real browsers, imitating real user interactions, waiting for network operations is anything but fast. We can't defeat it, unless we can scale, and run as much tests as possible at the same time.
  • Make it easy to run complete test suites or just a single test locally on the developer's machine, or in the cloud in multiple browsers, operating systems.

There are many javascript test frameworks out there, our choice was Mocha, because of it's clean BDD style interface, it's reporters and the clean way it handles asynchronous testing. Mocha follows the golden rule of doing one thing, and doing that thing well, therefore it excels in instrumenting tests and creating reports, but doesn't help us with running test cases simultaneously and aggregating their output.

Enter the world of Grunt, node.js and child processes. In our Grunt task we could spawn as many Mocha child processes as needed, let them run the tests parallel, sit back, relax and wait for the results. Sounds too easy, right? Well, there is a catch. Let me explain:

  • We have a finite amount of test machines, CPU cycles, RAM, so we can't run all the tests in parallel, most certainly there will be a waiting line, a queue for the tests, some will have to wait, until the others finish and exit.
  • Selenium tests are timing-sensitive. We are polling the DOM to see whether our modal dialog popped up after we've clicked on the registration link. If it doesn't show up in the given timeframe, we mark the test as a failure, and move on.
  • As soon as we execute a Mocha test, the clock starts ticking.

The last step needs some further explanation. A single tests lifecycle starts by executing it via Mocha, which starts measuring the duration from the start, until the teardown. But we are only interested in our application's performance and reliability, not how fast the test environment can start up a browser, so it must not imply our results. On a developers local machine it might be acceptable, but in the cloud, where many test cases might wait in the same queue, it could lead to test time-outs, even before they had a chance to execute.

The solution should be trivial: move the part of starting a browser outside of the tests scope, and let the test use it by injecting the appropriate session token in the test. This way we separate the test and it's environment.

Unfortunately by the time of writing this post, neither the official selenium-webdriver module, nor it's predecessor the wd module supports starting sessions based on existing tokens, they are starting from fresh always. There's a fork of the wd library on GitHub which enables this kind of functionality, and a pull request also (which we hope will be accepted soon).

After getting the timing issue out of the way, let's see what a test lifecycle looks like in our case:

  1. Invoking our Grunt task
  2. Collecting our test files
  3. Sending a request for a session token for each of the tests
  4. In the order they arrive, spawn child processes, and let them execute the test cases
  5. Wait for all the runners to finish, collect their results, and display it in the terminal, or create an xml or json file, needed for our CI server

Two oversimplified gists demonstrating the process:

The grunt task, running the tests:

A simple webdriver-mocha test file:

In the following posts, you can read more about creating the grunt task, running the tests in different environments (local machine, cloud-Saucelabs), having different test suites for different devices and platforms. Stay tuned!