UI and Integration Testing for a Full-Stack Javascript App

UI and Integration Testing for a Full-Stack Javascript App

While developing our new full-stack Javascript app, we needed to pick a technology for UI and integration testing.

For our Ruby apps, we use Capybara with RSpec and the Poltergeist bindings for PhantomJS. While the stack works, to simplify development and ramp-up time for new developers, we wanted a Javascript-based technology.

Criteria:

  • Write tests in Javascript
  • Compatibility with Jasmine
    • Our unit tests are in Jasmine, so we'd prefer not to juggle multiple test syntaxes
  • Supports ECMAScript6
  • Can run headless on a Jenkins server
  • As close to a real browser as possible
    • Officially we only support Chrome, so a Chrome/Webkit browser is preferred over Firefox/Gecko
  • Actively maintained, community support

Browser Testing Technology

Document Object Model (DOM) Emulation

Some technologies like JSDOM emulate the Document Object Model of a browser. They run your HTML and Javascript in that context and provide APIs to query the resulting DOM.

While this is useful for unit tests, we wanted something closer to a real browser for our integration tests.

Headless Browsers: PhantomJS and SlimerJS

PhantomJS and SlimerJS are headless (no display) browsers that use the Webkit (Safari, Chrome) and Gecko (Firefox) engines. Since they don't require a display, they are perfect for running on build servers that don't have monitors. However, they're often somewhat outdated or lacking in features compared to the desktop browsers whose engines they share.

Selenium

Selenium is a technology stack for controlling browsers by emulating mouse clicks, keyboard input, etc. It uses a client-server architecture. The server listens for instructions, and executes them on the browser. It's implemented in Java and is available as a JAR. The client ("driver" in Selenium lingo) sends instructions to the server, and there are drivers for a variety of languages. Nightwatch JS and WebdriverIO are the two main drivers for Javascript.

Using Selenium would give us the most accurate test interactions, but we need to be able to run on our Jenkins build server, which doesn't have a monitor. Chrome won't run without a display device. At the time we set up this system, Chrome hadn't announced their headless mode, so we needed to find another solution.

The blog post Run your tests headlessly with Xvfb describes how to use Xvfb to fake a display device. After installing Chrome on the Jenkins server, we successfully launched it with xvfb-run.

Conclusion

We decided to consider both PhantomJS and Selenium when choosing candidates.

Candidates

We gathered a list of Javascript integration testing technologies, read about them, and decided on a short list to evaluate in more detail.

We discarded these candidates because they didn't meet enough of our needs to warrant a closer look:

  • JSDom - not a full browser, just emulates a DOM
  • PhearJS - only returns a DOM, doesn't let you interact with it
  • Lotte - last update in 2013
  • WebSpecter - last update in 2015
  • PhantomJS + Jasmine + phantom-jasmine - not a full framework, just provides PhantomJS APIs in jasmine. phantom-jasmine last update in 2012.
  • SlimerJS - runs on Gecko (Firefox engine) but not truly headless. Would prefer Selenium + Xvfb + Firefox for Gecko tests, or PhantomJS for a headless browser.
  • ZombieJS - test framework that emulates a DOM (based on an outdated version of JSDom).
  • DalekJS - says "Don't use this for production!" on the website.
  • Chimp + Selenium + WebDriverIO - WebDriverIO has synchronous syntax, so no clear benefit to adding this on top.
  • Sahi Pro - not clear from the website what they actually provide, community support seems limited to their website

For in-depth evaluation, we wanted to try at least one dedicated PhantomJS technology, and at least one that used Selenium. For Selenium, we wanted to try running it against both PhantomJS and Xvfb + Chrome.

The final candidates:

Evaluation Plan

For each option on our shortlist, we decided to test the login page:

  • Load the login page
  • Log in with valid credentials and see the landing page
  • Log in with invalid credentials and see the error message

Casper.js

Casper.js is a navigation and test framework for PhantomJS (Webkit) or SlimerJS (Gecko). Since we are targeting Chrome, we chose to run it on PhantomJS.

We quickly ran into problems getting PhantomJS to play nice with our app. We use Babel to transpile to ES5, but PhantomJS refused to display anything except the container HTML. This is strange because our unit tests run on Jasmine with PhantomJS. After hammering at it for over a day, we decided to move on.

Another strike against Casper was the test API. They strongly recommend you include the planned or desired number of tests as an argument to the test suite function, which makes maintenance harder. The overall syntax was so different from Jasmine that we didn't think it was worth spending the time to make PhantomJS work.

Nightwatch JS

Nightwatch is a Javascript driver for Selenium.

It has first-class page object support. Their page object API makes it easy to organize code. As someone new to browser testing, the structure was great for learning.

There were a few minor problems. Neither of them were deal-breakers:

  • Slightly harder to set up than WebdriverIO
  • Test syntax is Mocha instead of Jasmine, so we'd have one test syntax for unit tests and a different one for browser tests. Mocha is pretty similar to Jasmine so it's a minor problem.

The big problem was test isolation. All the tests seemed to run in the same browser session. If we manually closed the session, the later tests would fail. Since some of our tests involve testing login functionality and user permissions, it was important to have clean browser sessions for them.

Test isolation was the deal-breaker, and why we went with the final choice, WebdriverIO.

WebdriverIO (Winner)

WebdriverIO (aka WDIO) is another Javascript driver for Selenium.

It supports both Jasmine and Mocha syntax. It has a setup wizard that can automatically install dependencies for you, like selenium-standalone and chromedriver.

In comparison to Nightwatch:

  • The website discusses page objects, but doesn't have built-in helpers. It recommends using Object.create or ES6 classes.
  • For test isolation, every test file is launched in a new browser session. There's also the possibility of parallelizing tests, with multiple files being run at once.
  • Project is more active than Nightwatch.

This made WebdriverIO the winner for our integration tests.

Conclusion

The folks at WebdriverIO have put together a great package for integration testing full-stack Javascript that's constantly being improved. It's a great all-around choice.

If you don't need session isolation for individual tests, Nightwatch will also work well for you. And if you're new to integration testing, check out Nightwatch's page objects for how to structure your tests. We created an ES6 base class for page objects in WebDriverIO based on the Nightwatch structure. It took a little more work but got us most of the benefits.

Contact us. Let's create magic together.

Our Newsletter is good. Sign up so we can deliver the goods. (Not bad, huh?)

Request a call