- Compatibility with Jasmine
- Our unit tests are in Jasmine, so we'd prefer not to juggle multiple test syntaxes
- Supports ECMAScript6
- Can run headless on a Jenkins server
- As close to a real browser as possible
- Officially we only support Chrome, so a Chrome/Webkit browser is preferred over Firefox/Gecko
- Actively maintained, community support
Browser Testing Technology
Document Object Model (DOM) Emulation
While this is useful for unit tests, we wanted something closer to a real browser for our integration tests.
Headless Browsers: PhantomJS and SlimerJS
PhantomJS and SlimerJS are headless (no display) browsers that use the Webkit (Safari, Chrome) and Gecko (Firefox) engines. Since they don't require a display, they are perfect for running on build servers that don't have monitors. However, they're often somewhat outdated or lacking in features compared to the desktop browsers whose engines they share.
Using Selenium would give us the most accurate test interactions, but we need to be able to run on our Jenkins build server, which doesn't have a monitor. Chrome won't run without a display device. At the time we set up this system, Chrome hadn't announced their headless mode, so we needed to find another solution.
The blog post Run your tests headlessly with Xvfb describes how to use Xvfb to fake a display device. After installing Chrome on the Jenkins server, we successfully launched it with xvfb-run.
We decided to consider both PhantomJS and Selenium when choosing candidates.
We discarded these candidates because they didn't meet enough of our needs to warrant a closer look:
- JSDom - not a full browser, just emulates a DOM
- PhearJS - only returns a DOM, doesn't let you interact with it
- Lotte - last update in 2013
- WebSpecter - last update in 2015
- PhantomJS + Jasmine + phantom-jasmine - not a full framework, just provides PhantomJS APIs in jasmine. phantom-jasmine last update in 2012.
- SlimerJS - runs on Gecko (Firefox engine) but not truly headless. Would prefer Selenium + Xvfb + Firefox for Gecko tests, or PhantomJS for a headless browser.
- ZombieJS - test framework that emulates a DOM (based on an outdated version of JSDom).
- DalekJS - says "Don't use this for production!" on the website.
- Chimp + Selenium + WebDriverIO - WebDriverIO has synchronous syntax, so no clear benefit to adding this on top.
- Sahi Pro - not clear from the website what they actually provide, community support seems limited to their website
For in-depth evaluation, we wanted to try at least one dedicated PhantomJS technology, and at least one that used Selenium. For Selenium, we wanted to try running it against both PhantomJS and Xvfb + Chrome.
The final candidates:
- Casper.js + PhantomJS
- Selenium-based stacks:
For each option on our shortlist, we decided to test the login page:
- Load the login page
- Log in with valid credentials and see the landing page
- Log in with invalid credentials and see the error message
Casper.js is a navigation and test framework for PhantomJS (Webkit) or SlimerJS (Gecko). Since we are targeting Chrome, we chose to run it on PhantomJS.
We quickly ran into problems getting PhantomJS to play nice with our app. We use Babel to transpile to ES5, but PhantomJS refused to display anything except the container HTML. This is strange because our unit tests run on Jasmine with PhantomJS. After hammering at it for over a day, we decided to move on.
Another strike against Casper was the test API. They strongly recommend you include the planned or desired number of tests as an argument to the test suite function, which makes maintenance harder. The overall syntax was so different from Jasmine that we didn't think it was worth spending the time to make PhantomJS work.
There were a few minor problems. Neither of them were deal-breakers:
- Slightly harder to set up than WebdriverIO
- Test syntax is Mocha instead of Jasmine, so we'd have one test syntax for unit tests and a different one for browser tests. Mocha is pretty similar to Jasmine so it's a minor problem.
The big problem was test isolation. All the tests seemed to run in the same browser session. If we manually closed the session, the later tests would fail. Since some of our tests involve testing login functionality and user permissions, it was important to have clean browser sessions for them.
Test isolation was the deal-breaker, and why we went with the final choice, WebdriverIO.
It supports both Jasmine and Mocha syntax. It has a setup wizard that can automatically install dependencies for you, like selenium-standalone and chromedriver.
In comparison to Nightwatch:
- The website discusses page objects, but doesn't have built-in helpers. It recommends using Object.create or ES6 classes.
- For test isolation, every test file is launched in a new browser session. There's also the possibility of parallelizing tests, with multiple files being run at once.
- Project is more active than Nightwatch.
This made WebdriverIO the winner for our integration tests.
If you don't need session isolation for individual tests, Nightwatch will also work well for you. And if you're new to integration testing, check out Nightwatch's page objects for how to structure your tests. We created an ES6 base class for page objects in WebDriverIO based on the Nightwatch structure. It took a little more work but got us most of the benefits.