Unit testing is great, but it doesn’t show you the whole picture. Integration tests are also nice, because you know that your components will work together as expected, but it is still missing some pieces.
The fact is, that when you code is in production, your users still encounter bugs no matter how many unit and integration tests you have.
You are still worried that when a real user hits your app she might immediately find a new bug.
Your integration and unit tests are just not interacting with the app the same way people do.
That is why there exist end-to-end tests, also known as functional tests, scenarios tests or simply as web or browser tests when it comes to web apps.
Imagine what would be
- Testing your work just like a real person will interact with it
- Knowing that every component from end to end will work together nicely
How it will all work?
To make it a reality you will need to setup several components before even writing the tests.
You are going to run your browser tests in a real browser. They can work even when your testing machine does not have GUI, like a Linux server for example.
We are going to use Chrome. The full blown version that so many people love and use, not the open source variant Chromium. We want to get as close to the real users as possible.
Then you are going to use ChromeDriver. It is a component which is developed by the same people who create Chrome & Chromium.
On one hand, it can control Chrome just like a real user will do. On the other hand it provides a standard API to which you can connect from Node or any other platform.
This is the WebDriver API. It’s a standard API with available libraries for almost any language out there.
You are also going to use webdriverio to interact with ChromeDriver from NodeJS. It implements the WebDriver API and makes interacting with it a breeze.
When you want to run Chrome on a Linux machine without GUI, like a server, you will need one more component XVFB.
To run on Linux, Chrome (and any other GUI app) needs an X server. However, you don’t want a full blown X server just to run a few tests. You need something fast and light.
XVFB is such fast and light X server, which actually doesn’t display anything. Instead it writes all commands to a buffer. It is developed by the same people who create the full blown X server.
You are NOT going to use Selenium
Whenever you read something about browser tests on the Internet with a real browser it always uses Selenium. People don’t even think about it twice.
However, when you are using ChromeDriver, you don’t require Selenium at all. ChromeDriver has everything you need and supports the complete the WebDriver API.
Let’s install and configure everything you need.
If you are going to run your tests on a Mac with OS X or on a Linux with GUI like Gnome, you don’t need XVFB and you can skip this part.
You only need it when your testing machine does not have a GUI.
To install XVFB on Debian or Ubuntu do the following
On CentOS or Red Hat you should
Once installed start it like this
This command runs the fake X server and creates a display for your programs to connect to at :99
You are not going to use Chromium but a full version of Chrome, just like your users.
If you are on a Mac, it is easy to install it from Chrome’s website.
Let’s see how it is done on Linux even without a GUI. You have to add the appropriate repository and then just install it as usual.
On Debian or Ubuntu
On CentOS or Red Hat
First create a file /etc/yum.repos.d/google-chrome.repo
Then install it with just one line
ChromeDriver will not only control Chrome but it will also start and stop it.
One instance of ChromeDriver can control multiple instances of Chrome simultaneously. It also supports multiple clients to connect to it and each of them to run its own independent tests.
On OS X you can install it from here http://chromedriver.storage.googleapis.com/index.html?path=2.20/
On Linux you can install it like this
Then run the following to start it
The exported variable instructs ChromeDriver which display to use to run Chrome. This is the fake display provided by XVFB. You don’t need to export this variable when you are not using XVFB.
You have everything in place to run your tests. Let’s have a look what your project might look like.
The project is very simple, just three files. app.js is the entry point of your application. tests.js contains all of your tests and views houses a few of your templates.
This structure is so simple because I want to you to focus on the web tests. To learn how a real Node & Express project can be set read my article on Best practices for Express app structure
Let’s install the project dependencies.
- express is the web framework that we are going to use to build the app.
- jade will provide us with a decent templating language.
- mocha will help us structure and run our tests.
- should is a great library for testing whether result is what you expect.
- webdriverio will help you control Chrome like a real user.
Let’s have a look at how each of the project files looks like.
Here is app.js
This is a simple Express web app. It defines two routes /funny and /sad, which just render the two templates.
Let’s see how each template looks like. This is funny.jade
Let’s now look at sad.jade
The two templates are almost identical with only a few differences.
Now let’s run the web app. You need it running to be able to access it from a real browser like when running the web tests.
Everything is ready, so let’s create your first tests.
There are a lot of things going on here. First, the options object contains the configuration for our browser.
By default webdriver expects Selenium which listens to http://127.0.0.1:4444/wd/hub. However ChromeDriver listens to http://127.0.0.1:9515/ and this is part of the configuration above.
You may also notice that there are many Chrome specific options. Their purpose is to make Chrome behaviour more predictable and faster, especially when starting.
Once configured you have to start the browser. This is the slowest operation, even with the parameters from above, and you want to do it inly once in a file.
The place for such operations is in the before function.
Once Chrome is started, it will not be closed automatically at the end of your tests. What’s more, when you run your tests with XVFB you will not even notice it but it will still eat your memory.
That is why you are using the method
after to close it after all your tests have
Finally, there are the two tests. Each of them begins on one of the two pages in the app, then checks whether an element specific to the page exists, then clicks a link to go to the other page and at the end checks the title to see whether it arrived at the right page.
You can see that we are using the url, getTitle, click and element methods.
- url loads a new page in the browser.
- getTitle reads the title in the browser.
- element looks for a specific element on the page.
- click clicks on an element.
I am using standard CSS selectors to select the elements I want, but there is also XPath if you are in that sort of thing. It is actually more powerful than CSS.
You can check the documentation at webdriveroi.
With the way the tests above are written you should not worry about any asynchronious actions.
As you can see we return the result from all the methods chained on
The returned result is a Promise. Mocha will wait for this promise to be resolved
before moving to the next test.
To run your tests, all you have to do is
There are mainly two ways to get feedback when you are running browser tests.
When you are running on a machine with with GUI you can just see how the page looks. When you are running on healdless machine you can still take a screenshot.
The other way to get some useful feedback is to look at the page source code.
Problems that you may encounter
There are a few problems which you might encounter when creating web tests. First,
sometimes the browser is not closing properly after the tests end. Even when you
call end in the
This can be fixed by restarting ChromeDriver. It will clear any window which is left and eating your memory.
There are many more similar problems that might happen, so be careful.
It is not all roses
You have just seen how wonderful web tests are. Unfortunately, they have a dark side, too.
The first problem is that they are slow, significantly slower than any other kind of tests.
The second problem is that their feedback is poor, just some element missing on a page, a screenshot and a source code. It may sound a lot, but it is not.
It is often hard to understand what really went wrong.
To make it worse, all your components, databases and services are taking part. Nothing is mocked and any of them can produce a problem. This makes it even harder to identify what went wrong.
Go on and create your first end-to-end tests. They might take a little bit more time to write, but the satisfaction of knowing that this is how the user will experience your site is great.
However, don’t make too many of them. They are hard to debug, and very slow. You should really on unit and integration tests to catch 99% of your problems and only then on web tests.
Once you have seen how it works with Chrome, you can check with other browsers. For example, you can try running Firefox with Selenium, or even Chrome on iPhone or Android.
The last thing I would suggest is to try to automate your browser testing with Jenkins.
Other articles that you may like