Building a Templating Language in Node.js - Part II: Gulp and Mocha and Travis! Oh My!

Now that we have a clear idea of what we’re building, let’s get our development environment and work flow set up. Since this is a longer post that contains a lot of introductory information, I’ve included a table of contents below. Feel free to skip over topics you’re familiar with!

Table of Contents

  1. Setting up the git repo
  2. Creating a package.json file
  3. Setting up Gulp
  4. Setting up Mocha
  5. Writing our first tests
  6. Continuous Integration and GitHub Flow
  7. Setting up Travis CI
  8. Conclusion

First There Was Git

We’ll start by creating our git repository, but before we do we’ll need to name our templating language. After much debate and deliberation I have settled on the name Sumo (I assure you, you don’t want to know the other options I considered).

I won’t go over how to set up a GitHub repo in this post, but if you’re unfamiliar with the process, GitHub has a handy guide.

After we’ve created our repo, let’s open up a terminal or command prompt and clone it onto our local machine.

git clone https://github.com/mgmeyers/sumo.git

⇡ To top

Setting up package.json

Next, we need to create a package.json file, which holds metadata about our project such as name, author, version, dependencies, and any build or test commands.

Let’s open up the project root directory in the terminal and execute:

npm init

This handy command will prompt us for information about our project and will generate a package.json file based on our responses.

Here is the information I entered. The values in parenthesis are defaults, as npm init will try to make some intelligent guesses.

name: (sumo) 
version: (0.0.0) 
description: An HTML-ish JavaScript templating system
entry point: (index.js) 
test command: gulp test
git repository: (https://github.com/mgmeyers/sumo.git) 
keywords: templating
author: Matthew Meyers <hello@matthewmeye.rs>
license: (ISC) MIT

This creates a package.json file containing:

{
  "name": "sumo",
  "version": "0.0.0",
  "description": "An HTML-ish JavaScript templating system",
  "main": "index.js",
  "scripts": {
    "test": "gulp test"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/mgmeyers/sumo.git"
  },
  "keywords": [
    "templating"
  ],
  "author": "Matthew Meyers <hello@matthewmeye.rs>",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/mgmeyers/sumo/issues"
  },
  "homepage": "https://github.com/mgmeyers/sumo"
}

You’ll notice that we’ve set our test command to gulp test. We haven’t set up gulp yet, but we will be setting up a test task which will run all of our Mocha tests. If we’d like, we can run this command using npm test, but more importantly this command will be run by Travis CI each time we push changes to GitHub.

We can always come back and modify package.json, so let’s call it done for now and commit our changes.

git add .
git commit -m "Adds package.json"

⇡ To top

Setting Up Gulp

For those of you unfamiliar, gulp is a tool for automating a wide array of development tasks, such as:

  • Running tests
  • Checking code for syntax errors
  • Compiling SASSLESS
  • Minifying CSSJS
  • Concatenating files

You can see a full list of gulp plugins on the gulp website.

Installing gulp is a two step process. First, we need to make sure gulp is installed globally using the -g flag with npm install. This will allow us to execute gulp tasks from the command line, like gulp test.

npm install -g gulp

Next, we’ll install gulp and save it as a dependency for our project. This will allow us to bring in gulp as a module when we write our gulp tasks.

npm install gulp --save-dev

If we take a look at package.json, it should now list gulp under devDependencies:

{
  "name": "sumo",
  "version": "0.0.0",
  // [...snip...]
  "devDependencies": {
    "gulp": "^3.8.7"
  }
}

Having our dependencies listed in package.json allows us to install all of them with a single command:

npm install

Note: there is also an option for saving modules as production dependencies: --save. The difference is in what the package is used for. If your application relies on an NPM module to function, then the module is a production dependency. If it relies on a module only for building, testing, or other development tasks, then it is a development dependency. Executing npm install --production will install only the production dependencies.

We’ll want to commit our changes before moving on, but first let’s make sure git ignores the node_modules folder, so we’re not checking thousands of files into our repo. To do this, we’ll need to create a .gitignore file in our project root containing:

# Ignore the `node_modules` folder
node_modules

Now we can commit our changes.

git add .
git commit -m "Adds gulp as a dependency"

Our First Gulp Task

Now that we have gulp set up, let’s create our first gulp task: linting. The purpose of a code linter is to analyze a piece of code and notify us if there are any potential syntax errors or poor coding conventions. The linter we’ll use is called JSHint, and is generally more forgiving and user friendly than the original JSLint.

Let’s start by installing the modules gulp-jshint and jshint-stylish.

npm install gulp-jshint jshint-stylish --save-dev

Note: the purpose of jshint-stylish is to make the output of jshint easier to read in the terminal.

Setting up gulpfile.js

Next, let’s create a file in our project’s root directory named gulpfile.js. This is the file gulp will look to for task definitions. For example, if we were to run gulp lint in the terminal, gulp would look for a lint task in gulpfile.js.

In our gulpfile, let’s bring the modules we’ll be using and setup the skeleton of our lint task.

var gulp = require('gulp');
var jshint = require('gulp-jshint');
var stylish = require('jshint-stylish');

gulp.task('lint', function () {
  // Perform some linting here
});

We want to make sure we lint everything, including our gulpfile. To do this we first need to tell gulp which files we want to process.

gulp.task('lint', function () {
  /**
   * We want to lint all of the JavaScript files in
   * the current directory as well as the `tests`
   * directory
   */
  return gulp.src(['./*.js', './tests/*.js']);
});

Now that we’ve determined which files we want lint, let’s specify what gulp should actually do with them.

gulp.task('lint', function () {
  return gulp.src(['./*.js', './tests/*.js'])
    /**
     * First, we pipe our files through `jshint`
     */
    .pipe(jshint())
    /**
     * Then we'll pipe the results through the
     * `jshint-stylish` module which will output
     * the results of `jshint` to the console in
     * a human friendly format.
     */
    .pipe(jshint.reporter(stylish))
    /**
     * Finally, we want `jshint` to terminate the
     * gulp task when it finds an error. This way
     * our tests won't run unless `jshint` passes.
     * To do this we'll pipe the results into the
     * `fail` reporter.
     */
    .pipe(jshint.reporter('fail'));
});

Now, if we execute gulp lint in the terminal it will run JSHint on all of our JavaScript files, including our gulp file!

At this stage, the output of gulp lint should look something like:

[gulp] Using gulpfile /home/matt/Documents/Projects/sumo/gulpfile.js
[gulp] Starting 'lint'...
[gulp] Finished 'lint' after 30 ms

And, for the sake of demonstration, if we delete the very last semicolon in gulpfile.js, and run gulp lint we should get something like:

[gulp] Using gulpfile /home/matt/Documents/Projects/sumo/gulpfile.js
[gulp] Starting 'lint'...

/home/matt/Documents/Projects/sumo/gulpfile.js
  line 20  col 3  Missing semicolon.

✖ 1 problem

[gulp] 'lint' errored after 71 ms JSHint failed for: /home/matt/Documents/Projects/sumo/gulpfile.js

/home/matt/Documents/Projects/sumo/node_modules/gulp/node_modules/orchestrator/index.js:153
      throw err;
            ^
Error: JSHint failed for: /home/matt/Documents/Projects/sumo/gulpfile.js

Let’s add the semicolon back and commit our changes.

git add .
git commit -m "Adds gulpfile.js and js linting"

⇡ To top

Setting Up Mocha

Now that gulp is set up, we can also use it to run our tests. By now you should be familiar with installing packages through npm, so go ahead and install the gulp-mocha package and save it as a dev dependency.

Next, at the top of our gulpfile, let’s bring in gulp-mocha.

var gulp = require('gulp');
var jshint = require('gulp-jshint');
var stylish = require('jshint-stylish');
var mocha = require('gulp-mocha');

Now we can create our test task. We’ll want to make sure our code is free of syntax errors before we test it, so let’s also make sure our lint task runs before our tests do.

/**
 * Let's designate `lint` as a dependency for
 * this task, so `test` will only run after
 * `lint` has completed.
 */
gulp.task('test', ['lint'], function () {
  /**
   * Set `read` to false so gulp passes the file
   * references straight to mocha without reading 
   * them. This will help speed things up.
   */
  return gulp.src('./tests/*.js', { read: false })
    /**
     * Let's pipe all of the JavaScript files
     * in our test folder into Mocha. Mocha
     * also has various reporters available.
     * I prefer `spec`, but feel free to choose
     * the one you like best.
     */
    .pipe(
      mocha({
        reporter: 'spec'
      })
    );
});

See the mocha documentation for a list of bundled reporters.

⇡ To top

Defining Our First Tests

Great, now that we have mocha set up we can write some tests! Our first will be simple IO tests. They will ensure that:

  1. A plain String will not be modified by sumo.
  2. Standard HTML will not be modified by sumo.
  3. Malformed input will not be modified by sumo.

While these tests may seem simple, they offer a great baseline. If any of them fail, it most likely means there is a serious error in our code, and we want to catch any serious errors as soon as possible.

Let’s create a tests folder in our project root, and inside of it create a file, 01-simple-io.js. There’s no specific reason I named the file this way other than making it easy for me to scan. Feel free to use a naming convention that suits you, but be consistent!

In 01-simple-io.js, let’s start by pulling in our dependencies.

/**
 * If we don't specify a file name, require will use 
 * `index.js`, which is our main file (or will be 
 * when we create it).
 */
var sumo = require('../');
/**
 * We'll use the assertion library built into node.
 */
var assert = require('assert');

A Note on Assertions

For those of you unfamiliar, the purpose of an assertion library is to throw errors when certain conditions are met. For example:

var a = 2;
var b = 1;

/**
 * assert.equal(actualValue, expectedValue, [message])
 * 
 * This assertion will pass, so nothing will happen.
 * The message will be passed along only if the assertion
 * fails.
 */
assert.equal(a - 1, b, 'a - 1 does not equal b');

/**
 * This assertion will NOT pass, so an error will
 * be thrown. The error text will be: 'a - 1 does equal b'
 */
assert.notEqual(a - 1, b, 'a - 1 does equal b');

/**
 * You can also simply pass in an expression
 * that evaluates to a boolean value
 * 
 * assert(value, message)
 */
assert(a > b, 'a is not greater than b');

Most test frameworks use these errors to determine if a test has passed or failed. There are many assertion libraries for node, some minimal and some complex. The built in assertion library is on the minimal side, which I prefer, but this is totally a personal preference. Check out the node.js documentation for more information about the assert module, and the methods available. It’s also worth looking into Chai, one of the more popular assertion libraries for node.

Now that we know some basic assertions, let’s start writing some tests!

Writing Our Tests

First, we need to describe our test suite. This allows us to group related tests together.

// `describe` is a function defined by mocha
describe('Simple IO', function () {
  // Tests go here...
});

Note: If you need to organize your tests further, you can nest multiple describe functions.

Next, we will describe our first test.

describe('Simple IO', function () {
  /**
   * the function `it` takes in a string describing 
   * the test or tests being performed and a function
   * that should throw an exeption if the test(s)
   * have failed
   */
  it(
    'should return an unmodified string when a plain string is passed in', 
    function () {
      var string = 'Hello!';

      assert.equal(
        /**
         * We haven't implemented `sumo.compile()` yet,
         * but this is how I imagine it will be used.
         * We can always come back and update our tests
         * if we need to.
         */
        sumo.compile(string, { name: 'Bob' }), 
        /**
         * The output of `sumo.compile(string)` should
         * be the same as `string`, since `string`
         * doesn't contain any templating logic.
         */
        string, 
        /**
         * This message will let us know exactly
         * which test failed, and will only be
         * displayed when the test fails.
         */
        'Plain string does not match output string'
      );
    }
  );
});

If the plain string we pass into sumo.compile() is modified in any way when it comes out, then this test will fail. Notice that we haven’t actually written any application code yet. This is an important part of the test driven development process. We write tests first, watch them fail, and then write the application code needed to make them pass. Before we can run our tests, we’ll need to create some boilerplate code defining all of the objects and functions used by the tests.

We’ll start by creating a file named index.js in our project root folder. In it, we’ll define sumo and define the compile() method.

var sumo = {
  // Nothing here yet...
};

sumo.compile = function () {
  // Nothing here yet...
};

/**
 * `module.exports` will be returned by `require('sumo')`
 */
module.exports = sumo;

Now we can actually run our test using gulp test, which outputs:

[gulp] Using gulpfile /home/matt/Documents/Projects/sumo/gulpfile.js
[gulp] Starting 'lint'...
[gulp] Finished 'lint' after 41 ms
[gulp] Starting 'test'...


  Simple IO
    1) should return unmodified string when plain string is passed in


  0 passing (3ms)
  1 failing

  1) Simple IO should return unmodified string when plain string is passed in:
     AssertionError: Plain string does not match output string
     [...Stack trace removed...]



[gulp] 'test' errored after 13 ms 1 test failed.

Success — kind of! Our test is correctly failing, so now we just need to add the application logic needed to make it pass.

Back in index.js, let’s flesh out sumo.compile().

sumo.compile = function (templateStr) {
  return templateStr;
};

Huh? That’s it?!

Like I said before, we only want to write enough code to make our tests pass. If we run our tests now, we should see that our tests do indeed pass!

[gulp] Using gulpfile /home/matt/Documents/Projects/sumo/gulpfile.js
[gulp] Starting 'lint'...
[gulp] Finished 'lint' after 43 ms
[gulp] Starting 'test'...


  Simple IO
    ✓ should return unmodified string when plain string is passed in 


  1 passing (3ms)

[gulp] Finished 'test' after 11 ms

If it seems strange to you that we wouldn’t write any code to start handling the template syntax, I ask you to bear with me until my next post were we’ll start writing some actual application code. This process of writing tests, watching them fail, and then implementing only enough code to get them to pass is the core of test driven development. As we get further into the project we’ll start to see the benefits of this system.

Let’s go ahead and add our two other tests to 01-simple-io.js.

describe('Simple IO', function () {
  it(
    'should return unmodified string when plain string is passed in', 
    function () {
      var string = 'Hello!';
      assert.equal(
        sumo.compile(string, { name: 'Bob' }),
        string,
        'Plain string does not match output string'
      );
    }
  );

  it(
    'should return unmodified HTML when plain HTML is passed in', 
    function () {
      var html = '<div>Hello!</div>';
      assert.equal( 
        sumo.compile(html, { name: 'Bob' }), 
        html, 
        'Plain HTML does not match output HTML'
      );
    }
  );

  it(
    'should return malformed input unchanged', 
    function () {
      var html = '<div>{ name</div>';
      var malformedCount = '<count collection="test"';

      assert.equal(
        sumo.compile(html, { name: 'Bob' }), 
        html, 
        'Malformed input does not match output'
      );

      assert.equal(
        sumo.compile(malformedCount, { test: [1,2,3] }), 
        malformedCount, 
        'Malformed count does not match output'
      );
    }
  );
});

We shouldn’t need to touch our application code to get this to work. Let’s see what gulp test says.

[gulp] Using gulpfile /home/matt/Documents/Projects/sumo/gulpfile.js
[gulp] Starting 'lint'...
[gulp] Finished 'lint' after 56 ms
[gulp] Starting 'test'...


  Simple IO
    ✓ should return unmodified string when plain string is passed in 
    ✓ should return unmodified HTML when plain HTML is passed in 
    ✓ should return malformed input unchanged 


  3 passing (3ms)

[gulp] Finished 'test' after 12 ms

Perfect! Try not to get to caught up on the tests in the beginning, we can always go back and add more. The important thing is to start defining the desired behavior of the application so we have a clear path for development.

A Note About Testing Small Projects vs. Large Projects

One thing I would like to bring up before we move on, is that the way tests are organized for this project is not necessarily how one would organize them for a larger project. This project is small enough that we can organize tests by the feature they are testing. On a larger project, however, it might make more sense to organize test by modules or components, and then by feature. For example, if you were to write tests for the entire JavaScript language, your tests would looks something like:

- Array
  - Array.prototype.find
    - Shouldn't throw a TypeError if IsCallable(predicate) is true
    - Find on empty array should return undefined
    - The length property of the find method should be 1
    - [...etc]
  - Array.prototype.indexOf
    - [...etc]
  - [...etc]
- Number
  - Number.isNaN
    - should return false if called with a boolean
    - should return true if called with NaN
    - should return false if called with a non-number Object
    - [...etc]
  - [...etc]

Let’s move on, but before we do let’s commit our changes.

git add .
git commit -m "Adds mocha and simple IO tests"

⇡ To top

Continuous Integration and GitHub Flow

Great, almost there! Our final task before we begin writing application code is to set up a continuous integration server.

Those of you familiar with continuous integration may be wondering why I am using it on such a small project with only one developer. The way this project will be set up when we’re done may seem like overkill to some, but at the very least it is an opportunity to practice good development habits, and that’s what this series is all about.

What Is Continuous Integration?

In the web development world, the term continuous integration is somewhat nebulous. A lot of people have a lot of opinions on what constitutes continuous integration, and they don’t always agree. Part of this disagreement seems to come from differing development ecosystems. For example, a C++ application will be built, tested, and deployed much differently than a node.js application.

At its core, continuous integration is a development methodology that attempts to reduce the friction caused by integrating code from multiple developers a master code base. This friction can come in the form of conflicts caused by two developers working on the same code, or bugs introduced by a developer’s code or during the integration process. The ultimate goal of continuous integration is for the master code base to never contain a failing build.

How does this apply to our project? In a larger project with many components and multiple developers, the need for continuous integration might be more obvious. In our case, however, we’ll just use a small portion of the continuous integration methodology, combined with GitHub flow to ensure our master branch is always production ready. We’ll be focusing on these core tenets of continuous integration:

  1. All code should be contained in a version controlled code repository, including the tools needed to build and test the main code base.
  2. Developers should commit to the main code base as frequently as possible.
  3. The code should be automatically built (if applicable) and tested after every commit.

What Is GitHub Flow?

GitHub flow is another methodology we’ll follow to ensure our master branch never contains a failing build. The core tenets of GitHub flow are:

  1. The master branch should always be deployable. For example, if you are working on a web app the master branch should always match what is on the live server.
  2. All work should be done on a descriptively named branch.
  3. Commit to your branch early and often.
  4. When work has been completed on a branch, it should be merged into the master branch through a pull request.

Bringing The Two Together

Here’s how our system will work:

  1. After we have all of the boiler plate for our project set up, we’ll never commit directly to the master branch.
  2. When we work on a new feature, we’ll start by creating a new branch.
  3. When we want to integrate that branch into the main repo, we will submit a pull request to a development branch.
  4. The development branch will be tested by Travis Ci, and if all of our tests pass, we will merge our development branch into master.

The whole purpose of this system is to prevent as many bugs as we can from making it into our master branch. The more developers you work with and the more complex your code base, the more you will see benefits from this system and ones like it.

⇡ To top

Setting Up Travis CI

So, what is Travis CI and where does it fit in to our work flow? The idea behind Travis CI is fairly simple. It watches a GitHub repo and will run any build or test scripts when we push changes to a specified branch. It will then notify us if our build or tests fail.

Setting up Travis CI is a breeze. To start, we’ll need to sign into Travis CI with our GitHub account. We can then “add new repository” to Travis which will allow us to point it to a repo of our choosing.

Let’s select sumo.

We’re now done on the Travis CI side. Easy, right? The last piece of the puzzle is our .travis.yml file which tells Travis the environment our project needs, and also which branches it should monitor for changes. Let’s create this .travis.yml file in our project’s root folder.

# First, we need to tell Travis which 
# language we're using, and what version
language: node_js
node_js:
  - "0.10"

# Then, let's specify which branches 
# we want Travis to monitor for changes.
# In our case, We only want our tests to 
# run when we push to the master or 
# development branches
branches:
  only:
    - master
    - development

That’s all there is to it! Let’s commit our changes and push to master, which will trigger Travis CI.

git add .
git commit -m "Adds Travis CI config"
git push origin master

Travis will now queue our project automatically and run the npm test command. When it finishes, Travis will email us the status of the build. We can also view the Travis CI dashboard for more detailed information.

Now for the best part: we get to put a status badge in our readme letting everyone know how awesome we are! Clicking on the ‘build passing’ icon in Travis CI will give us an option to select a branch, and will generate a code snippet that displays the build status of the project. Since this will go in our readme, let’s grab the markdown snippet:

[![Build Status](https://travis-ci.org/mgmeyers/sumo.svg?branch=master)](https://travis-ci.org/mgmeyers/sumo)

⇡ To top

Conclusion

That’s it! We now have a solid work flow and development environment in place that will, in theory, ensure a high quality code base. From here on out we will be writing actual application code. In the next post, we will start by implementing the <count /> tag.

As always, if you have any questions, comments, or see any mistakes, let me know!

Back To Top