ÆFLASH

I've written a lot about front-end build tools over the past year, and I have one more build-related topic to write about before I go back to talking about functional programming and javascript: Make.

Make is over 30 years old and a standard tool on unix-ey systems (e.g. Linux and OSX). You probably already have it installed on your machine. While it is ubiquitous for C builds, it is flexible enough that you can use it to automate just about anything build-related.

I have been a strong proponent of Grunt for the past few years, but I feel like it is too complicated for what it does. I had found myself fighting with it a bit, especially when I needed to define a complicated sequence of processing steps with intermediate artifacts. It also made me uneasy to see Gruntfiles that were hundreds of lines long, or Grunt configuration folders that contained dozens of small javascript files for large projects.

Meanwhile, I still was using Make. It had become standard at my company to use Makefiles as a sort of "command rolodex" that would list common operations to run on a project, such as make setup to run npm install and any other dependency management, make dev to start up Grunt or any other watchers, and make test or make ci to run unit or integration tests. But these were just a shallow usage of Make -- I wasn't taking advantage of Make's killer dependency management features. I eventually came to realize (with the help of others) that Make is actually better than Grunt in many regards.

If you've ever done any C or C++ programming in linux, you've probably seen a makefile full of rules like this:

foo.o: foo.c foo.h
  gcc foo.c -o foo.o

foo.o depends on foo.c and foo.h, so whenever any of those are newer thatn foo.o, run the gcc command to build it. If foo.o is newer, do nothing. Seems pretty basic on the surface, but within it is beautiful simplicity. Let's bring it to the javascript world:

dist/js/app.browserify.js: $(wildcard app/*.js app/**/*.js node_modules/*/package.json)
  browserify app/index.js --output $@

dist/js/app.min.js: dist/js/app.browserify.js
  uglifyjs --compress $< --output $@

We've just defined how and when to build our dev and production JS bundles. Our browserify bundle depends on anything in the app/ or node_modules/ folder, so when any of those update (or one of our node modules), just run Browserify. ($@ is an automatic variable in Make that just means "the name of the current target".) The minified version depends on the bundle as a prerequsite, so when the bundle updates, run Uglify. ($< is another special var that means the first of the list of prerequsites.) You can set up an alias for your JS files like so:

scripts: dist/js/app.browserify.js dist/js/app.min.js

# required so if you do have a file or folder called "scripts", it will
# always check the prereqs
.PHONY: scripts

Run make scripts and your JS will bundle and minify. If it needs to.

You can also build other things, and perform other tasks pretty easily:

dist/css/main.css: $(wildcard styles/*.scss styles/**/*.scss)
  node-sass --include-path styles/ styles/main.scss $@

styles: dist/css/main.css

static:
  mkdir -p dist
  cp -av public/* dist/

lint:
  jshint --config .jshintrc app/ test/
  jscs app/ test/

build: static styles lint scripts

.PHONY: scripts styles static lint build

Sass compilation and linting accomplished. We also set up a target that will copy any static images or html files we need to our dist/ folder, as well as a build target that will build everything with a single command.

It's also easy to make sure that you have the correct versions of your build tools on hand -- just install them normally as devDependencies with NPM, and add this line to the top of your Makefile to add the executables to your path:

export PATH := ./node_modules/.bin/:$(PATH)

This is the one caveat with a Make-based build system -- any tool will have to be its own separate program. However, you are not limited to javascript or node -- any program in any language will work as long as it is a unix process. This also means you can use any of the standard unix tools where applicable, including pipes!

dist/js/app.min.js: dist/js/app.browserify.js header.js footer.js
  cat header.js $< footer.js | sed 's/NODE_ENV/"production"/g' | uglifyjs -c -o $@

We added a header and footer around our bundle, did some variable substitution, and minified, all in one line. This is using standard unix tools that have existed for decades and are well understood. No need to use grunt-contrib-concat when you can just cat. With proper piping (and use of file descriptors, if things get complicated) you can also obviate the need for intermediate temporary files.

Watching

Make is awesome for defining a build, but it is still up to you to run make build after you modify a source file. For a nice development workflow, file system watching is still where Grunt shines. A very simple way to automate Make is to just hav grunt-contrib-watch watch your entire project folder, and then just have it invoke make with grunt-exec when anything changes. If you have your rules defined properly, with targets and prerequsites accurate, it will only build what it needs to.

grunt.initConfig({
  watch: {
    all: {
      files: ["app/**", "styles/**/*", "public/**/*", "node_modules/*/package.json"],
      tasks: ["exec:make"]
    }
  },
  exec: {
    make: {
      command: "make build"
    }
  }
});

A bit brute force, but it will work. I also like to hook up a dev server with livereload:

grunt.initConfig({
  watch: {
    all: {
      files: ["app/**", "styles/**/*", "public/**/*", "node_modules/*/package.json"],
      tasks: ["exec:make"]
    },
    livereload: {
      files: ["dist/**/*"],
      options: {
        livereload: 35729
      }
    }
  },
  exec: {
    make: {
      command: "make build"
    }
  }
  connect: {
    dist: {
      options: {
        base: ["dist", "node_modules"],
        port: 8888,
        livereload: 35729
      }
    }
  }
});

Browserify

The main drawback to the blunt-force-watch-everything strategy is that your browserify build will be slow. It will rebuild everything fron scratch each time. This will add several seconds to your edit → save → rebuild → livereload → debug → edit cycle, so I would recommend adding grunt-browserify with watch: true to your Gruntfile, and have the make task run when anything except your source javascript changes. This will get you back to sub-second rebuilds. See my previous article for more details on how to set up the Gruntfile with Watchify.

This will mean that you will have to duplicate your browserify configuration in the Gruntfile and in the Makefile. It can be a pain, but if you are doing things right, your custom configuration should be minimal, if non-existent.

I strongly reccommend against anything that requires custom Browserify configuration, or anything that deviates from the node.js module conventions. I've done too much mucking around with bizarre Browserify builds that all become more and more fragile until they break. Aliases, browser overrides, shims, multiple bundles -- they all over-complicate things, will make your build less robust, and are more headaches than they are worth. Manually place things in node_modules/, create your own modules were applicable, favor libraries that are CommonJS native, use the require("foo-browserify") wrapper if you have to, manually shim if there is no wrapper (and automate it with Make), and forget about multiple bundles unless they are absolutely necessary. I've wasted too much time dealing with the fallout from these techniques, and have concluded that they are more trouble than they are worth. The problems with the artifacts and build process are worse than the slightly increased ugliness while coding. Simple module setup also keeps the door open for the increasing numbers of tools that can grok the Node's require() logic. Complicated module setup inhibits interoperability and closes your app off to those special tools. I could write an entire article about this.

So define your Browserify build in both the Makefile and Gruntfile, but keep it simple.

Drawbacks

This won't work on Windows. You can probably get some basic things to work with MinGW, but you won't be able to easily use basic unix tools and pipes and I forsee lots of ugliness with PATHs. Gulp is probably your best bet if you have to support Windows. You trade unix pipes for node streams and will have to write a lot more code.

Make has no built-in watching or dev-server capabilities, so you'll still have to use Grunt or another watcher for those features.

Make has ugly syntax: Bash-like variables, esoteric automatic variables, and strangeness with .PHONY targets. It's less structured than JS, and does take some getting used to. On the other hand, the language is all well documented, follows simple rules, and there are lots of examples.

Other ideas

• You can make makefiles more readable by using variables:

BROWSERIFY_DEPS = $(wildcard app/*.js app/**/*.js node_modules/*/package.json)
JS_BUNDLE = dist/js/app.browserify.js

$(JS_BUNDLE): $(BROWSERIFY_DEPS)
  browserify app/index.js -o $@

• Make supports includes. You can put common tasks in a small file and import it. Store it in a small npm module and you can make repetative configuration portable:

include ./node_modules/build-tools/makefiles/CommonMakefile.mk

• Here's an easy way to run npm install only if you need to:

setup: .last_install

.last_install: package.json
  npm install
  touch .last_install

Make sure to add .last_install to your .gitignore and delete it on make clean.

• Use disc to see pretty analysis of your bundle's file size in your browser:

disc:
  browserify --full-paths app.index.js | discify -O

Summary

• Define your targets, dependencies, and build tasks.
• Use NPM to manage tool dependencies.
• Add ./node_modules/.bin/ to Make's $PATH.
• Write your own build scripts if you need to.
• Use standard unix utilities and pipes where applicable.
• Keep Browserify configs simple.
• Use Grunt for watching and livereload, and have it just invoke make.
• Use variables and includes to keep Makefiles clean.

Make, processes and pipes. Now get back to writing killer apps!

Update: I have written a small Grunt plugin that should ease interoperation between Grunt watchers and Make: grunt-make. It should definitely cut down the number of lines in your Gruntfile.