Plumbing in Node.js or How to Get Started with Streams

According to an article titled interestingly as "Are coders worth it?", web developers are just sophisticated plumbers. That's a fitting analogue for the topic of this post, streams that is. Streams are a concept originated from the early days of the Unix operating system. If you open up a terminal, you can grasp the power of streams.

Sources, Filters, Sinks

Pipe by darwin Bell
(CC BY)

The idea is very simple. You have sources, filters and sinks. Sometimes people might use a different naming scheme but this is enough for my purposes. Consider the definitions below:

• A source yields data. This could be text, video, audio or whatever you can think of.
• A filter accepts data, transforms it and passes it further as an output.
• A sink accepts input and converts it into some concrete output.

Using these three concepts you may build chains that perform series of transformations on data and output it. Generalized further you may define the transformation as a graph. As a result you end up with a dataflow architecture. If you have ever used a graph editor, say for video or audio, you know what I'm talking about.

The power of this type of scheme lies in the fact that even though the individual parts may be quite simple you can achieve powerful results by combining them.

Examples

I will give you a couple of simple examples based on my usage next:

• pbcopy < file.txt - Copies the contents of file.txt to OS X clipboard
• pbpaste > some_file.txt - Pastes the contents of clipboard to some_file.txt
• james | ./notify.js - Pipes output of "james", a build tool, to a notification script that displays the output on OS X notification center
• james > /dev/null - I could not care less of the output. Let's allow null to chomp that.

There are more of these type of tricks available in this brief Unix streaming tutorial. I've included the source of ./notify.js below:

#!/usr/bin/env node
var notifier = require('node-notifier');
var stdin = process.openStdin();
 
 
stdin.setEncoding('utf8');
stdin.on('data', function(chunk) {
    notifier.notify({message: chunk});
});

It is a simple script that uses node-notifier to stream the input. node-notifier uses terminal-notifier. In fact if terminal-notifier had been designed right, it would accept the stream straight away. It is a good idea to follow the standards when writing terminal tools as it allows other people to build upon your work more easily.

Conclusion

I hope this article gave you some idea of the power of streams. In case you wish to use them in your Node.js development, check out Five Minute Guide to Streams2. It explains the Node.js API in further detail. You'll find some familiar concepts there. substack has written a fine guide on the topic as well that goes even deeper.

A while ago we had a local event in which we discussed Node.js streams. As a result some interesting material was born. Check out node-nyanstream and streams-dojo in particular. Big thanks to Esa-Matti Suuronen for compiling those.

If you like adventures, you might want to check out substack's stream-adventure. It comes with a series of small tasks you ought to solve.

I am very interested to hear how you use streams in practice. Are there specific tasks for which they are particularly suitable?