A quick breakdown of the most useful JavaScript modules that I find myself using over and over again.
Intro
This is an opinionated article that focuses on general-purpose modules and utilities that I’ve found invaluable to NodeJS and frontend JavaScript development. It won’t be exhaustive or include any special-purpose modules, as those types of awesome lists are indeed awesome but tend to be a bit overwhelming.
Categories
Command Line Tools
Let’s get started with some extremely useful command line tools.
np - A better npm publish.
If you’re an npm author, I’d highly recommend checking out np, as it makes the process of bumping versions, adding git release tags, and publishing to npm a breeze, especially once you start having more than a couple of modules to maintain. It’s also worth noting release by Zeit as a solid alternative.
yarn - A better package manager compatible with npm.
Although npm v5 is a lot faster than previous versions, I still find yarn to be preferable to npm for local development for its speed and consistency. Either way, you’re working with the same database of npm modules under the hood, and imho there’s no clear winner between the two. You should pick whichever package manager best suits your project’s needs.
As a JS developer in 2018, I would, however, make sure that you’re at least familiar with both
npm and yarn and be comfortable switching off between them.prettier - An opinionated code formatter.
Prettier enforces a consistent style by parsing your code and re-printing it with its own rules that take the maximum line length into account, wrapping code when necessary.
I love eslint and have been a long-time user of JavaScript Standard Style in particular, but the idea behind automatic code formatters like prettier and gofmt is undeniably attractive.
As developers, we spend way too much time and mental energy worrying about code presentation and styling, whereas prettier alleviates the need for those thought processes and allows you to focus on what you’re writing instead of how you’re writing it.
now - Extremely simple deployments.
Now is absolutely the best free deployment system that exists today in terms of simplicity, reliability, and feature set. It‘s great for testing static and dynamic deployments and scales up nicely if and when you require more servers. Aaaaaaaaaand did I mention that it’s free until you want to scale up?!
It plays extremely well with Node.js and JS-powered webapps. I’d also highly recommend checking out the rest of Zeit’s offerings as well, as their team is comprised by some of the best JS devs the community has to offer.
asciinema - Free tool for recording high quality terminal sessions.
See my previous blog post “Making your Code Beautiful” for a breakdown of how you can take advantage of asciinema to produce quality code demos & screencasts like the pros.
Promises
This section really deserves an entire article unto itself, especially now that async & await have started becoming the de facto standard paradigm for concurrent programming in JavaScript. With that being said, I’d highly recommend checking out Sindre Sorhus’ excellent promise-fun module collection if you haven’t already. My only gripe with these modules is that they likely won’t work out-of-the-box with most frontend toolchains like create-react-app or rollup.
Here are a few of the most useful gems which stick out for working with promises and async-style code in Node:
pify - Promisify a callback-style function.
There are a lot of ways to convert functions from old-school callback-style to Promise-style, but I’ve found pify to be the best. It’s tiny and has some niceties like automatic method binding that the built-in util.promisify is lacking.
p-map - Map over promises concurrently.
Concurrency is great, but most of the time you want to set a practical limit on parallelism whether it be throttling network bandwidth or compute resources. This is where p-map really shines. I use it 99% of the time as a drop-in replacement for
Promise.all(…), which doesn’t support limiting parallelism.Before I was aware of p-map, I created my own version async-await-parallel, but you should use p-map as it’s better. 😛
p-retry - Retry a promise-returning or async function.
I generally wrap any HTTP requests and external service calls with p-retry to add a basic level of robustness to them. Combined with p-map, you can process large batches of external requests with controlled parallelism without having to worry too much about an occasional transport error, socket hang-up, or server timeout.
p-timeout - Timeout a promise after a specified amount of time.
Alongside p-retry, p-timeout is a must for robustly working with third-party APIs and services. You can also specify an optional fallback, as oftentimes returning something is better than hanging indefinitely or returning after an inordinate amount of time.
p-cache or p-memoize - Memoize the results of an async function via an LRU cache.
The aim of many of these Promise utilities reminds me a lot of architecting robust microservices, where every external dependency can be treated with a common interface supporting retrys, timeouts, caching, circuit breakers, fallbacks, etc.
Graceful degradation of functionality is generally preferable to overwhelming the system or not responding at all, so if you’re not too familiar with microservices, check them out and see if their design decisions can help improve your Promise handling abilities as well.
Scraping
There are a lot of great scraping utilities out there, some of which operate on raw HTML like cheerio, and some of which simulate a full browser environment like puppeteer. What you decide to use really depends on your use case, as working with raw HTML is a lot quicker and more lightweight, whereas automating a headless browser is more robust at the cost of being heavier to get started.
cheerio - Fast, flexible, and lean implementation of core jQuery designed specifically for the server.
Cheerio is really great for quick & dirty web scraping where you just want to operate against raw HTML. It provides a robust jQuery-like syntax for traversing & manipulating HTML documents. Cheerio pairs particularly well with request-promise-native below for fetching remote HTML documents.
puppeteer - Headless Chrome Node API
Unlike cheerio, puppeteer is a wrapper for automating headless chrome instances, which is really useful for working with modern JS-powered SPAs. Since you’re working with Chrome itself, it also has best-in-class support for parsing / rendering / scripting conformance. Headless Chrome is still relatively new, but it will likely phase out older approaches such as PhantomJS in the years to come.
If you need to faithfully scrape websites, automate web-based workflows, or capture screenshots, puppeteer is a clear winner that will only become more popular with time.
Node.js
dotenv-safe - Load environment variables from .env and ensure they’re all present.
This module extends the hugely popular dotenv module to enforce the existence of expected environment variables via a
.env.example file. Like the original, it provides fast, secure, and robust environment variable support for Node.request and request-promise-native - Simplified HTTP request client.
Making HTTP requests is an extremely common operation, and my goto module here is request-promise-native which wraps the original request module with native ES6 promise support. 95% of the time I want to await the result of a promisified HTTP request. The other 5% of the time I want to work with the response stream directly, in which case I use the underlying request module, foregoing Promise support.
For robustness, I will oftentimes wrap request-promise-native calls in some combination of p-retry, p-timeout, and p-cache.
It’s also worth mentioning got as a newer alternative to request with promise support baked in, although I haven’t used it much personally.
Example of downloading an HTML document with request-promise-native.
const request = require('request-promise-native') const html = await request('https://github.com') console.log(html)
consolidate - Template engine consolidation library for Node.
Consolidate is great for handling any type of backend templating (emails, tweets, raw html, etc.). I generally use handlebars as my templating engine of choice, but no matter what, I always wrap my template usage in consolidate, because it provides a simple & consistent interface to templating regardless of the template engine you decide to use under the hood.
For example, I used consolidate in create-react-library to render the boilerplate’s templates with library-specific variables.
execa - A better child_process.
Extremely useful if you need to run a shell command or spawn a child process in general.
fs-extra - A better fs with additional methods and Promise support.
It’s rare that I find myself using
fs directly anymore. Try fs-extra and you won’t look back.Math
D3 (Data-Driven Documents) is a popular frontend library for data visualization and animation. It also contains some of the best standalone packages for common math operations that I find myself consistently choosing over alternative modules.
d3-random — Generate random numbers from various distributions.
When
Math.random doesn’t cut it, give d3-random a try. It supports sampling from different common distributions, including uniform, normal, and exponential.d3-ease - Easing functions for smooth animations.
d3-interpolate - Interpolate numbers, colors, strings, arrays, objects, whatever!
This module provides a variety of interpolation methods for blending between two arbitrary values. Values may be numbers, colors, strings, arrays, or even deeply-nested objects.
Testing
ava - Awesome JS test runner.
It’s not surprising that my go-to for unit test runner for Node.js is yet another tool created by Sindre Sorhus. Ava is a newer unit test runner that takes a lot of what was good about mocha, tape, chai and other JS test runners, and bundles it all together into a quality project with sensible defaults that “just works”.
It’s worth noting that Ava’s tests are run in parallel by default, which you can disable at the file-level for use cases like database testing where the order your unit tests run may be important.
nock - HTTP mocking and expectations library.
Nock is great for testing modules that perform HTTP requests in isolation. If your Node module makes HTTP requests and you want to provide proper unit testing, then nock is the way to go.
sinon - Function spies, stubs and mocks for JS testing.
Sinon is a very useful utility library for writing isolated tests by taking advantage of dependency injection. It should be a part of every Node developer’s tool belt.
Wrapping Up
I hope you’ve found this breakdown helpful, even if it’s just learning about one quality module that you weren’t aware of before. I know a lot of aspiring and seasoned developers alike who end up rolling their own solutions to common problems, which can be a useful practice in & of itself, but it’s also good to know when there are quality, existing solutions you should be using instead of constantly reinventing the wheel.
The size & scope of NPM’s module library is unprecedented, and imho it’s one of JavaScript’s biggest advantages compared with other programming languages. The better you become at taking advantage of npm modules, the faster and more productive you’ll be as a developer. Higher-order “soft” skills like this are one of the hallmarks of becoming a mythical 10x programmer.
Have any favorite npm modules I left out? Let me know by sharing your favorite modules in the comments! ❤️
Follow me on twitter for more awesome stuff like this @transitive_bs