node-gyp and node.js on mobile platforms

As I’ve previously discussed I want to get node.js running on Android, iOS and WinRT. But to make that happen we need to understand the node.js ecosystem and that includes native add-ons and node-gyp. So I created a node package, node-gyp-counter, to heuristically determine how frequent node-gyp usage is in the node.js world. If my numbers are right then less than 2% of downloads of packages in 4/2016 involved node-gyp in any way. Of that 2%, just 10 packages account for 76% of node-gyp root package downloads and only 5 of those are relevant to mobile.
[Updated with figures from 5/24/2016]

1 Disclaimer

The code I used is untested, unreliable and probably conceptually incorrect. Seriously. You have been warned!

2 What I found

A tiny bit of terminology before I continue:
node-gyp root package A package that directly uses node-gyp
node-gyp package A package that is either a node-gyp root package or has one or more node-gyp root packages somewhere in its dependency tree.
Count Description
279,196 Total packages in node.js when I ran my query
3172 Total number of node-gyp root packages
28,863 Total number of node-gyp packages
10.3% Percentage of node.js packages that are node-gyp packages
4,230,547,186 Total downloads of node.js packages for 4/2016
71,735,382 Total downloads of node-gyp packages
1.7% Percentage of node.js package downloads that are node-gyp packages
Assuming my data isn’t completely wacked this argues that node-gyp is a tiny part of the node.js ecosystem. Only 1.7% of downloads in 4/2016 used node-gyp in any way, shape or form.
Next up I calculated the set of node-gyp packages that accounted for 80% of node-gyp package downloads in 4/2016. This returned 180 packages. I then calculated the union of the node-gyp root packages used by those 180 packages. The result had 64 entries (see A↓). In other words, if we support 64 node-gyp root packages we can support 80% of all packages that use node-gyp in node.js land. I went through those packages and I’d argue that only 41 of those are relevant to smart phones. Heck, if you look at the actual numbers the top 10 packages account for 76% of node-gyp related downloads and only 5 of those are relevant to mobile. So with literally 5 packages one could support almost 80% of all node-gyp downloads on mobile!
My take away is that if node.js ran on smart phones with no ability to run native add-ons we would still add huge value to the world supporting literally 98.3% of what people actually use node.js for. If we just figured out how to directly support 5 packages (possibly by building them straight into our mobile node.js images rather than using node-gyp to build and link) we would literally cover 98.7% of all node.js uses.

A Node-gyp root packages that account for 80% of node-gyp package use ordered by popularity

Package Name Description Relevant to mobile?
ws Websocket client/server/etc. Yes
fsevents Provides access to OS/X FSEvents. It’s used by Chokidar and Karma amongst others. It’s mostly used for build related stuff. No
NPM Well, um.. NPM. We don’t actually run NPM on mobile. We run it during build time when we create a mobile app. No
node-sass Node bindings to libsass, a stylesheet preprocessor. This is used at build time and so shouldn’t be needed on mobile. No
dtrace-provider Provide deep tracing into node.js. Cool but only runs on OS/X and Solaris systems. No
phantomjs A headless version of Webkit’s Javascript API. Used in node.js land for testing apps that have to run on Webkit. No
phantomjs-prebuilt The prebuild version of phantom-js No
bufferutil Used by WS, it’s actually optional I believe Yes
utf-8-validate Also used by WS, also optional I believe Yes
fibers Adds fibers to node.js. Yes
kerberos Kerberos library. Yes
thread-sleep Puts node to sleep for a specified period of time unless an event occurs Yes
v8-profiler Provides cpu profiling of v8. Yes
v8-debug Used by node-inspector. Yes
sse4_crc32 Accelerates crc32 calculations on Intel processors only No
buffertools Utilities to manipulate node.js buffers Yes
ttf2woff2 A node.js wrapper around Google’s WOFF2 file font format Yes
contextify A potentially cooler way to spin off separate v8 contexts. Seems primarily to be used by jsdom. Yes
hiredis A client for redis. Yes
bcrypt Used to store passwords in a way that is expensive to attack, so provides security if a password file is lost. Yes
sqlite3 Client for sqlite3 the ubiquitous mini-database. Yes
ws-pure Ironically this is a pure node.js implementation of WS except it has optional native dependencies Yes
prebuild Manages pre-built binaries, this is a build time tool, not run time. No
bunyan-syslog Sends bunyan records to a syslog server Yes
heapdump Captures a v8 heap dump Yes
node-expat Wrapper around expat XML parser Yes
websocket A mostly (but not entirely, there’s some C++ code, which is why it’s in this list) javascript implementation of Websockets. Yes
grpc Wrapper for Google’s gRPC framework Yes
microtime Provides current time to a microsecond accuracy. Yes
chromedriver This is a driver for the Selenium build time framework No
webdriver-http-sync A sync implementation of the WebDriver protocol used for testing from a PC No
appium-chromedriver A wrapper for the Chrome Driver used by Appium which uses the WebDriver protocol from a PC to a mobile device No
serialport Accessing local serial ports, I’m saying this isn’t mobile because mobile devices don’t have serial ports! No
leveldown Google’s name/value pair high perf database Yes
weak Allows you to get notified before an object is garbage collected. This is primarily used by dnode which is essentially dcom for node. Yes
runas Run command synchronously with admin privilege, not even workable unless the phone is rooted No
ursa Wrapper for RSA operations in OpenSSL Yes
phantomjs-ext An X86 specific version of the PhantomJS test environment No
pre-commit Makes sure your node.js code passes tests before committing, this is for development not run time No
snappy Bindings for Google’s snappy compression library Yes
iconv Proof that Unicode was a good idea, this library outputs to a bewildering array of country specific text encodings Yes
pathwatcher Watches for changes in files and directories Yes
usage Looks up cpu and memory usage on *nix’s. Yes
deasync Makes async Node.js calls into sync Yes
canvas Emulates canvas HTML element. I believe it’s primarily used for build and testing scenarios. No
execSync Executes shell commands synchronously, only meant for development, not production No
bignum Exposes the big int functionality in OpenSSL Yes
sha3 Implements SHA3 in C Yes
secp256k1 Implements ecdsa’s secp256k1 capability Yes
memwatch-next Detects memory leaks in node Yes
node-zopfli Trades off time for slightly better compression than gzip Yes
gc-stats Provides garbage collection stats for V8 Yes
zmq Provides binding to ZeroMQ Yes
libxmljs Binds to libxml Yes
node-syslog Provides interoperability with syslog deamon on unix systems. No
protagonist Supports specifying Web APIs using a high level language called apiblueprint No
ref Mostly used by FFI I’m going to argue it’s relevant to smart phones as it helps integrate with C code which runs fine on smart phones. No
marker-index Operations on text buffers Yes
bson-ext A binary JSON parser for MongoDB Yes
nodegit Git support for node Yes
oniguruma Native bindings to the oniguruma regular expression library Yes
husky Checks for bad commits No
mmmagic A binding to libmagic that detects content types of files Yes
appium-selendroid-driver Interface to Selendroid Android test system No

3 thoughts on “node-gyp and node.js on mobile platforms”

  1. I am currently using node-webkit on osx and a chrome app with browserify. I want to run my nodejs app on mobiles, iOS and Android to be specific. Of all the native modules I only need the net and dgram (udp) modules. Do you know a project that can help?

    1. About the only project I know that claims to ship iOS and Android with node is JXCore but they use a completely different API to talk to the Javascript engine so native add-ons will generally just not work. They claim that moving native add-ons over to their platform isn’t that difficult but I haven’t tried it. Beyond that I really don’t know of any shipping projects that can help. On iOS there are projects like node.app (which, near as I can tell, isn’t really node.js) and v8like (which is node.js but can’t run on stock JavaScriptCore so you can’t actually ship it) but neither is really going to solve your problem. On Android there is anode but it’s using an ancient fork of Node.js.

      So unfortunately the current situation is a real mess on mobile. I’m hoping to help make this all better but it’s going to take some time.

Leave a Reply

Your email address will not be published. Required fields are marked *