Node doesn’t really have an application model in the normal sense since Node sees itself as a server technology. But as Electron, NWJS, JXcore and JXcore-Cordova have shown there is a real need for Node based applications, both on desktop and mobile. This article is intended mostly as a primer to explore some of the issues anyone wanting to jump into building applications that embed Node.js need to think about. [Note: Updated on 4/18/2017 to add Termux, thanks to Jean-Jacques Debray for pointing it out.]
1 Setting the stage - how Node.js services are typically deployed
Node’s roots as a server technology means it has an interesting view on how one deploys a Node service. The process for deploying a Node service on a target machine is typically something like:
- Download the Node.js runtime of one’s choice
- Download a platform appropriate compiler if one is going to use native extensions, e.g. Visual Studio on Windows or GCC/Make on Linux, etc.
- Run ’npm install’ which will read the package.json, download all dependencies and if those dependencies involve native code use the compiler downloaded in step 2 to compile them
Note that tools like SLC have been around for awhile that will automate steps 2-4 so that one just needs to deploy the Node.js runtime and SLC itself at the target.
2 A Node Application is a different world
The need for this experience is even more acute on mobile platforms where typically it isn’t feasible to download much of anything and even if it is one certainly doesn’t want to depend on more than just a single download from the app store since connectivity can be iffy and expensive. And one certainly wants to avoid compiling anything directly on the mobile device if possible.
3 The players
These are the projects I’m currently aware of in the Node application space that at least rate as mostly (but not completely) dead or above. If I missed anything relevant please drop a comment.
NW.js This is probably the oldest of all the projects listed here. Started by Intel it is open source and provides a development environment that uses Chromium for the front end UX and Node.js v5 for the back end. NW.js also has some nifty tricks that make it easy to call from the DOM in the WebView to Node.js and since both are running V8 they can pass objects directly back and forth. Interestingly enough both Chromium and Node.js run on the same thread which could be an issue in terms of responsiveness but presumably one can use the Node.js process API to spawn other instances.
Electron This was created by github, is open source and has the same basic idea as NW.js, use Chromium for UX and Node.js for service logic. Where as NW.js starts in the browser and calls out to Node.js, Electron starts in Node.js and lets it call out to the browser. Note that Electron also has its own custom version of Node.js but it’s not clear to me how much that actually matters.
JXcore-Cordova This is an open source wrapper around JXcore that turns it into a Cordova plugin that can run on Android and iOS. It also has a very nice extension layer that lets one write Node.js extensions in Objective-C on iOS and Java on Android.
Node-ChakraCore This is an open source project from Microsoft that has built a shim layer between Node.js and V8. This shim layer redirects Node.js requests for V8 to ChakaCore instead. I mention it mostly because of its relevance to efforts like SpiderNode.
Positron This open source project attempts to be API compatible with Electron but is from Mozilla and so uses Mozilla’s tech stack so they use Gecko instead of Chromium and SpiderNode instead of Node.
SpiderNode This open source project is actually based on Microsoft’s Node-ChakraCore. Mozilla is taking Node-ChakraCore’s shim and pointing it at SpiderMonkey instead of ChakraCore. This is still a very new effort and they aren’t even close to up and running at the time I typed this. What I think makes it particularly compelling is that SpiderMonkey already runs on iOS and Android (as well as just about everywhere else). So if the project is successful it could run Node.js just about everywhere.
Termux is an open source project that runs a Linux terminal on Android. It supports its own APT-GET repository that contains node 6.10.2, Python, etc. In theory one could take the Termux code and embed it in an application running as a service and then manage the node instance. This doesn’t solve a bunch of the really hard problems like how you manage that node instance (there is no standard api), how you keep from repeating the (large) node modules both in the APK and on disk. It still requires connecting it to life cycle events. Etc. Basically all the work below. But hey, it at least runs mainline node on Android.
For the rest of the article I’m going to focus on projects that I know slightly better, NW.js, Electron, JXcore and JXcore-Cordova.
4 How do we put Node into an application? The embedding problem
Most applications are user focused and contain some kind of UX. Node has nothing to do with displaying a UX. So any time one wants to use Node in an application one immediately faces the issue that Node has to be used along side something else, something that can display things to the user (although I have seen people develop interactive character based GUIs using Node’s stdout support from a command line, how’s that for retro?).
So a Node application generally consists of three parts. The first part is some kind of loader that starts the application. It is the loader that then starts the UX (whatever it is) and initializes it as well as starting the Node.js instance and telling it what to do first (e.g. where the app.js is, where node_modules are, etc.).
To make this work one effectively needs to embed Node.js into one’s application so that the loader can do its thing. In the simplest case, say on a desktop OS, one could actually ship the Node.js executable for the platform and then have the loader call out to the shell and just start the standard Node.js executable. But it’s typically more useful to have a dedicated embedding API for Node.js so that the loader can manage things. How embedding gets dealt with depends largely on how one is building one’s Node.js application.
If the goal is to build one’s own application with its own UX and one just wants to add in Node.js then the only solution I know about today is JXcore whose embedding API runs on Linux, OS/X and Windows on the desktop, and Android and iOS on mobile.
For those that want to use WebViews as their UX there are several additional options including NW.js and Electron on desktop OS’s and JXcore-Cordova for Android and iOS. The advantage of these frameworks is that one can define one’s UX using HTML/JS/CSS and one’s service logic using Node and then have the results packaged up into stand alone executables that can run on the supported platforms. It’s a more “batteries included” approach but at the cost of requiring the use of a WebView for the UX.
5 How do we build Node packages to be used with the app? Especially native packages?
So what we are discussing is the exact inverse of the previous section. In the previous section we discussed how one builds a (presumably native) application that hosts Node. Now we are discussing how one extends Node so that it can, in effect, host native code.
Node has a standard solution for embedding native code called Node-Gyp.
Gyp is a build environment that uses a single build file format that Gyp can translate into platform specific build files such as Visual Studio projects, XCode projects, make files, etc. Node-gyp builds on Gyp to add node.js specific APIs that enable two-way communication between node and the native add-on.
The problem with node-gyp is that it generally just provides a way to talk to V8 and since V8 is constantly in flux the native code tends to become dependent on a particular version of V8. As Node.js updates its V8 engine this then breaks older native code.
The usual solution to this is something called Native Abstractions for Node (nan). This is a set of C macros that abstract away a lot of the V8 details so that one can write Node add-ons that won’t break every time Node.js changes the version of V8 it is using.
I find it interesting to note that Node-ChakraCore also supports nan. This means that if one writes a native extension using nan that it can run on either V8 or ChakraCore, which is pretty nifty.
In the case of Android where at least in theory one can run Node.js and V8 it should be possible, modulo the cross-compile problem mentioned next, to use nan or Node-Gyp modules without issue. And the same is true anywhere that ChakraCore runs. But what about say iOS where neither V8 nor ChakraCore runs?
Today the only solution I’m aware of is JXcore-Cordova. JXcore-Cordova introduces its own native extension API that makes it easy for the Node.js code to make both synchronous and asynchronous function calls to native code. It also supports allowing native code to push asynchronous events into Node.js. What is also nifty about it is that unlike Node-Gyp/nan which are C based, JXcore-Cordova uses Objective-C in iOS and Java in Android. Although to be fair one can also bridge to those languages from C using Node-Gyp/nan.
If one wanted to write an Android or iOS application that wasn’t based on Cordova but could use JXcore’s extension capabilities then one would only have to look at the hooks JXcore-Cordova uses to call the extension files and then replicate them in one’s own app.
6 How do we cross-compile native modules?
In theory this is all doable on desktop if supported by the build tools/compiler. For example, in theory one could build native Node.js packages on Linux for Windows if one configures the GCC environment variables correctly. But note that neither Node-gyp nor nan support cross compiling natively. Rather one has to go under them and tell the underlying compile framework what one is up to.
To make things even more interesting while GYP does support Android, it doesn't support the Android-NDK which is what one would want to use since nan is a C based API.
GYP does support iOS via XCode but that is less useful than it seems because what VM would you target the iOS build at? I know that question might seem strange so let’s take it apart.
And going back to Android, we don’t even have Node-Gyp support there, by design, so what do we do with native modules on Android?
The only Node.js environment I know that has addressed the cross-compile issues with Android and iOS is the previously mentioned JXcore-Cordova. But it doesn’t use Node-Gyp/nan to make this work. What JXcore-Cordova does is that it defines an Object-C file in iOS and a Java file in Android that one can overwrite to put in declarations of one’s own native code. This has absolutely nothing to do with Node-Gyp/nan.
So the bottom line is that if one wants to handle cross-compiling native Modules on Android and iOS then using JXcore-Cordova’s native (non Node-Gyp/nan) extension mechanism is probably the best bet currently.
7 How do we package our Node.js application?
For the “Batteries Included” approach of NW.js, Electron and JXcore-Cordova the answer is straight forward, use each platform’s build tool to create a final application. In the case of NW.js and Electron this mostly means zipping up a bunch of node_modules and copying them back to disk when the app is installed. If one is using JXcore (but not JXcore-Cordova) to embed just a pure Node.js engine then I believe one has to handle managing the node_modules oneself but that really shouldn’t be more than just zipping things up and unzipping in the right place. Unless one is on iOS or Android, there JXcore had to get fancy and create a virtual file system to get things to work in a sane way.
A naive way of handling this problem is that the first time the app is run it can read in the node_modules directory from the app’s ZIP file (since both APK and IPA are just zip files) and copy them to disk. But this isn’t great because it makes the first time start up of the app really slow (it takes times to unzip and copy all those files) and it wastes storage space as the same content is now available in both a zipped and unzipped form on the same device.
JXcore solved this problem by creating a virtual file system which makes the APK/IPA zip file look to Node.js like a file system. That way JXcore can run its node engine against the zip file directly. Of course nothing is free. A naive node package usually assumes that it can write in its own directory. That won’t work here since that “directory” is really just a virtual directory on top of a zip file and writing is obviously not supported. But in practice this hasn’t proven to be a huge deal.
So you can get there from here. On desktop and mobile, especially if you like to use WebViews, there are ways to build apps that embed Node.js. The demise of JXcore, which did all of this particularly well, is deeply unfortunate but I do see hope in the long run. For those who just want to embed Node.js directly (without using a WebView) things are a bit more complex but not unworkable. JXcore is probably still the best bet even there but hopefully not for too long.