Stuff Yaron Finds Interesting

Technology, Politics, Food, Finance, etc.

Looking at JXCore’s perf

Asking about a system’s performance is usually an invitation to a fight, not to useful data. But I did want some idea of how JXCore, a fork of node.js that runs on mobile OS’s, performed on iOS and Android. See here for more info on JXCore. Since the work I’m doing heavily involves PouchDB I decided to take the PouchDB performance tests and run them in six different environments and compare the results. Over all, one should expect node.js on a modern phone (Android or iOS) to be around 10x slower than on a PC. Which honestly, isn’t that bad when you think about it. I want to give a shout out to Brian Lambert for making the iOS tests happen and to Oguz Bastemur from Nubisa for helping us past various bugs. Oguz was very patient with Brian and I and we really appreciate it.

1 The machines

For the tests Brian and I used the following machines:
  • A Lenovo T440P laptop. This is an Intel core i7-4900MQ @ 2.80 Ghz. It has 4 physical cores and 8 virtual cores thanks to hyper threading. It also has 16 Gigs of RAM.
  • A Samsung Galaxy S4 running Android 4.3 with Quad-core 1.6 Ghz Cortex-A15 and 2 Gigs of RAM.
  • An iPhone 5 running iOS 8 with a Dual Core 1.3 Ghz Swift (ARM v7) processor and 1 Gig of RAM.
  • An iPhone 6 plus running iOS 8 with a Dual-core 1.4 GHz Cyclone (ARM v8) processor and 1 Gig of RAM.

2 The test setup

The test was to run the PouchDB perf tests using an in memory database. I eventually will want to run these tests against LevelDB but for now I didn’t want to deal with all the hassle of getting LevelDB to work against LevelDown as a JXCore native add-on. So instead I used memDown. This means that the tests are really just testing the Javascript engine and libuv, e.g. the stuff node.js does. It’s not testing the IO bus to local storage. We’ll get that covered when we put in LevelDB support.

2.1 Running the test on Node.js on a PC

  1. Run “npm install express pouchdb memdown express-pouchdb”
  2. CD’d into node_modules/pouchdb and ran “npm install” (this puts in the dev dependencies)
  3. Went inside of pouchdb/tests/performance/index.js and replaced
var PouchDB = process.browser ? window.PouchDB : require(’../..’); 
with
var PouchDB = require(’../..’).defaults({db: require("memdown")}); 
I then created the following as a node.js file and run it with node:
var express = require("express"); 
var pouchDBApp = express(); 
var PouchDB = require("pouchdb"); 
var InMemPouchDB = PouchDB.defaults({db: require("memdown")});

// Set up our test server 
pouchDBApp.use("/", require("express-pouchdb")(InMemPouchDB, {mode: "minimumForPouchDB"})); 
// Node tests by default will look at http://localhost:5984 
pouchDBApp.listen(5984);

require(’pouchdb/tests/performance/index.js’); 

2.2 Running the test on Android

I actually ran the test against two different node.js for Android code bases. One is just plain old node.js and v8 run along the lines described here. The other is JXCore. You can get the JXCore project here. Make sure to read the readme. I actually ran the JXCore project twice. Once using SpiderMonkey and once using V8.
Note that a bug in JXCore causes the Android perf numbers to be worse than what they will eventually be. The issue is that JXCore can run in two modes on Android. In one mode JXCore can run on the main thread but you have to keep calling it to run a loop. In the other mode JXCore just spawns a thread and lets the node.js code run to its heart’s content in that thread. The problem is that right now there is a bug that keeps the “thread” mode from properly seeing Node.js packages and so require() doesn’t work right. So for this test we just hammer the “loop” API and this adds overhead. Once the require bug is fixed we should see even better perf.

2.3 Running the test on iOS

You can see Brian’s code here. A similar bug to the require() bug described in the Android section also applied to iOS. But thankfully there was a quick and easy work around and so the perf one sees here is an honest picture of what one should expect on an iOS device. Also note that we can only run SpiderMonkey on iOS (no v8 support) and that when running SpiderMonkey on iOS we can’t use IonMonkey, the Firefox JIT’er. So in general the performance on iOS should not be as good as an equivalent Android device.

3 The test results

I provide the raw results in a table below but honestly I don’t think the actual timings are all that interesting. To me what’s more interesting is the relative performance of node.js on mobile versus the PC. Below is a totally unscientific average of the relative performance of the various PouchDB tests versus PC.
Platform Relative Average Performance to PC
Node.js/V8 on Android 12x
JXCore/SpiderMonkey on Android 31x
JXCore/V8 on Android 14x
JXCore/SpiderMonkey on iPhone 5 26x
JXCore/SpiderMonkey on iPhone 6 Plus 13x
Take these results with a very large grain of salt but they give a sense for how the perf looks. JXCore, with v8, on Android does a reasonable job of keeping up with plain ole v8 on Android. And JXCore’s numbers should get better once the require bug is fixed. But in either case running node.js on your Android phone is likely to be about 10x slower than a PC. Although to be fair I’m using a pretty old Android handset. I’d love to see the numbers on a Samsung Galaxy S5 or even better the upcoming S6. That would be a fairer comparison to the iPhone 6 Plus.
iPhone 5 is going to be really painful. SpiderMonkey without a JIT’er isn’t a pretty sight. So we should be looking for a 30x slowdown and I’m trying not to even think about what an iPhone 4 will look like. The iPhone 6, again without JIT, is on par with a two generation old Android handset. Given the JIT limitation that’s actually pretty impressive to me. So the future is bright for node.js on iOS but the path to get there could be ugly.
Here are the raw measurements. All numbers are in ms.
Test name Node.js/V8 on PC Node.js/V8 on Android JXCore/SpiderMonkey on Android JXCore/V8 on Android JXCore/SpiderMonkey on iOS (iPhone 5) JXCore/SpiderMonkey on iOS (iPhone 6 Plus)
basic-inserts 314 3167 6091 3572 4888 2776
bulk-inserts 858 12241 24427 11595 22396 10637
basic-gets 439 6914 13833 6487 13581 6869
all-docs-skip-limit 2449 40674 83278 47646 101229 46247
all-docs-startkey-endkey 157 1940 5845 2293 5573 2529
pull-replication-one-generation 459 6845 23896 7901 14022 6656
pull-replication-two-generation 1484 19551 45959 22277 30288 15322
temp-views 1360 16498 36803 18215 31513 14968
persisted-views 33 376 1405 443 1039 477
persisted-views-stale-ok 28 271 823 294 695 328
basic-attachments 1161 8124 16022 9381 12246 6285

4 Responses to Looking at JXCore’s perf

  1. Pingback: Thali Story 0 – The lights are officially on! |

Leave a Reply

Your email address will not be published. Required fields are marked *