emersonthis emersonthis - 6 months ago 26
Node.js Question

How to run binary on Heroku

I have a simple node.js app that runs great on my OSX dev environment. But when I push the same code to Heroku, I get

undefined
returned in unexpected places.

The library that is behaving unpredictably is called
pandoc
. It converts document formats. I integrating it into my node app using the wrapper package
node-pdc
. Pdc requires pandoc to be installed, but it accepts a path to specify the location of pandoc, which I'm doing locally and it works great. I uninstalled pandoc globally on my local machine to make sure I was pointing to the correct location of the executable file.

This works exactly as expected locally:

var pdc = require('pdc');
var path = require('path');
var fs = require('fs');
var Q = require('q');

// optional, if pandoc is not in PATH
pdc.path = path.resolve(__dirname +'/pandoc/1.15.0.6/bin/pandoc');

module.exports.mdToHtml = function (input, callback) {

//TEST
pdc('## Emerson', 'markdown', 'html', ['--template=smashingtemplate'], function(err, result){
console.log(result); // <h2>Emerson</h2>
});
...


But on Heroku, I see
undefined
in the logs. It does not throw any errors. Why does it work on my laptop but not on Heroku?

Both environments are running node v4.0.0

Update 1:
I tested to see if pandoc was available on Heroku...

$ heroku run bash

~ $ ls /app/pandoc/1.15.0.6/bin/
pandoc
~ $ /app/pandoc/1.15.0.6/bin/pandoc --version
bash: /app/pandoc/1.15.0.6/bin/pandoc: cannot execute binary file: Exec format error
~ $ ls /app/pandoc/1.15.0.6/bin/ -lah
total 73M
drwx------ 2 u35911 dyno 4.0K Jun 5 04:48 .
drwx------ 4 u35911 dyno 4.0K Jun 5 04:48 ..
-rwx------ 1 u35911 dyno 73M Jun 5 04:48 pandoc


The binary files are there. But I can't run them...

Update 2

~ $ objdump -i /app/pandoc/1.15.0.6/bin/pandoc
BFD header file version (GNU Binutils for Ubuntu) 2.24
elf64-x86-64
(header little endian, data little endian)
i386
elf32-i386
(header little endian, data little endian)
i386
elf32-x86-64
(header little endian, data little endian)
i386
a.out-i386-linux
(header little endian, data little endian)
i386
pei-i386
(header little endian, data little endian)
i386
pei-x86-64
(header little endian, data little endian)
i386
elf64-l1om
(header little endian, data little endian)
l1om
elf64-k1om
(header little endian, data little endian)
k1om
elf64-little
(header little endian, data little endian)
i386
l1om
k1om
plugin
elf64-big
(header big endian, data big endian)
i386
l1om
k1om
plugin
elf32-little
(header little endian, data little endian)
i386
l1om
k1om
plugin
elf32-big
(header big endian, data big endian)
i386
l1om
k1om
plugin
pe-x86-64
(header little endian, data little endian)
i386
pe-i386
(header little endian, data little endian)
i386
plugin
(header little endian, data little endian)
srec
(header endianness unknown, data endianness unknown)
i386
l1om
k1om
plugin
symbolsrec
(header endianness unknown, data endianness unknown)
i386
l1om
k1om
plugin
verilog
(header endianness unknown, data endianness unknown)
i386
l1om
k1om
plugin
tekhex
(header endianness unknown, data endianness unknown)
i386
l1om
k1om
plugin
binary
(header endianness unknown, data endianness unknown)
i386
l1om
k1om
plugin
ihex
(header endianness unknown, data endianness unknown)
i386
l1om
k1om
plugin

elf64-x86-64 elf32-i386 elf32-x86-64 a.out-i386-linux pei-i386 pei-x86-64 elf64-l1om elf64-k1om elf64-little
i386 elf64-x86-64 elf32-i386 elf32-x86-64 a.out-i386-linux pei-i386 pei-x86-64 ---------- ---------- elf64-little
l1om ------------ ---------- ------------ ---------------- -------- ---------- elf64-l1om ---------- elf64-little
k1om ------------ ---------- ------------ ---------------- -------- ---------- ---------- elf64-k1om elf64-little
plugin ------------ ---------- ------------ ---------------- -------- ---------- ---------- ---------- elf64-little

elf64-big elf32-little elf32-big pe-x86-64 pe-i386 plugin srec symbolsrec verilog tekhex binary ihex
i386 elf64-big elf32-little elf32-big pe-x86-64 pe-i386 ------ srec symbolsrec verilog tekhex binary ihex
l1om elf64-big elf32-little elf32-big --------- ------- ------ srec symbolsrec verilog tekhex binary ihex
k1om elf64-big elf32-little elf32-big --------- ------- ------ srec symbolsrec verilog tekhex binary ihex
plugin elf64-big elf32-little elf32-big --------- ------- ------ srec symbolsrec verilog tekhex binary ihex


And...

~ $ uname -a
Linux 7f4e1d74-8fc2-47d2-89b7-97bfc9db30dd 3.13.0-85-generic #129-Ubuntu SMP Thu Mar 17 20:50:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux


I might not be understanding it correctly, but it looks like they're both for 64 architecture.

~ $ file /app/pandoc/1.15.0.6/bin/pandoc
/app/pandoc/1.15.0.6/bin/pandoc: Mach-O 64-bit x86_64 executable


Update 3

Following @mbs1's suggestions...

Creating a relocatable binary
-----------------------------

It is possible to compile pandoc such that the data files
pandoc uses are embedded in the binary. The resulting binary
can be run from any directory and is completely self-contained.

cabal install hsb2hs # a required build tool
cabal install --flags="embed_data_files" citeproc-hs
cabal configure --flags="embed_data_files"
cabal build

You can find the pandoc executable in `dist/build/pandoc`. Copy this wherever
you please.


When I do
cabal build
it says:

cabal: No cabal file found.
Please create a package description file <pkgname>.cabal


Solution

I couldn't get it to work with any of the solutions requiring
cabal
. I did finally get it to work using
stack
(once I figured out how to get it installed).

$ stack install pandoc --flag pandoc:embed-data_files


Then I copied the generated exec file out of the vm and pushed it to Heroku.

Answer

You need to build a relocatable-binary of pandoc on the exact same OS as Heroku uses, so an Ubuntu Linux system (install it in a virtual machine if you're on Mac OS X) and then follow this or this. Basically the following should create a re-locatable binary:

stack install pandoc --flag pandoc:embed_data_files

Alternatively, you could also use Docverter, which is kind of pandoc as a service.