Martin Martin - 1 month ago 15
Node.js Question

CircleCI not caching my globally installed node module?

I am trying to cache a command line tool needed for my build process. The tool is made out of NodeJS. The build succeeds, but I need it to run faster.

The relevant parts of my circle.yml look like this :

dependencies:
post:
- npm -g list
- if [ $(npm -g list | grep -c starrynight) -lt 1 ]; then npm install -g starrynight; else echo "StarryNight seems to be cached"; fi

test:
override:
- npm -g list
- starrynight run-tests --framework nightwatch


The second
npm -g list
shows starrynight available for use, but the first one shows that it is not being cached.

echo $(npm prefix -g)


. . . gets me . . .

/home/ubuntu/nvm/v0.10.33


. . . so I am assuming CircleCI doesn't cache anything installed globally into nvm.

Nothing I have tried gets me my message, "StarryNight seems to be cached".

How can I cache starrynight?

Answer

Ok, I figured this out. Thanks to Hirokuni Kim of CircleCI for pointing me in the right direction.

The relevant bits of the new circle.yml looks like this :

machine:
  node:
    version: 0.10.33

dependencies:
  cache_directories:
    - ~/nvm/v0.10.33/lib/node_modules/starrynight
    - ~/nvm/v0.10.33/bin/starrynight
  pre:
    - if [ ! -e ~/nvm/v0.10.33/bin/starrynight ]; then npm install -g starrynight; else echo "Starrynight seems to be cached"; fi;

Hirokuni suggested caching ~/nvm but cache retrieval took as long as the build, since it restores every available version of nodejs.

I had tried previously to cache just ~/nvm/v0.10.33/lib/node_modules/starrynight on its own, without realizing that the sister 'directory' bin/starrynight is actually an essential symlink to the entry point of the module.

My working assumption is that NodeJS modules run from the command line through a series of symbolic references, probably as follows. . .

npm install -g starrynight creates two new artifacts:

  • an environment alias for npm named starrynight
  • a symlink in the ${prefix}/bin directory, which points to the entry point file, starrynight.js specified with the bin key in package.json.

When the user types starrynight as a CLI command the shell interprets it as an alias for npm and executes it. npm examines $0, gets starrynight, and starts up nodejs with the symlink ${prefix}/bin/starrynight as the module to execute. That symlink refers to ~/nvm/v0.10.33/lib/node_modules/starrynight where the real action takes place.

In short, it is necessary to cache both ${prefix}/lib/node_modules/xxx and ${prefix}/bin/xxx