Bumping package.json version without invalidating docker cache
Here's my take on this, based off other answers, but shorter and with usage of jq
:
Dockerfile:
FROM endeveit/docker-jq AS deps
# https://stackoverflow.com/a/58487433
# To prevent cache invalidation from changes in fields other than dependencies
COPY package.json /tmp
RUN jq '{ dependencies, devDependencies }' < /tmp/package.json > /tmp/deps.json
FROM node:12-alpine
WORKDIR /app
COPY --from=deps /tmp/deps.json ./package.json
COPY package-lock.json .
RUN npm ci
# https://docs.npmjs.com/cli/ci.html#description
COPY . .
RUN npm run build
LABEL maintainer="Alexey Vishnyakov <[email protected]>"
I extract dependencies
and devDependencies
fields to a separate file, then on next build step I copy it from the previous step as package.json
(COPY --from=deps /tmp/deps.json ./package.json
).
After RUN npm ci
, COPY . .
will overwrite gutted package.json
with the original one (you can test it by adding RUN cat package.json
after COPY . .
command.
Note that npm-scripts commands like postinstall
won't run since they're not present in the file during npm ci
and also if npm ci
is running from root
and without --unsafe-perm
Either run commands after COPY . .
or/and (if needed) include them via jq
(changing command will invalidate cache layer) or add --unsafe-perm
Dockerfile:
FROM endeveit/docker-jq AS deps
COPY package.json /tmp
RUN jq '{ dependencies, devDependencies, peerDependencies, scripts: (.scripts | { postinstall }) }' < /tmp/package.json > /tmp/deps.json
# keep postinstall script
FROM node:12-alpine
WORKDIR /app
COPY --from=deps /tmp/deps.json ./package.json
COPY package-lock.json .
# RUN npm ci --unsafe-perm
# allow postinstall to run from root (security risk)
RUN npm ci
# https://docs.npmjs.com/cli/ci.html#description
RUN npm run postinstall
...
I spent some time thinking about this. Fundamentally, I'm cheating because the package.json
file is, in fact, changed, which means anything that circumvents the cache invalidation technically makes the build not reproducible.
For my purposes, however, I care more about build time than strict cache correctness. Here's what I came up with:
build-artifacts.js
/*
Used to keep docker cache fresh despite package.json version bumps.
In this script
- copy package.json to package-artifact.json
- zero package.json version
In Docker
- copy package.json
- run npm install normal
- copy package-artifact.json to package.json (undo-build-artifacts.js accomplishes this with a conditional check that package-artifact exists)
*/
const fs = require('fs');
const package = fs.readFileSync('package.json', 'utf8');
fs.writeFileSync('package-artifact.json', package);
const modifiedPackage = { ...JSON.parse(package), version: '0.0.0' };
fs.writeFileSync('package.json', JSON.stringify(modifiedPackage));
const packageLock = fs.readFileSync('package-lock.json', 'utf8');
fs.writeFileSync('package-lock-artifact.json', packageLock);
const modifiedPackageLock = { ...JSON.parse(packageLock), version: '0.0.0' };
fs.writeFileSync('package-lock.json', JSON.stringify(modifiedPackageLock));
undo-build-artifacts.js
const fs = require('fs');
const hasBuildArtifacts = fs.existsSync('package-artifact.json');
if (hasBuildArtifacts) {
const package = fs.readFileSync('package-artifact.json', 'utf8');
const packageLock = fs.readFileSync('package-lock-artifact.json', 'utf8');
fs.writeFileSync('package.json', package);
fs.writeFileSync('package-lock.json', packageLock);
fs.unlinkSync('package-artifact.json');
fs.unlinkSync('package-lock-artifact.json');
}
These two files serve to relocate package.json
and package-lock.json
, replacing them with artifacts that have zeroed-out versions. These artifacts will be used in the docker build, and will be replaced with the original versions upon npm install
completion.
I run build-artifacts.js
in a Travis CI before_script
, and undo-build-artifacts.js
in the Dockerfile itself (after I npm install
). undo-build-artifacts.js
incorporates a check for the build artifacts, meaning the Docker container can still build if build-artifacts.js
hasn't run. That keeps the container portable enough in my books. :)
You can add an additional "preparation" step in your Dockerfile that creates a temporary package.json
where the "version"
field is fixed. This file is then used while installing dependencies and afterwards replaced by the "real" package.json
.
As all of this happens during the Docker build process, your actual source repository is not touched (so you can use the environment variable npm_package_version
both during your build and when running the docker script, e.g. to tag) and the solution is portable:
Dockerfile:
# PREPARATION
FROM node:lts-alpine as preparation
COPY package.json package-lock.json ./
# Create temporary package.json where version is set to 0.0.0
# – this way the cache of the build step won't be invalidated
# if only the version changed.
RUN ["node", "-e", "\
const pkg = JSON.parse(fs.readFileSync('package.json', 'utf-8'));\
const pkgLock = JSON.parse(fs.readFileSync('package-lock.json', 'utf-8'));\
fs.writeFileSync('package.json', JSON.stringify({ ...pkg, version: '0.0.0' }));\
fs.writeFileSync('package-lock.json', JSON.stringify({ ...pkgLock, version: '0.0.0' }));\
"]
# BUILD
FROM node:lts-alpine as build
# Install deps, using temporary package.json from preparation step
COPY --from=preparation package.json package-lock.json ./
RUN npm ci
# Copy source files (including "real" package.json) and build app
COPY . .
RUN npm run build
If you think inlining the Node script is iffy (I like it, because this way the entire Docker build process can be found in the Dockerfile), you can of course extract it to a separate JS file:
create-tmp-pkg.js:
const fs = require('fs');
const pkg = JSON.parse(fs.readFileSync('package.json', 'utf-8'));
const pkgLock = JSON.parse(fs.readFileSync('package-lock.json', 'utf-8'));
fs.writeFileSync('package.json', JSON.stringify({ ...pkg, version: '0.0.0' }));
fs.writeFileSync('package-lock.json', JSON.stringify({ ...pkgLock, version: '0.0.0' }));
and change your preparation step to:
# PREPARATION
FROM node:lts-alpine as preparation
COPY package.json package-lock.json create-tmp-pkg.js ./
# Create temporary package.json where version is set to "0.0.0"
# – this way the cache of the build step won't be invalidated
# if only the version changed.
RUN node create-tmp-pkg.js