subpackage installation does not always inherit credentials for the registry

[sonofhammer]

Symptom
When using private npm registry, on win10, x64 environment the node-win-x64 npm package will fail to install with a 401 error.

Environment Specifics
This was initially discovered in a CI environment using a Private Build agent in Azure DevOps Pipelines, but can be reproduced outside of the CI environment.
This was tried and tested across several Node and npm versions.

Root Cause
The npm install that is executed inside installArchSpecificPackage fails because it does not use the registry credentials stored in the project's .npmrc file

To Reproduce

1. Create a private registry of npm packages (I used Azure Artifacts) (and populate it with the appropriate packages)
2. Create a directory for your project, in it, configure the .npmrc to point to the private registry.
3. Do not configure your private registry credentials in the .npmrc file inside the users home directory. or any other three .npmrc files
4. Instead write out the credentials into the .npmrc file inside the project. (this is was Azure Pipelines does it on the Build agent. See NPM Authenticate and here for more info.)
5. run npm install node
6. observe authentication error for node-win-x64

Workarounds

1. Populate the .npmrc in the home directory with the credentials.
2. Poplulate the NPM_CONFIG_USERCONFIG environment variable with path of .npmrc that has the credentials.

Both of these are undesirable because the project's package.json and .npmrc should be all that is needed to install all of the needed npm packages. from either public or private npm registries.

Verbose log output below

PS C:\src> npm install node --verbose
npm info it worked if it ends with ok
npm verb cli [ 'C:\\Program Files\\nodejs\\node.exe',
npm verb cli   'C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js',
npm verb cli   'install',
npm verb cli   'node',
npm verb cli   '--verbose' ]
npm info using [email protected]
npm info using [email protected]
npm verb npm-session 8bb482269c29a83c
npm http fetch GET 200 https://pkgs.dev.azure.com/myproject_packaging/MyArtifacts/npm/registry/node 468ms
npm http fetch GET 200 https://pkgs.dev.azure.com/myproject_packaging/MyArtifacts/npm/registry/node/-/node-10.15.2.tgz 608ms
npm timing stage:loadCurrentTree Completed in 1903ms
npm timing stage:loadIdealTree:cloneCurrentTree Completed in 1ms
npm timing stage:loadIdealTree:loadShrinkwrap Completed in 14ms
npm http fetch GET 200 https://pkgs.dev.azure.com/myproject_packaging/MyArtifacts/npm/registry/node-bin-setup 222ms
npm timing stage:loadIdealTree:loadAllDepsIntoIdealTree Completed in 245ms
npm timing stage:loadIdealTree Completed in 268ms
npm timing stage:generateActionsToTake Completed in 11ms
npm verb correctMkdir C:\Users\MyUser\AppData\Roaming\npm-cache\_locks correctMkdir not in flight; initializing
npm timing audit compress Completed in 10ms
npm info audit Submitting payload of 247bytes
npm verb lock using C:\Users\MyUser\AppData\Roaming\npm-cache\_locks\staging-382712a43da04e93.lock for C:\src\node_modules\.staging
npm timing action:extract Completed in 75ms
npm http fetch POST 400 https://pkgs.dev.azure.com/myproject_packaging/MyArtifacts/npm/registry/-/npm/v1/security/audits/quick 297ms
npm timing action:finalize Completed in 44ms
npm timing action:refresh-package-json Completed in 55ms
npm info lifecycle [email protected]~preinstall: [email protected]
npm info lifecycle [email protected]~preinstall: [email protected]

> [email protected] preinstall C:\src\node_modules\node
> node installArchSpecificPackage

npm info it worked if it ends with ok
npm verb cli [ 'C:\\Program Files\\nodejs\\node.exe',
npm verb cli   'C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js',
npm verb cli   'prefix',
npm verb cli   '-g' ]
npm info using [email protected]
npm info using [email protected]
npm verb exit [ 0, true ]
npm timing npm Completed in 816ms
npm info ok
npm info it worked if it ends with ok
npm verb cli [ 'C:\\Program Files\\nodejs\\node.exe',
npm verb cli   'C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js',
npm verb cli   'install',
npm verb cli   '--no-save',
npm verb cli   '[email protected]' ]
npm info using [email protected]
npm info using [email protected]
npm verb npm-session 8b26d976588dc0f8
npm http fetch GET 401 https://pkgs.dev.azure.com/myproject_packaging/MyArtifacts/npm/registry/node-win-x64 220ms
npm timing stage:rollbackFailedOptional Completed in 1ms
npm timing stage:runTopLevelLifecycles Completed in 938ms
npm verb stack Error: 401 Unauthorized: [email protected]
npm verb stack     at fetch.then.res (C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\lib\fetchers\registry\fetch.js:42:19)
npm verb stack     at tryCatcher (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\util.js:16:23)
npm verb stack     at Promise._settlePromiseFromHandler (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\promise.js:512:31)
npm verb stack     at Promise._settlePromise (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\promise.js:569:18)
npm verb stack     at Promise._settlePromise0 (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\promise.js:614:10)
npm verb stack     at Promise._settlePromises (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\promise.js:693:18)
npm verb stack     at Async._drainQueue (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\async.js:133:16)
npm verb stack     at Async._drainQueues (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\async.js:143:10)
npm verb stack     at Immediate.Async.drainQueues [as _onImmediate] (C:\Program Files\nodejs\node_modules\npm\node_modules\bluebird\js\release\async.js:17:14)
npm verb stack     at runCallback (timers.js:705:18)
npm verb stack     at tryOnImmediate (timers.js:676:5)
npm verb stack     at processImmediate (timers.js:658:5)
npm verb cwd C:\src\node_modules\node
npm verb Windows_NT 10.0.17763
npm verb argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "--no-save" "[email protected]"
npm verb node v10.15.2
npm verb npm  v6.4.1
npm ERR! code E401
npm ERR! 401 Unauthorized: [email protected]
npm verb exit [ 1, true ]
npm timing npm Completed in 2782ms

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\MyUser\AppData\Roaming\npm-cache\_logs\2019-03-05T19_53_50_385Z-debug.log
internal/modules/cjs/loader.js:583
    throw err;
    ^

Error: Cannot find module 'node-win-x64/package.json'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
    at Function.resolve (internal/modules/cjs/helpers.js:32:19)
    at ChildProcess.<anonymous> (C:\src\node_modules\node-bin-setup\index.js:18:27)
    at ChildProcess.emit (events.js:189:13)
    at maybeClose (internal/child_process.js:970:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
npm verb lifecycle [email protected]~preinstall: unsafe-perm in lifecycle true
npm verb lifecycle [email protected]~preinstall: PATH: [REDACTED]
npm verb lifecycle [email protected]~preinstall: CWD: C:\src\node_modules\node
npm info lifecycle [email protected]~preinstall: Failed to exec preinstall script
npm timing action:preinstall Completed in 6118ms
npm verb unlock done using C:\Users\MyUser\AppData\Roaming\npm-cache\_locks\staging-382712a43da04e93.lock for C:\src\node_modules\.staging
npm WARN rollback Rolling back [email protected] failed (this is probably harmless): EPERM: operation not permitted, rmdir 'C:\src\node_modules'
npm timing stage:rollbackFailedOptional Completed in 21ms
npm timing stage:runTopLevelLifecycles Completed in 8791ms
npm WARN enoent ENOENT: no such file or directory, open 'C:\src\package.json'
npm verb enoent This is related to npm not being able to find a file.
npm verb enoent
npm WARN src No description
npm WARN src No repository field.
npm WARN src No README data
npm WARN src No license field.

npm verb stack Error: [email protected] preinstall: `node installArchSpecificPackage`
npm verb stack Exit status 1
npm verb stack     at EventEmitter.<anonymous> (C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\index.js:301:16)
npm verb stack     at EventEmitter.emit (events.js:189:13)
npm verb stack     at ChildProcess.<anonymous> (C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\lib\spawn.js:55:14)
npm verb stack     at ChildProcess.emit (events.js:189:13)
npm verb stack     at maybeClose (internal/child_process.js:970:16)
npm verb stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
npm verb pkgid [email protected]
npm verb cwd C:\src
npm verb Windows_NT 10.0.17763
npm verb argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "node" "--verbose"
npm verb node v10.15.2
npm verb npm  v6.4.1
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] preinstall: `node installArchSpecificPackage`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] preinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm verb exit [ 1, true ]
npm timing npm Completed in 10768ms

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\MyUser\AppData\Roaming\npm-cache\_logs\2019-03-05T19_53_50_761Z-debug.log

[VadneyK]

checkout solution to related issues here: #124

[VadneyK]

We really need this issue resolved: #126

[aredridel]

Does this still happen with current npm and node?

[Gerhut]

Yes

[aredridel]

Oh this is going to be interesting to fix. I need to figure out what normally injects those credentials and make sure the install script understands them or finds them.

[Gerhut]

My finding is that when running preinstall, the CWD is ./node_modules/node, but not ., so ./.npmrc does not provide credentials.

Edit: the fix is either making the preinstall running in . or link ./.npmrc to ./node_modules/node/.npmrc. I tried the former and failed because it's trying to retire and rename the node package itself during the preinstall in node package, that meets a EBUSY, I suspect it's due to the replacement of the binary in the node package with the one node-plat-arch, what is it used for in node?

Maybe the latter choice should be simpler

[Lonli-Lokli]

Still the problem