This commit re-implements `ext/fetch` and all dependent crates
using `hyper` and `hyper-util`, instead of `reqwest`.
The reasoning is that we want to have greater control and access
to low level `hyper` APIs when implementing `fetch` API as well
as `node:http` module.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
NPM inserts a default install script when a package has a `binding.gyp`
file.
It's possible, however, for the package to exclude the `binding.gyp`
file when they publish, and in this case the install script will never
succeed for a user of the package.
This happens with `fsevents`, for instance. They don't include the
`binding.gyp` file in their published tarball, but the default install
script appears in the manifest served by `npm`.
This causes us to warn that `fsevents` has an install script, but when
you try to run it it fails due to `binding.gyp` not existing.
Previously when we printed out the packages that skipped install
scripts, we didn't prefix them with `npm:`. When you pass
`--allow-scripts` though, we require `npm:`, which means you can't just
copy paste the package name from the warning message.
Adds support for running npm package lifecycle scripts, opted into via a
new `--allow-scripts` flag.
With this PR, when running `deno cache` (or `DENO_FUTURE=1 deno
install`) you can specify the `--allow-scripts=pkg1,pkg2` flag to run
lifecycle scripts attached to the given packages.
Note at the moment this only works when `nodeModulesDir` is true (using
the local resolver).
When a package with un-run lifecycle scripts is encountered, we emit a
warning suggesting things may not work and to try running lifecycle
scripts. Additionally, if a package script implicitly requires
`node-gyp` and it's not found on the system, we emit a warning.
Extra things in this PR:
- Extracted out bits of `task.rs` into a separate module for reuse
- Added a couple fields to `process.config` in order to support
`node-gyp` (it relies on a few variables being there)
- Drive by fix to downloading new npm packages to test registry
---
TODO:
- [x] validation for allow-scripts args (make sure it looks like an npm
package)
- [x] make allow-scripts matching smarter
- [ ] figure out what issues this closes
---
Review notes:
- This adds a bunch of deps to our test registry due to using
`node-gyp`, so it's pretty noisy
Adds much better support for the unstable Deno workspaces as well as
support for npm workspaces. npm workspaces is still lacking in that we
only install packages into the root node_modules folder. We'll make it
smarter over time in order for it to figure out when to add node_modules
folders within packages.
This includes a breaking change in config file resolution where we stop
searching for config files on the first found package.json unless it's
in a workspace. For the previous behaviour, the root deno.json needs to
be updated to be a workspace by adding `"workspace":
["./path-to-pkg-json-folder-goes-here"]`. See details in
https://github.com/denoland/deno_config/pull/66
Closes #24340
Closes #24159
Closes #24161
Closes #22020
Closes #18546
Closes #16106
Closes #24160
Closes #18296.
Adds a `--frozen` (alias `--frozen-lockfile`) flag that errors out if
the lockfile is out of date. This is useful for running in CI (where an
out of date lockfile is usually a mistake) or to prevent accidental
changes in dependencies.
As suggested in
https://github.com/denoland/deno/pull/24355#discussion_r1657875422.
I wasn't able to hide the mutex stuff as much as I'd like (ended up just
adding an escape hatch `inner()` method that locks the inner mutex),
because you can't return references to the inner fields through a mutex.
This is mostly motivated by the frozen lockfile changes
Also removes permissions being passed in for node resolution. It was
completely useless because we only checked it for reading package.json
files, but Deno reading package.json files for resolution is perfectly
fine.
My guess is this is also a perf improvement because Deno is doing less
work.
Introduces a `SyncReadAsyncWriteLock` to make it harder to write to the
npm resolution without first waiting async in a queue. For the npm
resolution, reading synchronously is fine, but when updating, someone
should wait async, clone the data, then write the data at the end back.
Fixes #24012.
In the case of multiple packages providing a binary with a same name, we
were basically leaving the results undefined (since we set up things in
parallel, and whichever got set up first won). In addition, we were
warning about these cases, even though it's a situation that's expected
to occur.
Instead, in the case of a collision in the binary names, we prefer the
binary provided by the package with the least depth in the dependency
tree.
While I was at it, I also took moved more code to `bin_entries.rs` since
it was starting to get a bit cluttered.
In https://github.com/denoland/deno/pull/23955 we changed the sqlite db
journal mode to WAL. This causes issues when someone is running an old
version of Deno using TRUNCATE and a new version because the two fight
against each other.
The same issue in two different places - doing blocking FS work in an
async task, limiting the amount of work that happens concurrently.
- When setting up node_modules, where we try to set up entries
concurrently but were blocking other tasks from actually running.
- When loading package info from the npm registry file cache, loading
and deserializing is expensive and prevents concurrency. This was
especially noticeable when loading an npm resolution snapshot from a
lockfile (`snapshot_from_lockfile` in `deno_npm`).
Installing deps in `deno-docs`:
```
❯ hyperfine -i -p 'rm -rf node_modules/' '../d7/deno-main i' '../d7/target/release/deno i'
Benchmark 1: ../d7/deno-main i
Time (mean ± σ): 2.193 s ± 0.027 s [User: 0.589 s, System: 1.033 s]
Range (min … max): 2.151 s … 2.242 s 10 runs
Benchmark 2: ../d7/target/release/deno i
Time (mean ± σ): 1.597 s ± 0.021 s [User: 0.977 s, System: 1.337 s]
Range (min … max): 1.550 s … 1.627 s 10 runs
Summary
../d7/target/release/deno i ran
1.37 ± 0.02 times faster than ../d7/deno-main i
```
Caching `npm:@11ty/eleventy`:
```
❯ hyperfine -i -p 'rm -rf node_modules/' --warmup 5 '../../d7/deno-main cache npm:@11ty/eleventy' '../../d7/target/release/deno cache npm:@11ty/eleventy'
Benchmark 1: ../../d7/deno-main cache npm:@11ty/eleventy
Time (mean ± σ): 129.9 ms ± 2.2 ms [User: 27.5 ms, System: 101.3 ms]
Range (min … max): 127.5 ms … 135.8 ms 10 runs
Benchmark 2: ../../d7/target/release/deno cache npm:@11ty/eleventy
Time (mean ± σ): 100.6 ms ± 1.3 ms [User: 38.8 ms, System: 233.8 ms]
Range (min … max): 99.3 ms … 103.2 ms 10 runs
Summary
../../d7/target/release/deno cache npm:@11ty/eleventy ran
1.29 ± 0.03 times faster than ../../d7/deno-main cache npm:@11ty/eleventy
```
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
Hard linking (`linkat`) is ridiculously slow on mac. `copyfile` is
better, but what's even faster is `clonefile`. It doesn't have the space
savings that comes with hardlinking, but the performance difference is
worth it imo.
```
❯ hyperfine -i -p 'rm -rf node_modules/' '../../d7/target/release/deno cache npm:@11ty/eleventy' 'deno cache npm:@11ty/eleventy'
Benchmark 1: ../../d7/target/release/deno cache npm:@11ty/eleventy
Time (mean ± σ): 115.4 ms ± 1.2 ms [User: 27.2 ms, System: 87.3 ms]
Range (min … max): 113.7 ms … 117.5 ms 10 runs
Benchmark 2: deno cache npm:@11ty/eleventy
Time (mean ± σ): 619.3 ms ± 6.4 ms [User: 34.3 ms, System: 575.6 ms]
Range (min … max): 612.2 ms … 633.3 ms 10 runs
Summary
../../d7/target/release/deno cache npm:@11ty/eleventy ran
5.37 ± 0.08 times faster than deno cache npm:@11ty/eleventy
```
This commit adds initial support for ".npmrc" files.
Currently we only discover ".npmrc" files next to "package.json" files
and discovering these files in user home dir is left for a follow up.
This pass supports "_authToken" and "_auth" configuration
for providing authentication.
LSP support has been left for a follow up PR.
Towards https://github.com/denoland/deno/issues/16105
This functionality was broken. The series of events was:
1. Load the npm resolution from the lockfile.
2. Discover only a subset of the specifiers in the documents.
3. Clear the npm snapshot.
4. Redo npm resolution with the new specifiers (~500ms).
What this now does:
1. Load the npm resolution from the lockfile.
2. Discover only a subset of the specifiers in the documents and take
into account the specifiers from the lockfile.
3. Do not redo resolution (~1ms).
This PR adds a new unstable "bring your own node_modules" (BYONM)
functionality currently behind a `--unstable-byonm` flag (`"unstable":
["byonm"]` in a deno.json).
This enables users to run a separate install command (ex. `npm install`,
`pnpm install`) then run `deno run main.ts` and Deno will respect the
layout of the node_modules directory as setup by the separate install
command. It also works with npm/yarn/pnpm workspaces.
For this PR, the behaviour is opted into by specifying
`--unstable-byonm`/`"unstable": ["byonm"]`, but in the future we may
make this the default behaviour as outlined in
https://github.com/denoland/deno/issues/18967#issuecomment-1761248941
This is an extremely rough initial implementation. Errors are
terrible in this and the LSP requires frequent restarts. Improvements
will be done in follow up PRs.
