Adds support for running npm package lifecycle scripts, opted into via a
new `--allow-scripts` flag.
With this PR, when running `deno cache` (or `DENO_FUTURE=1 deno
install`) you can specify the `--allow-scripts=pkg1,pkg2` flag to run
lifecycle scripts attached to the given packages.
Note at the moment this only works when `nodeModulesDir` is true (using
the local resolver).
When a package with un-run lifecycle scripts is encountered, we emit a
warning suggesting things may not work and to try running lifecycle
scripts. Additionally, if a package script implicitly requires
`node-gyp` and it's not found on the system, we emit a warning.
Extra things in this PR:
- Extracted out bits of `task.rs` into a separate module for reuse
- Added a couple fields to `process.config` in order to support
`node-gyp` (it relies on a few variables being there)
- Drive by fix to downloading new npm packages to test registry
---
TODO:
- [x] validation for allow-scripts args (make sure it looks like an npm
package)
- [x] make allow-scripts matching smarter
- [ ] figure out what issues this closes
---
Review notes:
- This adds a bunch of deps to our test registry due to using
`node-gyp`, so it's pretty noisy
This adds object config for the workspace config:
```json
{
"workspace": {
"members": ["./member-1", "./member-2"]
}
}
```
This is a more verbose version of `"workspace": ["./member-1",
"./member-2"]`. Although we don't need it at the moment, it makes the
naming of `"workspace"` more clear and leaves the object open for more
config in the future.
Closes https://github.com/denoland/deno/issues/24456
Adds much better support for the unstable Deno workspaces as well as
support for npm workspaces. npm workspaces is still lacking in that we
only install packages into the root node_modules folder. We'll make it
smarter over time in order for it to figure out when to add node_modules
folders within packages.
This includes a breaking change in config file resolution where we stop
searching for config files on the first found package.json unless it's
in a workspace. For the previous behaviour, the root deno.json needs to
be updated to be a workspace by adding `"workspace":
["./path-to-pkg-json-folder-goes-here"]`. See details in
https://github.com/denoland/deno_config/pull/66
Closes #24340
Closes #24159
Closes #24161
Closes #22020
Closes #18546
Closes #16106
Closes #24160
Closes #18296.
Adds a `--frozen` (alias `--frozen-lockfile`) flag that errors out if
the lockfile is out of date. This is useful for running in CI (where an
out of date lockfile is usually a mistake) or to prevent accidental
changes in dependencies.
As suggested in
https://github.com/denoland/deno/pull/24355#discussion_r1657875422.
I wasn't able to hide the mutex stuff as much as I'd like (ended up just
adding an escape hatch `inner()` method that locks the inner mutex),
because you can't return references to the inner fields through a mutex.
This is mostly motivated by the frozen lockfile changes
Also removes permissions being passed in for node resolution. It was
completely useless because we only checked it for reading package.json
files, but Deno reading package.json files for resolution is perfectly
fine.
My guess is this is also a perf improvement because Deno is doing less
work.
In https://github.com/denoland/deno/pull/23955 we changed the sqlite db
journal mode to WAL. This causes issues when someone is running an old
version of Deno using TRUNCATE and a new version because the two fight
against each other.
This commit adds initial support for ".npmrc" files.
Currently we only discover ".npmrc" files next to "package.json" files
and discovering these files in user home dir is left for a follow up.
This pass supports "_authToken" and "_auth" configuration
for providing authentication.
LSP support has been left for a follow up PR.
Towards https://github.com/denoland/deno/issues/16105
By default, uses a 60 second timeout, backing off 2x each time (can be
overridden using the hidden `DENO_SLOW_TEST_TIMEOUT` which we implement
only really for spec testing.
```
Deno.test(async function test() {
await new Promise(r => setTimeout(r, 130_000));
});
```
```
$ target/debug/deno test /tmp/test_slow.ts
Check file:///tmp/test_slow.ts
running 1 test from ../../../../../../tmp/test_slow.ts
test ...'test' is running very slowly (1m0s)
'test' is running very slowly (2m0s)
ok (2m10s)
ok | 1 passed | 0 failed (2m10s)
```
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Construct a new module graph container for workers instead of sharing it
with the main worker.
Fixes #17248
Fixes #23461
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
VScode will typically send a `textDocument/semanticTokens/full` request
followed by `textDocument/semanticTokens/range`, and occassionally
request semantic tokens even when we know nothing has changed. Semantic
tokens also get refreshed on each change. Computing semantic tokens is
relatively heavy in TSC, so we should avoid it as much as possible.
Caches the semantic tokens for open documents, to avoid making TSC do
unnecessary work. Results in a noticeable improvement in local
benchmarking
before:
```
Starting Deno benchmark
-> Start benchmarking lsp
- Simple Startup/Shutdown
(10 runs, mean: 383ms)
- Big Document/Several Edits
(5 runs, mean: 1079ms)
- Find/Replace
(10 runs, mean: 59ms)
- Code Lens
(10 runs, mean: 440ms)
- deco-cx/apps Multiple Edits + Navigation
(5 runs, mean: 9921ms)
<- End benchmarking lsp
```
after:
```
Starting Deno benchmark
-> Start benchmarking lsp
- Simple Startup/Shutdown
(10 runs, mean: 395ms)
- Big Document/Several Edits
(5 runs, mean: 1024ms)
- Find/Replace
(10 runs, mean: 56ms)
- Code Lens
(10 runs, mean: 438ms)
- deco-cx/apps Multiple Edits + Navigation
(5 runs, mean: 8927ms)
<- End benchmarking lsp
```
Moves sloppy import resolution from the loader to the resolver.
Also adds some test helper functions to make the lsp tests less verbose
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
Fixes the `Debug Failure` errors described in
https://github.com/denoland/deno/issues/23643#issuecomment-2094552765 .
The issue here was that we were passing diagnostic codes as strings but
TSC expects the codes to be numbers. This resulted in some quick fixes
not working (as illustrated by the test added here which fails before
this PR).
The first commit is the actual fix. The rest are just test related.
Fixes #23643.
We weren't catching the cancellation exception thrown by TSC on the JS
side, so the rust side was catching this exception and then attempting
to print out the exception via `toString`. That last bit resulted in a
cryptic `[object Object]` showing up in the logs like so:
```
Error during TS request "getCompletionEntryDetails":
[object Object]
```
I'm not 100% sure how we weren't seeing this in the past. My guess is
that #23409 and the subsequent PR to improve the exception catching and
logging surfaced this, but I'm still not quite clear on it.
My initial fix here returned `null` to rust when a server request was
cancelled, but this resulted in a deserialization error when we
attempted to deserialize that into the expected response type. So now,
as soon as the request's cancellation token signals we'll stop waiting
for a response and return an error (which will get swallowed as the LSP
request is being cancelled).
I was a bit surprised to find that [this
branch](0c671c9792/cli/lsp/tsc.rs (L1093))
actually executes sometimes, I believe due to the fact that aborting a
future may not [immediately stop its
execution](https://docs.rs/futures/latest/futures/stream/struct.AbortHandle.html#method.abort).
Before this PR, there would just be an uninformative "Error occurred"
message, after this PR you'll get a stack trace in the LSP output window
like this:
```text
Error during TS request "$getSupportedCodeFixes":
Error: i threw an exception
at serverRequest (ext:deno_tsc/99_main_compiler.js:1089:11)
```
The actual handling of `$projectChanged` is quick, but JS requests are
not. The cleared caches only get repopulated on the next actual request,
so just batch the change notification in with the next actual request.
No significant difference in benchmarks on my machine, but this speeds
up `did_change` handling and reduces our total number of JS requests (in
addition to coalescing multiple JS change notifs into one).
This PR enables V8 code cache for ES modules and for `require` scripts
through `op_eval_context`. Code cache artifacts are transparently stored
and fetched using sqlite db and are passed to V8. `--no-code-cache` can
be used to disable.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
I'm running into a node resolution bug in the lsp only and while
tracking it down I noticed this one.
Fixed by moving the project version out of `Documents`.
Fixes the regression described in
https://github.com/denoland/deno/pull/23293#issuecomment-2049819724.
This affected jupyter notebooks, as the LSP was passing in already
denormalized specifiers, while the jupyter kernel was not. We need to
denormalize the specifiers to evict the proper keys from our caches.
Currently we evict a lot of the caches on the JS side of things on every
request, namely script versions, script file names, and compiler
settings (as of #23283, it's not quite every request but it's still
unnecessarily often).
This PR reports changes to the JS side, so that it can evict exactly the
caches that it needs too. We might want to do some batching in the
future so as not to do 1 request per change.
Previously we locked the entire `FileSystemDocuments` even for lookups,
causing contention. This was particularly bad because some of the hot
ops (namely `op_resolve`) can end up hitting that lock under contention.
This PR replaces the mutex with synchronization internal to
`FileSystemDocuments` (an `AtomicBool` for the dirty flag, and then a
`DashMap` for the actual documents).
I need to think a bit more about whether or not this introduces any
problematic race conditions.
This functionality was broken. The series of events was:
1. Load the npm resolution from the lockfile.
2. Discover only a subset of the specifiers in the documents.
3. Clear the npm snapshot.
4. Redo npm resolution with the new specifiers (~500ms).
What this now does:
1. Load the npm resolution from the lockfile.
2. Discover only a subset of the specifiers in the documents and take
into account the specifiers from the lockfile.
3. Do not redo resolution (~1ms).
Fixes #23163.
The client-facing warning doesn't provide any value and is super
annoying. We still emit a warning message on the server side for format
errors, which should fulfill the same (less intrusive) purpose.
Was investigating a separate stack overflow (that I've now found in the
node resolution code) and came across this. We should avoid recursion
(this is very old code).
1. Stops `deno publish` using some custom include/exclude behaviour from
other sub commands
2. Takes ancestor directories into account when resolving gitignore
3. Backards compatible change that adds ability to unexclude an exclude
by using a negated glob at a more specific level for all sub commands
(see https://github.com/denoland/deno_config/pull/44).
The diagnostic was incorrect when importing a `.js` file with a
corresponding `.d.ts` file with sloppy imports because it would say to
change the `.js` extension to `.d.ts`, which is incorrect. We might as
well just hide this diagnostic.
This commit adds "deno add" subcommand that has a basic support for
adding "jsr:" packages to "deno.json" file.
This currently doesn't support "npm:" specifiers and specifying version
constraints.
Some `deno_std` tests were failing to print output that was resolved
after the last test finished. In addition, output printed before tests
began would sometimes appear above the "running X tests ..." line, and
sometimes below it depending on timing.
We now guarantee that all output is flushed before and after tests run,
making the output consistent.
Pre-test and post-test output are captured in `------ pre-test output
------` and `------ post-test output ------` blocks to differentiate
them from the regular output blocks.
Here's an example of a test (that is much noisier than normal, but an
example of what the output will look like):
```
Check ./load_unload.ts
------- pre-test output -------
load
----- output end -----
running 1 test from ./load_unload.ts
test ...
------- output -------
test
----- output end -----
test ... ok ([WILDCARD])
------- post-test output -------
unload
----- output end -----
```
As we add tracing to more types of runtime activity, `--trace-ops` is
less useful of a name. `--trace-leaks` better reflects that this feature
traces both ops and timers, and will eventually trace resource opening
as well.
This keeps `--trace-ops` as an alias for `--trace-leaks`, but prints a
warning to the console suggesting migration to `--trace-leaks`.
One test continues to use `--trace-ops` to test the deprecation warning.
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
- Removes the origin call, since all origins are the same for an isolate
(ie: the main module)
- Collects the `TestDescription`s and sends them all at the same time
inside of an Arc, allowing us to (later on) re-use these instead of
cloning.
Needs a follow-up pass to remove all the cloning, but that's a thread
that is pretty long to pull
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Gets us closer to solving #20707.
Rewrites the `TestEventSender`:
- Allow for explicit creation of multiple streams. This will allow for
one-std{out,err}-per-worker
- All test events are received along with a worker ID, allowing for
eventual, proper parallel threading of test events.
In theory this should open up proper interleaving of test output,
however that is left for a future PR.
I had some plans for a better performing synchronization primitive, but
the inter-thread communication is tricky. This does, however, speed up
the processing of large numbers of tests 15-25% (possibly even more on
100,000+).
Before
```
ok | 1000 passed | 0 failed (32ms)
ok | 10000 passed | 0 failed (276ms)
```
After
```
ok | 1000 passed | 0 failed (25ms)
ok | 10000 passed | 0 failed (230ms)
```
1. Renames zap/fast-check to instead be a `no-slow-types` lint rule.
1. This lint rule is automatically run when doing `deno lint` for
packages (deno.json files with a name, version, and exports field)
1. This lint rules still occurs on publish. It can be skipped by running
with `--no-slow-types`
This changes the lockfile to not store JSR specifiers in the "remote"
section. Instead a single JSR integrity is stored per package in the
lockfile, which is a hash of the version's `x.x.x_meta.json` file, which
contains hashes for every file in the package. The hashes in this file
are then compared against when loading.
Additionally, when using `{ "vendor": true }` in a deno.json, the files
can be modified without causing lockfile errors—the checksum is only
checked when copying into the vendor folder and not afterwards
(eventually we should add this behaviour for non-jsr specifiers as
well). As part of this change, the `vendor` folder creation is not
always automatic in the LSP and running an explicit cache command is
necessary. The code required to track checksums in the LSP would have
been too complex for this PR, so that all goes through deno_graph now.
The vendoring is still automatic when running from the CLI.
This implementation heavily depends on there being a lockfile, meaning
JSR specifiers will always diagnose as uncached unless it's there. In
practice this affects cases where a `deno.json` isn't being used. Our
NPM specifier support isn't subject to this.
The reason for this is that the version constraint solving code is
currently buried in `deno_graph` and not usable from the LSP, so the
only way to reuse that logic is the solved-version map in the lockfile's
`packages.specifiers`.
