Also removes permissions being passed in for node resolution. It was
completely useless because we only checked it for reading package.json
files, but Deno reading package.json files for resolution is perfectly
fine.
My guess is this is also a perf improvement because Deno is doing less
work.
This is the release commit being forwarded back to main for 1.44.1
Co-authored-by: devsnek <devsnek@users.noreply.github.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Introduces a `SyncReadAsyncWriteLock` to make it harder to write to the
npm resolution without first waiting async in a queue. For the npm
resolution, reading synchronously is fine, but when updating, someone
should wait async, clone the data, then write the data at the end back.
Factoring out `dlint` upgrade from
https://github.com/denoland/deno/pull/24034 as it
requires us to change the lint step on mac to use ARM runners.
---------
Co-authored-by: Luca Casonato <hello@lcas.dev>
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
The PEM file specified by DENO_CERT can contain multiple certificates
13924fdb1b/cli/args/mod.rs (L722-L742)
Signed-off-by: Andreas Kohn <andreas.kohn@gmail.com>
Fixes #24012.
In the case of multiple packages providing a binary with a same name, we
were basically leaving the results undefined (since we set up things in
parallel, and whichever got set up first won). In addition, we were
warning about these cases, even though it's a situation that's expected
to occur.
Instead, in the case of a collision in the binary names, we prefer the
binary provided by the package with the least depth in the dependency
tree.
While I was at it, I also took moved more code to `bin_entries.rs` since
it was starting to get a bit cluttered.
This commits adds the ability to set a would-be exit code
for the Deno process without forcing an immediate exit,
through the new `Deno.exitCode` API.
- **Implements `Deno.exitCode` getter and setter**: Adds support for
setting
and retrieving a would-be exit code via `Deno.exitCode`.
This allows for asynchronous cleanup before process termination
without immediately exiting.
- **Ensures type safety**: The setter for `Deno.exitCode` validates that
the provided value is a number, throwing a TypeError if not, to ensure
that
only valid exit codes are set.
Closes to #23605
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Brings in:
* More fully typed structures (for when we get to implementing more)
* `with_metadata`, `with_buffers`, etc. from
https://github.com/runtimed/runtimed/pull/99
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
In https://github.com/denoland/deno/pull/23955 we changed the sqlite db
journal mode to WAL. This causes issues when someone is running an old
version of Deno using TRUNCATE and a new version because the two fight
against each other.
The same issue in two different places - doing blocking FS work in an
async task, limiting the amount of work that happens concurrently.
- When setting up node_modules, where we try to set up entries
concurrently but were blocking other tasks from actually running.
- When loading package info from the npm registry file cache, loading
and deserializing is expensive and prevents concurrency. This was
especially noticeable when loading an npm resolution snapshot from a
lockfile (`snapshot_from_lockfile` in `deno_npm`).
Installing deps in `deno-docs`:
```
❯ hyperfine -i -p 'rm -rf node_modules/' '../d7/deno-main i' '../d7/target/release/deno i'
Benchmark 1: ../d7/deno-main i
Time (mean ± σ): 2.193 s ± 0.027 s [User: 0.589 s, System: 1.033 s]
Range (min … max): 2.151 s … 2.242 s 10 runs
Benchmark 2: ../d7/target/release/deno i
Time (mean ± σ): 1.597 s ± 0.021 s [User: 0.977 s, System: 1.337 s]
Range (min … max): 1.550 s … 1.627 s 10 runs
Summary
../d7/target/release/deno i ran
1.37 ± 0.02 times faster than ../d7/deno-main i
```
Caching `npm:@11ty/eleventy`:
```
❯ hyperfine -i -p 'rm -rf node_modules/' --warmup 5 '../../d7/deno-main cache npm:@11ty/eleventy' '../../d7/target/release/deno cache npm:@11ty/eleventy'
Benchmark 1: ../../d7/deno-main cache npm:@11ty/eleventy
Time (mean ± σ): 129.9 ms ± 2.2 ms [User: 27.5 ms, System: 101.3 ms]
Range (min … max): 127.5 ms … 135.8 ms 10 runs
Benchmark 2: ../../d7/target/release/deno cache npm:@11ty/eleventy
Time (mean ± σ): 100.6 ms ± 1.3 ms [User: 38.8 ms, System: 233.8 ms]
Range (min … max): 99.3 ms … 103.2 ms 10 runs
Summary
../../d7/target/release/deno cache npm:@11ty/eleventy ran
1.29 ± 0.03 times faster than ../../d7/deno-main cache npm:@11ty/eleventy
```
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
Hard linking (`linkat`) is ridiculously slow on mac. `copyfile` is
better, but what's even faster is `clonefile`. It doesn't have the space
savings that comes with hardlinking, but the performance difference is
worth it imo.
```
❯ hyperfine -i -p 'rm -rf node_modules/' '../../d7/target/release/deno cache npm:@11ty/eleventy' 'deno cache npm:@11ty/eleventy'
Benchmark 1: ../../d7/target/release/deno cache npm:@11ty/eleventy
Time (mean ± σ): 115.4 ms ± 1.2 ms [User: 27.2 ms, System: 87.3 ms]
Range (min … max): 113.7 ms … 117.5 ms 10 runs
Benchmark 2: deno cache npm:@11ty/eleventy
Time (mean ± σ): 619.3 ms ± 6.4 ms [User: 34.3 ms, System: 575.6 ms]
Range (min … max): 612.2 ms … 633.3 ms 10 runs
Summary
../../d7/target/release/deno cache npm:@11ty/eleventy ran
5.37 ± 0.08 times faster than deno cache npm:@11ty/eleventy
```
# Summary
This PR resolves about the issue.
fixes #10810
And the formerly context is in the PR.
#22582
Here is an expected behaviour example with this change.
- 🦕.test.ts
```ts
import { assertEquals } from "https://deno.land/std@0.215.0/assert/mod.ts";
Deno.test("example test", () => {
assertEquals("🍋", "🦕");
});
```
The mixed `number | bigint` representation was useful optimization for
pointers. Now, pointers are represented as V8 externals. As part of the
FFI stabilization effort we want to make `bigint` the only
representation for `u64` and `i64`.
BigInt representation performance is almost on par with mixed
representation with the added benefit that its less confusing and users
don't need manual checks and conversions for doing operations on the
value.
```
cpu: AMD Ryzen 5 7530U with Radeon Graphics
runtime: deno 1.43.6+92a8d09 (x86_64-unknown-linux-gnu)
file:///home/divy/gh/ffi/main.ts
benchmark time (avg) iter/s (min … max) p75 p99 p995
-------------------------------------------------------------------------- -----------------------------
nop 4.01 ns/iter 249,533,690.5 (3.97 ns … 10.8 ns) 3.97 ns 4.36 ns 9.03 ns
ret bigint 7.74 ns/iter 129,127,186.8 (7.72 ns … 10.46 ns) 7.72 ns 8.11 ns 8.82 ns
ret i32 7.81 ns/iter 128,087,100.5 (7.77 ns … 12.72 ns) 7.78 ns 8.57 ns 9.75 ns
ret bigint (add op) 15.02 ns/iter 66,588,253.2 (14.64 ns … 24.99 ns) 14.76 ns 19.13 ns 19.44 ns
ret i32 (add op) 12.02 ns/iter 83,209,131.8 (11.95 ns … 18.18 ns) 11.98 ns 13.11 ns 14.5 ns
```
Enhanced warning message for --env flag with run and eval subcommands.
The commit is specifically made to address issue #23674 by improving the
warning messages that appear when using the --env flag with run or eval
subcommands in the following scenarios:
1. Missing environment file.
2. Incorrect syntax in the environment file content.
**Changes made**
- Distinguishes between cases of missing environment file and wrong
syntax in the environment file content.
- Shows a concise warning message to convey the case/issue occurred.
**Code changes & enhancements**
- Implemented a match statement to handle different types of errors
received while getting and parsing the file content to display a concise
warning message, rather than simple error check and then displaying the
same warning message for whatever the type of error is.
- Updated the related existing tests to reflect the new warning
messages.
- Added two test cases to cover the wrong environment file content
syntax with both run and eval subcommands.
**Impact**
The use of --env flag with both run/eval would be more user-friendly as
it gives a precise description of what is not right when using
incorrectly.
If you could give it a look, @dsherret , I appreciate your feedback on
these changes.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds initial support for ".npmrc" files.
Currently we only discover ".npmrc" files next to "package.json" files
and discovering these files in user home dir is left for a follow up.
This pass supports "_authToken" and "_auth" configuration
for providing authentication.
LSP support has been left for a follow up PR.
Towards https://github.com/denoland/deno/issues/16105
Fixes #23571.
Previously, we required a `deno.json` to be present (or the `--lock`
flag) in order for us to resolve a `deno.lock` file. This meant that if
you were using deno in an npm-first project deno wouldn't use a
lockfile.
Additionally, while I was fixing that, I discovered there were a couple
bugs keeping the future `install` command from using a lockfile.
With this PR, `install` will actually resolve the lockfile (or create
one if not present), and update it if it's not up-to-date. This also
speeds up `deno install`, as we can use the lockfile to skip work during
npm resolution.
This PR removes the use of the custom `utc_now` function in favor of the
`chrono` implementation. It resolves #22864.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
While investigating poor cold start performance on my GCP VM (32 cores,
130GB SSD), I found that writing to the various sqlite databases in
DENO_DIR was quite slow. The slowness seems to primarily be caused by
excessive latency from a number of `fsync()` calls.
The performance difference is best demonstrated by deleting the sqlite
databases from DENO_DIR while leaving the downloaded sources in place.
The benchmark (see notes below):
```
piscisaureus@bert-us:~/erofs/source$ export DENO_DIR=./.deno
piscisaureus@bert-us:~/erofs/source$ hyperfine --warmup 3 \
--prepare "rm -rf .deno/*_v1*" \
"deno run -A --cached-only demo.ts" \
"eatmydata deno run -A --cached-only demo.ts" \
"~/deno/target/release/deno run -A --cached-only demo.ts"
Benchmark 1: deno run -A --cached-only demo.ts
Time (mean ± σ): 1.174 s ± 0.037 s [User: 0.153 s, System: 0.184 s]
Range (min … max): 1.104 s … 1.212 s 10 runs
Benchmark 2: eatmydata deno run -A --cached-only demo.ts
Time (mean ± σ): 265.5 ms ± 3.6 ms [User: 138.5 ms, System: 135.1 ms]
Range (min … max): 260.6 ms … 271.2 ms 11 runs
Benchmark 3: ~/deno/target/release/deno run -A --cached-only demo.ts
Time (mean ± σ): 226.2 ms ± 9.2 ms [User: 136.7 ms, System: 93.3 ms]
Range (min … max): 218.8 ms … 247.1 ms 13 runs
Summary
~/deno/target/release/deno run -A --cached-only demo.ts ran
1.17 ± 0.05 times faster than eatmydata deno run -A --cached-only demo.ts
5.19 ± 0.27 times faster than deno run -A --cached-only demo.ts
```
Notes:
* Benchmark 1: unmodified Deno 1.43.6
* Benchmark 2: unmodified Deno 1.43.6 wrapped with `eatmydata` (which is
a tool to neuter `fsync()` calls)
* Benchmark 3: this PR applied on top of Deno 1.43.6
The script that got benchmarked:
```typescript
// demo.ts
import * as express from "npm:express@4.16.3";
import * as postgres from "https://deno.land/x/postgres/mod.ts";
let _dummy = [express, postgres]; // Force use of imports.
console.log("hello world");
```
This is a primordialization effort to improve resistance against users
tampering with the global `Object` prototype.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
By default, uses a 60 second timeout, backing off 2x each time (can be
overridden using the hidden `DENO_SLOW_TEST_TIMEOUT` which we implement
only really for spec testing.
```
Deno.test(async function test() {
await new Promise(r => setTimeout(r, 130_000));
});
```
```
$ target/debug/deno test /tmp/test_slow.ts
Check file:///tmp/test_slow.ts
running 1 test from ../../../../../../tmp/test_slow.ts
test ...'test' is running very slowly (1m0s)
'test' is running very slowly (2m0s)
ok (2m10s)
ok | 1 passed | 0 failed (2m10s)
```
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This brings in [`runtimelib`](https://github.com/runtimed/runtimed) to
use:
## Fully typed structs for Jupyter Messages
```rust
let msg = connection.read().await?;
self
.send_iopub(
runtimelib::Status::busy().as_child_of(msg),
)
.await?;
```
## Jupyter paths
Jupyter paths are implemented in Rust, allowing the Deno kernel to be
installed completely via Deno without a requirement on Python or
Jupyter. Deno users will be able to install and use the kernel with just
VS Code or other editors that support Jupyter.
```rust
pub fn status() -> Result<(), AnyError> {
let user_data_dir = user_data_dir()?;
let kernel_spec_dir_path = user_data_dir.join("kernels").join("deno");
let kernel_spec_path = kernel_spec_dir_path.join("kernel.json");
if kernel_spec_path.exists() {
log::info!("✅ Deno kernel already installed");
Ok(())
} else {
log::warn!("ℹ️ Deno kernel is not yet installed, run `deno jupyter --install` to set it up");
Ok(())
}
}
```
Closes https://github.com/denoland/deno/issues/21619
https://github.com/denoland/deno/pull/23838 might accidentally disable
import assertions support because of V8 12.6 unshipping it, but we want
import assertions to be supported until Deno 2.
Construct a new module graph container for workers instead of sharing it
with the main worker.
Fixes #17248
Fixes #23461
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
Related: https://github.com/denoland/eszip/pull/181
eszip < v0.69.0 hashes all its contents to ensure data integrity. This
feature is not necessary in Deno CLI as the binary integrity guarantee
is deemed an external responsibility (ie it is to be assumed that, if
necessary, the compiled binary will be checksumed externally prior to
being executed).
eszip >= v0.69.0 no longer performs this checksum by default. This
reduces the cold-start time of the compiled binaries, proportionally to
their size.
VScode will typically send a `textDocument/semanticTokens/full` request
followed by `textDocument/semanticTokens/range`, and occassionally
request semantic tokens even when we know nothing has changed. Semantic
tokens also get refreshed on each change. Computing semantic tokens is
relatively heavy in TSC, so we should avoid it as much as possible.
Caches the semantic tokens for open documents, to avoid making TSC do
unnecessary work. Results in a noticeable improvement in local
benchmarking
before:
```
Starting Deno benchmark
-> Start benchmarking lsp
- Simple Startup/Shutdown
(10 runs, mean: 383ms)
- Big Document/Several Edits
(5 runs, mean: 1079ms)
- Find/Replace
(10 runs, mean: 59ms)
- Code Lens
(10 runs, mean: 440ms)
- deco-cx/apps Multiple Edits + Navigation
(5 runs, mean: 9921ms)
<- End benchmarking lsp
```
after:
```
Starting Deno benchmark
-> Start benchmarking lsp
- Simple Startup/Shutdown
(10 runs, mean: 395ms)
- Big Document/Several Edits
(5 runs, mean: 1024ms)
- Find/Replace
(10 runs, mean: 56ms)
- Code Lens
(10 runs, mean: 438ms)
- deco-cx/apps Multiple Edits + Navigation
(5 runs, mean: 8927ms)
<- End benchmarking lsp
```
This PR directly addresses the issue raised in #23282 where Deno panics
if `deno coverage` is called with `--include` regex that returns no
matches.
I've opted not to change the return value of `collect_summary` for
simplicity and return an empty `HashMap` instead
Moves sloppy import resolution from the loader to the resolver.
Also adds some test helper functions to make the lsp tests less verbose
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
1. Generally we should prefer to use the `log` crate.
2. I very often accidentally commit `eprintln`s.
When we should use `println` or `eprintln`, it's not too bad to be a bit
more verbose and ignore the lint rule.
**THIS PR HAS GIT CONFLICTS THAT MUST BE RESOLVED**
This is the release commit being forwarded back to main for 1.43.2
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.43.2 && git checkout -b forward_v1.43.2 upstream/forward_v1.43.2
```
Don't need this PR? Close it.
cc @nathanwhit
Co-authored-by: nathanwhit <nathanwhit@users.noreply.github.com>
Co-authored-by: Nathan Whitaker <nathan@deno.com>
This PR implements the changes we plan to make to `deno install` in deno
2.0.
- `deno install` without arguments caches dependencies from
`package.json` / `deno.json` and sets up the `node_modules` folder
- `deno install <pkg>` adds the package to the config file (either
`package.json` or `deno.json`), i.e. it aliases `deno add`
- `deno add` can also add deps to `package.json` (this is gated behind
`DENO_FUTURE` due to uncertainty around handling projects with both
`deno.json` and `package.json`)
- `deno install -g <bin>` installs a package as a globally available
binary (the same as `deno install <bin>` in 1.0)
---------
Co-authored-by: Nathan Whitaker <nathan@deno.com>
Fixes the `Debug Failure` errors described in
https://github.com/denoland/deno/issues/23643#issuecomment-2094552765 .
The issue here was that we were passing diagnostic codes as strings but
TSC expects the codes to be numbers. This resulted in some quick fixes
not working (as illustrated by the test added here which fails before
this PR).
The first commit is the actual fix. The rest are just test related.
Fixes #23643.
We weren't catching the cancellation exception thrown by TSC on the JS
side, so the rust side was catching this exception and then attempting
to print out the exception via `toString`. That last bit resulted in a
cryptic `[object Object]` showing up in the logs like so:
```
Error during TS request "getCompletionEntryDetails":
[object Object]
```
I'm not 100% sure how we weren't seeing this in the past. My guess is
that #23409 and the subsequent PR to improve the exception catching and
logging surfaced this, but I'm still not quite clear on it.
My initial fix here returned `null` to rust when a server request was
cancelled, but this resulted in a deserialization error when we
attempted to deserialize that into the expected response type. So now,
as soon as the request's cancellation token signals we'll stop waiting
for a response and return an error (which will get swallowed as the LSP
request is being cancelled).
I was a bit surprised to find that [this
branch](0c671c9792/cli/lsp/tsc.rs (L1093))
actually executes sometimes, I believe due to the fact that aborting a
future may not [immediately stop its
execution](https://docs.rs/futures/latest/futures/stream/struct.AbortHandle.html#method.abort).
This makes `create_runtime_snapshot` more useful for embedders who add
their own extension(s) to the runtime in build scripts.
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
Before this PR, there would just be an uninformative "Error occurred"
message, after this PR you'll get a stack trace in the LSP output window
like this:
```text
Error during TS request "$getSupportedCodeFixes":
Error: i threw an exception
at serverRequest (ext:deno_tsc/99_main_compiler.js:1089:11)
```
By default, `deno serve` will assign port 8000 (like `Deno.serve`).
Users may choose a different port using `--port`.
`deno serve /tmp/file.ts`
`server.ts`:
```ts
export default {
fetch(req) {
return new Response("hello world!\n");
},
};
```
When the response has been successfully send, we abort the
`Request.signal` property to indicate that all resources associated with
this transaction may be torn down.
This commit changes the workspace support to provide all workspace
members to be available as imports based on their names and versions.
Closes https://github.com/denoland/deno/issues/23343
<!--
Before submitting a PR, please read
https://docs.deno.com/runtime/manual/references/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
This PR wires up a new `jsxPrecompileSkipElements` option in
`compilerOptions` that can be used to exempt a list of elements from
being precompiled with the `precompile` JSX transform.
The actual handling of `$projectChanged` is quick, but JS requests are
not. The cleared caches only get repopulated on the next actual request,
so just batch the change notification in with the next actual request.
No significant difference in benchmarks on my machine, but this speeds
up `did_change` handling and reduces our total number of JS requests (in
addition to coalescing multiple JS change notifs into one).
Embedders may have special requirements around file opening, so we add a
new `check_open` permission check that is called as part of the file
open process.
Adds an `addr` field to `HttpServer` to simplify the pattern
`Deno.serve({ onListen({ port } => listenPort = port })`. This becomes:
`const server = Deno.serve({}); port = server.addr.port`.
Changes:
- Refactors `serve` overloads to split TLS out (in preparation for
landing a place for the TLS SNI information)
- Adds an `addr` field to `HttpServer` that matches the `addr` field of
the corresponding `Deno.Listener`s.
This PR enables V8 code cache for ES modules and for `require` scripts
through `op_eval_context`. Code cache artifacts are transparently stored
and fetched using sqlite db and are passed to V8. `--no-code-cache` can
be used to disable.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This allows people to use imports like:
```ts
import "./app.css";
```
...with `deno check` in systems where there's a bundle step (ex. Vite).
This will still error when using it with `deno run` or if the referenced
file does not exist.
See test cases for behaviour.
This PR adds a benchmark intended to measure how the LSP handles larger
repos, as well as its performance on a more realistic workload.
The repo being benchmarked is
[deco-cx/apps](https://github.com/deco-cx/apps) which has been vendored
along with its dependencies. It's included as a git submodule as its
fairly large. The LSP requests used in the benchmark are the actual
requests sent by VSCode as I opened, modified, and navigated around a
file (to simulate an actual user interaction).
The main motivation is to have a more realistic benchmark that measures
how we do with a large number of files and dependencies. The
improvements made from 1.42 to 1.42.3 mostly improved performance with
larger repos, so none of our existing benchmarks showed an improvement.
Here are the results for the changes made from 1.42 to 1.42.3 (the new
benchmark is the last one listed):
**1.42.0**
```test
Starting Deno benchmark
-> Start benchmarking lsp
- Simple Startup/Shutdown
(10 runs, mean: 379ms)
- Big Document/Several Edits
(5 runs, mean: 1142ms)
- Find/Replace
(10 runs, mean: 51ms)
- Code Lens
(10 runs, mean: 443ms)
- deco-cx/apps Multiple Edits + Navigation
(5 runs, mean: 25121ms)
<- End benchmarking lsp
```
**1.42.3**
```text
Starting Deno benchmark
-> Start benchmarking lsp
- Simple Startup/Shutdown
(10 runs, mean: 383ms)
- Big Document/Several Edits
(5 runs, mean: 1135ms)
- Find/Replace
(10 runs, mean: 55ms)
- Code Lens
(10 runs, mean: 440ms)
- deco-cx/apps Multiple Edits + Navigation
(5 runs, mean: 11675ms)
<- End benchmarking lsp
```
`TestEventSender` should not be Clone so we don't end up with multiple
copies of the same writer FD. This is probably not the cause of the test
channel lockups, but it's a lot easier to reason about.
Due to a terminating NUL that was placed in a `r#` string, we were not
actually NUL-terminating pipe names on Windows. While this has no
security implications due to the random nature of the prefix, it would
occasionally cause random failures when the trailing garbage would make
the pipe name invalid.
This commit moves logic of dispatching lifecycle events (
"load", "beforeunload", "unload") to be triggered from Rust.
Before that we were executing scripts from Rust, but now we
are storing references to functions from "99_main.js" and calling
them directly.
Prerequisite for https://github.com/denoland/deno/issues/23342
I'm running into a node resolution bug in the lsp only and while
tracking it down I noticed this one.
Fixed by moving the project version out of `Documents`.
…faces (#23296)"
This reverts commit e190acbfa8.
Reverting because it broke stable API type declarations. We will reland
it for v1.43 with updated interfaces
Fixes the regression described in
https://github.com/denoland/deno/pull/23293#issuecomment-2049819724.
This affected jupyter notebooks, as the LSP was passing in already
denormalized specifiers, while the jupyter kernel was not. We need to
denormalize the specifiers to evict the proper keys from our caches.
Currently we evict a lot of the caches on the JS side of things on every
request, namely script versions, script file names, and compiler
settings (as of #23283, it's not quite every request but it's still
unnecessarily often).
This PR reports changes to the JS side, so that it can evict exactly the
caches that it needs too. We might want to do some batching in the
future so as not to do 1 request per change.
This is PR a smaller retry of #23066 that simply ensures all async
`ext/fs` ops are accounted for if left hanging in tests. This also sorts
the `OP_DETAILS` in alphabetical order for easy future reading.
When reviewing, it might be best to look at the commits in order for
better understanding.
Removes the certificate options from all the interfaces and replaces
them with a new `TlsCertifiedKeyOptions`. This allows us to centralize
the documentation for TLS key management for both client and server, and
will allow us to add key object support in the future.
Also adds an option `keyFormat` field to the cert/key that must be
omitted or set to `pem`. This will allow us to load other format keys in
the future `der`, `pfx`, etc.
In a future PR, we will add a way to load a certified key object, and we
will add another option to `TlsCertifiedKeyOptions` like so:
```ts
export interface TlsCertifiedKeyOptions =
| TlsCertifiedKeyPem
| TlsCertifiedKeyFromFile
| TlsCertifiedKeyConnectTls
| { key: Deno.CertifiedKey }
```
Previously we locked the entire `FileSystemDocuments` even for lookups,
causing contention. This was particularly bad because some of the hot
ops (namely `op_resolve`) can end up hitting that lock under contention.
This PR replaces the mutex with synchronization internal to
`FileSystemDocuments` (an `AtomicBool` for the dirty flag, and then a
`DashMap` for the actual documents).
I need to think a bit more about whether or not this introduces any
problematic race conditions.
Changes `discreet` in the documentation for `discrete`
"Discreet" means careful to avoid being noticed, "discrete" means
separate parts, and is what the documentation refers to.
The TS language service requests source files via
[getSourceFile](7a25fd5ef0/cli/tsc/99_main_compiler.js (L560)).
In that function, we [unconditionally
add](7a25fd5ef0/cli/tsc/99_main_compiler.js (L613-L614))
the source file to our sourceFileCache. The issue is that we only remove
things from that cache if the source file [becomes out of
date](7a25fd5ef0/cli/tsc/99_main_compiler.js (L777-L783)).
For files that don't get changed, we keep them in the cache
indefinitely. So sometimes we keep SourceFile objects from being GC'ed
because they're retained in our cache, even though TS doesn't refer to
them any more. I see this in pretty much all of the heap snapshots I've
taken.
---
The fix here is pretty direct - just store weak references to the
sourcefiles in the cache. It doesn't really change our caching behavior,
it just prevents us from being the only retainer of a `SourceFile`. I
also split the `sourceFileCache` into a separate cache just for assets,
as we rely on those being alive.
The simpler fix is to only cache assets, but presumably that has a perf
impact.
---
In local testing, this PR reduced the size of the JS heap by about 1 GB
when using `deno lsp` in the Typescript repo.
This functionality was broken. The series of events was:
1. Load the npm resolution from the lockfile.
2. Discover only a subset of the specifiers in the documents.
3. Clear the npm snapshot.
4. Redo npm resolution with the new specifiers (~500ms).
What this now does:
1. Load the npm resolution from the lockfile.
2. Discover only a subset of the specifiers in the documents and take
into account the specifiers from the lockfile.
3. Do not redo resolution (~1ms).
The tests would deadlock if we tried to write the sync marker into a
pipe that was full because one test streamed just enough data to fill
the pipe, so when we went to actually write the sync marker we blocked
when nobody was reading.
We use a two-phase lock for sync markers now: one to indicate "ready to
sync" and the second to indicate that the sync bytes have been received.
This commit adds enum to "InstallFlags" and "UninstallFlags" that will
allow to support both local and global (un)installation.
Currently the local variant is not used.
Towards https://github.com/denoland/deno/issues/23062
Fixes #23163.
The client-facing warning doesn't provide any value and is super
annoying. We still emit a warning message on the server side for format
errors, which should fulfill the same (less intrusive) purpose.
When `DENO_FUTURE=1` env var is present, then BYONM
("bring your own node_modules") is enabled by default.
That means that is there's a `package.json` present, users
are expected to explicitly install dependencies from that file.
Towards https://github.com/denoland/deno/issues/23151
Was doing a bit of debugging on why some stuff is not working in a
personal project and ran a quick debug profile and saw it cloning the
pkg json a lot. We should put this in an Rc.
Unused locals and parameters don't make sense to surface in remote
modules. Additionally, fast check can cause these kind of diagnostics
when publishing, so they should be ignored.
Closes #22959
Was investigating a separate stack overflow (that I've now found in the
node resolution code) and came across this. We should avoid recursion
(this is very old code).
This PR introduces the ability to exclude certain paths from the file watcher
in Deno. This is particularly useful when running scripts in watch mode,
as it allows developers to prevent unnecessary restarts when changes are
made to files that do not affect the running script, or when executing
scripts that generate new files which results in an infinite restart
loop.
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
In preparation for upcoming changes to `deno install` in Deno 2.
If `-g` or `--global` flag is not provided a warning will be emitted:
```
⚠️ `deno install` behavior will change in Deno 2. To preserve the current behavior use `-g` or `--global` flag.
```
The same will happen for `deno uninstall` - unless `-g`/`--global` flag
is provided
a warning will be emitted.
Towards https://github.com/denoland/deno/issues/23062
---------
Signed-off-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
This commit changes "deno init" subcommand to use "jsr:" specifier for
standard library "assert" module. It is unversioned, but we will change
it to `@^1` once `@std/assert` release version 1.0.
This allows us to start decoupling `deno` and `deno_std` release. The
release scripts have been updated to take that into account.
Fixes #23053.
Two small bugs here:
- the existing condition for printing out the group header was broken.
it worked in the reproducer (in the issue above) without filtering only
by accident, due to setting `self.has_ungrouped = true` once we see the
warmup bench. Knowing that we sort benchmarks to put ungrouped benches
first, there are only two cases: 1) we are starting the first group 2)
we are ending the previous group and starting a new group
- when you passed `--filter` we were applying that filter to the warmup
bench (which is not visible to users), so we suffered from jit bias if
you were filtering (unless your filter was `<warmup>`)
TLDR;
Running
```bash
deno bench main.js --filter="G"
```
```js
// main.js
Deno.bench({
group: "G1",
name: "G1-A",
fn() {},
});
Deno.bench({
group: "G1",
name: "G1-B",
fn() {},
});
```
Before this PR:
```
benchmark time (avg) iter/s (min … max) p75 p99 p995
--------------------------------------------------------------- -----------------------------
G1-A 303.52 ps/iter3,294,726,102.1 (254.2 ps … 7.8 ns) 287.5 ps 391.7 ps 437.5 ps
G1-B 3.8 ns/iter 263,360,635.9 (2.24 ns … 8.36 ns) 3.84 ns 4.73 ns 4.94 ns
summary
G1-A
12.51x faster than G1-B
```
After this PR:
```
benchmark time (avg) iter/s (min … max) p75 p99 p995
--------------------------------------------------------------- -----------------------------
group G1
G1-A 3.85 ns/iter 259,822,096.0 (2.42 ns … 9.03 ns) 3.83 ns 4.62 ns 4.83 ns
G1-B 3.84 ns/iter 260,458,274.5 (3.55 ns … 7.05 ns) 3.83 ns 4.45 ns 4.7 ns
summary
G1-B
1x faster than G1-A
```
Before this PR, we didn't have any integration tests set up for the
`jupyter` subcommand.
This PR adds a basic jupyter client and helpers for writing integration
tests for the jupyter kernel. A lot of the code here is boilerplate,
mainly around the message format for jupyter.
This also adds a few basic integration tests, most notably for
requesting execution of a snippet of code and getting the correct
results.
This patch gets JUnit reporter to output more detailed information for
test steps (subtests).
## Issue with previous implementation
In the previous implementation, the test hierarchy was represented using
several XML tags like the following:
- `<testsuites>` corresponds to the entire test (one execution of `deno
test` has exactly one `<testsuites>` tag)
- `<testsuite>` corresponds to one file, such as `main_test.ts`
- `<testcase>` corresponds to one `Deno.test(...)`
- `<property>` corresponds to one `t.step(...)`
This structure describes the test layers but one problem is that
`<property>` tag is used for any use cases so some tools that can ingest
a JUnit XML file might not be able to interpret `<property>` as
subtests.
## How other tools address it
Some of the testing frameworks in the ecosystem address this issue by
fitting subtests into the `<testcase>` layer. For instance, take a look
at the following Go test file:
```go
package main_test
import "testing"
func TestMain(t *testing.T) {
t.Run("child 1", func(t *testing.T) {
// OK
})
t.Run("child 2", func(t *testing.T) {
// Error
t.Fatal("error")
})
}
```
Running [gotestsum], we can get the output like this:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="3" failures="2" errors="0" time="1.013694">
<testsuite tests="3" failures="2" time="0.510000" name="example/gosumtest" timestamp="2024-03-11T12:26:39+09:00">
<properties>
<property name="go.version" value="go1.22.1 darwin/arm64"></property>
</properties>
<testcase classname="example/gosumtest" name="TestMain/child_2" time="0.000000">
<failure message="Failed" type="">=== RUN TestMain/child_2
 main_test.go:12: error
--- FAIL: TestMain/child_2 (0.00s)
</failure>
</testcase>
<testcase classname="example/gosumtest" name="TestMain" time="0.000000">
<failure message="Failed" type="">=== RUN TestMain
--- FAIL: TestMain (0.00s)
</failure>
</testcase>
<testcase classname="example/gosumtest" name="TestMain/child_1" time="0.000000"></testcase>
</testsuite>
</testsuites>
```
This output shows that nested test cases are squashed into the
`<testcase>` layer by treating them as the same layer as their parent,
`TestMain`. We can still distinguish nested ones by their `name`
attributes that look like `TestMain/<subtest_name>`.
As described in #22795, [vitest] solves the issue in the same way as
[gotestsum].
One downside of this would be that one test failure that happens in a
nested test case will end up being counted multiple times, because not
only the subtest but also its wrapping container(s) are considered to be
failures. In fact, in the [gotestsum] output above, `TestMain/child_2`
failed (which is totally expected) while its parent, `TestMain`, was
also counted as failure. As
https://github.com/denoland/deno/pull/20273#discussion_r1307558757
pointed out, there is a test runner that offers flexibility to prevent
this, but I personally don't think the "duplicate failure count" issue
is a big deal.
## How to fix the issue in this patch
This patch fixes the issue with the same approach as [gotestsum] and
[vitest].
More specifically, nested test cases are put into the `<testcase>` level
and their names are now represented as squashed test names concatenated
by `>` (e.g. `parent 2 > child 1 > grandchild 1`). This change also
allows us to put a detailed error message as `<failure>` tag within the
`<testcase>` tag, which should be handled nicely by third-party tools
supporting JUnit XML.
## Extra fix
Also, file paths embedded into XML outputs are changed from absolute
path to relative path, which is helpful when running the test suites in
several different environments like CI.
Resolves #22795
[gotestsum]: https://github.com/gotestyourself/gotestsum
[vitest]: https://vitest.dev/
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This has been incorrect since the function adopted its (more intuitive)
current behavior in 9268df5f3. The same behavior change was backported
to v1.39.3 in 87e954f54.
Fixes #22941.
In that case, the only file with coverage was the `test.ts` file. The
coverage reporter filters out test files before compiling its report, so
after filtering we were left with an empty set of files. Later on it's
assumed that there is at least 1 file to be reported on, and we panic.
Instead of panicking, just issue an error after filtering.
In addition to the reasons for this outlined by @nayeemrmn in #14877
(which I think are reasons alone to not do this), this simplifies things
a lot because then we don't need to implement the following:
1. Need to handle a JSR module dynamically importing a module within it.
2. Need to handle importing an export of a JSR dep then another export
dynamically loaded later.
Additionally, people should be running `deno check dynamic_import.ts`
instead of relying on this behaviour.
Landing this as a fix because it's blocking people in some scenarios and
the current behaviour is broken (I didn't even have to change any tests
to remove this, which is bad).
Closes #22852
Closes #14877
Closes #22580
Skips the access check if the specific unary permission is in an
all-granted state. Generally prevents an allocation or two.
Hooks up a quiet "all" permission that is automatically inherited. This
permission will be used in the future to indicate that the user wishes
to accept all side-effects of the permissions they explicitly granted.
The "all" permission is an "ambient flag"-style permission that states
whether "allow-all" was passed on the command-line.
Issue https://github.com/denoland/deno/issues/22222
This PR moves `runtime/permissions` code to a upstream crate called
`deno_permissions`. The `deno_permissions::PermissionsContainer` is put
into the OpState and can be used instead of the current trait-based
permissions system.
For this PR, I've migrated `deno_fetch` to the new crate but kept the
rest of the trait-based system as a wrapper of `deno_permissions` crate.
Doing the migration all at once is error prone and hard to review.
Comparing incremental compile times for `ext/fetch` on Mac M1:
| profile | `cargo build --bin deno` | `cargo plonk build --bin deno` |
| --------- | ------------- | ------------------- |
| `debug` | 20 s | 0.8s |
| `release` | 4 mins 12 s | 1.4s |
This commit fixes race condition in "node:worker_threads" module were
the first message did a setup of "threadId", "workerData" and
"environmentData".
Now this data is passed explicitly during workers creation and is set up
before any user code is executed.
Closes https://github.com/denoland/deno/issues/22783
Closes https://github.com/denoland/deno/issues/22672
---------
Co-authored-by: Satya Rohith <me@satyarohith.com>
This is an unrealistic scenario, but it's still a good thing to fix and
have a test for because it probably fixes some other underlying issues
with how the gitignore was being resolved for the root directory.
From https://github.com/denoland/deno/pull/22720#issuecomment-1986134425
Previously the sloppy resolver could not resolve the following:
- foo/bar.ts
- foo.ts
- index.ts
Where `index.ts` contains `import "./foo"`, because it did not consider
`foo.ts` a valid target for this directory import.
This commit fixes this bug.
This is the release commit being forwarded back to main for 1.41.2
Signed-off-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
This allows explicitly overriding a .gitignore by specifying files and
directories in "include". This does not apply to globs in an include as
files matching those will still be gitignored. Additionally,
individually gitignored files within an included directory will still be
ignored.
1. Stops `deno publish` using some custom include/exclude behaviour from
other sub commands
2. Takes ancestor directories into account when resolving gitignore
3. Backards compatible change that adds ability to unexclude an exclude
by using a negated glob at a more specific level for all sub commands
(see https://github.com/denoland/deno_config/pull/44).
We emitted `import "./` rather than `import "./$NAME"`. This is now
fixed.
Also makes a cosmetic change so that `../` imports are now just imported
as `../`, not `./../`.
An undocumented "DENO_DISABLE_PEDANTIC_NODE_WARNINGS" env
var can be used to silence warnings for sloppy imports and node builtins
without `node:` prefix.
This was showing up on the flamegraph.
```
14:54 $ hyperfine -S none --warmup 25 '/tmp/deno run /tmp/empty.js' 'target/release/deno run /tmp/empty.js'
Benchmark 1: /tmp/deno run /tmp/empty.js
Time (mean ± σ): 17.2 ms ± 4.7 ms [User: 11.2 ms, System: 4.0 ms]
Range (min … max): 15.1 ms … 72.9 ms 172 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Benchmark 2: target/release/deno run /tmp/empty.js
Time (mean ± σ): 16.7 ms ± 1.1 ms [User: 11.1 ms, System: 4.0 ms]
Range (min … max): 15.0 ms … 20.1 ms 189 runs
Summary
'target/release/deno run /tmp/empty.js' ran
1.03 ± 0.29 times faster than '/tmp/deno run /tmp/empty.js'
✔ ~/Documents/github/deno/deno [faster_extract|…5⚑ 23]
```
The diagnostic was incorrect when importing a `.js` file with a
corresponding `.d.ts` file with sloppy imports because it would say to
change the `.js` extension to `.d.ts`, which is incorrect. We might as
well just hide this diagnostic.
Improves #19100
Fixes #20356
Replaces #20428
Changes made in deno_core to support this:
- [x] Errors must be handled in setTimeout callbacks
- [x] Microtask ordering is not-quite-right
- [x] Timer cancellation must be checked right before dispatch
- [x] Timer sanitizer
- [x] Move high-res timer to deno_core
- [x] Timers need opcall tracing
This commit adds "deno add" subcommand that has a basic support for
adding "jsr:" packages to "deno.json" file.
This currently doesn't support "npm:" specifiers and specifying version
constraints.
Some `deno_std` tests were failing to print output that was resolved
after the last test finished. In addition, output printed before tests
began would sometimes appear above the "running X tests ..." line, and
sometimes below it depending on timing.
We now guarantee that all output is flushed before and after tests run,
making the output consistent.
Pre-test and post-test output are captured in `------ pre-test output
------` and `------ post-test output ------` blocks to differentiate
them from the regular output blocks.
Here's an example of a test (that is much noisier than normal, but an
example of what the output will look like):
```
Check ./load_unload.ts
------- pre-test output -------
load
----- output end -----
running 1 test from ./load_unload.ts
test ...
------- output -------
test
----- output end -----
test ... ok ([WILDCARD])
------- post-test output -------
unload
----- output end -----
```
A security feature of JSR is that it is self contained other than npm
dependencies. At publish time, the registry rejects packages that write
code like this:
```ts
const data = await import("https://example.com/evil.js");
```
However, this can be trivially bypassed by writing code that the
registry cannot statically analyze for. This PR prevents Deno from
loading dynamic imports that do this.
As we add tracing to more types of runtime activity, `--trace-ops` is
less useful of a name. `--trace-leaks` better reflects that this feature
traces both ops and timers, and will eventually trace resource opening
as well.
This keeps `--trace-ops` as an alias for `--trace-leaks`, but prints a
warning to the console suggesting migration to `--trace-leaks`.
One test continues to use `--trace-ops` to test the deprecation warning.
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
- Removes the origin call, since all origins are the same for an isolate
(ie: the main module)
- Collects the `TestDescription`s and sends them all at the same time
inside of an Arc, allowing us to (later on) re-use these instead of
cloning.
Needs a follow-up pass to remove all the cloning, but that's a thread
that is pretty long to pull
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Supply chain security for JSR.
```
$ deno publish --provenance
Successfully published @divy/test_provenance@0.0.3
Provenance transparency log available at https://search.sigstore.dev/?logIndex=73657418
```
0. Package has been published.
1. Fetches the version manifest and verifies it's matching with uploaded
files and exports.
2. Builds the attestation SLSA payload using Github actions env.
3. Creates an ephemeral key pair for signing the github token
(aud=sigstore) and DSSE pre authentication tag.
4. Requests a X.509 signing certificate from Fulcio using the challenge
and ephemeral public key PEM.
5. Prepares a DSSE envelop for Rekor to witness. Posts an intoto entry
to Rekor and gets back the transparency log index.
6. Builds the provenance bundle and posts it to JSR.
Gets us closer to solving #20707.
Rewrites the `TestEventSender`:
- Allow for explicit creation of multiple streams. This will allow for
one-std{out,err}-per-worker
- All test events are received along with a worker ID, allowing for
eventual, proper parallel threading of test events.
In theory this should open up proper interleaving of test output,
however that is left for a future PR.
I had some plans for a better performing synchronization primitive, but
the inter-thread communication is tricky. This does, however, speed up
the processing of large numbers of tests 15-25% (possibly even more on
100,000+).
Before
```
ok | 1000 passed | 0 failed (32ms)
ok | 10000 passed | 0 failed (276ms)
```
After
```
ok | 1000 passed | 0 failed (25ms)
ok | 10000 passed | 0 failed (230ms)
```
<!--
Before submitting a PR, please read
https://docs.deno.com/runtime/manual/references/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
This PR enhances the `deno publish` command to infer dependencies from
`package.json` if present.
Updates dependent crates which includes an investigation fix by @irbull
in Deno's LSP when linting code. Huge thanks to Ian for tracking down
this issue.
Also includes Divy's deno_graph executor change, which reduces memory
usage when loading jsr specifiers and makes them faster.
Co-authored-by: irbull <irbull@users.noreply.github.com>
Co-authored-by: littledivy <littledivy@users.noreply.github.com>
When using a prefix or suffix containing an invalid filename character,
it's not entirely clear where the errors come from. We make these errors
more consistent across platforms.
In addition, all permission prompts for tempfile and tempdir were
printing the same API name.
We also take the opportunity to make the tempfile random space larger by
2x (using a base32-encoded u64 rather than a hex-encoded u32).
1. Renames zap/fast-check to instead be a `no-slow-types` lint rule.
1. This lint rule is automatically run when doing `deno lint` for
packages (deno.json files with a name, version, and exports field)
1. This lint rules still occurs on publish. It can be skipped by running
with `--no-slow-types`
This change deprecates
`Deno.CreateHttpClientOptions.{certChain,privateKey}` in favour of
`Deno.CreateHttpClientOptions.{cert,key}`.
Closes #22278
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
The format of the sanitizers will change a little bit:
- If multiple async ops leak and traces are on, we repeat the async op
header once per stack trace.
- All leaks are aggregated under a "Leaks detected:" banner as the new
timers are eventually going to be added, and these are neither ops nor
resources.
- `1 async op` is now `An async op`
- If ops and resources leak, we show both (rather than op leaks masking
resources)
Follow-on to https://github.com/denoland/deno/pull/22226
Splitting the sleep and interval ops allows us to detect an interval
timer. We also remove the use of the `op_async_void_deferred` call.
A future PR will be able to split the op sanitizer messages for timers
and intervals.
This changes the lockfile to not store JSR specifiers in the "remote"
section. Instead a single JSR integrity is stored per package in the
lockfile, which is a hash of the version's `x.x.x_meta.json` file, which
contains hashes for every file in the package. The hashes in this file
are then compared against when loading.
Additionally, when using `{ "vendor": true }` in a deno.json, the files
can be modified without causing lockfile errors—the checksum is only
checked when copying into the vendor folder and not afterwards
(eventually we should add this behaviour for non-jsr specifiers as
well). As part of this change, the `vendor` folder creation is not
always automatic in the LSP and running an explicit cache command is
necessary. The code required to track checksums in the LSP would have
been too complex for this PR, so that all goes through deno_graph now.
The vendoring is still automatic when running from the CLI.
Note: tests are not the only part of the codebase that uses `std`. Other
parts, like `tools/`, do too. So, it could be argued that this is a
little misleading. Either way, I'm doing this as discussed with
@mmastrac.
This introduces the `denort` binary - a slim version of deno without
tooling. The binary is used as the default for `deno compile`.
Improves `deno compile` final size by ~2.5x (141 MB -> 61 MB) on Linux
x86_64.
This implementation heavily depends on there being a lockfile, meaning
JSR specifiers will always diagnose as uncached unless it's there. In
practice this affects cases where a `deno.json` isn't being used. Our
NPM specifier support isn't subject to this.
The reason for this is that the version constraint solving code is
currently buried in `deno_graph` and not usable from the LSP, so the
only way to reuse that logic is the solved-version map in the lockfile's
`packages.specifiers`.
This looks like a massive PR, but it's only a move from cli/tests ->
tests, and updates of relative paths for files.
This is the first step towards aggregate all of the integration test
files under tests/, which will lead to a set of integration tests that
can run without the CLI binary being built.
While we could leave these tests under `cli`, it would require us to
keep a more complex directory structure for the various test runners. In
addition, we have a lot of complexity to ignore various test files in
the `cli` project itself (cargo publish exclusion rules, autotests =
false, etc).
And finally, the `tests/` folder will eventually house the `test_ffi`,
`test_napi` and other testing code, reducing the size of the root repo
directory.
For easier review, the extremely large and noisy "move" is in the first
commit (with no changes -- just a move), while the remainder of the
changes to actual files is in the second commit.
Removes the `FileFetcher`'s internal cache because I don't believe it's
necessary (we already cache this kind of stuff in places like deno_graph
or config files in different places). Removing it fixes this bug because
this functionality was already implemented in deno_graph and lowers
memory usage of the CLI a little bit.
This PR separates integration tests from CLI tests into a new project
named `cli_tests`. This is a prerequisite for an integration test runner
that can work with either the CLI binary in the current project, or one
that is built ahead of time.
## Background
Rust does not have the concept of artifact dependencies yet
(https://github.com/rust-lang/cargo/issues/9096). Because of this, the
only way we can ensure a binary is built before running associated tests
is by hanging tests off the crate with the binary itself.
Unfortunately this means that to run those tests, you _must_ build the
binary and in the case of the deno executable that might be a 10 minute
wait in release mode.
## Implementation
To allow for tests to run with and without the requirement that the
binary is up-to-date, we split the integration tests into a project of
their own. As these tests would not require the binary to build itself
before being run as-is, we add a stub integration `[[test]]` target in
the `cli` project that invokes these tests using `cargo test`.
The stub test runner we add has `harness = false` so that we can get
access to a `main` function. This `main` function's sole job is to
`execvp` the command `cargo test -p deno_cli`, effectively "calling"
another cargo target.
This ensures that the deno executable is always correctly rebuilt before
running the stub test runner from `cli`, and gets us closer to be able
to run the entire integration test suite on arbitrary deno executables
(and therefore split the build into multiple phases).
The new `cli_tests` project lives within `cli` to avoid a large PR. In
later PRs, the test data will be split from the `cli` project. As there
are a few thousand files, it'll be better to do this as a completely
separate PR to avoid noise.
- Move a workers test to js_unit_tests and make it work
- (slightly) repair the websocketstream_test and make it a JS unit test.
This test was being ignored and rotted quite a bit, but there's some
value in running as much of it as we can.
- Merge the two websocket test files
This removes the majority of `../../../../../../test_util` relative
imports from the codebase, allowing us to move this code more easily in
the future.
This commit removes some not really necessary FFI tests and in effect
removes them from being accessible from the user code.
This lowers the number of ops accessible to user code to 16.
Hacky quick fix. The real fix is a lot more work to do (move the
`SourceTextInfo` into all the diagnostics in order to make this less
error pone). I've already started on it, but it will require a lot of
downstream create changes.
Closes #22288
Auth tokens may be specified for one of the following:
- `abc123@deno.land`: `deno.land`, `www.deno.land`, etc
- `abc123@deno.land:8080`: `deno.land:8080`, `www.deno.land:8080`, etc
- `abc123@1.1.1.1`: IP `1.1.1.1` only
- `abc123@1.1.1.1:8080`: IP `1.1.1.1`, port 8080 only
- `abc123@[ipv6]`: IPv6 `[ipv6]` only
- `abc123@[ipv6]:8080`: IPv6 `[ipv6]`, port 8080 only
Leading dots are ignored, so `.deno.dev` is equivalent to `deno.dev`.
This moves the op sanitizer descriptions into Rust code and prepares for
eventual op import from `ext:core/ops`. We cannot import these ops from
`ext:core/ops` as the testing infrastructure ops are not always present.
Changes:
- Op descriptions live in `cli` code and are currently accessible via an
op for the older sanitizer code
- `phf` dep moved to workspace root so we can use it here
- `ops.op_XXX` changed to to `op_XXX` to prepare for op imports later
on.
This commit adds `ppid` getter for `node:process` to improve Node
compatibility one step further.
There is one problem though, which is that `Deno.ppid`, which
`process.ppid` internally calls, is actually of type `bigint` although
it's supposed to be `number`. I filed an issue for this (#22166). For
the time being, explciit type conversion from `bigint` to `number` is
applied to match the Node.js behavior.
Fixes https://github.com/denoland/deno/issues/22180
Matches the Node.js implementation more closely. Removed types, they do
not help just make it harder to debug with stack traces.
This commit adds a list of ops to `runtime/99_main.js` that are
currently relying on getting them from `Deno.core.ops`. All ops that are not
present in the list are removed from `Deno.core.ops` on startup (they are
imported from "virtual op module" - `ext:core/ops`) making them effectively
inaccessible to user code.
This change lowers the number of ops exposed to user code from 650 to
around 260. This number should be gradually decreased in follow-up PRs.
Step 1 of the Rustification of sanitizers, which unblocks the faster
timers.
This replaces the resource sanitizer with a Rust one, using the new APIs
in deno_core.
This commit adds automatic expansion of "imports" field in "deno.json"
file.
If "npm:" or "jsr:" imports are encountered we automatically try to add
a "directory" remapping.
Previously users had to specify entries for both `foo` and `foo/` to be
able to import like
`import { symbol1 } from "foo";` and `import { symbol2 } from
"foo/some_file.js"`:
```
{
"imports": {
"foo": "npm:@foo/bar",
"foo/": "npm:/@foo/bar/",
}
```
With this change users can only add entry for `foo`:
```
{
"imports": {
"foo": "npm:@foo/bar",
}
```
The entry for `foo/` will be provided automatically.
Similarly if user provides "directory" remapping explicitly, we will not
overwrite it.
For removal in Deno v2. There are two issues:
1. Any script being run causes the output of `warnOnDeprecatedApi()` to
be printed, even when none of the `rid` properties are called.
2. `.rid` of these classes is used in multiple tests. I'm not sure how
to account for that. I thought of having `STDIN_RID`, and friends,
constants, whose values can be shared between the tests and the classes
themselves. Should we go with that or do something else?
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
For removal in Deno v2. I've also updated the deprecation of
`Deno.FsWatcher.return()`, which, to be clear, I'm not in favour of
deprecating. I mention this in #15499. Either way, it's safe to merge
this PR, then decide against the deprecation.
- changed `Deno.UnsafeWindowSurface` typings from accepting
`Deno.UnsafePointerView` to `Deno.PointerValue`
- added width and height to `GPUCanvasConfiguration`
Removes weird "frame like" characters and simplifies the output:
```
warning: Use of deprecated "Deno.isatty()" API. This API will be removed in Deno 2.
Stack trace:
at file:///Users/ib/dev/deno/foo.js:2:8
hint: Use `stdStream.isTerminal()` instead.
warning: Use of deprecated "Deno.isatty()" API. This API will be removed in Deno 2.
Stack trace:
at file:///Users/ib/dev/deno/foo.js:7:8
hint: Use `stdStream.isTerminal()` instead.
```
https://github.com/denoland/deno/assets/13602871/7a6e24bf-44ec-4dbf-ac96-2af1db9f2ab9
This commit deprecates `window` global and adds deprecation
notice on each use of `window`.
We decided to proceed with removal of `window` global variable in Deno
2.0. There's a lot of code
in the wild that uses pattern like this:
```
if (typeof window !== "undefined) {
...
}
```
to check if the code is being run in browser. However, this check passes
fine in Deno and
most often libraries that do this check try to access some browser API
that is not available
in Deno, or use DOM APIs (which are also not available in Deno).
This situation has occurred multiple times already
and it's unfeasible to expect the whole ecosystem to migrate to new
check (and even if that
happened there's a ton of code that's already shipped and won't change).
The migration is straightfoward - replace all usages of `window` with
`globalThis` or `self`.
When Deno encounters use of `window` global it will now issue a warning,
steering users
towards required changes:
```
Warning
├ Use of deprecated "window" API.
│
├ This API will be removed in Deno 2.0. Make sure to upgrade to a stable API before then.
│
├ Suggestion: Use `globalThis` or `self` instead.
│
├ Suggestion: You can provide `window` in the current scope with: `const window = globalThis`.
│
└ Stack trace:
└─ at file:///Users/ib/dev/deno/foo.js:7:1
```
Ref https://github.com/denoland/deno/issues/13367.
This commit adds support for [TC39 Decorator
Proposal](https://github.com/tc39/proposal-decorators).
These decorators are only available in transpiled sources - ie.
non-JavaScript files (because of lack of support in V8).
This entails that "experimental TypeScript decorators" are not available
by default
and require to be configured, with a configuration like this:
```
{
"compilerOptions": {
"experimentalDecorators": true
}
}
```
Closes https://github.com/denoland/deno/issues/19160
---------
Signed-off-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: crowlkats <crowlkats@toaxl.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
This change sets the removal version for the `deno bundle` sub-command
for Deno v2. The warnings appear when `deno bundle` is run and in the
`--help` menu.
This change:
1. Implements `Deno.FsFile.sync()` and `Deno.FsFile.syncSync()`.
2. Deprecates `Deno.fsync()` and `Deno.fsyncSync()` for removal in Deno
v2, in favour of the above corresponding methods.
Related #21995
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This change:
1. Implements `Deno.FsFile.dataSync()` and `Deno.FsFile.dataSyncSync()`.
2. Deprecates `Deno.fdatasync()` and `Deno.fdatasyncSync()` for removal
in Deno v2, in favour of the above corresponding methods.
3. Replaces use of `Deno.fdatasync()` and `Deno.fdatasyncSync()` with
the above instance methods.
Related #21995
Most uses of `Deno.resources()` within tests, as they previously checked
for leaked resources. This is not needed as the test runner does this
automatically. Other internal uses of this API have been replaced with
the internal `Deno[Deno.internal].core.resources()`.
This change:
1. Implements `Deno.stdin.isTerminal()`, `Deno.stdout.isTerminal()` and
`Deno.stderr.isTerminal()`.
2. Deprecates `Deno.isatty()` for removal in Deno v2, in favour of the
above instance methods.
3. Replaces use of `Deno.isatty()` with the above instance methods.
Related #21995
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
This commit adds `import.meta.filename` and `import.meta.dirname` APIs.
These APIs return string representation of filename and containing
dirname.
They are only defined for local modules (modules that have `file:///`
scheme).
Example:
```ts
console.log(import.meta.filename, import.meta.dirname)
```
Unix:
```
$ deno run /dev/my_module.ts
/dev/my_module.ts /dev/
```
Windows:
```
$ deno run C:\dev\my_module.ts
C:\dev\my_module.ts C:\
```
This change sets the removal version of `Deno.customInspect` for Deno
v2.
Towards #22021
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
It appears the `--prompt` flag has done nothing for some time. Perhaps,
since #13650. Classifying this as a dead functionality removal for this
reason.
Did this while working on #22021.
This change:
1. Sets the removal version for `Deno.ListenTlsOptions.certFile`,
`Deno.ListenTlsOptions.keyFile` and `Deno.ConnectTlsOptions.certFile`
for Deno v2, in favour of the `cert`, `key` and `caCerts` options,
respectively.
2. Replaces use of the deprecated options with the new recommended
options.
Towards #22021
This initially uses the new diagnostic printer in `deno lint`,
`deno doc` and `deno publish`. In the limit we should also update
`deno check` to use this printer.
This commit deprecates "--unstable" flag.
When "--unstable" flag is encountered a warning like this is printed:
```
The `--unstable` flag is deprecated, use granular `--unstable-*` flags instead.
Learn more at: https://docs.deno.com/runtime/manual/tools/unstable_flags
```
When "--unstable" flag is used and an unstable API is called an
additional warning like this is printed for each API call:
```
The `Deno.dlopen` API was used with `--unstable` flag. The `--unstable` flag is deprecated, use granular `--unstable-ffi` instead.
Learn more at: https://docs.deno.com/runtime/manual/tools/unstable_flags
```
When no "--unstable-*" flag is provided and an unstable API is called
following
warning is issued before exiting:
```
Unstable API 'Deno.dlopen'. The `--unstable-ffi` flag must be provided.
```
---------
Signed-off-by: Divy Srivastava <dj.srivastava23@gmail.com>
Signed-off-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Asher Gomez <ashersaupingomez@gmail.com>
This change removes the currently deprecated `Deno.cron()` overload with
`options` as a potential last argument.
This might be fine to do now, in a major release, as `Deno.cron()` is an
unstable API. I thought of doing this while working on #22021. If this
is not ready to remove, I can instead set the removal version of this
overload for Deno v2.
Note: this overload was deprecated in Deno v1.38.2 (#21225). So it's
been deprecated for over 2 months.
This change:
1. Sets the removal version for `Deno.RequestEvent`, `Deno.HttpConn` and
`Deno.serveHttp()` for Deno v2. I thought it might be worth calling
`warnOnDeprecatedApi()` within `Deno.Request` and `Deno.HttpConn`
methods, but I thought just having it called within `Deno.serveHttp()`
might be sufficient.
2. Removes some possibly unneeded related benchmarks.
Towards #22021
Closes https://github.com/denoland/deno/issues/21828.
This API is a huge footgun. And given that "Deno.serveHttp" is a
deprecated API that is discouraged to use (use "Deno.serve()"
instead); it makes no sense to keep this API around.
This is a step towards fully migrating to Hyper 1.
Fixes #21928.
We have a code path which empties the extension sources because they're
expected to be pre-executed in the snapshot. Instead of using
conditional compilation for that, we now just check if a snapshot was
provided.
Removes the `dont_use_runtime_snapshot` feature. We didn't allow not
providing a snapshot unless this feature was provided, now we always do.
Adds the `only_snapshotted_js_sources` feature for us to use in CLI.
This asserts that a snapshot is provided and gates the runtime
transpilation code so it isn't included in the executable.
This change takes advantage of explicit resources management for
`FsFile` instances and tweaks documentation to encourage the use of it.
---------
Signed-off-by: Asher Gomez <ashersaupingomez@gmail.com>
<!--
Before submitting a PR, please read https://deno.com/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
We were missing the `constants` export in the promise `fs` API which is
available in node.
```ts
import { constants, promises } from "node:fs";
import { constants as fsPromiseConstants } from "node:fs/promises";
console.log(constants === promises.constants); // logs: true
console.log(constants === fsPromiseConstants); // logs: true
```
Fixes https://github.com/denoland/deno/issues/21994
<!--
Before submitting a PR, please read https://deno.com/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
Ultimately, it came down to using incorrect property names in the
`FakeSocket` constructor. The values are passed under `remoteAddress`,
not `hostname` and `remotePort` instead of `port`.
Fixes https://github.com/denoland/deno/issues/21980
This commit introduces deprecation warnings for "Deno.*" APIs.
This is gonna be quite noisy, but should tremendously help with user
code updates to ensure
smooth migration to Deno 2.0. The warning is printed at each unique call
site to help quickly
identify where code needs to be adjusted. There's some stack frame
filtering going on to
remove frames that are not useful to the user and would only cause
confusion.
The warning can be silenced using "--quiet" flag or
"DENO_NO_DEPRECATION_WARNINGS" env var.
"Deno.run()" API is now using this warning. Other deprecated APIs will
start warning
in follow up PRs.
Example:
```js
import { runEcho as runEcho2 } from "http://localhost:4545/run/warn_on_deprecated_api/mod.ts";
const p = Deno.run({
cmd: [
Deno.execPath(),
"eval",
"console.log('hello world')",
],
});
await p.status();
p.close();
async function runEcho() {
const p = Deno.run({
cmd: [
Deno.execPath(),
"eval",
"console.log('hello world')",
],
});
await p.status();
p.close();
}
await runEcho();
await runEcho();
for (let i = 0; i < 10; i++) {
await runEcho();
}
await runEcho2();
```
```
$ deno run --allow-read foo.js
Warning
├ Use of deprecated "Deno.run()" API.
│
├ This API will be removed in Deno 2.0. Make sure to upgrade to a stable API before then.
│
├ Suggestion: Use "Deno.Command()" API instead.
│
└ Stack trace:
└─ at file:///Users/ib/dev/deno/foo.js:3:16
hello world
Warning
├ Use of deprecated "Deno.run()" API.
│
├ This API will be removed in Deno 2.0. Make sure to upgrade to a stable API before then.
│
├ Suggestion: Use "Deno.Command()" API instead.
│
└ Stack trace:
├─ at runEcho (file:///Users/ib/dev/deno/foo.js:8:18)
└─ at file:///Users/ib/dev/deno/foo.js:13:7
hello world
Warning
├ Use of deprecated "Deno.run()" API.
│
├ This API will be removed in Deno 2.0. Make sure to upgrade to a stable API before then.
│
├ Suggestion: Use "Deno.Command()" API instead.
│
└ Stack trace:
├─ at runEcho (file:///Users/ib/dev/deno/foo.js:8:18)
└─ at file:///Users/ib/dev/deno/foo.js:14:7
hello world
Warning
├ Use of deprecated "Deno.run()" API.
│
├ This API will be removed in Deno 2.0. Make sure to upgrade to a stable API before then.
│
├ Suggestion: Use "Deno.Command()" API instead.
│
└ Stack trace:
├─ at runEcho (file:///Users/ib/dev/deno/foo.js:8:18)
└─ at file:///Users/ib/dev/deno/foo.js:17:9
hello world
hello world
hello world
hello world
hello world
hello world
hello world
hello world
hello world
hello world
Warning
├ Use of deprecated "Deno.run()" API.
│
├ This API will be removed in Deno 2.0. Make sure to upgrade to a stable API before then.
│
├ Suggestion: Use "Deno.Command()" API instead.
│
├ Suggestion: It appears this API is used by a remote dependency.
│ Try upgrading to the latest version of that dependency.
│
└ Stack trace:
├─ at runEcho (http://localhost:4545/run/warn_on_deprecated_api/mod.ts:2:18)
└─ at file:///Users/ib/dev/deno/foo.js:20:7
hello world
```
Closes #21839
Transpiler doing type checking such as the ones used in dnt or bundler
fail because of incompatible Worker types if env like browser are
targeted.
Signed-off-by: Jérôme Benoit <jerome.benoit@sap.com>
This commit removes the requirement for `--unstable` flag in `deno
jupyter` subcommand. The process will no longer exit if this flag is not
provided, however the subcommand itself is still considered unstable
and might change in the future.
Required for https://github.com/denoland/deno/pull/21452
This commit adds support for [Stage 3 Temporal API
proposal](https://tc39.es/proposal-temporal/docs/).
The API is available when `--unstable-temporal` flag is passed.
---------
Signed-off-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: David Sherret <dsherret@gmail.com>
Co-authored-by: Kenta Moriuchi <moriken@kimamass.com>
This commit removes conditional type-checking of unstable APIs.
Before this commit `deno check` (or any other type-checking command and
the LSP) would error out if there was an unstable API in the code, but not
`--unstable` flag provided.
This situation hinders DX and makes it harder to configure Deno. Failing
during runtime unless `--unstable` flag is provided is enough in this case.
This was incorrectly being stored so the AOT cache for JSR specifiers
wasn't being used. I added a wrapper type to help prevent the API being
used incorrectly.
This commit adds support for "rejectionhandled" Web Event and
"rejectionHandled" Node event.
```js
import process from "node:process";
process.on("rejectionHandled", (promise) => {
console.log("rejectionHandled", reason, promise);
});
window.addEventListener("rejectionhandled", (event) => {
console.log("rejectionhandled", event.reason, event.promise);
});
```
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
<!--
Before submitting a PR, please read https://deno.com/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
Ensures the Deno process is brought down even when the runtime gets hung
up on something.
Marvin found that the lsp was running without a parent vscode around so
this is maybe/probably related.
Fixes https://github.com/denoland/deno/issues/21840
The problem was hard to reproduce as its a race condition. I've added a
test that reproduces the problem 1/10 tries. We should move the
idleTimeout handling to Rust (maybe even built into fastwebsocket).
We were calling `expand_glob` on our excludes, which is very expensive
and unnecessary because we can pattern match while traversing instead.
1. Doesn't expand "exclude" globs. Instead pattern matches while walking
the directory.
2. Splits up the "include" into base paths and applicable file patterns.
This causes less pattern matching to occur because we're only pattern
matching on patterns that might match and not ones in completely
unrelated directories.
This defines the removal version of v2 for the following APIs that
favour the Streams API:
* `Deno.copy()`
* `Deno.File`
* `Deno.iter()`
* `Deno.Buffer`
* `Deno.readAll()`
* `Deno.readAllSync()`
* `Deno.writeAll()`
* `Deno.writeAllSync()`
* `Deno.FsWatcher.return()`
Part 1 of a potential 3 part series. Ref #13449
The current implementation passes key material back and forth RustCrypto
group of crates and ring. ring does not implement p521 yet.
This PR adds support for P521 named curve in `generateKey` and
`importKey` where we use RustCrypto. Other parts should be moved over to
the RustGroup group of crates for consistency.
This PR fixes #21658.
- `check` subcommand sees `exclude` option in `deno.json`. When some
paths passed with `check` command listed in `exclude`, they are ignored.
- When some files are listed in `exclude` and imported indirectly among
module graph, they are checked.
Harden the code that does permission checks to protect against
re-opening of stdin.
Code that runs FFI is vulnerable to an attack where fd 0 is closed
during a permission check and re-opened with a file that contains a
positive response (ie: `y` or `A`). While FFI code is dangerous in
general, we can make it more difficult for FFI-enabled code to bypass
additional permission checks.
- Checks to see if the underlying file for stdin has changed from the
start to the end of the permission check (detects races)
- Checks to see if the message is excessively long (lowering the window
for races)
- Checks to see if stdin and stderr are still terminals at the end of
the function (making races more difficult)
Fixes #20516
Follow up to #21747 and #21746
This tackles the last point of #20516 where certain inputs weren't
accepted in the other zlib methods
This adds the `toU8` conversion of `_brotli` to `_zlib.mjs`, when we
create the ZLibBuffer, we'll sanitize the input. I noticed that the
async had no handler for `string` input so I added that as well.
Fixes https://github.com/denoland/deno/issues/21734
Changes:
- Use default encode when options do not provide a encode callback.
- Remove internal TS for `node:querystring`. Its not helping catching
bugs like this because of invalid type assumptions and use of `any`,
more of a maintenance burden.
This fixes point 3 of https://github.com/denoland/deno/issues/20516
This PR creates consistency between the sync and async versions of the
brotli compress where we will always return a buffer like Node.
This fixes point 2 of #20516
This adds a conversion from Dataview/Buffer by returning `obj.buffer`
which can be converted to a `UInt8Array`.
Question: Regarding point 4 of the mentioned issue would it be
appropriate to copy the toU8 helper to the `zlib.mjs` methods?
These tests started failing on CI on Dec 28th, 2023 returning
ENOTFOUND. It's unclear what's going on, since `dig -x
8.8.8.8.in-addr.arpa`.
It needs a deeper investigation, but I want to unblock main branch
in the meantime.
Main change is that:
- "hyper" has been renamed to "hyper_v014" to signal that it's legacy
- "hyper1" has been renamed to "hyper" and should be the default
`opAsync` requires a lookup by name on each async call. This is a
mechanical translation of all opAsync calls to ensureFastOps.
The `opAsync` API on Deno.core will be removed at a later time.
When we migrate to op-import-per-extension, we will want to ensure that
ops have one and only one place where they are imported. This tackles
the ops that are imported via `ensureFastOps`, but does not yet tackle
direct `ops` imports.
Landing ahead of https://github.com/denoland/deno_core/pull/393
Node HTTP/2 was using the default h2 `Bytes` datatype when we can be
making using of `BufView` like we do in `Deno.serve`.
`fetch` and `Deno.serverHttp` can't make use of `BufView` because they
are using `reqwest` which is stuck on hyper 0.x at this time.
This commit adds a way to connect to the TS compiler host that is run
as part of the "deno lsp" subcommand. This can be done by specifying
"DENO_LSP_INSPECTOR" variable.
---------
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
This PR implements the child_process IPC pipe between parent and child.
The implementation uses Windows named pipes created by parent and passes
the inheritable file handle to the child.
I've also replace parts of the initial implementation which passed the
raw parent fd to JS with resource ids instead. This way no file handle
is exposed to the JS land (both parent and child).
`IpcJsonStreamResource` can stream upto 800MB/s of JSON data on Win 11
AMD Ryzen 7 16GB (without `memchr` vectorization)
`napi_open_handle_scope` was returning a bogus handle_scope and we were
trying to close it in `napi_close_handle_scope`.
This is a bit of a challenge to test, but the following testcase comes
from #21601 and appears to be fixed by this.
```
import { decode } from "https://deno.land/std@0.209.0/encoding/base64.ts";
import sharp from "npm:sharp";
const img = 'iVBORw0KGgoAAAANSUhEUgAAAQAAAAEAAQMAAABmvDolAAAAA1BMVEWq09/P7Lz1AAAAH0lEQVRoge3BAQ0AAADCoPdPbQ43oAAAAAAAAAAAvg0hAAABmmDh1QAAAABJRU5ErkJggg==';
Deno.test("async", async () => {
const id = setTimeout(() => Deno.exit(1), 1000);
await sharp(decode(img)).toBuffer();
await sharp(decode(img)).toBuffer();
clearTimeout(id);
});
```
Fixes https://github.com/denoland/deno/issues/21594
I verified locally that this fixes the problem. I'm working on testing
harness for Jupyter kernel to catch regressions like this and will
add it in a follow up PR.
Bumped versions for 1.39.0
Please ensure:
- [x] Target branch is correct (`vX.XX` if a patch release, `main` if
minor)
- [x] Crate versions are bumped correctly
- [x] deno_std version is incremented in the code (see
`cli/deno_std.rs`)
- [x] Releases.md is updated correctly (think relevancy and remove
reverts)
To make edits to this PR:
```shell
git fetch upstream release_1_39.0 && git checkout -b release_1_39.0 upstream/release_1_39.0
```
cc @mmastrac
---------
Co-authored-by: mmastrac <mmastrac@users.noreply.github.com>
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
This change deprecates `Deno.Reader`, `Deno.ReaderSync`, `Deno.Writer`,
`Deno.WriterSync` and `Deno.Closer` in favour of the [Web Streams
API](https://developer.mozilla.org/en-US/docs/Web/API/Streams_API).
After discussing with Yoshiya, we both thought now might be the right
time to deprecate these interfaces with v2 getting closer.
This PR implements the Node child_process IPC functionality in Deno on
Unix systems.
For `fd > 2` a duplex unix pipe is set up between the parent and child
processes. Currently implements data passing via the channel in the JSON
serialization format.
Part 2 of removing middleware.
This is somewhat awkward because `V8CrossThreadTaskSpawner` requires
tasks to be `Send`, but NAPI makes heavy use of `!Send` pointers. In
addition, Rust causes a closure to be `!Send` if you pull a `!Send`
value out of a struct.
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Adds performance measurements for all ops used by the LSP. Also changes
output of "Language server status" page to include more precise
information.
Current suspicion is that computing "script version" takes a long time
for some users.
This problem occurred trying to load tensorflow.js
```
> import * as tf from 'npm:@tensorflow/tfjs';
Uncaught SyntaxError: Identifier 'mod' has already been declared at file:///Users/ry/Library/Caches/deno/npm/registry.npmjs.org/@tensorflow/tfjs/4.14.0/dist/tf.node.js:167:14
at async <anonymous>:1:33
```
This commit adds a no-op flushHeaders method to the ServerResponse
object. It is a nop because the ServerResponse implementation is based
on top of the Deno server API instead of the Node `OutgoingMessage`
base.
Fixes #21509
This fixes #21434 for `BroadcastChannel` and `WebSocketStream`.
`--unstable` still enable both, but granular unstable flags now also
work:
* `--unstable-net` now enables `WebSocketStream`.
* `--unstable-broadcast-channel` now enables `BroadcastChannel`.
* Additionally, there are now tests for all granular unstable flags.
Since `unsafe-proto` already had tests, so I didn't add any for this
one.
It also introduces a map to keep track of granular unstable ids without
having to sync multiple places.
This commit brings back usage of primordials in "40_testing.js" by
turning it back into an ES module and using new "lazy loading" functionality
of ES modules coming from "deno_core".
The same approach was applied to "40_jupyter.js".
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Adds an `--unstable-sloppy-imports` flag which supports the
following for `file:` specifiers:
* Allows writing `./mod` in a specifier to do extension probing.
- ex. `import { Example } from "./example"` instead of `import { Example
} from "./example.ts"`
* Allows writing `./routes` to do directory extension probing for files
like `./routes/index.ts`
* Allows writing `./mod.js` for *mod.ts* files.
This functionality is **NOT RECOMMENDED** for general use with Deno:
1. It's not as optimal for perf:
https://marvinh.dev/blog/speeding-up-javascript-ecosystem-part-2/
1. It makes tooling in the ecosystem more complex in order to have to
understand this.
1. The "Deno way" is to be explicit about what you're doing. It's better
in the long run.
1. It doesn't work if published to the Deno registry because doing stuff
like extension probing with remote specifiers would be incredibly slow.
This is instead only recommended to help with migrating existing
projects to Deno. For example, it's very useful for getting CJS projects
written with import/export declaration working in Deno without modifying
module specifiers and for supporting TS ESM projects written with
`./mod.js` specifiers.
This feature will output warnings to guide the user towards correcting
their specifiers. Additionally, quick fixes are provided in the LSP to
update these specifiers:
Landing changes required for
https://github.com/denoland/deno_core/pull/359
We needed to update 99_main.js and a whole load of tests.
API changes:
- setPromiseRejectCallback becomes setUnhandledPromiseRejectionHandler.
The function is now called from eventLoopTick.
- The promiseRejectMacrotaskCallback no longer exists, as this is
automatically handled in eventLoopTick.
- ops.op_dispatch_exception now takes a second parameter: in_promise.
The preferred way to call this op is now reportUnhandledException or
reportUnhandledPromiseRejection.
