Issue https://github.com/denoland/deno/issues/22222
This PR moves `runtime/permissions` code to a upstream crate called
`deno_permissions`. The `deno_permissions::PermissionsContainer` is put
into the OpState and can be used instead of the current trait-based
permissions system.
For this PR, I've migrated `deno_fetch` to the new crate but kept the
rest of the trait-based system as a wrapper of `deno_permissions` crate.
Doing the migration all at once is error prone and hard to review.
Comparing incremental compile times for `ext/fetch` on Mac M1:
| profile | `cargo build --bin deno` | `cargo plonk build --bin deno` |
| --------- | ------------- | ------------------- |
| `debug` | 20 s | 0.8s |
| `release` | 4 mins 12 s | 1.4s |
Fixes #19214.
We were using the `idna` crate to implement our polyfill for
`punycode.toASCII` and `punycode.toUnicode`. The `idna` crate is
correct, and adheres to the IDNA2003/2008 spec, but it turns out
`node`'s implementations don't really follow any spec! Instead, node
splits the domain by `'.'` and punycode encodes/decodes each part. This
means that node's implementations will happily work on codepoints that
are disallowed by the IDNA specs, causing the error in #19214.
While fixing this, I went ahead and matched the node behavior on all of
the punycode functions and enabled node's punycode test in our
`node_compat` suite.
This commit fixes race condition in "node:worker_threads" module were
the first message did a setup of "threadId", "workerData" and
"environmentData".
Now this data is passed explicitly during workers creation and is set up
before any user code is executed.
Closes https://github.com/denoland/deno/issues/22783
Closes https://github.com/denoland/deno/issues/22672
---------
Co-authored-by: Satya Rohith <me@satyarohith.com>
Modify `_http_outgoing.ts` to support the extended signature of
`validateHeaderName()` used since node v19.5.0/v18.14.0 by adding the
`label` parameter. (see:
https://nodejs.org/api/http.html#httpvalidateheadernamename-label)
Making both validation functions accessible as public exports of
`node:http`
Fixes: #22614
Fixes #21660
Adds a basic `Immediate` class to mirror `NodeJS.Immediate`, and changes
`setImmediate` and `clearImmediate` to return and accept (respectively)
`Immediate` objects.
Note that for now {ref,unref,hasRef} are effectively stubs, as deno_core
doesn't really natively support immediates (they're currently modeled as
timers with delay of 0). Eventually we probably want to actually
implement these properly.
This commit changes how we figure out if we're running on main
thread in `node:worker_threads` module. Instead of relying on quirky
"magic variable" for a name to check if we're on main thread, we are
now explicitly passing this information during bootstrapping of the
runtime. As a side effect, `WorkerOptions.name` is more useful
and matches what Node.js does more closely (though not fully).
Towards https://github.com/denoland/deno/issues/22783
This is the release commit being forwarded back to main for 1.41.2
Signed-off-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
1. Stops `deno publish` using some custom include/exclude behaviour from
other sub commands
2. Takes ancestor directories into account when resolving gitignore
3. Backards compatible change that adds ability to unexclude an exclude
by using a negated glob at a more specific level for all sub commands
(see https://github.com/denoland/deno_config/pull/44).
<!--
Before submitting a PR, please read
https://docs.deno.com/runtime/manual/references/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
Fixes https://github.com/denoland/deno/issues/18127https://github.com/denoland/deno/issues/17248
SvelteKit works now!
```
$ deno run -A npm:create-svelte@latest my-app
create-svelte version 6.0.9
┌ Welcome to SvelteKit!
│
◇ Which Svelte app template?
│ SvelteKit demo app
│
◇ Add type checking with TypeScript?
│ Yes, using JavaScript with JSDoc comments
│
◇ Select additional options (use arrow keys/space bar)
│ none
│
└ Your project is ready!
✔ Type-checked JavaScript
https://www.typescriptlang.org/tsconfig#checkJs
Install community-maintained integrations:
https://github.com/svelte-add/svelte-add
Next steps:
1: cd my-app
2: npm install
3: git init && git add -A && git commit -m "Initial commit" (optional)
4: npm run dev -- --open
To close the dev server, hit Ctrl-C
Stuck? Visit us at https://svelte.dev/chat
$ cd my-app/
$ deno task dev
Task dev vite dev
VITE v5.1.4 ready in 1632 ms
➜ Local: http://localhost:5173/
➜ Network: use --host to expose
➜ press h + enter to show help
```
<!--
Before submitting a PR, please read
https://docs.deno.com/runtime/manual/references/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
The issue seems to be already fixed since we upped to a newer 18.x
version string in [another
PR](https://github.com/denoland/deno/pull/20366). Updating to latest
node LTS version can't hurt though.
Fixes https://github.com/denoland/deno/issues/21515
Improves #19100
Fixes #20356
Replaces #20428
Changes made in deno_core to support this:
- [x] Errors must be handled in setTimeout callbacks
- [x] Microtask ordering is not-quite-right
- [x] Timer cancellation must be checked right before dispatch
- [x] Timer sanitizer
- [x] Move high-res timer to deno_core
- [x] Timers need opcall tracing
fixes #22627
This PR fixes a node compat issue that is preventing `serverless-http`
and `serverless-express` npm modules from working with Deno. These
modules are useful for running apps on AWS Lambda (and other serverless
infra).
---------
Signed-off-by: Igor Zinkovsky <igor@deno.com>
Investigating https://github.com/denoland/deno/issues/22574
Unable to reproduce with a unit test, but assuming that it's a name
collision or create pipe/open pipe race, and adding some additional
diagnostics.
Gets us closer to solving #20707.
Rewrites the `TestEventSender`:
- Allow for explicit creation of multiple streams. This will allow for
one-std{out,err}-per-worker
- All test events are received along with a worker ID, allowing for
eventual, proper parallel threading of test events.
In theory this should open up proper interleaving of test output,
however that is left for a future PR.
I had some plans for a better performing synchronization primitive, but
the inter-thread communication is tricky. This does, however, speed up
the processing of large numbers of tests 15-25% (possibly even more on
100,000+).
Before
```
ok | 1000 passed | 0 failed (32ms)
ok | 10000 passed | 0 failed (276ms)
```
After
```
ok | 1000 passed | 0 failed (25ms)
ok | 10000 passed | 0 failed (230ms)
```
When using a prefix or suffix containing an invalid filename character,
it's not entirely clear where the errors come from. We make these errors
more consistent across platforms.
In addition, all permission prompts for tempfile and tempdir were
printing the same API name.
We also take the opportunity to make the tempfile random space larger by
2x (using a base32-encoded u64 rather than a hex-encoded u32).
This change deprecates
`Deno.CreateHttpClientOptions.{certChain,privateKey}` in favour of
`Deno.CreateHttpClientOptions.{cert,key}`.
Closes #22278
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
Splitting the sleep and interval ops allows us to detect an interval
timer. We also remove the use of the `op_async_void_deferred` call.
A future PR will be able to split the op sanitizer messages for timers
and intervals.
Based on #21074 and #20741 I was looking for further potential use cases
of `TransformStream` `cancel()` method, so here go `CompressionStream`
and `DecompressionStream`.
Fixes #14212
`certFile` was being passed to `op_tls_start()` when it's unused and
undocumented.
Signed-off-by: Asher Gomez <ashersaupingomez@gmail.com>
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
Note: tests are not the only part of the codebase that uses `std`. Other
parts, like `tools/`, do too. So, it could be argued that this is a
little misleading. Either way, I'm doing this as discussed with
@mmastrac.
This PR separates integration tests from CLI tests into a new project
named `cli_tests`. This is a prerequisite for an integration test runner
that can work with either the CLI binary in the current project, or one
that is built ahead of time.
## Background
Rust does not have the concept of artifact dependencies yet
(https://github.com/rust-lang/cargo/issues/9096). Because of this, the
only way we can ensure a binary is built before running associated tests
is by hanging tests off the crate with the binary itself.
Unfortunately this means that to run those tests, you _must_ build the
binary and in the case of the deno executable that might be a 10 minute
wait in release mode.
## Implementation
To allow for tests to run with and without the requirement that the
binary is up-to-date, we split the integration tests into a project of
their own. As these tests would not require the binary to build itself
before being run as-is, we add a stub integration `[[test]]` target in
the `cli` project that invokes these tests using `cargo test`.
The stub test runner we add has `harness = false` so that we can get
access to a `main` function. This `main` function's sole job is to
`execvp` the command `cargo test -p deno_cli`, effectively "calling"
another cargo target.
This ensures that the deno executable is always correctly rebuilt before
running the stub test runner from `cli`, and gets us closer to be able
to run the entire integration test suite on arbitrary deno executables
(and therefore split the build into multiple phases).
The new `cli_tests` project lives within `cli` to avoid a large PR. In
later PRs, the test data will be split from the `cli` project. As there
are a few thousand files, it'll be better to do this as a completely
separate PR to avoid noise.
This moves the op sanitizer descriptions into Rust code and prepares for
eventual op import from `ext:core/ops`. We cannot import these ops from
`ext:core/ops` as the testing infrastructure ops are not always present.
Changes:
- Op descriptions live in `cli` code and are currently accessible via an
op for the older sanitizer code
- `phf` dep moved to workspace root so we can use it here
- `ops.op_XXX` changed to to `op_XXX` to prepare for op imports later
on.
Migrations:
- Error registration no longer required for Interrupted or BadResource
(these are core exception)
- `include_js_files!`/`ExtensionFileSource` changes
This commit adds `ppid` getter for `node:process` to improve Node
compatibility one step further.
There is one problem though, which is that `Deno.ppid`, which
`process.ppid` internally calls, is actually of type `bigint` although
it's supposed to be `number`. I filed an issue for this (#22166). For
the time being, explciit type conversion from `bigint` to `number` is
applied to match the Node.js behavior.
Fixes https://github.com/denoland/deno/issues/22180
Matches the Node.js implementation more closely. Removed types, they do
not help just make it harder to debug with stack traces.
Follow up to #22157.
This leaves us with 4 usages of `ensureFastOps()` in `deno` itself.
There's also about 150 usages of `Deno.core.ops.<op_name>` left as well.
For removal in Deno v2. There are two issues:
1. Any script being run causes the output of `warnOnDeprecatedApi()` to
be printed, even when none of the `rid` properties are called.
2. `.rid` of these classes is used in multiple tests. I'm not sure how
to account for that. I thought of having `STDIN_RID`, and friends,
constants, whose values can be shared between the tests and the classes
themselves. Should we go with that or do something else?
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This change:
1. Implements `Deno.FsFile.sync()` and `Deno.FsFile.syncSync()`.
2. Deprecates `Deno.fsync()` and `Deno.fsyncSync()` for removal in Deno
v2, in favour of the above corresponding methods.
Related #21995
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This change:
1. Implements `Deno.FsFile.dataSync()` and `Deno.FsFile.dataSyncSync()`.
2. Deprecates `Deno.fdatasync()` and `Deno.fdatasyncSync()` for removal
in Deno v2, in favour of the above corresponding methods.
3. Replaces use of `Deno.fdatasync()` and `Deno.fdatasyncSync()` with
the above instance methods.
Related #21995
Most uses of `Deno.resources()` within tests, as they previously checked
for leaked resources. This is not needed as the test runner does this
automatically. Other internal uses of this API have been replaced with
the internal `Deno[Deno.internal].core.resources()`.
This change:
1. Implements `Deno.stdin.isTerminal()`, `Deno.stdout.isTerminal()` and
`Deno.stderr.isTerminal()`.
2. Deprecates `Deno.isatty()` for removal in Deno v2, in favour of the
above instance methods.
3. Replaces use of `Deno.isatty()` with the above instance methods.
Related #21995
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
This change:
1. Sets the removal version for `Deno.ListenTlsOptions.certFile`,
`Deno.ListenTlsOptions.keyFile` and `Deno.ConnectTlsOptions.certFile`
for Deno v2, in favour of the `cert`, `key` and `caCerts` options,
respectively.
2. Replaces use of the deprecated options with the new recommended
options.
Towards #22021
This commit deprecates "--unstable" flag.
When "--unstable" flag is encountered a warning like this is printed:
```
The `--unstable` flag is deprecated, use granular `--unstable-*` flags instead.
Learn more at: https://docs.deno.com/runtime/manual/tools/unstable_flags
```
When "--unstable" flag is used and an unstable API is called an
additional warning like this is printed for each API call:
```
The `Deno.dlopen` API was used with `--unstable` flag. The `--unstable` flag is deprecated, use granular `--unstable-ffi` instead.
Learn more at: https://docs.deno.com/runtime/manual/tools/unstable_flags
```
When no "--unstable-*" flag is provided and an unstable API is called
following
warning is issued before exiting:
```
Unstable API 'Deno.dlopen'. The `--unstable-ffi` flag must be provided.
```
---------
Signed-off-by: Divy Srivastava <dj.srivastava23@gmail.com>
Signed-off-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Asher Gomez <ashersaupingomez@gmail.com>
This change removes the currently deprecated `Deno.cron()` overload with
`options` as a potential last argument.
This might be fine to do now, in a major release, as `Deno.cron()` is an
unstable API. I thought of doing this while working on #22021. If this
is not ready to remove, I can instead set the removal version of this
overload for Deno v2.
Note: this overload was deprecated in Deno v1.38.2 (#21225). So it's
been deprecated for over 2 months.
Closes https://github.com/denoland/deno/issues/21828.
This API is a huge footgun. And given that "Deno.serveHttp" is a
deprecated API that is discouraged to use (use "Deno.serve()"
instead); it makes no sense to keep this API around.
This is a step towards fully migrating to Hyper 1.
<!--
Before submitting a PR, please read https://deno.com/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
We were missing the `constants` export in the promise `fs` API which is
available in node.
```ts
import { constants, promises } from "node:fs";
import { constants as fsPromiseConstants } from "node:fs/promises";
console.log(constants === promises.constants); // logs: true
console.log(constants === fsPromiseConstants); // logs: true
```
Fixes https://github.com/denoland/deno/issues/21994
Introduces the first cppgc backed Resource into Deno.
This fixes the memory leak when using `X509Certificate`
**Comparison**:
```js
import { X509Certificate } from 'node:crypto';
const r = Deno.readFileSync('cli/tests/node_compat/test/fixtures/keys/agent1-cert.pem');
setInterval(() => {
for (let i = 0; i < 10000; i++) {
const cert = new X509Certificate(r);
}
}, 1000);
```
Memory usage after 5 secs
`main`: 1692MB
`cppgc`: peaks at 400MB
<!--
Before submitting a PR, please read https://deno.com/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
Ultimately, it came down to using incorrect property names in the
`FakeSocket` constructor. The values are passed under `remoteAddress`,
not `hostname` and `remotePort` instead of `port`.
Fixes https://github.com/denoland/deno/issues/21980
This commit adds support for [Stage 3 Temporal API
proposal](https://tc39.es/proposal-temporal/docs/).
The API is available when `--unstable-temporal` flag is passed.
---------
Signed-off-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: David Sherret <dsherret@gmail.com>
Co-authored-by: Kenta Moriuchi <moriken@kimamass.com>
This commit adds support for "rejectionhandled" Web Event and
"rejectionHandled" Node event.
```js
import process from "node:process";
process.on("rejectionHandled", (promise) => {
console.log("rejectionHandled", reason, promise);
});
window.addEventListener("rejectionhandled", (event) => {
console.log("rejectionhandled", event.reason, event.promise);
});
```
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
<!--
Before submitting a PR, please read https://deno.com/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
Fixes https://github.com/denoland/deno/issues/21840
The problem was hard to reproduce as its a race condition. I've added a
test that reproduces the problem 1/10 tries. We should move the
idleTimeout handling to Rust (maybe even built into fastwebsocket).
<!--
Before submitting a PR, please read https://deno.com/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
Fixes https://github.com/denoland/deno/issues/21836
Part 1 of a potential 3 part series. Ref #13449
The current implementation passes key material back and forth RustCrypto
group of crates and ring. ring does not implement p521 yet.
This PR adds support for P521 named curve in `generateKey` and
`importKey` where we use RustCrypto. Other parts should be moved over to
the RustGroup group of crates for consistency.
Part 1 of #21713
Changes:
- Remove `.present()` and add a `presentGPUCanvasContext` (not exposed
yet to users)
- Move lazy load logic to `00_init.js`. This can be used to use webgpu
on-demand from future code (OffScreenCanvas)
Fixes #20516
Follow up to #21747 and #21746
This tackles the last point of #20516 where certain inputs weren't
accepted in the other zlib methods
This adds the `toU8` conversion of `_brotli` to `_zlib.mjs`, when we
create the ZLibBuffer, we'll sanitize the input. I noticed that the
async had no handler for `string` input so I added that as well.
Fixes https://github.com/denoland/deno/issues/21734
Changes:
- Use default encode when options do not provide a encode callback.
- Remove internal TS for `node:querystring`. Its not helping catching
bugs like this because of invalid type assumptions and use of `any`,
more of a maintenance burden.
This fixes point 3 of https://github.com/denoland/deno/issues/20516
This PR creates consistency between the sync and async versions of the
brotli compress where we will always return a buffer like Node.
This fixes point 2 of #20516
This adds a conversion from Dataview/Buffer by returning `obj.buffer`
which can be converted to a `UInt8Array`.
Question: Regarding point 4 of the mentioned issue would it be
appropriate to copy the toU8 helper to the `zlib.mjs` methods?
Main change is that:
- "hyper" has been renamed to "hyper_v014" to signal that it's legacy
- "hyper1" has been renamed to "hyper" and should be the default
`opAsync` requires a lookup by name on each async call. This is a
mechanical translation of all opAsync calls to ensureFastOps.
The `opAsync` API on Deno.core will be removed at a later time.
When we migrate to op-import-per-extension, we will want to ensure that
ops have one and only one place where they are imported. This tackles
the ops that are imported via `ensureFastOps`, but does not yet tackle
direct `ops` imports.
Landing ahead of https://github.com/denoland/deno_core/pull/393
Node HTTP/2 was using the default h2 `Bytes` datatype when we can be
making using of `BufView` like we do in `Deno.serve`.
`fetch` and `Deno.serverHttp` can't make use of `BufView` because they
are using `reqwest` which is stuck on hyper 0.x at this time.
This PR implements the child_process IPC pipe between parent and child.
The implementation uses Windows named pipes created by parent and passes
the inheritable file handle to the child.
I've also replace parts of the initial implementation which passed the
raw parent fd to JS with resource ids instead. This way no file handle
is exposed to the JS land (both parent and child).
`IpcJsonStreamResource` can stream upto 800MB/s of JSON data on Win 11
AMD Ryzen 7 16GB (without `memchr` vectorization)
Bumped versions for 1.39.0
Please ensure:
- [x] Target branch is correct (`vX.XX` if a patch release, `main` if
minor)
- [x] Crate versions are bumped correctly
- [x] deno_std version is incremented in the code (see
`cli/deno_std.rs`)
- [x] Releases.md is updated correctly (think relevancy and remove
reverts)
To make edits to this PR:
```shell
git fetch upstream release_1_39.0 && git checkout -b release_1_39.0 upstream/release_1_39.0
```
cc @mmastrac
---------
Co-authored-by: mmastrac <mmastrac@users.noreply.github.com>
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
This PR implements the Node child_process IPC functionality in Deno on
Unix systems.
For `fd > 2` a duplex unix pipe is set up between the parent and child
processes. Currently implements data passing via the channel in the JSON
serialization format.
Part 2 of removing middleware.
This is somewhat awkward because `V8CrossThreadTaskSpawner` requires
tasks to be `Send`, but NAPI makes heavy use of `!Send` pointers. In
addition, Rust causes a closure to be `!Send` if you pull a `!Send`
value out of a struct.
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
This problem occurred trying to load tensorflow.js
```
> import * as tf from 'npm:@tensorflow/tfjs';
Uncaught SyntaxError: Identifier 'mod' has already been declared at file:///Users/ry/Library/Caches/deno/npm/registry.npmjs.org/@tensorflow/tfjs/4.14.0/dist/tf.node.js:167:14
at async <anonymous>:1:33
```
This commit adds a no-op flushHeaders method to the ServerResponse
object. It is a nop because the ServerResponse implementation is based
on top of the Deno server API instead of the Node `OutgoingMessage`
base.
Fixes #21509
This significantly optimizes URLPattern in the case where the same
URL is matched against many patterns (like in a router).
Also minor speedups to other use-cases.
This PR is an attempt to fix
https://github.com/denoland/deno/issues/20293, in which node modules
connecting to databases fail due to TLS errors. I ran into this
attempting to use
[node-postgres](https://github.com/brianc/node-postgres) to connect to a
[Neon](https://neon.tech) database.
Investigating via `--inspect-brk` led me to notice that the hostname
eventually passed to `Deno.startTls` was null. The hostname is
determined by the following code:
f6b889b432/ext/node/polyfills/_tls_wrap.ts (L87-L89)
This logic doesn't appear to be correct. I couldn't find reference to
`servername` existing on the `secureContext` in either Node's or Deno's
docs. There's a lot of scope here, and it's my first time reading
through this code, so I could be missing something!
Node uses [the following
logic](2e458d9736/lib/_tls_wrap.js (L1679-L1682)
) to determine the hostname for certificate validation:
```
const hostname = options.servername ||
options.host ||
(options.socket && options.socket._host) ||
'localhost';
```
This PR updates the `TLSSocket` polyfill to use behave similarly (though
I omitted the default to `localhost` at the end; I'm not sure if
including it is necessary or correct). With this change, `node-postgres`
connects to my TLS endpoint successfully (aside: Neon requires SNI,
which also works as expected).
---
I tried to update the tests in
https://github.com/denoland/deno/blob/main/cli/tests/unit_node/tls_test.ts
to exercise this change, but the test fails for me on `main` on Linux. I
investigated briefly and noticed that the test fixture
`cli/tests/testdata/tls/localhost.crt` doesn't appear to include the
`subjectAltName` specified in `domains.txt`. I believe the certificate
isn't matching `localhost`, but that's where I ended investigating.
This commit refactors how we access "core", "internals" and
"primordials" objects coming from `deno_core`, in our internal JavaScript code.
Instead of capturing them from "globalThis.__bootstrap" namespace, we
import them from recently added "ext:core/mod.js" file.
Landing changes required for
https://github.com/denoland/deno_core/pull/359
We needed to update 99_main.js and a whole load of tests.
API changes:
- setPromiseRejectCallback becomes setUnhandledPromiseRejectionHandler.
The function is now called from eventLoopTick.
- The promiseRejectMacrotaskCallback no longer exists, as this is
automatically handled in eventLoopTick.
- ops.op_dispatch_exception now takes a second parameter: in_promise.
The preferred way to call this op is now reportUnhandledException or
reportUnhandledPromiseRejection.
This commit stabilizes "Deno.HttpServer.shutdown" API as well as
Unix socket support in "Deno.serve" API.
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This commit adds a method of `Symbol.dispose` to the object returned
from `Deno.createHttpClient`, so we can make use of [explicit resource
management](https://github.com/tc39/proposal-explicit-resource-management)
by declaring it with `using`.
This commit adds support for a new `kv.watch()` method that allows
watching for changes to a key-value pair. This is useful for cases
where you want to be notified when a key-value pair changes, but
don't want to have to poll for changes.
---------
Co-authored-by: losfair <zhy20000919@hotmail.com>
Add support for signing with a RSA PEM private key: `pkcs8` and `pkcs1`.
Fixes https://github.com/denoland/deno/issues/18972
Ref #21124
Verified fix with `npm:sshpk`. Unverfied but fixes
`npm:google-auth-library`, `npm:web-push` & `oracle/oci-typescript-sdk`
---------
Signed-off-by: Divy Srivastava <dj.srivastava23@gmail.com>
Switch `ext/fetch` over to `resourceForReadableStream` to simplify and
unify implementation with `ext/serve`. This allows us to work in Rust
with resources only.
Two additional changes made to `resourceForReadableStream` were
required:
- Add an optional length to `resourceForReadableStream` which translates
to `size_hint`
- Fix a bug where writing to a closed stream that was full would panic
Rust 1.74 may have made this code temporarily valid in [#113126 Replace
old private-in-public diagnostic with type privacy
lints](https://github.com/rust-lang/rust/pull/113126), so we didn't
catch it at build time.
It fails in 1.73 and +nightly, however.
Follow-up to #20822. cc @lrowe
The `httpServerExplicitResourceManagement` tests were randomly failing
on CI because of a race.
The `drain` waker was missing wakeup events if the listeners shut down
after the last HTTP response finished. If we lost the race (rare), the
server Rc would be dropped and we wouldn't poll it again.
This replaces the drain waker system with a signalling Rc that always
resolves when the refcount is about to become 1.
Fix verified by running serve tests in a loop:
```
for i in {0..100}; do cargo run --features=__http_tracing -- test
-A --unstable '/Users/matt/Documents/github/deno/deno/cli/tests/unit/ser
ve_test.ts' --filter httpServerExplicitResourceManagement; done;
```
This PR changes the `Deno.cron` API:
* Marks the existing function as deprecated
* Introduces 2 new overloads, where the handler arg is always last:
```ts
Deno.cron(
name: string,
schedule: string,
handler: () => Promise<void> | void,
)
Deno.cron(
name: string,
schedule: string,
options?: { backoffSchedule?: number[]; signal?: AbortSignal },
handler: () => Promise<void> | void,
)
```
This PR also fixes a bug, when other crons continue execution after one
of the crons was closed using `signal`.
Fixes #21121 and #19498
Migrates fully to rustls_tokio_stream. We no longer need to maintain our
own TlsStream implementation to properly support duplex.
This should fix a number of errors with TLS and websockets, HTTP and
"other" places where it's failing.
Use HttpRecord as response body so requests can be tracked all the way
to response body completion.
This allows Request properties to be accessed while the response body is
streaming.
Graceful shutdown now awaits a future instead of async spinning waiting
for requests to finish.
On the minimal benchmark this refactor improves performance an
additional 2% over pooling alone for a net 3% increase over the previous
deno main branch.
Builds upon https://github.com/denoland/deno/pull/20809 and
https://github.com/denoland/deno/pull/20770.
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
Reuse existing existing allocations for HttpRecord and response
HeaderMap where possible.
At request end used allocations are returned to the pool and the pool
and the pool sized to 1/8th the current number of inflight requests.
For http1 hyper will reuse the response HeaderMap for the following
request on the connection.
Builds upon https://github.com/denoland/deno/pull/20770
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
Makes the JavaScript Request use a v8:External opaque pointer to
directly refer to the Rust HttpRecord.
The HttpRecord is now reference counted. To avoid leaks the strong count
is checked at request completion.
Performance seems unchanged on the minimal benchmark. 118614 req/s this
branch vs 118564 req/s on main, but variance between runs on my laptop
is pretty high.
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
This PR uses the new `cancel` method of `TransformStream` to properly
clean up the internal `TextDecoder` used in `TextDecoderStream` if the
stream is cancelled.
Fixes #13142
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
We only want one zlib dependency.
Zlib dependencies are reorganized so they use a hidden
`__vendored_zlib_ng` flag in cli that enables zlib-ng for both libz-sys
(used by ext/node) and flate2 (used by deno_web).
This is the release commit being forwarded back to main for 1.38.1
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: littledivy <littledivy@users.noreply.github.com>
We can move all promise ID knowledge to deno_core, allowing us to better
experiment with promise implementation in deno_core.
`{un,}refOpPromise(promise)` is equivalent to
`{un,}refOp(promise[promiseIdSymbol])`
This commit adds granular `--unstable-*` flags:
- "--unstable-broadcast-channel"
- "--unstable-ffi"
- "--unstable-fs"
- "--unstable-http"
- "--unstable-kv"
- "--unstable-net"
- "--unstable-worker-options"
- "--unstable-cron"
These flags are meant to replace a "catch-all" flag - "--unstable", that
gives a binary control whether unstable features are enabled or not. The
downside of this flag that allowing eg. Deno KV API also enables the FFI
API (though the latter is still gated with a permission).
These flags can also be specified in `deno.json` file under `unstable`
key.
Currently, "--unstable" flag works the same way - I will open a follow
up PR that will print a warning when using "--unstable" and suggest to use
concrete "--unstable-*" flag instead. We plan to phase out "--unstable"
completely in Deno 2.
Implements `WebSocket` over http/2. This requires a conformant http/2
server supporting the extended connect protocol.
Passes approximately 100 new WPT tests (mostly `?wpt_flags=h2` versions
of existing websockets APIs).
This is implemented as a fallback when http/1.1 fails, so a server that
supports both h1 and h2 WebSockets will still end up on the http/1.1
upgrade path.
The patch also cleas up the websockets handshake to split it up into
http, https+http1 and https+http2, making it a little less intertwined.
This uncovered a likely bug in the WPT test server:
https://github.com/web-platform-tests/wpt/issues/42896
This PR adds unstable `Deno.cron` API to trigger execution of cron jobs.
* State: All cron state is in memory. Cron jobs are scheduled according
to the cron schedule expression and the current time. No state is
persisted to disk.
* Time zone: Cron expressions specify time in UTC.
* Overlapping executions: not permitted. If the next scheduled execution
time occurs while the same cron job is still executing, the scheduled
execution is skipped.
* Retries: failed jobs are automatically retried until they succeed or
until retry threshold is reached. Retry policy can be optionally
specified using `options.backoffSchedule`.
Use new https://github.com/denoland/rustls-tokio-stream project instead
of tokio-rustls for direct websocket connections. This library was
written from the ground up to be more reliable and should help with
various bugs that may occur due to underlying bugs in the old library.
Believed to fix #20355, #18977, #20948
Towards #20996
`deno_whoami` is lightweight on unix and has zero framework dependency
on macOS. https://github.com/denoland/deno_whoami
---------
Signed-off-by: Divy Srivastava <dj.srivastava23@gmail.com>
Workaround the circular references issue by using a initializer function
to give tty stream class to `initStdin`.
Fixes https://github.com/denoland/deno/issues/21024
Fixes https://github.com/denoland/deno/issues/20611
Fixes https://github.com/denoland/deno/issues/20890
Fixes https://github.com/denoland/deno/issues/20336
`create-svelte` works now:
```
divy@mini /t/a> ~/gh/deno/target/debug/deno run -A --unstable --reload npm:create-svelte@latest sveltekit-deno
create-svelte version 5.1.1
┌ Welcome to SvelteKit!
│
◇ Which Svelte app template?
│ Skeleton project
│
◇ Add type checking with TypeScript?
│ Yes, using JavaScript with JSDoc comments
│
◇ Select additional options (use arrow keys/space bar)
│ none
│
└ Your project is ready!
✔ Type-checked JavaScript
https://www.typescriptlang.org/tsconfig#checkJs
Install community-maintained integrations:
https://github.com/svelte-add/svelte-add
Next steps:
1: cd sveltekit-deno
2: npm install
3: git init && git add -A && git commit -m "Initial commit" (optional)
4: npm run dev -- --open
To close the dev server, hit Ctrl-C
Stuck? Visit us at https://svelte.dev/chat
```
---------
Signed-off-by: Divy Srivastava <dj.srivastava23@gmail.com>
This commit updates the ext/kv module to use the denokv_* crates for
the protocol and the sqlite backend. This also fixes a couple of bugs in
the sqlite backend, and updates versionstamps to be updated less
linearly.
I'm not sure what was the purpose of trying to be so clever with the
args were (maybe an optimization?), but it breaks variadic args as
pointed out in #20054.
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
This commit improves "node:http2" module implementation, by enabling
to use "options.createConnection" callback when starting an HTTP2
session.
This change enables to pass basic client-side test with "grpc-js/grpc"
package.
Smaller fixes like "Http2Session.unref()" and "Http2Session.setTimeout()"
were handled as well.
Fixes #16647
Otherwise you can not return `Deno.Server` from async functions.
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Migrate to op2. Making a few decisions to get this across the line:
- Empty slices, no matter where the come from, are null pointers. The v8
bugs (https://bugs.chromium.org/p/v8/issues/detail?id=13489) and
(https://bugs.chromium.org/p/v8/issues/detail?id=13488) make passing
around zero-length slice pointers too dangerous as they might be
uninitialized or null data.
- Offsets and lengths are `#[number] isize` and `#[number] usize`
respectively -- 53 bits should be enough for anyone
- Pointers are bigints. This is a u64 in the fastcall world, and can
accept Integer/Int32/Number/BigInt v8 types in the slow world.
fixes #20454
Current KV queues implementation assumes that `enqueue` and
`listenQueue` are called on the same instance of `Deno.Kv`. It's
possible that the same Deno process opens multiple KV instances pointing
to the same fs path, and in that case `listenQueue` should still get
notified of messages enqueued through a different KV instance.
For the following example, if I set the encoding to `base64url`, it'll
throw an unexpected TypeError:
```ts
import { Buffer } from "node:buffer";
Buffer.from("IntcImhlbGxvXCI6XCJoZGQvZStpXCJ9Ig", "base64url").toString();
// error: Uncaught TypeError: src.subarray is not a function
// const buf = Buffer.from(
// ^
// at blitBuffer (ext:deno_node/internal/buffer.mjs:1779:15)
// at Uint8Array.base64urlWrite (ext:deno_node/internal/buffer.mjs:691:10)
// at Object.write (ext:deno_node/internal/buffer.mjs:2195:11)
// at Uint8Array.write (ext:deno_node/internal/buffer.mjs:794:14)
// at fromString (ext:deno_node/internal/buffer.mjs:214:22)
// at _from (ext:deno_node/internal/buffer.mjs:119:12)
// at Function.from (ext:deno_node/internal/buffer.mjs:157:10)
// at file:///Users/foodieats/temp/buffer1.ts:3:20
```
The error caused by `base64urlWrite` function, it should call
`forgivingBase64UrlDecode` not `forgivingBase64UrlEncode`
Also fixed #20563 .
We can go one level down in abstraction and avoid using the public
`ReadableStream` APIs.
This patch ~5% perf boost on small ReadableStream:
```
Running 10s test @ http://localhost:8080/
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 148.32us 108.95us 3.88ms 95.71%
Req/Sec 33.24k 2.68k 37.94k 73.76%
668188 requests in 10.10s, 77.74MB read
Requests/sec: 66162.91
Transfer/sec: 7.70MB
```
main:
```
Running 10s test @ http://localhost:8080/
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 150.23us 67.61us 4.39ms 94.80%
Req/Sec 31.81k 1.55k 35.56k 83.17%
639078 requests in 10.10s, 74.36MB read
Requests/sec: 63273.72
Transfer/sec: 7.36MB
```
Fixes: #20569 by introducing a custom replacement for the tokio mpsc
channel that is byte-size backpressure-aware.
Using the testcase in the linked bug, we see all the small writes
aggregated into a single packet and HTTP frame.
```
10:39 $ nc localhost 8000
GET / HTTP/1.1
HTTP/1.1 200 OK
content-type: text/plain
vary: Accept-Encoding
transfer-encoding: chunked
date: Tue, 19 Sep 2023 16:39:13 GMT
A
0
1
2
3
4
```
This patch:
```
Running 10s test @ http://localhost:8080/
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 157.47us 194.89us 9.53ms 98.97%
Req/Sec 31.37k 1.56k 34.73k 85.15%
630407 requests in 10.10s, 73.35MB read
Requests/sec: 62428.12
Transfer/sec: 7.26MB
```
main:
```
Running 10s test @ http://localhost:8080/
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 343.75us 200.48us 10.41ms 98.25%
Req/Sec 14.64k 806.52 16.98k 84.65%
294018 requests in 10.10s, 39.82MB read
Requests/sec: 29109.91
Transfer/sec: 3.94MB
```
---------
Co-authored-by: Bert Belder <bertbelder@gmail.com>
This commit improves async op sanitizer speed by only delaying metrics
collection if there are pending ops. This
results in a speedup of around 30% for small CPU bound unit tests.
It performs this check and possible delay on every collection now,
fixing an issue with parent test leaks into steps.
This commit adds "deno jupyter" subcommand which
provides a Deno kernel for Jupyter notebooks.
The implementation is mostly based on Deno's REPL and
reuses large parts of it (though there's some clean up that
needs to happen in follow up PRs). Not all functionality of
Jupyter kernel is implemented and some message type
are still not implemented (eg. "inspect_request") but
the kernel is fully working and provides all the capatibilities
that the Deno REPL has; including TypeScript transpilation
and npm packages support.
Closes https://github.com/denoland/deno/issues/13016
---------
Co-authored-by: Adam Powers <apowers@ato.ms>
Co-authored-by: Kyle Kelley <rgbkrk@gmail.com>
This commit improves compatibility of "node:http2" module by polyfilling
"connect" method and "ClientHttp2Session" class. Basic operations like
streaming, header and trailer handling are working correctly.
Refing/unrefing is still a TODO and "npm:grpc-js/grpc" is not yet working
correctly.
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
