This moves the op sanitizer descriptions into Rust code and prepares for
eventual op import from `ext:core/ops`. We cannot import these ops from
`ext:core/ops` as the testing infrastructure ops are not always present.
Changes:
- Op descriptions live in `cli` code and are currently accessible via an
op for the older sanitizer code
- `phf` dep moved to workspace root so we can use it here
- `ops.op_XXX` changed to to `op_XXX` to prepare for op imports later
on.
Migrations:
- Error registration no longer required for Interrupted or BadResource
(these are core exception)
- `include_js_files!`/`ExtensionFileSource` changes
Introduces the first cppgc backed Resource into Deno.
This fixes the memory leak when using `X509Certificate`
**Comparison**:
```js
import { X509Certificate } from 'node:crypto';
const r = Deno.readFileSync('cli/tests/node_compat/test/fixtures/keys/agent1-cert.pem');
setInterval(() => {
for (let i = 0; i < 10000; i++) {
const cert = new X509Certificate(r);
}
}, 1000);
```
Memory usage after 5 secs
`main`: 1692MB
`cppgc`: peaks at 400MB
Part 1 of #21713
Changes:
- Remove `.present()` and add a `presentGPUCanvasContext` (not exposed
yet to users)
- Move lazy load logic to `00_init.js`. This can be used to use webgpu
on-demand from future code (OffScreenCanvas)
Main change is that:
- "hyper" has been renamed to "hyper_v014" to signal that it's legacy
- "hyper1" has been renamed to "hyper" and should be the default
Bumped versions for 1.39.0
Please ensure:
- [x] Target branch is correct (`vX.XX` if a patch release, `main` if
minor)
- [x] Crate versions are bumped correctly
- [x] deno_std version is incremented in the code (see
`cli/deno_std.rs`)
- [x] Releases.md is updated correctly (think relevancy and remove
reverts)
To make edits to this PR:
```shell
git fetch upstream release_1_39.0 && git checkout -b release_1_39.0 upstream/release_1_39.0
```
cc @mmastrac
---------
Co-authored-by: mmastrac <mmastrac@users.noreply.github.com>
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
This commit brings back usage of primordials in "40_testing.js" by
turning it back into an ES module and using new "lazy loading" functionality
of ES modules coming from "deno_core".
The same approach was applied to "40_jupyter.js".
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Landing changes required for
https://github.com/denoland/deno_core/pull/359
We needed to update 99_main.js and a whole load of tests.
API changes:
- setPromiseRejectCallback becomes setUnhandledPromiseRejectionHandler.
The function is now called from eventLoopTick.
- The promiseRejectMacrotaskCallback no longer exists, as this is
automatically handled in eventLoopTick.
- ops.op_dispatch_exception now takes a second parameter: in_promise.
The preferred way to call this op is now reportUnhandledException or
reportUnhandledPromiseRejection.
This commit adds support for a new `kv.watch()` method that allows
watching for changes to a key-value pair. This is useful for cases
where you want to be notified when a key-value pair changes, but
don't want to have to poll for changes.
---------
Co-authored-by: losfair <zhy20000919@hotmail.com>
Rust 1.74 may have made this code temporarily valid in [#113126 Replace
old private-in-public diagnostic with type privacy
lints](https://github.com/rust-lang/rust/pull/113126), so we didn't
catch it at build time.
It fails in 1.73 and +nightly, however.
Fixes #21121 and #19498
Migrates fully to rustls_tokio_stream. We no longer need to maintain our
own TlsStream implementation to properly support duplex.
This should fix a number of errors with TLS and websockets, HTTP and
"other" places where it's failing.
We only want one zlib dependency.
Zlib dependencies are reorganized so they use a hidden
`__vendored_zlib_ng` flag in cli that enables zlib-ng for both libz-sys
(used by ext/node) and flate2 (used by deno_web).
This is the release commit being forwarded back to main for 1.38.1
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: littledivy <littledivy@users.noreply.github.com>
We can move all promise ID knowledge to deno_core, allowing us to better
experiment with promise implementation in deno_core.
`{un,}refOpPromise(promise)` is equivalent to
`{un,}refOp(promise[promiseIdSymbol])`
Remove tokio-rustls as a direct dependency of Deno and refactor
test_server to reduce code duplication.
All tcp and tls listener paths go through the same streams now, with the
exception of the simpler Hyper http-only handlers (those can be done in
a later follow-up).
Minor bugs fixed:
- gRPC server should only serve h2
- WebSocket over http/2 had a port overlap
- Restored missing eye-catchers for some servers (still missing on Hyper
ones)
Implements `WebSocket` over http/2. This requires a conformant http/2
server supporting the extended connect protocol.
Passes approximately 100 new WPT tests (mostly `?wpt_flags=h2` versions
of existing websockets APIs).
This is implemented as a fallback when http/1.1 fails, so a server that
supports both h1 and h2 WebSockets will still end up on the http/1.1
upgrade path.
The patch also cleas up the websockets handshake to split it up into
http, https+http1 and https+http2, making it a little less intertwined.
This uncovered a likely bug in the WPT test server:
https://github.com/web-platform-tests/wpt/issues/42896
This PR adds unstable `Deno.cron` API to trigger execution of cron jobs.
* State: All cron state is in memory. Cron jobs are scheduled according
to the cron schedule expression and the current time. No state is
persisted to disk.
* Time zone: Cron expressions specify time in UTC.
* Overlapping executions: not permitted. If the next scheduled execution
time occurs while the same cron job is still executing, the scheduled
execution is skipped.
* Retries: failed jobs are automatically retried until they succeed or
until retry threshold is reached. Retry policy can be optionally
specified using `options.backoffSchedule`.
Use new https://github.com/denoland/rustls-tokio-stream project instead
of tokio-rustls for direct websocket connections. This library was
written from the ground up to be more reliable and should help with
various bugs that may occur due to underlying bugs in the old library.
Believed to fix #20355, #18977, #20948
This commit updates the ext/kv module to use the denokv_* crates for
the protocol and the sqlite backend. This also fixes a couple of bugs in
the sqlite backend, and updates versionstamps to be updated less
linearly.
Adds an experimental unstable built-in package manager to Deno, but it is
currently not usable because the registry infrastructure hasn't been
setup and it points to a non-existent url by default. The default
registry url can be configured via the `DENO_REGISTRY_URL` environment
variable.
This commit adds "deno jupyter" subcommand which
provides a Deno kernel for Jupyter notebooks.
The implementation is mostly based on Deno's REPL and
reuses large parts of it (though there's some clean up that
needs to happen in follow up PRs). Not all functionality of
Jupyter kernel is implemented and some message type
are still not implemented (eg. "inspect_request") but
the kernel is fully working and provides all the capatibilities
that the Deno REPL has; including TypeScript transpilation
and npm packages support.
Closes https://github.com/denoland/deno/issues/13016
---------
Co-authored-by: Adam Powers <apowers@ato.ms>
Co-authored-by: Kyle Kelley <rgbkrk@gmail.com>
This commit improves compatibility of "node:http2" module by polyfilling
"connect" method and "ClientHttp2Session" class. Basic operations like
streaming, header and trailer handling are working correctly.
Refing/unrefing is still a TODO and "npm:grpc-js/grpc" is not yet working
correctly.
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
This adds the ability to pattern match unordered lines. For example, the
downloading messages may appear in any order
```
[UNORDERED_START]
Download https://localhost:4546/a.ts
Download https://localhost:4546/b.ts
[UNORDERED_END]
Hello!
```
Additionally, I've made the pattern matching slightly more strict and the output better.
This PR implements a graceful shutdown API for Deno.serve, allowing all
current connections to drain from the server before shutting down, while
preventing new connections from being started or new transactions on
existing connections from being created.
We split the cancellation handle into two parts: a listener handle, and
a connection handle. A graceful shutdown cancels the listener only,
while allowing the connections to drain. The connection handle aborts
all futures. If the listener handle is cancelled, we put the connections
into graceful shutdown mode, which disables keep-alive on http/1.1 and
uses http/2 mechanisms for http/2 connections.
In addition, we now guarantee that all connections are complete or
cancelled, and all resources are cleaned up when the server `finished`
promise resolves -- we use a Rust-side server refcount for this.
Performance impact: does not appear to affect basic serving performance
by more than 1% (~126k -> ~125k)
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This allows us to opt in to extremely detailed tracing from dependency
libraries, like so:
```
cargo run --features tracing/log,tracing/max_level_trace -- test --log-level=trace -A --unstable ./cli/tests/unit/serve_test.ts
```
It will not impact normal operation as it requires the
`tracing/max_level_trace` and `tracing/log` to be active.
Note that tracing is already a dependency -- this just makes it a direct
dep of cli so we can access its features more easily.
This patch adds a `remote` backend for `ext/kv`. This supports
connection to Deno Deploy and potentially other services compatible with
the KV Connect protocol.
This commit moves `snapshot_from_lockfile` function to [deno_npm
crate](https://github.com/denoland/deno_npm). This allows this function
to be called outside Deno CLI (in particular, Deno Deploy).
This changes the design of the manifest.json file to have a separate
"folders" map for mapping hashed directories. This allows, for example,
to add files in a folder like `http_localhost_8000/#testing_5de71/` and
have them be resolved automatically as long as their remaining
components are identity-mappable to the file system (not hashed). It
also saves space in the manifest.json file by only including the hashed
directory instead of each descendant file.
```
// manifest.json
{
"folders": {
"https://localhost/NOT_MAPPABLE/": "localhost/#not_mappable_5cefgh"
},
"modules": {
"https://localhost/folder/file": {
"headers": {
"content-type": "application/javascript"
}
},
}
}
// folder structure
localhost
- folder
- #file_2defn (note: I've made up the hashes in these examples)
- #not_mappable_5cefgh
- mod.ts
- etc.ts
- more_files.ts
```
This bumps `async-compression` dependency in `deno_http` to latest, in
order to avoid having multiple duplicate versions.
Related, it also unpin a stale `flate2` dependency so that the whole
chain of `async-compression` -> `flate2` -> `miniz_oxide` can surface up
to current versions.
The lockfile entries for all of the above crates have been update
accordingly; the new tree of dependencies looks like this:
```
$ cargo tree -i -p miniz_oxide
miniz_oxide v0.7.1
└── flate2 v1.0.26
└── async-compression v0.4.1
```
This is a new op system that will eventually replace `#[op]`.
Features
- More maintainable, generally less-coupled code
- More modern Rust proc-macro libraries
- Enforces correct `fast` labelling for fast ops, allowing for visual
scanning of fast ops
- Explicit marking of `#[string]`, `#[serde]` and `#[smi]` parameters.
This first version of op2 supports integer and Option<integer>
parameters only, and allows us to start working on converting ops and
adding features.
This commit adds ability to print metrics of the Tokio
runtime to the console by passing "DENO_TOKIO_METRICS=1"
env var.
Metrics will be printed every second, but this can be changed
by "DENO_TOKIO_METRICS_INTERVAL=500" env var.
This runs our `js_unit_tests` and `node_unit_tests` in parallel, one
rust test per JS unit test file. Some of our JS tests don't like running
in parallel due to port requirements, so this also makes those use a
specific port-per-file. This does not attempt to make the node-compat
tests work.
This reverts commit 798c1ad0f1.
Reverting because this change caused a spike in memory usage, but we
can't fully realise gains from lower GC pressure from more optimal
malloc/ free provided by "jemalloc".
We might revisit the topic in future.
**THIS PR HAS GIT CONFLICTS THAT MUST BE RESOLVED**
This is the release commit being forwarded back to main for 1.33.4
Please ensure:
- [x] Everything looks ok in the PR
- [ ] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.33.4 && git checkout -b forward_v1.33.4 upstream/forward_v1.33.4
```
Don't need this PR? Close it.
cc @levex
Co-authored-by: levex <levex@users.noreply.github.com>
Co-authored-by: Levente Kurusa <lkurusa@kernelstuff.org>
Note: If the package information has already been cached, then this
requires running with `--reload` or for the registry information to be
fetched some other way (ex. the cache busting).
Closes #15544
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
If a timer is requested with <=100ms resolution, request the high-res
timer. Since the default Windows timer period is 15ms, this means a
100ms timer could fire at 115ms (15% late). We assume that timers longer
than 100ms are a reasonable cutoff here.
The high-res timers on Windows are still limited. Unfortuntely this
means that our shortest duration 4ms timers can still be 25% late, but
without a more complex timer system or spinning on the clock itself,
we're somewhat bounded by the OS' scheduler itself.
**THIS PR HAS GIT CONFLICTS THAT MUST BE RESOLVED**
This is the release commit being forwarded back to main for 1.33.3
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.33.3 && git checkout -b forward_v1.33.3 upstream/forward_v1.33.3
```
Don't need this PR? Close it.
cc @levex
Co-authored-by: Levente Kurusa <lkurusa@kernelstuff.org>
Upgrade `thiserror` to `1.40.0`.
Remove version pinning so that consumers of deno crates can install
newer versions of `thiserrors` without waiting for us to upgrade our
Cargo.toml.
Upgrade `deno_lockfile` to `0.14.0` to bring in `thiserror` upgrade, see
https://github.com/denoland/deno_lockfile/pull/1.
1. Boxed `File` and `FileSystem` to allow more easily passing this
through the CLI code (as shown within this pr).
2. `StdFileResource` is now `FileResource`. `FileResource` now contains
an `Rc<dyn File>`.
**THIS PR HAS GIT CONFLICTS THAT MUST BE RESOLVED**
This is the release commit being forwarded back to main for 1.33.2
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.33.2 && git checkout -b forward_v1.33.2 upstream/forward_v1.33.2
```
Don't need this PR? Close it.
cc @levex
Co-authored-by: levex <levex@users.noreply.github.com>
Co-authored-by: Levente Kurusa <lkurusa@kernelstuff.org>
This is a rewrite of the `Deno.serve` API to live on top of hyper
1.0-rc3. The code should be more maintainable long-term, and avoids some
of the slower mpsc patterns that made the older code less efficient than
it could have been.
Missing features:
- `upgradeHttp` and `upgradeHttpRaw` (`upgradeWebSocket` is available,
however).
- Automatic compression is unavailable on responses.
This is just a straight refactor and I didn't do any cleanup in
ext/node. After this PR we can start to clean it up and make things
private that don't need to be public anymore.
- bump deps: the newest `lazy-regex` need newer `oncecell` and
`regex`
- reduce `unwrap`
- remove dep `lazy_static`
- make more regex cached
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds new "op_http_upgrade_early", that allows to hijack
existing "Deno.HttpConn" acquired from "Deno.serveHttp" API
and performing a Websocket upgrade on this connection.
This is not a public API and is meant to be used internally in the
"ext/node" polyfills for "http" module.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
1. Rewrites the tests to be more back and forth rather than getting the
output all at once (which I believe was causing the hangs on linux and
maybe mac)
2. Runs the pty tests on the linux ci.
3. Fixes a bunch of tests that were just wrong.
4. Adds timeouts on the pty tests.
Towards #18455
This commit implements `checkPrimeSync` and `checkPrime` in node:crypto
using the Miller-Rabin primality test (fun fact: it actually is a test
for composite numbers)
It first compares the candidate against many known small primes and if
not, proceeds to run the Miller-Rabin primality test.
http://nickle.org/examples/miller-rabin.5c used as reference
implementation.
This commit adds unstable "Deno.openKv()" API that allows to open
a key-value database at a specified path.
---------
Co-authored-by: Luca Casonato <hello@lcas.dev>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit disables compression of the TSC snapshot.
The compression only decreased the size of snapshot by 0.5Mb
and it took about 40s during release build to compress.
With recent gains in TS 5.0 upgrade in terms of size and performance
it makes sense to remove this compression.
This commit removes compression for the runtime JS code.
It means that we will have a bigger binary, but faster startup. After
several discussion in the CLI team we decided it's worth to trade
about 3Mb of binary size for 2ms faster startup time. With WebGPU
removed in 35196eab27
it shouldn't have such a big impact on the binary size.
This PR _**temporarily**_ removes WebGPU (which has behind the
`--unstable` flag in Deno), due to performance complications due to its
presence.
It will be brought back in the future; as a point of reference, Chrome
will ship WebGPU to stable on 26/04/2023.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This is implemented in such a way that it should still allow processes
to go through when a file lock wasn't properly cleaned up and the OS
hasn't released it yet (but with a 200ms-ish delay).
Closes #18039
This is a super basic initial implementation. We don't create a
`node_modules/.bin` folder at the moment and add it to the PATH like we
should which is necessary to make command name resolution in the
subprocess work properly (ex. you run a script that launches another
script that then tries to launch an "npx command"... this won't work
atm).
Closes #17492
Bump the rsa crate to 0.7.0
The API for the `rsa` crate has changed significantly, but I have
verified that tests continue to pass throughout this update.
This PR implements the NAPI for loading native modules into Deno.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: DjDeveloper <43033058+DjDeveloperr@users.noreply.github.com>
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
