Rewrites 3 ops that used "op(deferred)" to use "op2(async(lazy))"
instead.
This will allow us to remove codepath for handling "deferred" ops in
"deno_core".
The fix for #20188 was not entirely correct -- we were unlocking the
global buffer incorrectly. This PR introduces a lock state that ensures
we only unlock a lock we have taken out.
When a TCP connection is force-closed (ie: browser refresh), the
underlying future we pass to Hyper is dropped which may cause us to try
to drop the body resource while the OpState lock is still held.
Preconditions for this bug to trigger:
- The body resource must have been taken
- The response must return a resource (which requires us to take the
OpState lock)
- The TCP connection must have been dropped before this
Fixes #20315 and #20298
<!--
Before submitting a PR, please read https://deno.com/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
As the title.
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
This patch adds a `remote` backend for `ext/kv`. This supports
connection to Deno Deploy and potentially other services compatible with
the KV Connect protocol.
Deno.serve's fast streaming implementation was not keeping the request
body resource ID alive. We were taking the `Rc<Resource>` from the
resource table during the response, so a hairpin duplex response that
fed back the request body would work.
However, if any JS code attempted to read from the request body (which
requires the resource ID to be valid), the response would fail with a
difficult-to-diagnose "EOF" error.
This was affecting more complex duplex uses of `Deno.fetch` (though as
far as I can tell was unreported).
Simple test:
```ts
const reader = request.body.getReader();
return new Response(
new ReadableStream({
async pull(controller) {
const { done, value } = await reader.read();
if (done) {
controller.close();
} else {
controller.enqueue(value);
}
},
}),
```
And then attempt to use the stream in duplex mode:
```ts
async function testDuplex(
reader: ReadableStreamDefaultReader<Uint8Array>,
writable: WritableStreamDefaultWriter<Uint8Array>,
) {
await writable.write(new Uint8Array([1]));
const chunk1 = await reader.read();
assert(!chunk1.done);
assertEquals(chunk1.value, new Uint8Array([1]));
await writable.write(new Uint8Array([2]));
const chunk2 = await reader.read();
assert(!chunk2.done);
assertEquals(chunk2.value, new Uint8Array([2]));
await writable.close();
const chunk3 = await reader.read();
assert(chunk3.done);
}
```
In older versions of Deno, this would just lock up. I believe after
23ff0e722e, it started throwing a more
explicit error:
```
httpServerStreamDuplexJavascript => ./cli/tests/unit/serve_test.ts:1339:6
error: TypeError: request or response body error: error reading a body from connection: Connection reset by peer (os error 54)
at async Object.pull (ext:deno_web/06_streams.js:810:27)
```
Properly handle the `SQLITE_BUSY` error code by retrying the
transaction.
Also wraps database initialization logic in a transaction to protect
against incomplete/concurrent initializations.
Fixes https://github.com/denoland/deno/issues/20116.
The goal of this PR is to address issue #20106 where a `TypeError`
occurs when the variables `uid` and `gid` from `userInfo()` in `node:os`
are reassigned if the user is on Windows. Both `uid` and `gid` are
marked as `const` therefore producing a `TypeError` when the two are
reassigned.
This PR achieves that goal by marking `uid` and `gid` as `let`
To fix bugs around detection of when node emulation is required, we will
just eagerly initialize it. The improvements we make to reduce the
impact of the startup time:
- [x] Process stdin/stdout/stderr are lazily created
- [x] node.js global proxy no longer allocates on each access check
- [x] Process checks for `beforeExit` listeners before doing expensive
shutdown work
- [x] Process should avoid adding global event handlers until listeners
are added
Benchmarking this PR (`89de7e1ff`) vs main (`41cad2179`)
```
12:36 $ third_party/prebuilt/mac/hyperfine --warmup 100 -S none './deno-41cad2179 run ./empty.js' './deno-89de7e1ff run ./empty.js'
Benchmark 1: ./deno-41cad2179 run ./empty.js
Time (mean ± σ): 24.3 ms ± 1.6 ms [User: 16.2 ms, System: 6.0 ms]
Range (min … max): 21.1 ms … 29.1 ms 115 runs
Benchmark 2: ./deno-89de7e1ff run ./empty.js
Time (mean ± σ): 24.0 ms ± 1.4 ms [User: 16.3 ms, System: 5.6 ms]
Range (min … max): 21.3 ms … 28.6 ms 126 runs
```
Fixes https://github.com/denoland/deno/issues/20142
Fixes https://github.com/denoland/deno/issues/15826
Fixes https://github.com/denoland/deno/issues/20028
The goal of this PR is to address issue #19520 where Deno panics when
encountering an invalid SSL certificate.
This PR achieves that goal by removing an `.expect()` statement and
implementing a match statement on `tsl_config` (found in
[/ext/net/ops_tsl.rs](e071382768/ext/net/ops_tls.rs (L1058)))
to check whether the desired configuration is valid
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
The original implementation of `Cache` used a custom `shutdown` method
on the resource, but to simplify fast streams work we're going to move
this to an op of its own.
While we're in here, we're going to replace `opAsync` with
`ensureFastOps`. `op2` work will have to wait because of some
limitations to our async support, however.
Rename some of the helper methods on the Fs trait to be suffixed with
`_sync` / `_async`, in preparation of the introduction of more async
methods for some helpers.
Also adds a `read_text_file_async` helper to complement the renamed
`read_text_file_sync` helper.
This PR ensures that the original signal event is fired before any
dependent signal events.
---
The enabled tests fail on `main`:
```
assert_array_equals: Abort events fired in correct order expected property 0 to be
"original-aborted" but got "clone-aborted" (expected array ["original-aborted", "clone-aborted"]
got ["clone-aborted", "original-aborted"])
```
Closes #19399 (running without snapshots at all was suggested as an
alternative solution).
Adds a `__runtime_js_sources` pseudo-private feature to load extension
JS sources at runtime for faster development, instead of building and
loading snapshots or embedding sources in the binary. Will only work in
a development environment obviously.
Try running `cargo test --features __runtime_js_sources
integration::node_unit_tests::os_test`. Then break some behaviour in
`ext/node/polyfills/os.ts` e.g. make `function cpus() {}` return an
empty array, and run it again. Fix and then run again. No more build
time in between.
This bumps `async-compression` dependency in `deno_http` to latest, in
order to avoid having multiple duplicate versions.
Related, it also unpin a stale `flate2` dependency so that the whole
chain of `async-compression` -> `flate2` -> `miniz_oxide` can surface up
to current versions.
The lockfile entries for all of the above crates have been update
accordingly; the new tree of dependencies looks like this:
```
$ cargo tree -i -p miniz_oxide
miniz_oxide v0.7.1
└── flate2 v1.0.26
└── async-compression v0.4.1
```
This tweaks the HTTP response-writer in order to align the two possible
execution flows into using the same gzip default compression level, that
is `1` (otherwise the implicit default level is `6`).
This PR fixes some crashing WPT tests due to an unresolved promise.
---
This could be a [stream spec](https://streams.spec.whatwg.org) bug
When `controller.close` is called on a byob stream, there's no cleanup
of pending `readIntoRequests`. The only cleanup of pending
`readIntoRequests` happen when `.byobRequest.respond(0)` is called, it
happens
here:6ba245fe25/ext/web/06_streams.js (L2026)
which ends up calling `readIntoRequest.closeSteps(chunk);` in
6ba245fe25/ext/web/06_streams.js (L2070)
To reproduce:
```js
async function byobRead() {
const input = [new Uint8Array([8, 241, 48, 123, 151])];
const stream = new ReadableStream({
type: "bytes",
async pull(controller) {
if(input.length === 0) {
controller.close();
// controller.byobRequest.respond(0); // uncomment for fix
return
}
controller.enqueue(input.shift())
},
});
const reader = stream.getReader({ mode: 'byob' });
const r1 = await reader.read(new Uint8Array(64));
console.log(r1);
const r2 = await reader.read(new Uint8Array(64));
console.log(r2);
}
await byobRead();
```
Running the script triggers:
```
error: Top-level await promise never resolved
```
This is a prerequisite for fast streams work -- this particular resource
used a custom `mpsc`-style stream, and this work will allow us to unify
it with the streams in `ext/http` in time.
Instead of using Option as an internal semaphore for "correctly
completed EOF", we allow code to propagate errors into the channel which
can be picked up by downstream sinks like Hyper. EOF is signalled using
a more standard sender drop.
This commit adds new "--deny-*" permission flags. These are complimentary to
"--allow-*" flags.
These flags can be used to restrict access to certain resources, even if they
were granted using "--allow-*" flags or the "--allow-all" ("-A") flag.
Eg. specifying "--allow-read --deny-read" will result in a permission error,
while "--allow-read --deny-read=/etc" will allow read access to all FS but the
"/etc" directory.
Runtime permissions APIs ("Deno.permissions") were adjusted as well, mainly
by adding, a new "PermissionStatus.partial" field. This field denotes that
while permission might be granted to requested resource, it's only partial (ie.
a "--deny-*" flag was specified that excludes some of the requested resources).
Eg. specifying "--allow-read=foo/ --deny-read=foo/bar" and then querying for
permissions like "Deno.permissions.query({ name: "read", path: "foo/" })"
will return "PermissionStatus { state: "granted", onchange: null, partial: true }",
denoting that some of the subpaths don't have read access.
Closes #18804.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
This commit provides basic polyfill for "node:test" module. Currently
only top-level "test" function is polyfilled, all remaining functions from
that module throw not implemented errors.
This addresses issue #19918.
## Issue description
Event messages have the wrong isTrusted value when they are not
triggered by user interaction, which differs from the browser. In
particular, all MessageEvents created by Deno have isTrusted set to
false, even though it should be true.
This is my first ever contribution to Deno, so I might be missing
something.
Includes a lightly-modified version of hyper-util's `TokioIo` utility.
Hyper changes:
v1.0.0-rc.4 (2023-07-10)
Bug Fixes
http1:
http1 server graceful shutdown fix (#3261)
([f4b51300](f4b513009d))
send error on Incoming body when connection errors (#3256)
([52f19259](52f192593f),
closes https://github.com/hyperium/hyper/issues/3253)
properly end chunked bodies when it was known to be empty (#3254)
([fec64cf0](fec64cf0ab),
closes https://github.com/hyperium/hyper/issues/3252)
Features
client: Make clients able to use non-Send executor (#3184)
([d977f209](d977f209bc),
closes https://github.com/hyperium/hyper/issues/3017)
rt:
replace IO traits with hyper::rt ones (#3230)
([f9f65b7a](f9f65b7aa6),
closes https://github.com/hyperium/hyper/issues/3110)
add downcast on Sleep trait (#3125)
([d92d3917](d92d3917d9),
closes https://github.com/hyperium/hyper/issues/3027)
service: change Service::call to take &self (#3223)
([d894439e](d894439e00),
closes https://github.com/hyperium/hyper/issues/3040)
Breaking Changes
Any IO transport type provided must not implement hyper::rt::{Read,
Write} instead of tokio::io traits. You can grab a helper type from
hyper-util to wrap Tokio types, or implement the traits yourself, if
it's a custom type.
([f9f65b7a](f9f65b7aa6))
client::conn::http2 types now use another generic for an Executor. Code
that names Connection needs to include the additional generic parameter.
([d977f209](d977f209bc))
The Service::call function no longer takes a mutable reference to self.
The FnMut trait bound on the service::util::service_fn function and the
trait bound on the impl for the ServiceFn struct were changed from FnMut
to Fn.
This PR fixes #19818. The problem was that the new InnerRequest class does not initialize the fields urlList and urlListProcessed that are used during a request clone. The solution aims to be straightforward by simply initializing the missing properties during the clone process. I also implemented a "cache" to the url getter of the new InnerRequest, avoiding the cost of calling op_http_get_request_method_and_url.
V8 doesn't like having internal slots on the "real" globalThis object.
This commit works around this limitation by storing the inner globalThis
objects for segregated globals in a context slot.
Benchmarking shows numbers are pretty close, however this is recommended
for the best possible thread-local performance and may improve in future
Rust compiler revisions.
Code run within Deno-mode and Node-mode should have access to a
slightly different set of globals. Previously this was done through a
compile time code-transform for Node-mode, but this is not ideal and has
many edge cases, for example Node's globalThis having a different
identity than Deno's globalThis.
This commit makes the `globalThis` of the entire runtime a semi-proxy.
This proxy returns a different set of globals depending on the caller's
mode. This is not a full proxy, because it is shadowed by "real"
properties on globalThis. This is done to avoid the overhead of a full
proxy for all globalThis operations.
The globals between Deno-mode and Node-mode are now properly segregated.
This means that code running in Deno-mode will not have access to Node's
globals, and vice versa. Deleting a managed global in Deno-mode will
NOT delete the corresponding global in Node-mode, and vice versa.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: Aapo Alasuutari <aapo.alasuutari@gmail.com>
Fixes #19737 by adding brotli compression parameters.
Time after:
`Accept-Encoding: gzip`:
```
real 0m0.214s
user 0m0.005s
sys 0m0.013s
```
`Accept-Encoding: br`:
Before:
```
real 0m10.303s
user 0m0.005s
sys 0m0.010s
```
After:
```
real 0m0.127s
user 0m0.006s
sys 0m0.014s
```
This makes the implementation of "AsyncLocalStorage" from
"node:async_hooks" 3.5x faster than before for noop benchmark
(measuring baseline overhead). It's still 3.5x slower than not
using `AsyncLocalStorage` and 1.64x slower than using
noop promise hooks.
This commit stabilizes "Deno.serve()", which becomes the
preferred way to create HTTP servers in Deno.
Documentation was adjusted for each overload of "Deno.serve()"
API and the API always binds to "127.0.0.1:8000" by default.
This PR changes Web IDL interfaces to be declared with `var` instead of
`class`, so that accessing them via `globalThis` does not raise type
errors.
Closes #13390.
Fixes #19687 by adding a rejection handler to the write inside the
setTimeout. There is a small window where the promise is actually not
awaited and may reject without a handler.
Implementation of generics for `#[op2]`, along with some refactoring to
improve the ergonomics of ops with generics parameters:
- The ops have generics on the struct rather than the associated
methods, which allows us to trait-ify ops (impossible when they are on
the methods)
- The decl() method can become a trait-associated const field which
unlocks future optimizations
Callers of ops need to switch from:
`op_net_connect_tcp::call::<TestPermission>(conn_state, ip_addr)` to
`op_net_connect_tcp::<TestPermission>::call(conn_state, ip_addr)`.
Fixes the WPT tests that test w/invalid codes. Also explicitly ignoring
some h2 tests to hopefully prevent flakes.
The previous changes to WebSocketStream introduced a bug where the close
errors were not made available if the `pull` method was re-entrant.
This is a fix for issue #19644, concerning the `parseCssColor` function
in the file `ext/console/01_console.js`. Changes made on lines
2756-2758. To sum it up:
> The internal `parseCssColor` function currently parses 3/4-digit hex
colors incorrectly. For example, it parses the string `#FFFFFF` as
`[255, 255, 255]` (as expected), but returns `[240, 240, 240]` for
`#FFF`, when it should return the same triplet as the former.
While it's not going to cause a fatal runtime error, it did bug me
enough to fix it real quick.
…nclusion" (#19519)"
This reverts commit 28a4f3d0f5.
This change causes failures when used outside Deno repo:
```
============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.
Platform: linux x86_64
Version: 1.34.3+b37b286
Args: ["/opt/hostedtoolcache/deno/0.0.0-b37b286f7fa68d5656f7c180f6127bdc38cf2cf5/x64/deno", "test", "--doc", "--unstable", "--allow-all", "--coverage=./cov"]
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Failed to read "/home/runner/work/deno/deno/core/00_primordials.js"
Caused by:
No such file or directory (os error 2)', core/runtime/jsruntime.rs:699:8
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
Relands #19463. This time the `ExtensionFileSourceCode` enum is
preserved, so this effectively just splits feature
`include_js_for_snapshotting` into `exclude_js_sources` and
`runtime_js_sources`, adds a `force_include_js_sources` option on
`extension!()`, and unifies `ext::Init_ops_and_esm()` and
`ext::init_ops()` into `ext::init()`.
This is a new op system that will eventually replace `#[op]`.
Features
- More maintainable, generally less-coupled code
- More modern Rust proc-macro libraries
- Enforces correct `fast` labelling for fast ops, allowing for visual
scanning of fast ops
- Explicit marking of `#[string]`, `#[serde]` and `#[smi]` parameters.
This first version of op2 supports integer and Option<integer>
parameters only, and allows us to start working on converting ops and
adding features.
The WHATWG DOM specification has corrected the spelling of "slotable" to
"slottable".[1] This commit aligns our implementation accordingly.
[1]: https://github.com/whatwg/dom/pull/845
Fixes #19568
Values are not coerced to the desired type during deserialisation. This
makes serde_v8 stricter.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
`ZeroCopyBuf` was convenient to use, but sometimes it did hide details
that some copies were necessary in certain cases. Also it made it way to easy
for the caller to pass around and convert into different values. This commit
splits `ZeroCopyBuf` into `JsBuffer` (an array buffer coming from V8) and
`ToJsBuffer` (a Rust buffer that will be converted into a V8 array buffer).
As a result some magical conversions were removed (they were never used)
limiting the API surface and preparing for changes in #19534.
Fixes a bug I noticed when deriving a key based from `ECDH`. Similar
issue is also mentioned in #14693, where they derive a key using
`PBKDF2`
- In the WebCrypto API, `deriveKey()` is equivalent to `deriveBits()`
followed by `importKey()`
- But, `deriveKey()` requires just `deriveKey` in the `usages` of the
`baseKey` parameter. The `deriveBits` usage is not required to be
allowed. This is the uniform behaviour in Node, Chrome and Firefox.
- The impl currently has userland-accessible `SubtleCrypto.deriveKey()`
and `SubtleCrypto.deriveBits()`, as well as an internal `deriveBits()`
(this is the one that accesses the ffi).
- Also, `SubtleCrypto.deriveKey()` checks if `deriveKey` is an allowed
usage and `SubtleCrypto.deriveBits()` checks if `deriveBits` is an
allowed usage, as required.
- However, the impl currently calls the userland accessible
`SubtleCrypto.deriveBits()` in `SubtleCrypto.deriveKey()`, leading to an
error being thrown if the `deriveBits` usage isn't present.
- Fixed this by making it call the internal `deriveBits()`
instead.
… (#19463)"
This reverts commit ceb03cfb03.
This is being reverted because it causes 3.5Mb increase in the binary
size,
due to runtime JS code being included in the binary, even though it's
already snapshotted.
CC @nayeemrmn
Use `Map` to cache validated HTTP headers. Cache
has a capacity of 4096 elements and it's cleared
once that capacity is reached.
In `preactssr` benchmark it lowers the time spent
when adding headers from 180ms to 2.5ms.
Remove `ExtensionFileSourceCode::LoadedFromFsDuringSnapshot` and feature
`include_js_for_snapshotting` since they leak paths that are only
applicable in this repo to embedders. Replace with feature
`exclude_js_sources`. Additionally the feature
`force_include_js_sources` allows negating it, if both features are set.
We need both of these because features are additive and there must be a
way of force including sources for snapshot creation while still having
the `exclude_js_sources` feature. `force_include_js_sources` is only set
for build deps, so sources are still excluded from the final binary.
You can also specify `force_include_js_sources` on any extension to
override the above features for that extension. Towards #19398.
But there was still the snapshot-from-snapshot situation where code
could be executed twice, I addressed that by making `mod_evaluate()` and
scripts like `core/01_core.js` behave idempotently. This allowed
unifying `ext::init_ops()` and `ext::init_ops_and_esm()` into
`ext::init()`.
Reduce the GC pressure from the websocket event method by splitting it
into an event getter and a buffer getter.
Before:
165.9k msg/sec
After:
169.9k msg/sec
This switches syscall used in HTTP and WS server from "writev"
to "sendto".
"DENO_USE_WRITEV=1" can be used to enable using "writev" syscall.
Doing this for easier testing of various setups.
This commit migrates "deno_core" from using "FuturesUnordered" to
"tokio::task::JoinSet". This makes every op to be a separate Tokio task
and should unlock better utilization of kqueue/epoll.
There were two quirks added to this PR:
- because of the fact that "JoinSet" immediately polls spawn tasks,
op sanitizers can give false positives in some cases, this was
alleviated by polling event loop once before running a test with
"deno test", which gives canceled ops an opportunity to settle
- "JsRuntimeState::waker" was moved to "OpState::waker" so that FFI
API can still use threadsafe functions - without this change the
registered wakers were wrong as they would not wake up the
whole "JsRuntime" but the task associated with an op
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
Using `deopt-explorer` I found that a bunch of fields on `WebSocket`
class were polymorphic.
Fortunately it was enough to initialize them to `undefined`
to fix the problem.
No need to go through the async machinery for `send(String | Buffer)` --
we can fire and forget, and then route any send errors into the async
call we're already making (`op_ws_next_event`).
Early benchmark on MacOS:
Before: 155.8k msg/sec
After: 166.2k msg/sec (+6.6%)
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds basic support for "node:http2" module. Not
all APIs have been yet implemented, but this change already
allows to use this module for some basic functions.
The "grpc" package is still not working, but it's a good stepping
stone.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
## WHY
ref: https://github.com/denoland/deno/issues/19165
The FileHandle class has many missing methods compared to node.
Add these.
## WHAT
- Add close method
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
Related issue: https://github.com/denoland/deno/issues/19358.
This is a regression that seems to have been introduced in
https://github.com/denoland/deno/pull/18905. It looks to have been a
performance optimization.
The issue is probably easiest described with some code:
```ts
const target = new EventTarget();
const event = new Event("foo");
target.addEventListener("foo", () => {
console.log('base');
target.addEventListener("foo", () => {
console.log('nested');
});
});
target.dispatchEvent(event);
```
Essentially, the second event listener is being attached while the `foo`
event is still being dispatched. It should then not fire that second
event listener, but Deno currently does.
`rusqlite` does not support async operations; with this PR SQLite
operations will run through `spawn_blocking` to ensure that the event
loop does not get blocked.
There is still only a single SQLite connection. So all operations will
do an async wait on the connection. In the future we can add a
connection pool if needed.
This PR attempts to resolve the first item on the list from
https://github.com/denoland/deno/issues/19330 which is about using a
flat list of interleaved key/value pairs, instead of a nested array of
tuples.
I can tackle some more if you can provide a quick example of using raw
v8 arrays, cc @mmastrac
<!--
Before submitting a PR, please read https://deno.com/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
Internally, `node-tap` spawns a child process with `stdio: [0, 1, 2]`.
Whilst we don't support passing fd numbers as an argument so far, it
turns out that `[0, 1, 2]` is equivalent to `"inherit"` which we already
support. See: https://nodejs.org/api/child_process.html#optionsstdio
Mapping it to `"inherit"` is fine for us and gets us one step closer in
getting `node-tap` working. I'm now at the stage where already the
coverage table is shown 🎉
<!--
Before submitting a PR, please read https://deno.com/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
## WHY
ref: https://github.com/denoland/deno/issues/19165
Node's fs/promises includes a FileHandle class, but deno does not. The
open function in Node's fs/promises returns a FileHandle, which provides
an IO interface to the file. However, deno's open function returns a
resource id.
### deno
```js
> const fs = await import("node:fs/promises");
undefined
> const file3 = await fs.open("./README.md");
undefined
> file3
3
> file3.read
undefined
Node:
```
### Node
```js
> const fs = await import("fs/promises");
undefined
> const file3 = await fs.open("./tests/e2e_unit/testdata/file.txt");
undefined
> file3
FileHandle {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
close: [Function: close],
[Symbol(kCapture)]: false,
[Symbol(kHandle)]: FileHandle {},
[Symbol(kFd)]: 24,
[Symbol(kRefs)]: 1,
[Symbol(kClosePromise)]: null
}
> file3.read
[Function: read]
```
To be compatible with Node, deno's open function should also return a
FileHandle.
## WHAT
I have implemented the first step in adding a FileHandle.
- Changed the return value of the open function to a FileHandle object
- Implemented the readFile method in FileHandle
- Add test code
## What to do next
This PR is the first step in adding a FileHandle, and there are things
that should be done next.
- Add functionality equivalent to Node's FileHandle to FileHandle
(currently there is only readFile)
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
For the first implementation of node:http2, we'll use the internal
version of `Deno.serve` which allows us to listen on a raw TCP
connection rather than a listener.
This is mostly a refactoring, and hooking up of `op_http_serve_on` that
was never previously exposed (but designed for this purpose).
This PR adds the missing `process.reallyExit()` method to node's
`process` object.
Was [pinged on
twitter](https://twitter.com/biwanczuk/status/1663326659787862017)
regarding running the `fastify` test suite in node. They use `node-tap`
which has been around arguably the longest of the test frameworks and
relies on a couple of old APIs. They have `signal-exit` as a dependency
which in turn [makes use of
`process.reallyExit()`](8fa7fc9a9c/src/index.ts (L19)).
That function cannot be found anywhere in their documentation, but
exists at runtime. See
6a6b3c5402/lib/internal/bootstrap/node.js (L172)
This doesn't yet make `node-tap` work, but gets us one step closer.
Under heavy load, we often have requests queued up that don't need an
async call to retrieve. We can use a fast path sync op to drain this set
of ready requests, and then fall back to the async op once we run out of
work.
This is a .5-1% bump in req/s on an M2 mac. About 90% of the handlers go
through this sync phase (based on a simple instrumentation that is not
included in this PR) and skip the async machinery entirely.
Rather than disallowing `ext:` resolution, clear the module map after
initializing extensions so extension modules are anonymized. This
operation is explicitly called in `deno_runtime`. Re-inject `node:`
specifiers into the module map after doing this.
Fixes #17717.
This commit fixes problem with loading N-API modules that use
the "old" way of registration (using "napi_module_register" API).
The slot was not cleared after loading modules, causing subsequent
calls that use the new way of registration (using
"napi_register_module_v1" API) to try and load the previous module.
Ref https://github.com/denoland/deno/issues/16460
---------
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
`isFile`, `isDirectory`, `isSymlink` are defined in `Deno.FileInfo`, but
`isBlockDevice`, `isCharacterDevice`, `isFIFO`, `isSocket` are not
defined.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit fixes emitting "unhandledrejection" event when there are
"node:" or "npm:" imports.
Before this commit the Node "unhandledRejection" event was emitted
using a regular listener for Web "unhandledrejection" event. This
listener was installed before any user listener had a chance to be
installed which effectively prevent emitting "unhandledrejection"
events to user code.
Closes https://github.com/denoland/deno/issues/16928
Noticed that we're checking more module paths than necessary. In
particular the module path array contains a couple of entries with a
duplicated `node_modules/node_modules` suffix.
```js
[
// ... more entries before here, where some also contain duplicate suffixes
"/Users/marvinhagemeister/dev/preact-render-to-string/node_modules/.deno/node_modules",
"/Users/marvinhagemeister/dev/preact-render-to-string/node_modules/node_modules", // <-- duplicate suffix
"/Users/marvinhagemeister/dev/preact-render-to-string/node_modules",
"/Users/marvinhagemeister/dev/node_modules",
"/Users/marvinhagemeister/node_modules",
"/Users/node_modules",
"/node_modules",
"/node_modules" // <-- duplicate entry
]
```
This was caused by a misunderstanding in how Rust's
[`Path::ends_with()`](https://doc.rust-lang.org/std/path/struct.Path.html#method.ends_with)
works. It's designed to match on whole path segments and the suffix
`/node_modules` is not that, except for the root entry. This meant that
our check for if the path already ended with `node_module` always
returned `false`. Removing the leading slash fixes that.
While we're at it, we can remove the last condition where we explicitly
added the root `/node_modules` entry since the while loop prior to that
takes care of it already.
This commit changes implementation of "setImmediate"
from "node:timers" module to 0ms timer that is never
clamped to 4ms no matter how many nested calls there are.
Closes https://github.com/denoland/deno/issues/19034
This commit changes the implementation of `ext/web` timers, by using
"op_void_async_deferred" for timeouts of 0ms.
0ms timeout is meant to be run at the end of the event loop tick and
currently Tokio timers that we use to back timeouts have at least 1ms
resolution. That means that 0ms timeout actually take >1ms. This
commit changes that and runs 0ms timeout at the end of the event
loop tick.
One consequence is that "unrefing" a 0ms timer will actually keep
the event loop alive (which I believe actually makes sense, the test
we had only worked because the timeout took more than 1ms).
Ref https://github.com/denoland/deno/issues/19034
This commit changes the return type of an unstable `Deno.serve()` API
to instead return a `Deno.Server` object that has a `finished` field.
This change is done in preparation to be able to ref/unref the HTTP
server.
**THIS PR HAS GIT CONFLICTS THAT MUST BE RESOLVED**
This is the release commit being forwarded back to main for 1.33.4
Please ensure:
- [x] Everything looks ok in the PR
- [ ] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.33.4 && git checkout -b forward_v1.33.4 upstream/forward_v1.33.4
```
Don't need this PR? Close it.
cc @levex
Co-authored-by: levex <levex@users.noreply.github.com>
Co-authored-by: Levente Kurusa <lkurusa@kernelstuff.org>
This is a bit bare bones but gets `npm-run-all` working. For full stdio
compatibility with node more work is needed which is probably better
done in follow up PRs.
Fixes #19159
If a timer is requested with <=100ms resolution, request the high-res
timer. Since the default Windows timer period is 15ms, this means a
100ms timer could fire at 115ms (15% late). We assume that timers longer
than 100ms are a reasonable cutoff here.
The high-res timers on Windows are still limited. Unfortuntely this
means that our shortest duration 4ms timers can still be 25% late, but
without a more complex timer system or spinning on the clock itself,
we're somewhat bounded by the OS' scheduler itself.
This commit reimplements most of "node:http" client APIs using
"ext/fetch".
There is some duplicated code and two removed Node compat tests that
will be fixed in follow up PRs.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Fixes for various `Attemped to access invalid request` bugs (#19058,
#15427, #17213).
We did not wait for both a drop event and a completion event before
removing items from the slab table. This ensures that we do so.
In addition, the slab methods are refactored out into `slab.rs` for
maintainability.
Currently the `multipart/form-data` parser in
`Request.prototype.formData` and `Response.prototype.formData` decodes
non-ASCII filenames incorrectly, as if they were encoded in Latin-1
rather than UTF-8. This happens because the header section of each
`multipart/form-data` entry is decoded as Latin-1 in order to be parsed
with `Headers`, which only allows `ByteString`s, but the names and
filenames are never decoded correctly. This PR fixes this as a
post-processing step.
Note that the `multipart/form-data` parsing for this APIs in the Fetch
spec is very much underspecified, and it does not specify that names and
filenames must be decoded as UTF-8. However, it does require that the
bodies of non-`File` entries are decoded as UTF-8, and in browsers,
names and filenames always use the same encoding as the body.
Closes #19142.
This commit shams "performance.markResourceTiming" API by
using a noop function. It is done to provide compatibility with
"npm:undici" package. We should look into actually implementing
this API properly, but I wanted to unblock support for "undici" and
"astro" for now.
Ref https://github.com/denoland/deno/issues/19065
Merges `op_http_upgrade_next` and `op_ws_server_create`, significantly
simplifying websocket construction in ext/http (next), and removing one
JS -> Rust call. Also WS server now doesn't bypass
`HttpPropertyExtractor`.
Improve abstractions around listeners to support listener + connection
network stream combinations not previously possible (for example a
listener exposed as a Tcp, creating Unix network streams).
Partially supersedes #19016.
This migrates `spawn` and `spawn_blocking` to `deno_core`, and removes
the requirement for `spawn` tasks to be `Send` given our single-threaded
executor.
While we don't need to technically do anything w/`spawn_blocking`, this
allows us to have a single `JoinHandle` type that works for both cases,
and allows us to more easily experiment with alternative
`spawn_blocking` implementations that do not require tokio (ie: rayon).
Async ops (+~35%):
Before:
```
time 1310 ms rate 763358
time 1267 ms rate 789265
time 1259 ms rate 794281
time 1266 ms rate 789889
```
After:
```
time 956 ms rate 1046025
time 954 ms rate 1048218
time 924 ms rate 1082251
time 920 ms rate 1086956
```
HTTP serve (+~4.4%):
Before:
```
Running 10s test @ http://localhost:4500
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 68.78us 19.77us 1.43ms 86.84%
Req/Sec 68.78k 5.00k 73.84k 91.58%
1381833 requests in 10.10s, 167.36MB read
Requests/sec: 136823.29
Transfer/sec: 16.57MB
```
After:
```
Running 10s test @ http://localhost:4500
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 63.12us 17.43us 1.11ms 85.13%
Req/Sec 71.82k 3.71k 77.02k 79.21%
1443195 requests in 10.10s, 174.79MB read
Requests/sec: 142921.99
Transfer/sec: 17.31MB
```
Suggested-By: alice@ryhl.io
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
**THIS PR HAS GIT CONFLICTS THAT MUST BE RESOLVED**
This is the release commit being forwarded back to main for 1.33.3
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.33.3 && git checkout -b forward_v1.33.3 upstream/forward_v1.33.3
```
Don't need this PR? Close it.
cc @levex
Co-authored-by: Levente Kurusa <lkurusa@kernelstuff.org>
This PR ensures that node's `worker_threads` module exports
`MessageChannel`, `MessagePort` and the `BroadcastChannel` API. Fixing
these won't make `esbuild` work, but brings us one step closer 🎉
Fixes #19028 .
This is the initial support for npm and node specifiers in `deno
compile`. The npm packages are included in the binary and read from it via
a virtual file system. This also supports the `--node-modules-dir` flag,
dependencies specified in a package.json, and npm binary commands (ex.
`deno compile --unstable npm:cowsay`)
Closes #16632
`Content-Encoding: gzip` support for `Deno.serve`. This doesn't support
Brotli (`br`) yet, however it should not be difficult to add. Heuristics
for compression are modelled after those in `Deno.serveHttp`.
Tests are provided to ensure that the gzip compression is correct. We
chunk a number of different streams (zeros, hard-to-compress data,
already-gzipped data) in a number of different ways (regular, random,
large/small, small/large).
This now allows circular imports across extensions.
Instead of load + eval of all ESM files in declaration order, all files
are only loaded. Eval is done recursively by V8, only evaluating
files that are listed in `Extension::esm_entry_point` fields.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Fixes #16699 and #18960 by ensuring that we release our HTTP
`spawn_local` tasks when the HTTP resource is dropped.
Because our cancel handle was being projected from the resource via
`RcMap`, the resource was never `Drop`ped. By splitting the handle out
into its own `Rc`, we can avoid keeping the resource alive and let it
drop to cancel everything.
