Partially supersedes #19016.
This migrates `spawn` and `spawn_blocking` to `deno_core`, and removes
the requirement for `spawn` tasks to be `Send` given our single-threaded
executor.
While we don't need to technically do anything w/`spawn_blocking`, this
allows us to have a single `JoinHandle` type that works for both cases,
and allows us to more easily experiment with alternative
`spawn_blocking` implementations that do not require tokio (ie: rayon).
Async ops (+~35%):
Before:
```
time 1310 ms rate 763358
time 1267 ms rate 789265
time 1259 ms rate 794281
time 1266 ms rate 789889
```
After:
```
time 956 ms rate 1046025
time 954 ms rate 1048218
time 924 ms rate 1082251
time 920 ms rate 1086956
```
HTTP serve (+~4.4%):
Before:
```
Running 10s test @ http://localhost:4500
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 68.78us 19.77us 1.43ms 86.84%
Req/Sec 68.78k 5.00k 73.84k 91.58%
1381833 requests in 10.10s, 167.36MB read
Requests/sec: 136823.29
Transfer/sec: 16.57MB
```
After:
```
Running 10s test @ http://localhost:4500
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 63.12us 17.43us 1.11ms 85.13%
Req/Sec 71.82k 3.71k 77.02k 79.21%
1443195 requests in 10.10s, 174.79MB read
Requests/sec: 142921.99
Transfer/sec: 17.31MB
```
Suggested-By: alice@ryhl.io
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
**THIS PR HAS GIT CONFLICTS THAT MUST BE RESOLVED**
This is the release commit being forwarded back to main for 1.33.3
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.33.3 && git checkout -b forward_v1.33.3 upstream/forward_v1.33.3
```
Don't need this PR? Close it.
cc @levex
Co-authored-by: Levente Kurusa <lkurusa@kernelstuff.org>
**THIS PR HAS GIT CONFLICTS THAT MUST BE RESOLVED**
This is the release commit being forwarded back to main for 1.33.2
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.33.2 && git checkout -b forward_v1.33.2 upstream/forward_v1.33.2
```
Don't need this PR? Close it.
cc @levex
Co-authored-by: levex <levex@users.noreply.github.com>
Co-authored-by: Levente Kurusa <lkurusa@kernelstuff.org>
This implements HTTP/2 prior-knowledge connections, allowing clients to
request HTTP/2 over plaintext or TLS-without-ALPN connections. If a
client requests a specific protocol via ALPN (`h2` or `http/1.1`),
however, the protocol is forced and must be used.
This is a rewrite of the `Deno.serve` API to live on top of hyper
1.0-rc3. The code should be more maintainable long-term, and avoids some
of the slower mpsc patterns that made the older code less efficient than
it could have been.
Missing features:
- `upgradeHttp` and `upgradeHttpRaw` (`upgradeWebSocket` is available,
however).
- Automatic compression is unavailable on responses.
This function was inherently racy which showed on slow machines -
the connect future started before the spawned task was first polled.
This change makes it so we're already accepting a connection when
the connect future is first polled.
Follow-up to #18210:
* we are passing the generated `cfg` object into the state function
rather than passing individual config fields
* reduce cloning dramatically by making the state_fn `FnOnce`
* `take` for `ExtensionBuilder` to avoid more unnecessary copies
* renamed `config` to `options`
This implements two macros to simplify extension registration and centralize a lot of the boilerplate as a base for future improvements:
* `deno_core::ops!` registers a block of `#[op]`s, optionally with type
parameters, useful for places where we share lists of ops
* `deno_core::extension!` is used to register an extension, and creates
two methods that can be used at runtime/snapshot generation time:
`init_ops` and `init_ops_and_esm`.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit splits "<ext_name>::init" functions into "init_ops" and
"init_ops_and_esm". That way we don't have to construct list of
ESM sources on each startup if we're running with a snapshot.
In a follow up commit "deno_core" will be changed to not have a split
between "extensions" and "extensions_with_js" - it will be embedders'
responsibility to pass appropriately configured extensions.
Prerequisite for https://github.com/denoland/deno/pull/18080
This commit renames "deno_core::InternalModuleLoader" to
"ExtModuleLoader" and changes the specifiers used by the
modules loaded from this loader to "ext:".
"internal:" scheme was really ambiguous and it's more characters than
"ext:", which should result in slightly smaller snapshot size.
Closes https://github.com/denoland/deno/issues/18020
There's no point for this API to expect result. If something fails it should
result in a panic during build time to signal to embedder that setup is
wrong.
This PR refactors all internal js files (except core) to be written as
ES modules.
`__bootstrap`has been mostly replaced with static imports in form in
`internal:[path to file from repo root]`.
To specify if files are ESM, an `esm` method has been added to
`Extension`, similar to the `js` method.
A new ModuleLoader called `InternalModuleLoader` has been added to
enable the loading of internal specifiers, which is used in all
situations except when a snapshot is only loaded, and not a new one is
created from it.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Updated third_party dlint to v0.37.0 for GitHub Actions. This PR
includes following changes:
* fix(prefer-primordials): Stop using array pattern assignments
* fix(prefer-primordials): Stop using global intrinsics except for
`SharedArrayBuffer`
* feat(guard-for-in): Apply new guard-for-in rule
The whole point of creating this alternative operation was to allow
usage in node, without `--unstable` flag.
Introduced and I believe missed in
https://github.com/denoland/deno/pull/16520/
This commit adds "Deno.Conn.ref()" and "Deno.Conn.unref()" methods.
These methods can be used to make connection block or not block the
event loop from finishing. Refing/unrefing only influences "read"
operations - ie. scheduling writes to a connection _do_ keep event
loop alive.
Required for https://github.com/denoland/deno/issues/16710
This PR adds copies of several unstable APIs that are available
in "Deno[Deno.internal].nodeUnstable" namespace.
These copies do not perform unstable check (ie. don't require
"--unstable" flag to be present). Otherwise they work exactly
the same, including permission checks.
These APIs are not meant to be used by users directly and
can change at any time.
Copies of following APIs are available in that namespace:
- Deno.spawnChild
- Deno.spawn
- Deno.spawnSync
- Deno.serve
- Deno.upgradeHttpRaw
- Deno.listenDatagram
Previously `op_net_listen`, `op_net_accept`, and various other ops in
ext/net where variadic on the transport. This created a lot of code
bloat. This commit updates the code to instead have separate ops for
each transport.
This commit adds a `reuseAddress` option for UDP sockets. When this
option is enabled, one can listen on an address even though it is
already being listened on from a different process or thread. The new
socket will steal the address from the existing socket.
On Windows and Linux this uses the `SO_REUSEADDR` option, while on other
Unixes this is done with `SO_REUSEPORT`.
This behavior aligns with what libuv does.
TCP sockets still unconditionally set the `SO_REUSEADDR` flag - this
behavior matches Node.js and Go. This PR does not change this behaviour.
Co-authored-by: Luca Casonato <hello@lcas.dev>
When listening on a UNIX socket path, Deno currently tries to unlink
this path prior to actually listening. The implementation of this
behaviour is VERY racy, involves 2 additional syscalls, and does not
match the behaviour of any other runtime (Node.js, Go, Rust, etc).
This commit removes this behaviour. If a user wants to listen on an
existing socket, they must now unlink the file themselves prior to
listening.
This change in behaviour only impacts --unstable APIs, so it is not
a breaking change.
This commit removes the calls to `expect()` on `std::rc::Rc`, which caused
Deno to panic under certain situations. We now return an error if `Rc`
is referenced by other variables.
Fixes #9360
Fixes #13345
Fixes #13926
Fixes #16241
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: bartlomieju <bartlomieju@users.noreply.github.com>
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
Co-authored-by: denobot <33910674+denobot@users.noreply.github.com>
Co-authored-by: bartlomieju <bartlomieju@users.noreply.github.com>
This commit adds a new op_write_all to core that allows writing an
entire chunk in a single async op call. Internally this calls
`Resource::write_all`.
The `writableStreamForRid` has been moved to `06_streams.js` now, and
uses this new op. Various other code paths now also use this new op.
Closes #16227
This commit introduces two new buffer wrapper types to `deno_core`. The
main benefit of these new wrappers is that they can wrap a number of
different underlying buffer types. This allows for a more flexible read
and write API on resources that will require less copying of data
between different buffer representations.
- `BufView` is a read-only view onto a buffer. It can be backed by
`ZeroCopyBuf`, `Vec<u8>`, and `bytes::Bytes`.
- `BufViewMut` is a read-write view onto a buffer. It can be cheaply
converted into a `BufView`. It can be backed by `ZeroCopyBuf` or
`Vec<u8>`.
Both new buffer views have a cursor. This means that the start point of
the view can be constrained to write / read from just a slice of the
view. Only the start point of the slice can be adjusted. The end point
is fixed. To adjust the end point, the underlying buffer needs to be
truncated.
Readable resources have been changed to better cater to resources that
do not support BYOB reads. The basic `read` method now returns a
`BufView` instead of taking a `ZeroCopyBuf` to fill. This allows the
operation to return buffers that the resource has already allocated,
instead of forcing the caller to allocate the buffer. BYOB reads are
still very useful for resources that support them, so a new `read_byob`
method has been added that takes a `BufViewMut` to fill. `op_read`
attempts to use `read_byob` if the resource supports it, which falls
back to `read` and performs an additional copy if it does not. For
Rust->JS reads this change should have no impact, but for Rust->Rust
reads, this allows the caller to avoid an additional copy in many
scenarios. This combined with the support for `BufView` to be backed by
`bytes::Bytes` allows us to avoid one data copy when piping from a
`fetch` response into an `ext/http` response.
Writable resources have been changed to take a `BufView` instead of a
`ZeroCopyBuf` as an argument. This allows for less copying of data in
certain scenarios, as described above. Additionally a new
`Resource::write_all` method has been added that takes a `BufView` and
continually attempts to write the resource until the entire buffer has
been written. Certain resources like files can override this method to
provide a more efficient `write_all` implementation.
This is the release commit being forwarded back to main for 1.26.1
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.26.1 && git checkout -b forward_v1.26.1 upstream/forward_v1.26.1
```
Don't need this PR? Close it.
cc @cjihrig
Co-authored-by: cjihrig <cjihrig@users.noreply.github.com>
Stop allowing clippy::derive-partial-eq-without-eq and fix warnings
about deriving PartialEq without also deriving Eq.
In one case I removed the PartialEq because it a) wasn't necessary,
and b) sketchy because it was comparing floating point numbers.
IMO, that's a good argument for enforcing the lint rule, because it
would most likely have been caught during review if it had been enabled.
Welcome to better optimised op calls! Currently opSync is called with parameters of every type and count. This most definitely makes the call megamorphic. Additionally, it seems that spread params leads to V8 not being able to optimise the calls quite as well (apparently Fast Calls cannot be used with spread params).
Monomorphising op calls should lead to some improved performance. Now that unwrapping of sync ops results is done on Rust side, this is pretty simple:
```
opSync("op_foo", param1, param2);
// -> turns to
ops.op_foo(param1, param2);
```
This means sync op calls are now just directly calling the native binding function. When V8 Fast API Calls are enabled, this will enable those to be called on the optimised path.
Monomorphising async ops likely requires using callbacks and is left as an exercise to the reader.
This commit adds "Deno.upgradeHttp" API, which
allows to "hijack" connection and switch protocols, to eg.
implement WebSocket required for Node compat.
Co-authored-by: crowlkats <crowlkats@toaxl.com>
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds `readable` and `writable` properties to `Deno.File` and
`Deno.Conn`. This makes it very simple to use files and network sockets
with fetch or the native HTTP server.
std/http/server knows how to handle "Listener has been closed"
exceptions but not "operation canceled" errors.
Make "unix" listen sockets throw the same exception as "tcp" listen
sockets when the socket is closed and has a pending accept operation.
There is still a discrepancy when multiple accept requests are posted
but that's probably a less visible issue and something for another day.
Fixes #13033
This allows resources to be "streams" by implementing read/write/shutdown. These streams are implicit since their nature (read/write/duplex) isn't known until called, but we could easily add another method to explicitly tag resources as streams.
`op_read/op_write/op_shutdown` are now builtin ops provided by `deno_core`
Note: this current implementation is simple & straightforward but it results in an additional alloc per read/write call
Closes #12556
A `handshake()` method was added that returns when the TLS handshake is
complete. The `TlsListener` and `TlsConn` interfaces were added to
accomodate this new method.
Closes: #11759.
A bug was fixed that could cause a hang when a method was
called on a TlsConn object that had thrown an exception earlier.
Additionally, a bug was fixed that caused TlsConn.write() to not
completely flush large buffers (>64kB) to the socket.
The public `TlsConn.handshake()` API is scheduled for inclusion in the
next minor release. See https://github.com/denoland/deno/pull/12467.
Currently all async ops are polled lazily, which means that op
initialization code is postponed until control is yielded to the event
loop. This has some weird consequences, e.g.
```js
let listener = Deno.listen(...);
let conn_promise = listener.accept();
listener.close();
// `BadResource` is thrown. A reasonable error would be `Interrupted`.
let conn = await conn_promise;
```
JavaScript promises are expected to be eagerly evaluated. This patch
makes ops actually do that.
These are confusing. They say they are "for users that don't care about
permissions", but that isn't correct. `NoTimersPermissions` disables
permissions instead of enabling them.
I would argue that implementors should decide what permissions they want
themselves, and not take our opinionated permissions struct.
This adds support for using in memory CA certificates for
`Deno.startTLS`, `Deno.connectTLS` and `Deno.createHttpClient`.
`certFile` is deprecated in `startTls` and `connectTls`, and removed
from `Deno.createHttpClient`.
Default to None if UnsafelyIgnoreCertificateErrors is not present in the
OpState.
Embedders may not have a need for restricting outgoing TLS connections
and having them hunt through the source code for the magic incantation
that makes the borrow panics go away, is less user friendly.
