Note: If the package information has already been cached, then this
requires running with `--reload` or for the registry information to be
fetched some other way (ex. the cache busting).
Closes #15544
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit reimplements most of "node:http" client APIs using
"ext/fetch".
There is some duplicated code and two removed Node compat tests that
will be fixed in follow up PRs.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Fixes for various `Attemped to access invalid request` bugs (#19058,
#15427, #17213).
We did not wait for both a drop event and a completion event before
removing items from the slab table. This ensures that we do so.
In addition, the slab methods are refactored out into `slab.rs` for
maintainability.
Currently the `multipart/form-data` parser in
`Request.prototype.formData` and `Response.prototype.formData` decodes
non-ASCII filenames incorrectly, as if they were encoded in Latin-1
rather than UTF-8. This happens because the header section of each
`multipart/form-data` entry is decoded as Latin-1 in order to be parsed
with `Headers`, which only allows `ByteString`s, but the names and
filenames are never decoded correctly. This PR fixes this as a
post-processing step.
Note that the `multipart/form-data` parsing for this APIs in the Fetch
spec is very much underspecified, and it does not specify that names and
filenames must be decoded as UTF-8. However, it does require that the
bodies of non-`File` entries are decoded as UTF-8, and in browsers,
names and filenames always use the same encoding as the body.
Closes #19142.
Partially supersedes #19016.
This migrates `spawn` and `spawn_blocking` to `deno_core`, and removes
the requirement for `spawn` tasks to be `Send` given our single-threaded
executor.
While we don't need to technically do anything w/`spawn_blocking`, this
allows us to have a single `JoinHandle` type that works for both cases,
and allows us to more easily experiment with alternative
`spawn_blocking` implementations that do not require tokio (ie: rayon).
Async ops (+~35%):
Before:
```
time 1310 ms rate 763358
time 1267 ms rate 789265
time 1259 ms rate 794281
time 1266 ms rate 789889
```
After:
```
time 956 ms rate 1046025
time 954 ms rate 1048218
time 924 ms rate 1082251
time 920 ms rate 1086956
```
HTTP serve (+~4.4%):
Before:
```
Running 10s test @ http://localhost:4500
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 68.78us 19.77us 1.43ms 86.84%
Req/Sec 68.78k 5.00k 73.84k 91.58%
1381833 requests in 10.10s, 167.36MB read
Requests/sec: 136823.29
Transfer/sec: 16.57MB
```
After:
```
Running 10s test @ http://localhost:4500
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 63.12us 17.43us 1.11ms 85.13%
Req/Sec 71.82k 3.71k 77.02k 79.21%
1443195 requests in 10.10s, 174.79MB read
Requests/sec: 142921.99
Transfer/sec: 17.31MB
```
Suggested-By: alice@ryhl.io
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
**THIS PR HAS GIT CONFLICTS THAT MUST BE RESOLVED**
This is the release commit being forwarded back to main for 1.33.3
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.33.3 && git checkout -b forward_v1.33.3 upstream/forward_v1.33.3
```
Don't need this PR? Close it.
cc @levex
Co-authored-by: Levente Kurusa <lkurusa@kernelstuff.org>
Adds a `deno.preloadLimit` option (ex. `"deno.preloadLimit": 2000`)
which specifies how many file entries to traverse on the file system
when the lsp loads or its configuration changes.
Closes #18955
This PR ensures that node's `worker_threads` module exports
`MessageChannel`, `MessagePort` and the `BroadcastChannel` API. Fixing
these won't make `esbuild` work, but brings us one step closer 🎉
Fixes #19028 .
This is the initial support for npm and node specifiers in `deno
compile`. The npm packages are included in the binary and read from it via
a virtual file system. This also supports the `--node-modules-dir` flag,
dependencies specified in a package.json, and npm binary commands (ex.
`deno compile --unstable npm:cowsay`)
Closes #16632
`Content-Encoding: gzip` support for `Deno.serve`. This doesn't support
Brotli (`br`) yet, however it should not be difficult to add. Heuristics
for compression are modelled after those in `Deno.serveHttp`.
Tests are provided to ensure that the gzip compression is correct. We
chunk a number of different streams (zeros, hard-to-compress data,
already-gzipped data) in a number of different ways (regular, random,
large/small, small/large).
This now allows circular imports across extensions.
Instead of load + eval of all ESM files in declaration order, all files
are only loaded. Eval is done recursively by V8, only evaluating
files that are listed in `Extension::esm_entry_point` fields.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Fixes #16699 and #18960 by ensuring that we release our HTTP
`spawn_local` tasks when the HTTP resource is dropped.
Because our cancel handle was being projected from the resource via
`RcMap`, the resource was never `Drop`ped. By splitting the handle out
into its own `Rc`, we can avoid keeping the resource alive and let it
drop to cancel everything.
1. Boxed `File` and `FileSystem` to allow more easily passing this
through the CLI code (as shown within this pr).
2. `StdFileResource` is now `FileResource`. `FileResource` now contains
an `Rc<dyn File>`.
**THIS PR HAS GIT CONFLICTS THAT MUST BE RESOLVED**
This is the release commit being forwarded back to main for 1.33.2
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.33.2 && git checkout -b forward_v1.33.2 upstream/forward_v1.33.2
```
Don't need this PR? Close it.
cc @levex
Co-authored-by: levex <levex@users.noreply.github.com>
Co-authored-by: Levente Kurusa <lkurusa@kernelstuff.org>
Fixes #18979.
This changes the predicate for allowing `ext:` specifier resolution from
`snapshot_loaded_and_not_snapshotting` to `ext_resolution_allowed` which
is only set to true during the extension module loading phase. Module
loaders as used in core
are now declared as `ExtModuleLoader` rather than `dyn ModuleLoader`.
`new Deno.KvU64(1n) + 2n == 3n` is now true.
`new Deno.KvU64(1n)` is now inspected as `[Deno.KvU64: 1n]`
(`Object(1n)` is inspected as `[BigInt: 1n]`).
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit changes how paths for npm packages are handled,
by canonicalizing them when resolving. This is done so that instead
of returning
"node_modules/<package_name>@<version>/node_modules/<dep>/index.js"
(which is a symlink) we "node_modules/<dep>@<dep_version>/index.js.
Fixes https://github.com/denoland/deno/issues/18924
Fixes https://github.com/bluwy/create-vite-extra/issues/31
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
This removes `ProcState` and replaces it with a new `CliFactory` which
initializes our "service structs" on demand. This isn't a performance
improvement at the moment for `deno run`, but might unlock performance
improvements in the future.
- Do not use `ReflectHas` in `isNode`.
- Avoid copying handler array when handlers.length == 1
- Avoid searching for path target when path.length == 1
```
Linux divy-2 5.19.0-1022-gcp #24~22.04.1-Ubuntu SMP Sun Apr 23 09:51:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
32GiB System memory
Intel(R) Xeon(R) CPU @ 3.10GHz
# main + https://github.com/denoland/deno/pull/18904
Msg/sec: 89326.750000
Msg/sec: 90320.000000
Msg/sec: 89576.250000
# this patch
Msg/sec: 97250.000000
Msg/sec: 97125.500000
Msg/sec: 97964.500000
```
Fixes regression from #18878 where `Promise.reject()`,
`Promise.reject(undefined)` and `reportError(undefined)` panic in the
REPL.
Fixes `throw undefined` printing `Uncaught Unknown exception` instead of
`Uncaught undefined`.
Fixes #8858.
Fixes #8869.
```
$ target/debug/deno
Deno 1.32.5
exit using ctrl+d, ctrl+c, or close()
REPL is running with all permissions allowed.
To specify permissions, run `deno repl` with allow flags.
> Promise.reject(new Error("bar"));
Promise { <rejected> Error: bar
at <anonymous>:2:16 }
Uncaught (in promise) Error: bar
at <anonymous>:2:16
> reportError(new Error("baz"));
undefined
Uncaught Error: baz
at <anonymous>:2:13
>
Towards https://github.com/denoland/deno/issues/18455
`safe`, `add` and `rem` options are not implemented because there is no
rust crate that provides this functionality (except rust-openssl maybe)
and its just not clear if this API is used widely.
Hey there! I took a crack at improving these embedded docs [as requested
here](https://github.com/denoland/deno/issues/18685). These should
accurately reflect the functionality of the permission-related flags for
`deno run`.
### Highlights
* Adds human-readable argument string in the format [prescribed in the
docs](https://docs.rs/clap/latest/clap/struct.Arg.html#method.value_name)
* Keeps text description terse, but includes a relevant copy/pasteable
docs link
* Includes example argument usage/formatting
This commit adds `@deprecated` comments to `Deno.run` API declarations.
Since stabilization of `Deno.Command` API in [Deno
v1.31](https://deno.com/blog/v1.31#api-stabilizations), `Deno.Command`
is the preferred (more reliable) API to interact with subprocesses.
This is the preparation for the removal of `Deno.run` API in Deno 2.0.
We can make `NodePermissions` rely on interior mutability (which the
`PermissionsContainer` is already doing) in order to not have to clone
everything all the time. This also reduces the chance of an accidental
`borrow` while `borrrow_mut`.
This implements HTTP/2 prior-knowledge connections, allowing clients to
request HTTP/2 over plaintext or TLS-without-ALPN connections. If a
client requests a specific protocol via ALPN (`h2` or `http/1.1`),
however, the protocol is forced and must be used.
We were indeterministically including packages in the top level
`node_modules/` folder when using a local node_modules directory. This
change aligns with pnpm and only includes top level packages in this
folder. This should be faster for initializing the folder, but may
expose issues in packages that reference other packages not defined in
their dependencies. That said, the behaviour previously was previously
broken.
This has exposed a bug in the require implementation where it doesn't
find a package (which is the main underlying issue here). There is a
failing test already for this in the test suite after this change.
Closes #18822
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This is a rewrite of the `Deno.serve` API to live on top of hyper
1.0-rc3. The code should be more maintainable long-term, and avoids some
of the slower mpsc patterns that made the older code less efficient than
it could have been.
Missing features:
- `upgradeHttp` and `upgradeHttpRaw` (`upgradeWebSocket` is available,
however).
- Automatic compression is unavailable on responses.
This is just a straight refactor and I didn't do any cleanup in
ext/node. After this PR we can start to clean it up and make things
private that don't need to be public anymore.
1. Adds cli/standalone folder
2. Writes the bytes directly to the output file. When adding npm
packages this might get quite large, so let's not keep the final output
in memory just in case.
Towards #18455
This commit implements the keypair generation for asymmetric keys for
the `generateKeyPair` API.
See how key material is managed in this implementation:
https://www.notion.so/denolandinc/node-crypto-design-99fc33f568d24e47a5e4b36002c5325d?pvs=4
Private and public key encoding depend on `KeyObject#export` which is
not implemented. I've also skipped ED448 and X448 since we need a crate
for that in WebCrypto too.
```
./target/release/deno run cli/bench/async_ops_deferred.js
time 794 ms rate 1259445
time 786 ms rate 1272264
time 770 ms rate 1298701
time 784 ms rate 1275510
time 775 ms rate 1290322
time 786 ms rate 1272264
time 773 ms rate 1293661
time 771 ms rate 1297016
time 774 ms rate 1291989
time 767 ms rate 1303780
time 764 ms rate 1308900
time 768 ms rate 1302083
time 763 ms rate 1310615
time 761 ms rate 1314060
time 761 ms rate 1314060
time 762 ms rate 1312335
time 763 ms rate 1310615
time 759 ms rate 1317523
time 760 ms rate 1315789
time 761 ms rate 1314060
time 769 ms rate 1300390
time 763 ms rate 1310615
time 760 ms rate 1315789
time 763 ms rate 1310615
time 761 ms rate 1314060
time 759 ms rate 1317523
time 765 ms rate 1307189
time 760 ms rate 1315789
time 764 ms rate 1308900
time 763 ms rate 1310615
time 760 ms rate 1315789
time 757 ms rate 1321003
time 763 ms rate 1310615
time 759 ms rate 1317523
time 771 ms rate 1297016
time 759 ms rate 1317523
time 759 ms rate 1317523
time 763 ms rate 1310615
time 754 ms rate 1326259
time 755 ms rate 1324503
time 762 ms rate 1312335
time 752 ms rate 1329787
time 755 ms rate 1324503
time 754 ms rate 1326259
time 759 ms rate 1317523
time 754 ms rate 1326259
time 749 ms rate 1335113
time 753 ms rate 1328021
time 756 ms rate 1322751
time 753 ms rate 1328021
```
```
samply record -r 20000 target/release/deno run cli/bench/async_ops_deferred.js
```
https://share.firefox.dev/43Efvm6
1. Breaks up functionality within `ProcState` into several other structs
to break out the responsibilities (`ProcState` is only a data struct
now).
2. Moves towards being able to inject dependencies more easily and have
functionality only require what it needs.
3. Exposes `Arc<T>` around the "service structs" instead of it being
embedded within them. The idea behind embedding them was to reduce the
verbosity of needing to pass around `Arc<...>`, but I don't think it was
exactly working and as we move more of these structs to be more
injectable I don't think the extra verbosity will be a big deal.
Fixes https://github.com/denoland/deno/issues/18700
Timeline of the events that lead to the bug.
1. WebSocket handshake complete
2. Server on `read_frame` holding an AsyncRefCell borrow of the
WebSocket stream.
3. Client sends a TXT frame after a some time
4. Server recieves the frame and goes back to `read_frame`.
5. After some time, Server starts a `write_frame` but `read_frame` is
still holding a borrow!
^--- Locked. read_frame needs to complete so we can resume the write.
This commit changes all writes to directly borrow the
`fastwebsocket::WebSocket` resource under the assumption that it won't
affect ongoing reads.
This commit refactors "deno_core" to do fewer boundary crossings
from Rust to V8. In other words we are now calling V8 from Rust fewer
times.
This is done by merging 3 distinct callbacks into a single one. Instead
of having "op resolve" callback, "next tick" callback and "macrotask
queue" callback, we now have only "Deno.core.eventLoopTick" callback,
which is responsible for doing the same actions previous 3 callbacks.
On each of the event loop we were doing at least 2 boundary crosses
(timers macrotask queue callback and unhandled promise rejection
callback) and up to 4 crosses if there were op response and next tick
callbacks coming from Node.js compatibility layer. Now this is all done
in a single callback.
Closes https://github.com/denoland/deno/issues/18620
Removes the functions in the `emit` module and replaces them with an
`Emitter` struct that can have "ctor dependencies" injected rather than
using functions to pass along the dependencies.
This is part of a long term refactor to move more functionality out of
proc state.
Stores the test/bench functions in rust op state during registration.
The functions are wrapped in JS first so that they return a directly
convertible `TestResult`/`BenchResult`. Test steps are still mostly
handled in JS since they are pretty much invoked by the user. Allows
removing a bunch of infrastructure for communicating between JS and
rust. Allows using rust utilities for things like shuffling tests
(`Vec::shuffle`). We can progressively move op and resource sanitization
to rust as well.
Fixes #17122.
Fixes #17312.
This commit changes "eager ops" to directly return a response value
instead of calling "opresponse" callback in JavaScript. This saves
one boundary crossing and has a fantastic impact on the "async_ops.js"
benchmark:
```
v1.32.4
$ deno run cli/bench/async_ops.js
time 329 ms rate 3039513
time 322 ms rate 3105590
time 307 ms rate 3257328
time 301 ms rate 3322259
time 303 ms rate 3300330
time 306 ms rate 3267973
time 300 ms rate 3333333
time 301 ms rate 3322259
time 301 ms rate 3322259
time 301 ms rate 3322259
time 302 ms rate 3311258
time 301 ms rate 3322259
time 302 ms rate 3311258
time 302 ms rate 3311258
time 303 ms rate 3300330
```
```
this branch
$ ./target/release/deno run -A cli/bench/async_ops.js
time 257 ms rate 3891050
time 248 ms rate 4032258
time 251 ms rate 3984063
time 246 ms rate 4065040
time 238 ms rate 4201680
time 227 ms rate 4405286
time 228 ms rate 4385964
time 229 ms rate 4366812
time 228 ms rate 4385964
time 226 ms rate 4424778
time 226 ms rate 4424778
time 227 ms rate 4405286
time 228 ms rate 4385964
time 227 ms rate 4405286
time 228 ms rate 4385964
time 227 ms rate 4405286
time 229 ms rate 4366812
time 228 ms rate 4385964
```
Prerequisite for https://github.com/denoland/deno/pull/18652
The CLI docs suggested that all deno subcommands no longer type-check by
default. This is only the case for some subcommands, and this PR
clarifies the CLI docs in this regard.
- bump deps: the newest `lazy-regex` need newer `oncecell` and
`regex`
- reduce `unwrap`
- remove dep `lazy_static`
- make more regex cached
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
`current_dir().unwrap()` joined with a Path is equivalent to the
implementation in `resolve_from_cwd()`. Manually tested on Ubuntu 22.04
and Windows 11.
Signed-off-by: Elijah Conners <business@elijahpepe.com>
This commit abstracts out the specifics of the underlying system calls
FS operations behind a new `FileSystem` and `File` trait in the
`ext/fs` extension.
This allows other embedders to re-use ext/fs, but substituting in a
different FS backend.
This is likely not the final form of these traits. Eventually they will
be entirely `deno_core::Resource` agnostic, and will live in a seperate
crate.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Fixes #6259.
Adds the location for v8 syntax errors to the message (`message += " at
{location}"`) when rethrowing them for dynamic imports.
Discussing with @bartlomieju on discord I proposed just preserving v8's
error and not reconstructing it, allowing the standard stack trace to
just point to the syntax error instead of the dynamic import. But on
further thought this way has parity with SWC's syntax errors + has the
advantage of showing both the syntax error and dynamic import location.
```ts
// temp.js
await import("./temp2.js");
// temp2.js
function foo() {
await Promise.resolve();
}
// Before:
// error: Uncaught (in promise) SyntaxError: Unexpected reserved word
// await import("./temp2.js");
// ^
// at async file:///.../temp.js:1:1
// After:
// error: Uncaught (in promise) SyntaxError: Unexpected reserved word at file:///.../temp2.js:2:3
// await import("./temp2.js");
// ^
// at async file:///.../temp.js:1:1
```
This reloads an npm package's dependency's information when a
version/version req/tag is not found.
This PR applies only to dependencies of npm packages. It does NOT yet
cause npm specifiers to have their dependency information cache busted.
That requires a different solution, but this should help cache bust in
more scenarios.
Part of #16901, but doesn't close it yet
Turns out `autoprefixer` is a better reproduction case then
`microbundle`.
Fixes #18535
Fixes #18600
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This introduces a new built-in `unset` command to `deno task` by bumping
deno_task_shell to the latest version 0.11.0. Also this includes a fix
on how `deno task` handles empty environment variables (see
https://github.com/denoland/deno_task_shell/pull/80 for details).
This is a follow-on to the earlier work in reducing string copies,
mainly focused on ensuring that ASCII strings are easy to provide to the
JS runtime.
While we are replacing a 16-byte reference in a number of places with a
24-byte structure (measured via `std::mem::size_of`), the reduction in
copies wins out over the additional size of the arguments passed into
functions.
Benchmarking shows approximately the same if not slightly less wallclock
time/instructions retired, but I believe this continues to open up
further refactoring opportunities.
This commit updates the `Deno.Kv` API to return the new commited
versionstamp for the mutated data from `db.set` and `ao.commit`. This is
returned in the form of a `Deno.KvCommitResult` object that has a
`versionstamp` property.
This reverts commit a3529d0232.
This change made debugging Node tests very hard - `AssertionError` is
now printed as `[Circular *1]` giving no visibility what failed.
We need to align two implementations together and remove this one then.
1. Fixes a cosmetic issue in the repl where it would display lsp warning
messages.
2. Lazily loads dependencies from the package.json on use.
3. Supports using bare specifiers from package.json in the REPL.
Closes #17929
Closes #18494
We currently run the all test cases in `parallel` category at the same
time, which invokes hundreds process at the same time, and that seems
causing some flakiness in CI. (maybe related to #18487)
This PR limits the concurrency to the number of cpu cores. This is more
aligned to how Node.js run their `parallel` test in their repository.
42c4a35952/Makefile (L356)
