This is a rewrite of the `Deno.serve` API to live on top of hyper
1.0-rc3. The code should be more maintainable long-term, and avoids some
of the slower mpsc patterns that made the older code less efficient than
it could have been.
Missing features:
- `upgradeHttp` and `upgradeHttpRaw` (`upgradeWebSocket` is available,
however).
- Automatic compression is unavailable on responses.
This is just a straight refactor and I didn't do any cleanup in
ext/node. After this PR we can start to clean it up and make things
private that don't need to be public anymore.
1. Adds cli/standalone folder
2. Writes the bytes directly to the output file. When adding npm
packages this might get quite large, so let's not keep the final output
in memory just in case.
Towards #18455
This commit implements the keypair generation for asymmetric keys for
the `generateKeyPair` API.
See how key material is managed in this implementation:
https://www.notion.so/denolandinc/node-crypto-design-99fc33f568d24e47a5e4b36002c5325d?pvs=4
Private and public key encoding depend on `KeyObject#export` which is
not implemented. I've also skipped ED448 and X448 since we need a crate
for that in WebCrypto too.
```
./target/release/deno run cli/bench/async_ops_deferred.js
time 794 ms rate 1259445
time 786 ms rate 1272264
time 770 ms rate 1298701
time 784 ms rate 1275510
time 775 ms rate 1290322
time 786 ms rate 1272264
time 773 ms rate 1293661
time 771 ms rate 1297016
time 774 ms rate 1291989
time 767 ms rate 1303780
time 764 ms rate 1308900
time 768 ms rate 1302083
time 763 ms rate 1310615
time 761 ms rate 1314060
time 761 ms rate 1314060
time 762 ms rate 1312335
time 763 ms rate 1310615
time 759 ms rate 1317523
time 760 ms rate 1315789
time 761 ms rate 1314060
time 769 ms rate 1300390
time 763 ms rate 1310615
time 760 ms rate 1315789
time 763 ms rate 1310615
time 761 ms rate 1314060
time 759 ms rate 1317523
time 765 ms rate 1307189
time 760 ms rate 1315789
time 764 ms rate 1308900
time 763 ms rate 1310615
time 760 ms rate 1315789
time 757 ms rate 1321003
time 763 ms rate 1310615
time 759 ms rate 1317523
time 771 ms rate 1297016
time 759 ms rate 1317523
time 759 ms rate 1317523
time 763 ms rate 1310615
time 754 ms rate 1326259
time 755 ms rate 1324503
time 762 ms rate 1312335
time 752 ms rate 1329787
time 755 ms rate 1324503
time 754 ms rate 1326259
time 759 ms rate 1317523
time 754 ms rate 1326259
time 749 ms rate 1335113
time 753 ms rate 1328021
time 756 ms rate 1322751
time 753 ms rate 1328021
```
```
samply record -r 20000 target/release/deno run cli/bench/async_ops_deferred.js
```
https://share.firefox.dev/43Efvm6
1. Breaks up functionality within `ProcState` into several other structs
to break out the responsibilities (`ProcState` is only a data struct
now).
2. Moves towards being able to inject dependencies more easily and have
functionality only require what it needs.
3. Exposes `Arc<T>` around the "service structs" instead of it being
embedded within them. The idea behind embedding them was to reduce the
verbosity of needing to pass around `Arc<...>`, but I don't think it was
exactly working and as we move more of these structs to be more
injectable I don't think the extra verbosity will be a big deal.
Fixes https://github.com/denoland/deno/issues/18700
Timeline of the events that lead to the bug.
1. WebSocket handshake complete
2. Server on `read_frame` holding an AsyncRefCell borrow of the
WebSocket stream.
3. Client sends a TXT frame after a some time
4. Server recieves the frame and goes back to `read_frame`.
5. After some time, Server starts a `write_frame` but `read_frame` is
still holding a borrow!
^--- Locked. read_frame needs to complete so we can resume the write.
This commit changes all writes to directly borrow the
`fastwebsocket::WebSocket` resource under the assumption that it won't
affect ongoing reads.
This commit refactors "deno_core" to do fewer boundary crossings
from Rust to V8. In other words we are now calling V8 from Rust fewer
times.
This is done by merging 3 distinct callbacks into a single one. Instead
of having "op resolve" callback, "next tick" callback and "macrotask
queue" callback, we now have only "Deno.core.eventLoopTick" callback,
which is responsible for doing the same actions previous 3 callbacks.
On each of the event loop we were doing at least 2 boundary crosses
(timers macrotask queue callback and unhandled promise rejection
callback) and up to 4 crosses if there were op response and next tick
callbacks coming from Node.js compatibility layer. Now this is all done
in a single callback.
Closes https://github.com/denoland/deno/issues/18620
Removes the functions in the `emit` module and replaces them with an
`Emitter` struct that can have "ctor dependencies" injected rather than
using functions to pass along the dependencies.
This is part of a long term refactor to move more functionality out of
proc state.
Stores the test/bench functions in rust op state during registration.
The functions are wrapped in JS first so that they return a directly
convertible `TestResult`/`BenchResult`. Test steps are still mostly
handled in JS since they are pretty much invoked by the user. Allows
removing a bunch of infrastructure for communicating between JS and
rust. Allows using rust utilities for things like shuffling tests
(`Vec::shuffle`). We can progressively move op and resource sanitization
to rust as well.
Fixes #17122.
Fixes #17312.
This commit changes "eager ops" to directly return a response value
instead of calling "opresponse" callback in JavaScript. This saves
one boundary crossing and has a fantastic impact on the "async_ops.js"
benchmark:
```
v1.32.4
$ deno run cli/bench/async_ops.js
time 329 ms rate 3039513
time 322 ms rate 3105590
time 307 ms rate 3257328
time 301 ms rate 3322259
time 303 ms rate 3300330
time 306 ms rate 3267973
time 300 ms rate 3333333
time 301 ms rate 3322259
time 301 ms rate 3322259
time 301 ms rate 3322259
time 302 ms rate 3311258
time 301 ms rate 3322259
time 302 ms rate 3311258
time 302 ms rate 3311258
time 303 ms rate 3300330
```
```
this branch
$ ./target/release/deno run -A cli/bench/async_ops.js
time 257 ms rate 3891050
time 248 ms rate 4032258
time 251 ms rate 3984063
time 246 ms rate 4065040
time 238 ms rate 4201680
time 227 ms rate 4405286
time 228 ms rate 4385964
time 229 ms rate 4366812
time 228 ms rate 4385964
time 226 ms rate 4424778
time 226 ms rate 4424778
time 227 ms rate 4405286
time 228 ms rate 4385964
time 227 ms rate 4405286
time 228 ms rate 4385964
time 227 ms rate 4405286
time 229 ms rate 4366812
time 228 ms rate 4385964
```
Prerequisite for https://github.com/denoland/deno/pull/18652
The CLI docs suggested that all deno subcommands no longer type-check by
default. This is only the case for some subcommands, and this PR
clarifies the CLI docs in this regard.
- bump deps: the newest `lazy-regex` need newer `oncecell` and
`regex`
- reduce `unwrap`
- remove dep `lazy_static`
- make more regex cached
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
`current_dir().unwrap()` joined with a Path is equivalent to the
implementation in `resolve_from_cwd()`. Manually tested on Ubuntu 22.04
and Windows 11.
Signed-off-by: Elijah Conners <business@elijahpepe.com>
This commit abstracts out the specifics of the underlying system calls
FS operations behind a new `FileSystem` and `File` trait in the
`ext/fs` extension.
This allows other embedders to re-use ext/fs, but substituting in a
different FS backend.
This is likely not the final form of these traits. Eventually they will
be entirely `deno_core::Resource` agnostic, and will live in a seperate
crate.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Fixes #6259.
Adds the location for v8 syntax errors to the message (`message += " at
{location}"`) when rethrowing them for dynamic imports.
Discussing with @bartlomieju on discord I proposed just preserving v8's
error and not reconstructing it, allowing the standard stack trace to
just point to the syntax error instead of the dynamic import. But on
further thought this way has parity with SWC's syntax errors + has the
advantage of showing both the syntax error and dynamic import location.
```ts
// temp.js
await import("./temp2.js");
// temp2.js
function foo() {
await Promise.resolve();
}
// Before:
// error: Uncaught (in promise) SyntaxError: Unexpected reserved word
// await import("./temp2.js");
// ^
// at async file:///.../temp.js:1:1
// After:
// error: Uncaught (in promise) SyntaxError: Unexpected reserved word at file:///.../temp2.js:2:3
// await import("./temp2.js");
// ^
// at async file:///.../temp.js:1:1
```
This reloads an npm package's dependency's information when a
version/version req/tag is not found.
This PR applies only to dependencies of npm packages. It does NOT yet
cause npm specifiers to have their dependency information cache busted.
That requires a different solution, but this should help cache bust in
more scenarios.
Part of #16901, but doesn't close it yet
Turns out `autoprefixer` is a better reproduction case then
`microbundle`.
Fixes #18535
Fixes #18600
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This introduces a new built-in `unset` command to `deno task` by bumping
deno_task_shell to the latest version 0.11.0. Also this includes a fix
on how `deno task` handles empty environment variables (see
https://github.com/denoland/deno_task_shell/pull/80 for details).
This is a follow-on to the earlier work in reducing string copies,
mainly focused on ensuring that ASCII strings are easy to provide to the
JS runtime.
While we are replacing a 16-byte reference in a number of places with a
24-byte structure (measured via `std::mem::size_of`), the reduction in
copies wins out over the additional size of the arguments passed into
functions.
Benchmarking shows approximately the same if not slightly less wallclock
time/instructions retired, but I believe this continues to open up
further refactoring opportunities.
This commit updates the `Deno.Kv` API to return the new commited
versionstamp for the mutated data from `db.set` and `ao.commit`. This is
returned in the form of a `Deno.KvCommitResult` object that has a
`versionstamp` property.
This reverts commit a3529d0232.
This change made debugging Node tests very hard - `AssertionError` is
now printed as `[Circular *1]` giving no visibility what failed.
We need to align two implementations together and remove this one then.
1. Fixes a cosmetic issue in the repl where it would display lsp warning
messages.
2. Lazily loads dependencies from the package.json on use.
3. Supports using bare specifiers from package.json in the REPL.
Closes #17929
Closes #18494
We currently run the all test cases in `parallel` category at the same
time, which invokes hundreds process at the same time, and that seems
causing some flakiness in CI. (maybe related to #18487)
This PR limits the concurrency to the number of cpu cores. This is more
aligned to how Node.js run their `parallel` test in their repository.
42c4a35952/Makefile (L356)
This commit adds associated type to "NodeEnv" trait, called "Fs".
The "Fs" type has a trait bound on "NodeFs", which specifies APIs
required for all ops and resolution APIs to function.
A "RealFs" implementation of "NodeFs" is exported from the "deno_node"
crate, that provides a default implementation for the trait.
All code in "deno_node" extension was changed to use the "NodeFs" trait
to handle file system operations, instead of relying on APIs from the
standard library.
This commit changes the type parameter for "deno_node" extension, from
`P: NodePermission` to `Env: NodeEnv`.
`NodeEnv` is a new trait that has associated type `P: NodePermission`.
This is a stepping stone to support swappable file system for the
extension, that will be added as a second associated type to the
`NodeEnv` trait.
1. Rewrites the tests to be more back and forth rather than getting the
output all at once (which I believe was causing the hangs on linux and
maybe mac)
2. Runs the pty tests on the linux ci.
3. Fixes a bunch of tests that were just wrong.
4. Adds timeouts on the pty tests.
Fast-follow on #18401 -- the reason that some tests were panicking in
the `CacheDB` `impl Drop` was that the cache itself was being dropped
during panic and the runtime may or may not still exist at that point.
We can reduce the actual tokio runtime testing to where it's needed.
In addition, we return the journal mode to `TRUNCATE` to avoid the risk
of data corruption.
Towards #18455
This commit implements `checkPrimeSync` and `checkPrime` in node:crypto
using the Miller-Rabin primality test (fun fact: it actually is a test
for composite numbers)
It first compares the candidate against many known small primes and if
not, proceeds to run the Miller-Rabin primality test.
http://nickle.org/examples/miller-rabin.5c used as reference
implementation.
This gets SQLite off the flamegraph and reduces initialization time by
somewhere between 0.2ms and 0.5ms. In addition, I took the opportunity
to move all the cache management code to a single place and reduce
duplication. While the PR has a net gain of lines, much of that is just
being a bit more deliberate with how we're recovering from errors.
The existing caches had various policies for dealing with cache
corruption, so I've unified them and tried to isolate the decisions we
make for recovery in a single place (see `open_connection` in
`CacheDB`). The policy I chose was:
1. Retry twice to open on-disk caches
2. If that fails, try to delete the file and recreate it on-disk
3. If we fail to delete the file or re-create a new cache, use a
fallback strategy that can be chosen per-cache: InMemory (temporary
cache for the process run), BlackHole (ignore writes, return empty
reads), or Error (fail on every operation).
The caches all use the same general code now, and share the cache
failure recovery policy.
In addition, it cleans up a TODO in the `NodeAnalysisCache`.
This internal node hook is used by libraries such as `ts-node` when used
as a require hook `node -r ts-node/register`. That combination is often
used with test frameworks like `mocha` or `jasmine`.
We had a reference to `Module._preloadModules` in our code, but the
implementation was missing. While fixing this I also noticed that the
`fakeParent` module that we create internally always threw because of
the `pathDirname` check on the module id in the constructor of `Mdoule`.
So this code path was probably broken for a while.
```txt
✖ ERROR: Error: Empty filepath.
at pathDirname (ext:deno_node/01_require.js:245:11)
at new Module (ext:deno_node/01_require.js:446:15)
at Function.Module._resolveFilename (ext:deno_node/01_require.js:754:28)
at Function.resolve (ext:deno_node/01_require.js:1015:19)
```
This commit adds the `crypto.createSecretKey` API.
Key management: This follows the same approach as our WebCrypto
CryptoKey impl where we use WeakMap for storing key material and a
handle is passed around, such that (only internal) JS can access the key
material and we don't have to explicitly close a Rust resource.
As a result, `createHmac` now accepts a secret KeyObject.
Closes https://github.com/denoland/deno/issues/17844
This will improve diagnostics and catch any non-ASCII extension code
early.
This will use `debug_assert!` rather than `assert!` to avoid runtime
costs, and ensures (in debug_assert mode only) that all extension source
files are ASCII as we load them.
This will make it a bit harder to accidentally use a client url in the
wrong place. I don't fully understand why we do this mapping, but this
will help prevent bugs like #18373
Closes #18374
No need for two almost identical implementations of the same thing
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
Co-authored-by: Aapo Alasuutari <aapo.alasuutari@gmail.com>
Previously the mapping between `AnyValue::Bool` and `KeyPart::Bool` was
inverted.
This patch also allows using the empty key `[]` as range start/end to
`snapshot_read`.
Currently `Deno.openKv(":memory:")` requests read+write permissions for
`./:memory:` even though no file is read or written. Also added some
guards for special sqlite paths that were unintentionally opted into.
This commit adds unstable "Deno.openKv()" API that allows to open
a key-value database at a specified path.
---------
Co-authored-by: Luca Casonato <hello@lcas.dev>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
…'t automatically re-created (#18330)"
This reverts commit 2ef8269fdb.
Printing these messages by default (instead of requiring `-L debug`
flag) caused
various tests to start printing it and mismatch in output assertions.
Reduce the number of copies and allocations of script code by carrying
around ownership/reference information from creation time.
As an advantage, this allows us to maintain the identity of `&'static
str`-based scripts and use v8's external 1-byte strings (to avoid
incorrectly passing non-ASCII strings, debug `assert!`s gate all string
reference paths).
Benchmark results:
Perf improvements -- ~0.1 - 0.2ms faster, but should reduce garbage
w/external strings and reduces data copies overall. May also unlock some
more interesting optimizations in the future.
This requires adding some generics to functions, but manual
monomorphization has been applied (outer/inner function) to avoid code
bloat.
This commit disables compression of the TSC snapshot.
The compression only decreased the size of snapshot by 0.5Mb
and it took about 40s during release build to compress.
With recent gains in TS 5.0 upgrade in terms of size and performance
it makes sense to remove this compression.
This PR changes the inspect result of anonymous functions from
`[Function]` to `[Function (anonymous)]`. This behavior is aligned
to `util.inspect` of Node.js.
Moving some code around in `ext/node` is it's a bit better well defined
and makes it possible for others to embed it.
I expect to see no difference in startup perf with this change.
These functions don't need to be async, as they are only calling
synchronous JavaScript code. As a follow up, all 3 functions
should be merge together - this will reduce roundtrips for
calling V8 from Rust, which is somewhat expensive
This commit adds support for spawning Web Workers in self-contained
binaries created with "deno compile" subcommand.
As long as module requested in "new Worker" constructor is part of the
eszip (by means of statically importing it beforehand, or using "--include"
flag), then the worker can be spawned.
This change will enable dynamic imports and web workers to use modules
not reachable from the main module, by passing a list of extra side
module roots as options to `deno compile`.
This can be done by specifying "--include" flag that accepts a file path or a
URL. This flag can be specified multiple times, to include several modules.
The modules specified with "--include" flag, will be added to the produced
"eszip".
This commit changes the build process in a way that preserves already
registered ops in the snapshot. This allows us to skip creating hundreds of
"v8::String" on each startup, but sadly there is still some op registration
going on startup (however we're registering 49 ops instead of >200 ops).
This situation could be further improved, by moving some of the ops
from "runtime/" to a separate extension crates.
---------
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Follow-up to #18210:
* we are passing the generated `cfg` object into the state function
rather than passing individual config fields
* reduce cloning dramatically by making the state_fn `FnOnce`
* `take` for `ExtensionBuilder` to avoid more unnecessary copies
* renamed `config` to `options`
This implements two macros to simplify extension registration and centralize a lot of the boilerplate as a base for future improvements:
* `deno_core::ops!` registers a block of `#[op]`s, optionally with type
parameters, useful for places where we share lists of ops
* `deno_core::extension!` is used to register an extension, and creates
two methods that can be used at runtime/snapshot generation time:
`init_ops` and `init_ops_and_esm`.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit removes compression for the runtime JS code.
It means that we will have a bigger binary, but faster startup. After
several discussion in the CLI team we decided it's worth to trade
about 3Mb of binary size for 2ms faster startup time. With WebGPU
removed in 35196eab27
it shouldn't have such a big impact on the binary size.
This PR _**temporarily**_ removes WebGPU (which has behind the
`--unstable` flag in Deno), due to performance complications due to its
presence.
It will be brought back in the future; as a point of reference, Chrome
will ship WebGPU to stable on 26/04/2023.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Closes #17831. This change hides the indices of any indexed collection
when triggering tab completion for object properties in the REPL.
An example is shown in the issue, but for verbosity here is another.
Before the change:
```
> const arr = new Uint8ClampedArray([1, 2, 3])
undefined
> arr.
0 map
1 reverse
2 reduce
...
```
After the change:
```
> const arr = new Uint8ClampedArray([1, 2, 3])
undefined
> arr.
constructor reduce
BYTES_PER_ELEMENT reduceRight
buffer set
...
```
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
To be able to preserve "Deno.core.ops" we need to ensure that
ops are registered in the same order in various places, otherwise
we will get mismatch in external references ordering.
Prerequisite for https://github.com/denoland/deno/pull/18080
This commit adds support for retrieving `dev` information
when stating files on Windows.
Additionally `Deno.FileInfo` interfaces was changed to always
return 0 for fields that we don't retrieve information for on Windows.
Closes https://github.com/denoland/deno/issues/18053
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
These methods are confusing because the arguments are backwards. I feel
like they should have never been added to `Option<T>` and that clippy
should suggest rewriting to
`map(...).unwrap_or(...)`/`map(...).unwrap_or_else(|| ...)`
https://github.com/rust-lang/rfcs/issues/1025
Chrono's `clock` feature pulls in `iana-time-zone` which links to macOS
core_foundation. This PR itself is not enough to get rid of
CoreFoundation. Removal depends on getting rid of security framework,
see #18071
This commit changes "deno_core" to not rely on implicitly calling
"std::env::current_dir()" when resolving module specifiers using
APIs from "deno_core::modules_specifier".
Supersedes https://github.com/denoland/deno/pull/15454
Remove remaining usages of "resolve_url_or_path_deprecated" in favor
of "resolve_url_or_path" with explicit calls to
"std::env::current_dir()".
Towards landing https://github.com/denoland/deno/pull/15454
This commit changes various CLI subcommands that have support for
the "--watch" flag to use initial current working directory when
resolving "main module".
This is part of migration towards explicitly passing current working
directory to "deno_core::resolve_url_or_path" API.
As a side effect this makes "deno <subcommand> --watch" more aligned to
user expectations, where calling "Deno.chdir()" during program doesn't
break watcher.
Towards landing https://github.com/denoland/deno/pull/15454
This commit changes current "deno_core::resolve_url_or_path" API to
"resolve_url_or_path_deprecated" and adds new "resolve_url_or_path"
API that requires to explicitly pass the directory from which paths
should be resolved to.
Some of the call sites were updated to use the new API, the reminder
of them will be updated in a follow up PR.
Towards landing https://github.com/denoland/deno/pull/15454
Creating the node_modules folder when the packages are already
downloaded can take a bit of time and not knowing what is going on can
be confusing. It's better to show a progress bar.
This has been bothering me for a while and it became more painful while
working on #18136 because injecting the shared progress bar became very
verbose. Basically we should move the creation of all these npm structs
up to a higher level.
This is a stepping stone for a future refactor where we can improve how
we create all our structs.
This commit removes "deno_core::RuntimeOptions::extensions_with_js".
Now it's embedders' responsibility to properly register extensions
that will not contains JavaScript sources when running from an existing
snapshot.
Prerequisite for https://github.com/denoland/deno/pull/18080
This commit partially reverts changes from
https://github.com/denoland/deno/pull/18095.
Turns out I made a mistake that became apparent when working
on removing "RuntimeOptions::extensions_with_js" in a follow up.
This commit splits "<ext_name>::init" functions into "init_ops" and
"init_ops_and_esm". That way we don't have to construct list of
ESM sources on each startup if we're running with a snapshot.
In a follow up commit "deno_core" will be changed to not have a split
between "extensions" and "extensions_with_js" - it will be embedders'
responsibility to pass appropriately configured extensions.
Prerequisite for https://github.com/denoland/deno/pull/18080
This improves peer dependency resolution yet again. We did not handle
scenarios like the following:
```
// a -> b -> c -> d -> c -> b (peer)
```
...which would maybe work ok the first time its run in some cases, but
then lead to a lockfile that would error on load.
This now keeps track of circular dependencies and updates nodes
accordingly. That said, there is still a lurking bug in this code
somewhere that I've added a comment for (there is a mitigation on the
tail end that seems to work well). The current state is much better than
before and I can look into it later. I think it's something small that's
incorrect.
```
Benchmark 1: deno run -A ../empty.js
Time (mean ± σ): 20.5 ms ± 0.5 ms [User: 13.4 ms, System: 5.1 ms]
Range (min … max): 19.8 ms … 24.0 ms 119 runs
Benchmark 2: target/release/deno run -A ../empty.js
Time (mean ± σ): 18.8 ms ± 0.3 ms [User: 13.0 ms, System: 4.9 ms]
Range (min … max): 18.3 ms … 19.9 ms 129 runs
```
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
