We can make `NodePermissions` rely on interior mutability (which the
`PermissionsContainer` is already doing) in order to not have to clone
everything all the time. This also reduces the chance of an accidental
`borrow` while `borrrow_mut`.
This implements HTTP/2 prior-knowledge connections, allowing clients to
request HTTP/2 over plaintext or TLS-without-ALPN connections. If a
client requests a specific protocol via ALPN (`h2` or `http/1.1`),
however, the protocol is forced and must be used.
We were indeterministically including packages in the top level
`node_modules/` folder when using a local node_modules directory. This
change aligns with pnpm and only includes top level packages in this
folder. This should be faster for initializing the folder, but may
expose issues in packages that reference other packages not defined in
their dependencies. That said, the behaviour previously was previously
broken.
This has exposed a bug in the require implementation where it doesn't
find a package (which is the main underlying issue here). There is a
failing test already for this in the test suite after this change.
Closes #18822
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This is a rewrite of the `Deno.serve` API to live on top of hyper
1.0-rc3. The code should be more maintainable long-term, and avoids some
of the slower mpsc patterns that made the older code less efficient than
it could have been.
Missing features:
- `upgradeHttp` and `upgradeHttpRaw` (`upgradeWebSocket` is available,
however).
- Automatic compression is unavailable on responses.
This is just a straight refactor and I didn't do any cleanup in
ext/node. After this PR we can start to clean it up and make things
private that don't need to be public anymore.
1. Adds cli/standalone folder
2. Writes the bytes directly to the output file. When adding npm
packages this might get quite large, so let's not keep the final output
in memory just in case.
Towards #18455
This commit implements the keypair generation for asymmetric keys for
the `generateKeyPair` API.
See how key material is managed in this implementation:
https://www.notion.so/denolandinc/node-crypto-design-99fc33f568d24e47a5e4b36002c5325d?pvs=4
Private and public key encoding depend on `KeyObject#export` which is
not implemented. I've also skipped ED448 and X448 since we need a crate
for that in WebCrypto too.
```
./target/release/deno run cli/bench/async_ops_deferred.js
time 794 ms rate 1259445
time 786 ms rate 1272264
time 770 ms rate 1298701
time 784 ms rate 1275510
time 775 ms rate 1290322
time 786 ms rate 1272264
time 773 ms rate 1293661
time 771 ms rate 1297016
time 774 ms rate 1291989
time 767 ms rate 1303780
time 764 ms rate 1308900
time 768 ms rate 1302083
time 763 ms rate 1310615
time 761 ms rate 1314060
time 761 ms rate 1314060
time 762 ms rate 1312335
time 763 ms rate 1310615
time 759 ms rate 1317523
time 760 ms rate 1315789
time 761 ms rate 1314060
time 769 ms rate 1300390
time 763 ms rate 1310615
time 760 ms rate 1315789
time 763 ms rate 1310615
time 761 ms rate 1314060
time 759 ms rate 1317523
time 765 ms rate 1307189
time 760 ms rate 1315789
time 764 ms rate 1308900
time 763 ms rate 1310615
time 760 ms rate 1315789
time 757 ms rate 1321003
time 763 ms rate 1310615
time 759 ms rate 1317523
time 771 ms rate 1297016
time 759 ms rate 1317523
time 759 ms rate 1317523
time 763 ms rate 1310615
time 754 ms rate 1326259
time 755 ms rate 1324503
time 762 ms rate 1312335
time 752 ms rate 1329787
time 755 ms rate 1324503
time 754 ms rate 1326259
time 759 ms rate 1317523
time 754 ms rate 1326259
time 749 ms rate 1335113
time 753 ms rate 1328021
time 756 ms rate 1322751
time 753 ms rate 1328021
```
```
samply record -r 20000 target/release/deno run cli/bench/async_ops_deferred.js
```
https://share.firefox.dev/43Efvm6
1. Breaks up functionality within `ProcState` into several other structs
to break out the responsibilities (`ProcState` is only a data struct
now).
2. Moves towards being able to inject dependencies more easily and have
functionality only require what it needs.
3. Exposes `Arc<T>` around the "service structs" instead of it being
embedded within them. The idea behind embedding them was to reduce the
verbosity of needing to pass around `Arc<...>`, but I don't think it was
exactly working and as we move more of these structs to be more
injectable I don't think the extra verbosity will be a big deal.
Fixes https://github.com/denoland/deno/issues/18700
Timeline of the events that lead to the bug.
1. WebSocket handshake complete
2. Server on `read_frame` holding an AsyncRefCell borrow of the
WebSocket stream.
3. Client sends a TXT frame after a some time
4. Server recieves the frame and goes back to `read_frame`.
5. After some time, Server starts a `write_frame` but `read_frame` is
still holding a borrow!
^--- Locked. read_frame needs to complete so we can resume the write.
This commit changes all writes to directly borrow the
`fastwebsocket::WebSocket` resource under the assumption that it won't
affect ongoing reads.
This commit refactors "deno_core" to do fewer boundary crossings
from Rust to V8. In other words we are now calling V8 from Rust fewer
times.
This is done by merging 3 distinct callbacks into a single one. Instead
of having "op resolve" callback, "next tick" callback and "macrotask
queue" callback, we now have only "Deno.core.eventLoopTick" callback,
which is responsible for doing the same actions previous 3 callbacks.
On each of the event loop we were doing at least 2 boundary crosses
(timers macrotask queue callback and unhandled promise rejection
callback) and up to 4 crosses if there were op response and next tick
callbacks coming from Node.js compatibility layer. Now this is all done
in a single callback.
Closes https://github.com/denoland/deno/issues/18620
Removes the functions in the `emit` module and replaces them with an
`Emitter` struct that can have "ctor dependencies" injected rather than
using functions to pass along the dependencies.
This is part of a long term refactor to move more functionality out of
proc state.
Stores the test/bench functions in rust op state during registration.
The functions are wrapped in JS first so that they return a directly
convertible `TestResult`/`BenchResult`. Test steps are still mostly
handled in JS since they are pretty much invoked by the user. Allows
removing a bunch of infrastructure for communicating between JS and
rust. Allows using rust utilities for things like shuffling tests
(`Vec::shuffle`). We can progressively move op and resource sanitization
to rust as well.
Fixes #17122.
Fixes #17312.
This commit changes "eager ops" to directly return a response value
instead of calling "opresponse" callback in JavaScript. This saves
one boundary crossing and has a fantastic impact on the "async_ops.js"
benchmark:
```
v1.32.4
$ deno run cli/bench/async_ops.js
time 329 ms rate 3039513
time 322 ms rate 3105590
time 307 ms rate 3257328
time 301 ms rate 3322259
time 303 ms rate 3300330
time 306 ms rate 3267973
time 300 ms rate 3333333
time 301 ms rate 3322259
time 301 ms rate 3322259
time 301 ms rate 3322259
time 302 ms rate 3311258
time 301 ms rate 3322259
time 302 ms rate 3311258
time 302 ms rate 3311258
time 303 ms rate 3300330
```
```
this branch
$ ./target/release/deno run -A cli/bench/async_ops.js
time 257 ms rate 3891050
time 248 ms rate 4032258
time 251 ms rate 3984063
time 246 ms rate 4065040
time 238 ms rate 4201680
time 227 ms rate 4405286
time 228 ms rate 4385964
time 229 ms rate 4366812
time 228 ms rate 4385964
time 226 ms rate 4424778
time 226 ms rate 4424778
time 227 ms rate 4405286
time 228 ms rate 4385964
time 227 ms rate 4405286
time 228 ms rate 4385964
time 227 ms rate 4405286
time 229 ms rate 4366812
time 228 ms rate 4385964
```
Prerequisite for https://github.com/denoland/deno/pull/18652
The CLI docs suggested that all deno subcommands no longer type-check by
default. This is only the case for some subcommands, and this PR
clarifies the CLI docs in this regard.
- bump deps: the newest `lazy-regex` need newer `oncecell` and
`regex`
- reduce `unwrap`
- remove dep `lazy_static`
- make more regex cached
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
`current_dir().unwrap()` joined with a Path is equivalent to the
implementation in `resolve_from_cwd()`. Manually tested on Ubuntu 22.04
and Windows 11.
Signed-off-by: Elijah Conners <business@elijahpepe.com>
This commit abstracts out the specifics of the underlying system calls
FS operations behind a new `FileSystem` and `File` trait in the
`ext/fs` extension.
This allows other embedders to re-use ext/fs, but substituting in a
different FS backend.
This is likely not the final form of these traits. Eventually they will
be entirely `deno_core::Resource` agnostic, and will live in a seperate
crate.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Fixes #6259.
Adds the location for v8 syntax errors to the message (`message += " at
{location}"`) when rethrowing them for dynamic imports.
Discussing with @bartlomieju on discord I proposed just preserving v8's
error and not reconstructing it, allowing the standard stack trace to
just point to the syntax error instead of the dynamic import. But on
further thought this way has parity with SWC's syntax errors + has the
advantage of showing both the syntax error and dynamic import location.
```ts
// temp.js
await import("./temp2.js");
// temp2.js
function foo() {
await Promise.resolve();
}
// Before:
// error: Uncaught (in promise) SyntaxError: Unexpected reserved word
// await import("./temp2.js");
// ^
// at async file:///.../temp.js:1:1
// After:
// error: Uncaught (in promise) SyntaxError: Unexpected reserved word at file:///.../temp2.js:2:3
// await import("./temp2.js");
// ^
// at async file:///.../temp.js:1:1
```
This reloads an npm package's dependency's information when a
version/version req/tag is not found.
This PR applies only to dependencies of npm packages. It does NOT yet
cause npm specifiers to have their dependency information cache busted.
That requires a different solution, but this should help cache bust in
more scenarios.
Part of #16901, but doesn't close it yet
Turns out `autoprefixer` is a better reproduction case then
`microbundle`.
Fixes #18535
Fixes #18600
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This introduces a new built-in `unset` command to `deno task` by bumping
deno_task_shell to the latest version 0.11.0. Also this includes a fix
on how `deno task` handles empty environment variables (see
https://github.com/denoland/deno_task_shell/pull/80 for details).
This is a follow-on to the earlier work in reducing string copies,
mainly focused on ensuring that ASCII strings are easy to provide to the
JS runtime.
While we are replacing a 16-byte reference in a number of places with a
24-byte structure (measured via `std::mem::size_of`), the reduction in
copies wins out over the additional size of the arguments passed into
functions.
Benchmarking shows approximately the same if not slightly less wallclock
time/instructions retired, but I believe this continues to open up
further refactoring opportunities.
This commit updates the `Deno.Kv` API to return the new commited
versionstamp for the mutated data from `db.set` and `ao.commit`. This is
returned in the form of a `Deno.KvCommitResult` object that has a
`versionstamp` property.
This reverts commit a3529d0232.
This change made debugging Node tests very hard - `AssertionError` is
now printed as `[Circular *1]` giving no visibility what failed.
We need to align two implementations together and remove this one then.
1. Fixes a cosmetic issue in the repl where it would display lsp warning
messages.
2. Lazily loads dependencies from the package.json on use.
3. Supports using bare specifiers from package.json in the REPL.
Closes #17929
Closes #18494
We currently run the all test cases in `parallel` category at the same
time, which invokes hundreds process at the same time, and that seems
causing some flakiness in CI. (maybe related to #18487)
This PR limits the concurrency to the number of cpu cores. This is more
aligned to how Node.js run their `parallel` test in their repository.
42c4a35952/Makefile (L356)
This commit adds associated type to "NodeEnv" trait, called "Fs".
The "Fs" type has a trait bound on "NodeFs", which specifies APIs
required for all ops and resolution APIs to function.
A "RealFs" implementation of "NodeFs" is exported from the "deno_node"
crate, that provides a default implementation for the trait.
All code in "deno_node" extension was changed to use the "NodeFs" trait
to handle file system operations, instead of relying on APIs from the
standard library.
This commit changes the type parameter for "deno_node" extension, from
`P: NodePermission` to `Env: NodeEnv`.
`NodeEnv` is a new trait that has associated type `P: NodePermission`.
This is a stepping stone to support swappable file system for the
extension, that will be added as a second associated type to the
`NodeEnv` trait.
1. Rewrites the tests to be more back and forth rather than getting the
output all at once (which I believe was causing the hangs on linux and
maybe mac)
2. Runs the pty tests on the linux ci.
3. Fixes a bunch of tests that were just wrong.
4. Adds timeouts on the pty tests.
Fast-follow on #18401 -- the reason that some tests were panicking in
the `CacheDB` `impl Drop` was that the cache itself was being dropped
during panic and the runtime may or may not still exist at that point.
We can reduce the actual tokio runtime testing to where it's needed.
In addition, we return the journal mode to `TRUNCATE` to avoid the risk
of data corruption.
Towards #18455
This commit implements `checkPrimeSync` and `checkPrime` in node:crypto
using the Miller-Rabin primality test (fun fact: it actually is a test
for composite numbers)
It first compares the candidate against many known small primes and if
not, proceeds to run the Miller-Rabin primality test.
http://nickle.org/examples/miller-rabin.5c used as reference
implementation.
This gets SQLite off the flamegraph and reduces initialization time by
somewhere between 0.2ms and 0.5ms. In addition, I took the opportunity
to move all the cache management code to a single place and reduce
duplication. While the PR has a net gain of lines, much of that is just
being a bit more deliberate with how we're recovering from errors.
The existing caches had various policies for dealing with cache
corruption, so I've unified them and tried to isolate the decisions we
make for recovery in a single place (see `open_connection` in
`CacheDB`). The policy I chose was:
1. Retry twice to open on-disk caches
2. If that fails, try to delete the file and recreate it on-disk
3. If we fail to delete the file or re-create a new cache, use a
fallback strategy that can be chosen per-cache: InMemory (temporary
cache for the process run), BlackHole (ignore writes, return empty
reads), or Error (fail on every operation).
The caches all use the same general code now, and share the cache
failure recovery policy.
In addition, it cleans up a TODO in the `NodeAnalysisCache`.
This internal node hook is used by libraries such as `ts-node` when used
as a require hook `node -r ts-node/register`. That combination is often
used with test frameworks like `mocha` or `jasmine`.
We had a reference to `Module._preloadModules` in our code, but the
implementation was missing. While fixing this I also noticed that the
`fakeParent` module that we create internally always threw because of
the `pathDirname` check on the module id in the constructor of `Mdoule`.
So this code path was probably broken for a while.
```txt
✖ ERROR: Error: Empty filepath.
at pathDirname (ext:deno_node/01_require.js:245:11)
at new Module (ext:deno_node/01_require.js:446:15)
at Function.Module._resolveFilename (ext:deno_node/01_require.js:754:28)
at Function.resolve (ext:deno_node/01_require.js:1015:19)
```
This commit adds the `crypto.createSecretKey` API.
Key management: This follows the same approach as our WebCrypto
CryptoKey impl where we use WeakMap for storing key material and a
handle is passed around, such that (only internal) JS can access the key
material and we don't have to explicitly close a Rust resource.
As a result, `createHmac` now accepts a secret KeyObject.
Closes https://github.com/denoland/deno/issues/17844
This will improve diagnostics and catch any non-ASCII extension code
early.
This will use `debug_assert!` rather than `assert!` to avoid runtime
costs, and ensures (in debug_assert mode only) that all extension source
files are ASCII as we load them.
This will make it a bit harder to accidentally use a client url in the
wrong place. I don't fully understand why we do this mapping, but this
will help prevent bugs like #18373
Closes #18374
No need for two almost identical implementations of the same thing
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
Co-authored-by: Aapo Alasuutari <aapo.alasuutari@gmail.com>
Previously the mapping between `AnyValue::Bool` and `KeyPart::Bool` was
inverted.
This patch also allows using the empty key `[]` as range start/end to
`snapshot_read`.
Currently `Deno.openKv(":memory:")` requests read+write permissions for
`./:memory:` even though no file is read or written. Also added some
guards for special sqlite paths that were unintentionally opted into.
This commit adds unstable "Deno.openKv()" API that allows to open
a key-value database at a specified path.
---------
Co-authored-by: Luca Casonato <hello@lcas.dev>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
…'t automatically re-created (#18330)"
This reverts commit 2ef8269fdb.
Printing these messages by default (instead of requiring `-L debug`
flag) caused
various tests to start printing it and mismatch in output assertions.
Reduce the number of copies and allocations of script code by carrying
around ownership/reference information from creation time.
As an advantage, this allows us to maintain the identity of `&'static
str`-based scripts and use v8's external 1-byte strings (to avoid
incorrectly passing non-ASCII strings, debug `assert!`s gate all string
reference paths).
Benchmark results:
Perf improvements -- ~0.1 - 0.2ms faster, but should reduce garbage
w/external strings and reduces data copies overall. May also unlock some
more interesting optimizations in the future.
This requires adding some generics to functions, but manual
monomorphization has been applied (outer/inner function) to avoid code
bloat.
This commit disables compression of the TSC snapshot.
The compression only decreased the size of snapshot by 0.5Mb
and it took about 40s during release build to compress.
With recent gains in TS 5.0 upgrade in terms of size and performance
it makes sense to remove this compression.
This PR changes the inspect result of anonymous functions from
`[Function]` to `[Function (anonymous)]`. This behavior is aligned
to `util.inspect` of Node.js.
Moving some code around in `ext/node` is it's a bit better well defined
and makes it possible for others to embed it.
I expect to see no difference in startup perf with this change.
These functions don't need to be async, as they are only calling
synchronous JavaScript code. As a follow up, all 3 functions
should be merge together - this will reduce roundtrips for
calling V8 from Rust, which is somewhat expensive
This commit adds support for spawning Web Workers in self-contained
binaries created with "deno compile" subcommand.
As long as module requested in "new Worker" constructor is part of the
eszip (by means of statically importing it beforehand, or using "--include"
flag), then the worker can be spawned.
This change will enable dynamic imports and web workers to use modules
not reachable from the main module, by passing a list of extra side
module roots as options to `deno compile`.
This can be done by specifying "--include" flag that accepts a file path or a
URL. This flag can be specified multiple times, to include several modules.
The modules specified with "--include" flag, will be added to the produced
"eszip".
This commit changes the build process in a way that preserves already
registered ops in the snapshot. This allows us to skip creating hundreds of
"v8::String" on each startup, but sadly there is still some op registration
going on startup (however we're registering 49 ops instead of >200 ops).
This situation could be further improved, by moving some of the ops
from "runtime/" to a separate extension crates.
---------
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Follow-up to #18210:
* we are passing the generated `cfg` object into the state function
rather than passing individual config fields
* reduce cloning dramatically by making the state_fn `FnOnce`
* `take` for `ExtensionBuilder` to avoid more unnecessary copies
* renamed `config` to `options`
This implements two macros to simplify extension registration and centralize a lot of the boilerplate as a base for future improvements:
* `deno_core::ops!` registers a block of `#[op]`s, optionally with type
parameters, useful for places where we share lists of ops
* `deno_core::extension!` is used to register an extension, and creates
two methods that can be used at runtime/snapshot generation time:
`init_ops` and `init_ops_and_esm`.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit removes compression for the runtime JS code.
It means that we will have a bigger binary, but faster startup. After
several discussion in the CLI team we decided it's worth to trade
about 3Mb of binary size for 2ms faster startup time. With WebGPU
removed in 35196eab27
it shouldn't have such a big impact on the binary size.
This PR _**temporarily**_ removes WebGPU (which has behind the
`--unstable` flag in Deno), due to performance complications due to its
presence.
It will be brought back in the future; as a point of reference, Chrome
will ship WebGPU to stable on 26/04/2023.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Closes #17831. This change hides the indices of any indexed collection
when triggering tab completion for object properties in the REPL.
An example is shown in the issue, but for verbosity here is another.
Before the change:
```
> const arr = new Uint8ClampedArray([1, 2, 3])
undefined
> arr.
0 map
1 reverse
2 reduce
...
```
After the change:
```
> const arr = new Uint8ClampedArray([1, 2, 3])
undefined
> arr.
constructor reduce
BYTES_PER_ELEMENT reduceRight
buffer set
...
```
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
To be able to preserve "Deno.core.ops" we need to ensure that
ops are registered in the same order in various places, otherwise
we will get mismatch in external references ordering.
Prerequisite for https://github.com/denoland/deno/pull/18080
This commit adds support for retrieving `dev` information
when stating files on Windows.
Additionally `Deno.FileInfo` interfaces was changed to always
return 0 for fields that we don't retrieve information for on Windows.
Closes https://github.com/denoland/deno/issues/18053
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
These methods are confusing because the arguments are backwards. I feel
like they should have never been added to `Option<T>` and that clippy
should suggest rewriting to
`map(...).unwrap_or(...)`/`map(...).unwrap_or_else(|| ...)`
https://github.com/rust-lang/rfcs/issues/1025
Chrono's `clock` feature pulls in `iana-time-zone` which links to macOS
core_foundation. This PR itself is not enough to get rid of
CoreFoundation. Removal depends on getting rid of security framework,
see #18071
This commit changes "deno_core" to not rely on implicitly calling
"std::env::current_dir()" when resolving module specifiers using
APIs from "deno_core::modules_specifier".
Supersedes https://github.com/denoland/deno/pull/15454
Remove remaining usages of "resolve_url_or_path_deprecated" in favor
of "resolve_url_or_path" with explicit calls to
"std::env::current_dir()".
Towards landing https://github.com/denoland/deno/pull/15454
This commit changes various CLI subcommands that have support for
the "--watch" flag to use initial current working directory when
resolving "main module".
This is part of migration towards explicitly passing current working
directory to "deno_core::resolve_url_or_path" API.
As a side effect this makes "deno <subcommand> --watch" more aligned to
user expectations, where calling "Deno.chdir()" during program doesn't
break watcher.
Towards landing https://github.com/denoland/deno/pull/15454
This commit changes current "deno_core::resolve_url_or_path" API to
"resolve_url_or_path_deprecated" and adds new "resolve_url_or_path"
API that requires to explicitly pass the directory from which paths
should be resolved to.
Some of the call sites were updated to use the new API, the reminder
of them will be updated in a follow up PR.
Towards landing https://github.com/denoland/deno/pull/15454
Creating the node_modules folder when the packages are already
downloaded can take a bit of time and not knowing what is going on can
be confusing. It's better to show a progress bar.
This has been bothering me for a while and it became more painful while
working on #18136 because injecting the shared progress bar became very
verbose. Basically we should move the creation of all these npm structs
up to a higher level.
This is a stepping stone for a future refactor where we can improve how
we create all our structs.
This commit removes "deno_core::RuntimeOptions::extensions_with_js".
Now it's embedders' responsibility to properly register extensions
that will not contains JavaScript sources when running from an existing
snapshot.
Prerequisite for https://github.com/denoland/deno/pull/18080
This commit partially reverts changes from
https://github.com/denoland/deno/pull/18095.
Turns out I made a mistake that became apparent when working
on removing "RuntimeOptions::extensions_with_js" in a follow up.
This commit splits "<ext_name>::init" functions into "init_ops" and
"init_ops_and_esm". That way we don't have to construct list of
ESM sources on each startup if we're running with a snapshot.
In a follow up commit "deno_core" will be changed to not have a split
between "extensions" and "extensions_with_js" - it will be embedders'
responsibility to pass appropriately configured extensions.
Prerequisite for https://github.com/denoland/deno/pull/18080
This improves peer dependency resolution yet again. We did not handle
scenarios like the following:
```
// a -> b -> c -> d -> c -> b (peer)
```
...which would maybe work ok the first time its run in some cases, but
then lead to a lockfile that would error on load.
This now keeps track of circular dependencies and updates nodes
accordingly. That said, there is still a lurking bug in this code
somewhere that I've added a comment for (there is a mitigation on the
tail end that seems to work well). The current state is much better than
before and I can look into it later. I think it's something small that's
incorrect.
```
Benchmark 1: deno run -A ../empty.js
Time (mean ± σ): 20.5 ms ± 0.5 ms [User: 13.4 ms, System: 5.1 ms]
Range (min … max): 19.8 ms … 24.0 ms 119 runs
Benchmark 2: target/release/deno run -A ../empty.js
Time (mean ± σ): 18.8 ms ± 0.3 ms [User: 13.0 ms, System: 4.9 ms]
Range (min … max): 18.3 ms … 19.9 ms 129 runs
```
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This is implemented in such a way that it should still allow processes
to go through when a file lock wasn't properly cleaned up and the OS
hasn't released it yet (but with a 200ms-ish delay).
Closes #18039
This commit renames "deno_core::InternalModuleLoader" to
"ExtModuleLoader" and changes the specifiers used by the
modules loaded from this loader to "ext:".
"internal:" scheme was really ambiguous and it's more characters than
"ext:", which should result in slightly smaller snapshot size.
Closes https://github.com/denoland/deno/issues/18020
Since we are snapshotting extension source at build time, there's no
need to define list of sources for the extension at runtime.
This commit changes "deno_node" extension by removing "init_polyfill"
function in favor of "init_polyfill_ops_and_esm()" and "init_polyfill_ops()".
The former is used during snapshot and when "deno_runtime" is compiled
with "dont_create_runtime_snapshot" cargo feature flag. The latter is used
when running a worker from an existing snapshot.
This is a start of a bigger refactor to all extensions - thanks to this
change, we don't have to iterate over all defined source files for extension at
runtime, and because of that we don't have to create a filepath for each of the
source files. It's not a big deal, but we are iterating over 300 files on each start,
and concatenating 3 strings before creating a "PathBuf" for ~200 of them.
This is already visible on the startup flamegraphs and should be avoided.
There's no point for this API to expect result. If something fails it should
result in a panic during build time to signal to embedder that setup is
wrong.
We use information about build in several extension crates like
"ext/node" or "runtime/". In an effort to move "fs" APIs to a separate
crate it is a prerequisite to have this information available outside
of the "runtime/" crate.
This commit moves definition of "build" object to "Deno.core" that is
later forwarded to "Deno.build".
JavaScript APIs from "runtime/js/40_files.js" combined abstractions
for stdio streams ("Stdout", "Stderr", "Stdin") and file system file
("File", "FsFile"). APIs from "runtime/js/40_read_file.js" and
"runtime/js/40_write_file.js" were implemented using ops from
"runtime/ops/fs.rs".
This file was removed and relevant APIs were moved to "deno_io/12_io.js"
and "runtime/js/30_fs.js".
This work is meant to enable factoring out "deno_fs" crate.
This commit changes "include_js_files!" macro from "deno_core"
in a way that "dir" option doesn't cause specifiers to be rewritten
to include it.
Example:
```
include_js_files! {
dir "js",
"hello.js",
}
```
The above definition required embedders to use:
`import ... from "internal:<ext_name>/js/hello.js"`.
But with this change, the "js" directory in which the files are stored
is an implementation detail, which for embedders results in:
`import ... from "internal:<ext_name>/hello.js"`.
The directory the files are stored in, is an implementation detail and
in some cases might result in a significant size difference for the
snapshot. As an example, in "deno_node" extension, we store the
source code in "polyfills" directory; which resulted in each specifier
to look like "internal:deno_node/polyfills/<module_name>", but with
this change it's "internal:deno_node/<module_name>".
Given that "deno_node" has over 100 files, many of them having
several import specifiers to the same extension, this change removes
10 characters from each import specifier.
denoland/eszip#115 added support for statically-analyzed dynamic imports
in eszip, which made `deno compile` support dynamic imports starting
from #17858. This PR adds a test for it.
----
This test is adapted from PR #17663.
Closes #17908
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
This commit updates deno_lint crate to 0.41.0. The new version contains
a braking change that requries a minor code fix here, which is also
addressed in this commit.
This lazily does an "npm install" when any package name matches what's
found in the package.json or when running a script from package.json
with deno task.
Part of #17916
Closes #17928
This is a super basic initial implementation. We don't create a
`node_modules/.bin` folder at the moment and add it to the PATH like we
should which is necessary to make command name resolution in the
subprocess work properly (ex. you run a script that launches another
script that then tries to launch an "npx command"... this won't work
atm).
Closes #17492
This commit enables resolution of "bare specifiers" (eg. "import express
from 'express';") if a "package.json" file is discovered.
It's a step towards being able to run projects authored for Node.js
without any changes.
With this commit we are able to successfully run Vite projects without
any changes to the user code.
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
This commit adds new "A" option to the interactive permission prompt, that will
allow all subsequent permissions for given group (domain). Ie. when querying for
permissions to access eg. env variables responding with "A" will allow access
to all environmental variables.
This works for all permission domains and should make permission prompts
more ergonomic for users.
This changes npm specifiers to be handled by deno_graph and resolved to
an npm package name and version when the specifier is encountered. It
also slightly changes how npm specifier resolution occurs—previously it
would collect all the npm specifiers and resolve them all at once, but
now it resolves them on the fly as they are encountered in the module
graph.
https://github.com/denoland/deno_graph/pull/232
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Deno.Command needs to be stabilized first and allow people to upgrade to
it before we can deprecate Deno.run. Otherwise lint will suddenly fail
with deprecated errors without giving people a chance to update.
This PR fixes peer dependency resolution to only resolve peers based on
the current graph traversal path. Previously, it would resolve a peers
by looking at a graph node's ancestors, which is not correct because
graph nodes are shared by different resolutions.
It also stores more information about peer dependency resolution in the
lockfile.
This allows to not include source code into the binary (because
it will already be included in the V8 snapshot).
Nothing changes for the embedders - everything should still build the
same.
This commit brings the binary size from 87Mb to 82Mb on M1.
Alternative to https://github.com/denoland/deno/pull/17820 and
https://github.com/denoland/deno/pull/17653
---------
Co-authored-by: Leo Kettmeir <crowlkats@toaxl.com>
This commits adds auto-discovery of "package.json" file when running
"deno run" and "deno task" subcommands. In case of "deno run" the
"package.json" is being looked up starting from the directory of the
script that is being run, stopping early if "deno.json(c)" file is found
(ie. FS tree won't be traversed "up" from "deno.json").
When "package.json" is discovered the "--node-modules-dir" flag is
implied, leading to creation of local "node_modules/" directory - we
did that, because most tools relying on "package.json" will expect
"node_modules/" directory to be present (eg. Vite). Additionally
"dependencies" and "devDependencies" specified in the "package.json"
are downloaded on startup.
This is a stepping stone to supporting bare specifier imports, but
the actual integration will be done in a follow up commit.
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
This PR adds the remaining ~650 Node.js compat test cases from std/node.
Among these 650 cases, about 130 cases are now failing. These failing
cases are prefixed with `TODO:` in `tests/node_compat/config.json`.
These will be addressed in later PRs.
This commit changes definition of "ExtensionFileSource", by changing
"code" field to being "ExtensionFileSourceCode" enum. Currently the enum
has only a single variant "IncludedInBinary". It is done in preparation
to allow embedders to decide if they want to include the source code in the
binary when snapshotting (in most cases they shouldn't do that).
In the follow up commit we'll add more variants to
"ExtensionFileSourceCode".
"include_js_files_dir!" macro was removed in favor "include_js_files!"
macro which can now accept "dir" option.
The `NpmPackageId` struct is being renamed to `NpmPackageNodeId`. In a
future PR it will be moved down into only npm dependency resolution and
a `NpmPackageId` struct will be introduced in `deno_graph` that only has
the name and version of the package (no peer dependency identifier
information). So a `NpmPackageReq` will map to an `NpmPackageId`, which
will map to an `NpmPackageNodeId` in the npm resolution.
Adds two test files: "cli/tests/unit_node/process_test.ts" and
"cli/tests/unit_node/child_process_test.ts"
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This PR changes Node.js/npm compatibility layer to use polyfills for
built-in Node.js
embedded in the snapshot (that are coming from "ext/node" extension).
As a result loading `std/node`, either from
"https://deno.land/std@<latest>/" or
from "DENO_NODE_COMPAT_URL" env variable were removed. All code that is
imported via "npm:" specifiers now uses code embedded in the snapshot.
Several fixes were applied to various modules in "ext/node" to make
tests pass.
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
This commit moves "deno_std/node" in "ext/node" crate. The code is
transpiled and snapshotted during the build process.
During the first pass a minimal amount of work was done to create the
snapshot, a lot of code in "ext/node" depends on presence of "Deno"
global. This code will be gradually fixed in the follow up PRs to migrate
it to import relevant APIs from "internal:" modules.
Currently the code from snapshot is not used in any way, and all
Node/npm compatibility still uses code from
"https://deno.land/std/node" (or from the location specified by
"DENO_NODE_COMPAT_URL"). This will also be handled in a follow
up PRs.
---------
Co-authored-by: crowlkats <crowlkats@toaxl.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This commit stabilizes Node-API, the "--unstable" flag is no longer
required to load native extensions. "--allow-ffi" permission is still
required to load them.
Compressing the TSC snapshot in debug build took
~45s on M1 MacBook Pro; without compression it took ~1s.
Thus we're not not using compressed snapshot, trading off
a lot of build time for some startup time in debug build.
This was not caught in the previous test case, as the response body was
smaller than the size of `HEAD` response.
This made `nwritten < responseLen` check in `writeFixedResponse` to
fail, and not trigger `op_flash_respond_async` as a result.
When the response body is larger than the `HEAD` though, as in the
updated test case (`HEAD` i 120 bytes, where our response is 300 bytes),
it would think that we still have something to send, and effectively
panic, as `op_flash_respond` already removed the request from the pool.
This change, makes the `handleResponse` function always calculate the
number of bytes to transmit when `HEAD` request is encountered.
Effectively ignoring `Content-Length` of the body, but still setting it
correctly in the request header itself.
Fixes https://github.com/denoland/deno/issues/17737
This commit moves some code around from "cli/node/mod.rs" to
"ext/node". Additionally "ext/node" was changed to factor out
"ops.rs" and "polyfill.rs" modules.
This change makes absolute urls, that contain no path like `deno install
https://my-cli.io` to follow redirects and extract the name from it.
It allows modifies `test_util` server listener on port `4550`
(`REDIRECT_ABSOLUTE_PORT`) to allow for specifying `redirect_to` query
param, that fill use that value for it's next redirect.
Fixes https://github.com/denoland/deno/issues/17409
This commit does preparatory work to allow snapshotting Node.js
compatibility layer, that currently lives in `std/node`. The logic was
changed to allow loading some modules from the snapshot and
some from the remote URL.
Additionally "module_es_shim.js" that provides exports for "node:module"
is now snapshotted.
This is a proof of concept for being able to snapshot TypeScript files.
Currently only a single runtime file is authored in TypeScript -
"runtime/js/01_version.ts".
Not needed infrastructure was removed from "core/snapshot_util.rs".
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This PR refactors all internal js files (except core) to be written as
ES modules.
`__bootstrap`has been mostly replaced with static imports in form in
`internal:[path to file from repo root]`.
To specify if files are ESM, an `esm` method has been added to
`Extension`, similar to the `js` method.
A new ModuleLoader called `InternalModuleLoader` has been added to
enable the loading of internal specifiers, which is used in all
situations except when a snapshot is only loaded, and not a new one is
created from it.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
- Generalizes the npm version code (ex. `NpmVersion` -> `Version`,
`NpmVersionReq` -> `VersionReq`). This is a slow refactor towards
extracting out this code for deno specifiers and better usage in
deno_graph.
- Removes `SpecifierVersionReq`. Consolidates `NpmVersionReq` and
`SpecifierVersionReq` to just `VersionReq`
- Removes `NpmVersionMatcher`. This now just looks at `VersionReq`.
- Paves the way to allow us to create `NpmPackageReference`'s from a
package.json's dependencies/dev dependencies
(`VersionReq::parse_from_npm`).
This commit changes handling of config file to enable
specifying "imports" and "scopes" objects effectively making
the configuration file an import map.
"imports" and "scopes" take precedence over "importMap" configuration,
but have lower priority than "--importmap" CLI flag.
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
Co-authored-by: David Sherret <dsherret@gmail.com>
This commit adds sync versions of async APIs to "Deno.permissions"
namespace.
Following APIs were added:
- "Deno.permissions.querySync"
- "Deno.permissions.requestSync"
- "Deno.permissions.revokeSync"
Allows to change behavior of `deno fmt` to use "ASI" setting for
semicolons instead of always prefering them, this is done
by "--options-semi=asi" flag or `"semi": "asi"` setting
in the config file.
This commit removes "Deno.core" namespace. It is strictly private API
that has no stability guarantees, we were supposed to remove it long time ago.
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This change prints a link to the release notes to `deno upgrade` output
and its variations.
Release notes aren't printed for commands relating to canary versions.
Closes #16350.
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
The way the standalone mode handles the `--cert` flag is different to
all other modes. This is because `--cert` takes a path to the
certificate file, which is directly added to the root cert store; except
for compile mode, where its byte contents are stored in the standalone
metadata, and they are added to the root cert store after the
`ProcState` is created.
This change instead changes `Flags::ca_file` (an `Option<String>`) into
`Flags::ca_data`, which can represent a `String` file path or a
`Vec<u8>` with the certificate contents. That way, standalone mode can
create a `ProcState` whose root cert store alreay contains the
certificate.
This change also adds a tests for certificates in standalone mode, since
there weren't any before.
This refactor will help with implementing web workers in standalone mode
in the future.
Updated third_party dlint to v0.37.0 for GitHub Actions. This PR
includes following changes:
* fix(prefer-primordials): Stop using array pattern assignments
* fix(prefer-primordials): Stop using global intrinsics except for
`SharedArrayBuffer`
* feat(guard-for-in): Apply new guard-for-in rule
I'm not sure this properly handles scenarios where an npm package uses
an alias that resolves to itself, we can fix that if we find a package
that actually depends on that behavior.
Closes #17420
The leading cause of the problem was that `handleResponse` has
`tryRespondChunked` passed as an argument, which in turn is implemented
as a call to `core.ops.op_try_flash_respond_chuncked`, that throws in
the repro code.
`handleResponse` was not handled correctly, as it not returned any
value, and had no `catch` attached to it.
It also effectively was never correctly handled inside two other blocks
with `resp.then` and `PromisePrototypeCatch(PromisePrototypeThen(resp,
"..."))` as well, as it just short-circuited the promise with an empty
resolve, instead of relying on the last `(async () => {})` block.
This change makes `handleResponse` return a correct value and attach
`onError` handler to the "non-thenable" variant of response handling
code.
This commit removes the libuv pollyfills introduced with Node-API
support.
It is too much Node-specific. Most Node-API modules that depend on libuv
are already giving up the benefits of Node-API. We should rather
encourage modules to use `ThreadSafeFunction` or `AsyncWork` to
interface with the event loop.
Relevant discussion a few months ago:
https://github.com/denoland/deno/pull/13633#discussion_r904916178
cc @bartlomieju
This commit changes "ProcState" to store "file_fetcher" field in an "Arc",
allowing it to be preserved between restarts and thus keeping the state
alive between the restarts. File watchers for "deno test" and "deno bench"
now reset "ProcState" between restarts.
This commit changes signature of "deno_core::ModuleLoader::resolve" to pass
an enum indicating whether or not we're resolving a specifier for dynamic import.
Additionally "CliModuleLoader" was changes to store both "parent permissions" (or
"root permissions") as well as "dynamic permissions" that allow to check for permissions
in top-level module load an dynamic imports.
Then all code paths that have anything to do with Node/npm compat are now checking
for permissions which are passed from module loader instance associated with given
worker.
This PR updates the name used in `clap::Arg::value_name` for the
`--inspect*` flags from `HOST:PORT` to `HOST_AND_PORT` because the
former causes an arguments error when using shell completions in the
`zsh` shell.
Adds support for passing and returning structs as buffers to FFI. This does not implement fastapi support for structs. Needed for certain system APIs such as AppKit on macOS.
Turns out we were cloning permissions which after prompting were discarded,
so the state of permissions was never preserved. To handle that we need to store
all permissions behind "Arc<Mutex<>>" (because there are situations where we
need to send them to other thread).
Testing and benching code still uses "Permissions" in most places - it's undesirable
to share the same permission set between various test/bench files - otherwise
granting or revoking permissions in one file would influence behavior of other test
files.
If an optional peer dependency entry previously wasn't resolved and it's
now being resolved, then it will add it as if it were a dependency of
the previously resolved package instead of creating a new "copy package"
(seems to be what npm and pnpm does).
Closes #17240
