This API is required by several extensions like "ext/node", "ext/ffi"
and also FS APIs that we want to move to a separate crate. Because
of that "pathFromURL" API was moved to "deno_web" extension so
other extension crates can rely on it.
We use information about build in several extension crates like
"ext/node" or "runtime/". In an effort to move "fs" APIs to a separate
crate it is a prerequisite to have this information available outside
of the "runtime/" crate.
This commit moves definition of "build" object to "Deno.core" that is
later forwarded to "Deno.build".
This commit merges "runtime/js/40_spawn.js" into
"runtime/js/40_process.js", and "runtime::ops::spawn"
into "runtime::ops::process".
It makes little sense to have them separated given that we want to
factor out these APIs into a separate extension crate.
JavaScript APIs from "runtime/js/40_files.js" combined abstractions
for stdio streams ("Stdout", "Stderr", "Stdin") and file system file
("File", "FsFile"). APIs from "runtime/js/40_read_file.js" and
"runtime/js/40_write_file.js" were implemented using ops from
"runtime/ops/fs.rs".
This file was removed and relevant APIs were moved to "deno_io/12_io.js"
and "runtime/js/30_fs.js".
This work is meant to enable factoring out "deno_fs" crate.
This file was not used anywhere - even though it exposed an unstable
"Deno.DiagnosticCategory" API. Seems like a left over from when we
had "Deno.emit" API.
This commit changes "include_js_files!" macro from "deno_core"
in a way that "dir" option doesn't cause specifiers to be rewritten
to include it.
Example:
```
include_js_files! {
dir "js",
"hello.js",
}
```
The above definition required embedders to use:
`import ... from "internal:<ext_name>/js/hello.js"`.
But with this change, the "js" directory in which the files are stored
is an implementation detail, which for embedders results in:
`import ... from "internal:<ext_name>/hello.js"`.
The directory the files are stored in, is an implementation detail and
in some cases might result in a significant size difference for the
snapshot. As an example, in "deno_node" extension, we store the
source code in "polyfills" directory; which resulted in each specifier
to look like "internal:deno_node/polyfills/<module_name>", but with
this change it's "internal:deno_node/<module_name>".
Given that "deno_node" has over 100 files, many of them having
several import specifiers to the same extension, this change removes
10 characters from each import specifier.
Runtime generation of async op wrappers contributed to increased startup
time and core became unusable with
`--disallow-code-generation-from-strings` flag. The optimization only
affects very small microbenchmarks so this revert will not cause any
regressions.
This commit adds new "A" option to the interactive permission prompt, that will
allow all subsequent permissions for given group (domain). Ie. when querying for
permissions to access eg. env variables responding with "A" will allow access
to all environmental variables.
This works for all permission domains and should make permission prompts
more ergonomic for users.
This commit further improves startup time by:
- no relying on "JsRuntime::execute_script" for runtime bootstrapping,
this is instead done using V8 APIs directly
- registering error classes during the snapshot time, instead of on
startup
Further improvements can be made, mainly around removing
"core.initializeAsyncOps()" which takes around 2ms.
This commit should result in ~1ms startup time improvement.
This allows to not include source code into the binary (because
it will already be included in the V8 snapshot).
Nothing changes for the embedders - everything should still build the
same.
This commit brings the binary size from 87Mb to 82Mb on M1.
Alternative to https://github.com/denoland/deno/pull/17820 and
https://github.com/denoland/deno/pull/17653
---------
Co-authored-by: Leo Kettmeir <crowlkats@toaxl.com>
This commit changes definition of "ExtensionFileSource", by changing
"code" field to being "ExtensionFileSourceCode" enum. Currently the enum
has only a single variant "IncludedInBinary". It is done in preparation
to allow embedders to decide if they want to include the source code in the
binary when snapshotting (in most cases they shouldn't do that).
In the follow up commit we'll add more variants to
"ExtensionFileSourceCode".
"include_js_files_dir!" macro was removed in favor "include_js_files!"
macro which can now accept "dir" option.
Adds two test files: "cli/tests/unit_node/process_test.ts" and
"cli/tests/unit_node/child_process_test.ts"
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This commit moves "deno_std/node" in "ext/node" crate. The code is
transpiled and snapshotted during the build process.
During the first pass a minimal amount of work was done to create the
snapshot, a lot of code in "ext/node" depends on presence of "Deno"
global. This code will be gradually fixed in the follow up PRs to migrate
it to import relevant APIs from "internal:" modules.
Currently the code from snapshot is not used in any way, and all
Node/npm compatibility still uses code from
"https://deno.land/std/node" (or from the location specified by
"DENO_NODE_COMPAT_URL"). This will also be handled in a follow
up PRs.
---------
Co-authored-by: crowlkats <crowlkats@toaxl.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This commit stabilizes Node-API, the "--unstable" flag is no longer
required to load native extensions. "--allow-ffi" permission is still
required to load them.
This is a proof of concept for being able to snapshot TypeScript files.
Currently only a single runtime file is authored in TypeScript -
"runtime/js/01_version.ts".
Not needed infrastructure was removed from "core/snapshot_util.rs".
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This PR refactors all internal js files (except core) to be written as
ES modules.
`__bootstrap`has been mostly replaced with static imports in form in
`internal:[path to file from repo root]`.
To specify if files are ESM, an `esm` method has been added to
`Extension`, similar to the `js` method.
A new ModuleLoader called `InternalModuleLoader` has been added to
enable the loading of internal specifiers, which is used in all
situations except when a snapshot is only loaded, and not a new one is
created from it.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds sync versions of async APIs to "Deno.permissions"
namespace.
Following APIs were added:
- "Deno.permissions.querySync"
- "Deno.permissions.requestSync"
- "Deno.permissions.revokeSync"
This commit removes "Deno.core" namespace. It is strictly private API
that has no stability guarantees, we were supposed to remove it long time ago.
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
Updated third_party dlint to v0.37.0 for GitHub Actions. This PR
includes following changes:
* fix(prefer-primordials): Stop using array pattern assignments
* fix(prefer-primordials): Stop using global intrinsics except for
`SharedArrayBuffer`
* feat(guard-for-in): Apply new guard-for-in rule
Turns out we were cloning permissions which after prompting were discarded,
so the state of permissions was never preserved. To handle that we need to store
all permissions behind "Arc<Mutex<>>" (because there are situations where we
need to send them to other thread).
Testing and benching code still uses "Permissions" in most places - it's undesirable
to share the same permission set between various test/bench files - otherwise
granting or revoking permissions in one file would influence behavior of other test
files.
Previously, `Deno.permissions.[revoke|request]()` wouldn't correctly
process the `path: URL` when `name` was `ffi`. This change fixes that
behaviour and adds a new function, `formDescriptor()`, to ensure `URL`
arguments are consistently handled across
`Deno.permissions.[query|revoke|request]()`.
This commit changes permission prompt to show that "import()" API
is requesting permissions.
Given "dynamic_import.js" like so:
```
import("https://deno.land/std@0.170.0/version.ts");
```
Before:
```
deno run dynamic_import.js
⚠️ ┌ Deno requests net access to "deno.land".
├ Run again with --allow-net to bypass this prompt.
└ Allow? [y/n] (y = yes, allow; n = no, deny) >
```
After:
```
deno run dynamic_import.js
⚠️ ┌ Deno requests net access to "deno.land".
├ Requested by `import()` API
├ Run again with --allow-net to bypass this prompt.
└ Allow? [y/n] (y = yes, allow; n = no, deny) >
```
This commit refactors several things in "runtime/permissions" module:
- splits it into "mod.rs" and "prompter.rs"
- adds "PermissionPrompter" trait with two implementations:
* "TtyPrompter"
* "TestPrompter"
- adds "before" and "after" prompt callback which can be used to hide
progress bar in the CLI (to be done in a follow up)
- "permissions_prompt" API returns "PromptResponse" enum, instead
of a boolean; this allows to add "allow all"/"deny all" functionality
for the prompt
This commit changes implementation of "Deno.memoryUsage()" to return
correct value for "rss" field. To do that we implement a specialized function
per os to retrieve this information.
This commit changes "--allow-ffi" flag to support "parent paths",
ie. if an FFI library is loaded we are checking if the library has an
ancestor path in the allowlist for the FFI permission descriptor.
This commit adds new "--inspect-wait" flag which works similarly
to "--inspect-brk" in that it waits for inspector session to be
established before running code. However it doesn't break on the first
statement of user code, but instead runs it as soon as a session
is established.
This commit removes three unstable Deno APIs:
- "Deno.spawn()"
- "Deno.spawnSync()"
- "Deno.spawnChild()"
These APIs were replaced by a unified "Deno.Command" API.
Uses SeqOneByteString optimization to do zero-copy `&str` arguments in
fast calls.
- [x] Depends on https://github.com/denoland/rusty_v8/pull/1129
- [x] Depends on
https://chromium-review.googlesource.com/c/v8/v8/+/4036884
- [x] Disable in async ops
- [x] Make it work with owned `String` with an extra alloc in fast path.
- [x] Support `Cow<'_, str>`. Owned for slow case, Borrowed for fast
case
```rust
#[op]
fn op_string_len(s: &str) -> u32 {
str.len() as u32
}
```
Refactors the `Deno.Command` class to not handle any state, but only being an intermediary to calling its methods, and as such any methods and properties besides `output`, `outputSync` & `spawn` have been removed. Interracting with a `spawn`ed subprocess now works by using the methods and properties on the returned class of the `spawn` method.
This commit allows to execute more JS code from extensions when
creating a snapshot from an existing snapshot.
"deno_core::RuntimeOptions::extensions_with_js" field was added
that is used to pass a list of extensions whose both "ops" and
associated JS source should be executed upon start.
Co-authored-by: crowlkats <crowlkats@toaxl.com>
With trial and error I found that most debuggers expect "isDefault" to be sent
in "auxData" field of "executionContextCreated" notification. This stems from
the fact that Node.js sends this data and eg. VSCode requires it to close
connection to the debugger when the program finishes execution.
**This patch**
```
benchmark time (avg) (min … max) p75 p99 p995
------------------------------------------------- -----------------------------
echo deno 23.99 ms/iter (22.51 ms … 33.61 ms) 23.97 ms 33.61 ms 33.61 ms
cat 16kb 24.27 ms/iter (22.5 ms … 35.21 ms) 24.2 ms 35.21 ms 35.21 ms
cat 1mb 25.88 ms/iter (25.04 ms … 30.28 ms) 26.12 ms 30.28 ms 30.28 ms
cat 15mb 38.41 ms/iter (35.7 ms … 50 ms) 38.31 ms 50 ms 50 ms
```
**main**
```
benchmark time (avg) (min … max) p75 p99 p995
------------------------------------------------- -----------------------------
echo deno 35.66 ms/iter (34.53 ms … 41.84 ms) 35.79 ms 41.84 ms 41.84 ms
cat 16kb 35.99 ms/iter (34.52 ms … 44.94 ms) 36.05 ms 44.94 ms 44.94 ms
cat 1mb 38.68 ms/iter (36.67 ms … 50.44 ms) 37.95 ms 50.44 ms 50.44 ms
cat 15mb 48.4 ms/iter (46.19 ms … 58.41 ms) 49.16 ms 58.41 ms 58.41 ms
```
This PR adds copies of several unstable APIs that are available
in "Deno[Deno.internal].nodeUnstable" namespace.
These copies do not perform unstable check (ie. don't require
"--unstable" flag to be present). Otherwise they work exactly
the same, including permission checks.
These APIs are not meant to be used by users directly and
can change at any time.
Copies of following APIs are available in that namespace:
- Deno.spawnChild
- Deno.spawn
- Deno.spawnSync
- Deno.serve
- Deno.upgradeHttpRaw
- Deno.listenDatagram
This commit makes "npm:" specifiers not require "--unstable" flag.
At the moment some APIs used by Node polyfills still require
"--unstable" which will be addressed in follow up PRs.
This PR makes it possible for applications to create workers from custom
snapshots to improve runtime performance (without having to fork/copy
`runtime/workers.rs`).
V8's JIT can do a better job knowing the argument count and also enable
fast call path (in future).
This also lets us call async ops without `opAsync`:
```js
const { ops } = Deno.core;
await ops.op_void_async();
```
this patch: 4405286 ops/sec
main: 3508771 ops/sec
This commit stabilizes "Deno.consoleSize()" API.
There is one change compared to previous unstable API,
in that the API doesn't accept any arguments. Console size
is established by querying syscalls for stdio streams at fd
0, 1 and 2.
Previously `op_net_listen`, `op_net_accept`, and various other ops in
ext/net where variadic on the transport. This created a lot of code
bloat. This commit updates the code to instead have separate ops for
each transport.
This commit removes the calls to `expect()` on `std::rc::Rc`, which caused
Deno to panic under certain situations. We now return an error if `Rc`
is referenced by other variables.
Fixes #9360
Fixes #13345
Fixes #13926
Fixes #16241
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: bartlomieju <bartlomieju@users.noreply.github.com>
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
Co-authored-by: denobot <33910674+denobot@users.noreply.github.com>
Co-authored-by: bartlomieju <bartlomieju@users.noreply.github.com>
This change adds `windowsRawArguments` to `SpawnOptions`. The option enables
skipping the default quoting and escaping while creating the command on
windows.
The option works in a similar way as `windowsVerbatimArguments` in
child_process.spawn options in Node.js, and is necessary for simulating
it in `std/node`.
closes #8852
This adds an implementation of `Default` for both `WorkerOptions` and
`BootstrapOptions`. Since both of these structs are rather big, this
should make it easier for people unfamiliar with the internals to focus
on the options relevant to them.
As a user of `deno_runtime` I feel like these should serve as good
defaults, getting people them started without having to tweak the
runtime. Additionally even if some changes are made, the usage of
`..Default::default()` will significantly help with code clarity and
verbosity.
This commit adds a new op_write_all to core that allows writing an
entire chunk in a single async op call. Internally this calls
`Resource::write_all`.
The `writableStreamForRid` has been moved to `06_streams.js` now, and
uses this new op. Various other code paths now also use this new op.
Closes #16227
This commit introduces two new buffer wrapper types to `deno_core`. The
main benefit of these new wrappers is that they can wrap a number of
different underlying buffer types. This allows for a more flexible read
and write API on resources that will require less copying of data
between different buffer representations.
- `BufView` is a read-only view onto a buffer. It can be backed by
`ZeroCopyBuf`, `Vec<u8>`, and `bytes::Bytes`.
- `BufViewMut` is a read-write view onto a buffer. It can be cheaply
converted into a `BufView`. It can be backed by `ZeroCopyBuf` or
`Vec<u8>`.
Both new buffer views have a cursor. This means that the start point of
the view can be constrained to write / read from just a slice of the
view. Only the start point of the slice can be adjusted. The end point
is fixed. To adjust the end point, the underlying buffer needs to be
truncated.
Readable resources have been changed to better cater to resources that
do not support BYOB reads. The basic `read` method now returns a
`BufView` instead of taking a `ZeroCopyBuf` to fill. This allows the
operation to return buffers that the resource has already allocated,
instead of forcing the caller to allocate the buffer. BYOB reads are
still very useful for resources that support them, so a new `read_byob`
method has been added that takes a `BufViewMut` to fill. `op_read`
attempts to use `read_byob` if the resource supports it, which falls
back to `read` and performs an additional copy if it does not. For
Rust->JS reads this change should have no impact, but for Rust->Rust
reads, this allows the caller to avoid an additional copy in many
scenarios. This combined with the support for `BufView` to be backed by
`bytes::Bytes` allows us to avoid one data copy when piping from a
`fetch` response into an `ext/http` response.
Writable resources have been changed to take a `BufView` instead of a
`ZeroCopyBuf` as an argument. This allows for less copying of data in
certain scenarios, as described above. Additionally a new
`Resource::write_all` method has been added that takes a `BufView` and
continually attempts to write the resource until the entire buffer has
been written. Certain resources like files can override this method to
provide a more efficient `write_all` implementation.
This is the release commit being forwarded back to main for 1.26.1
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.26.1 && git checkout -b forward_v1.26.1 upstream/forward_v1.26.1
```
Don't need this PR? Close it.
cc @cjihrig
Co-authored-by: cjihrig <cjihrig@users.noreply.github.com>
This PR implements the NAPI for loading native modules into Deno.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: DjDeveloper <43033058+DjDeveloperr@users.noreply.github.com>
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
We can use Resource::read_return & op_read instead. This allows HTTP
request bodies to participate in FastStream.
To make this work, `readableStreamForRid` required a change to allow non
auto-closing resources to be handled. This required some minor changes
in our FastStream paths in ext/http and ext/flash.
v0.1.3 contains code that will stop working with newer versions of
libstd because the layout of some std::net types changed.
Refs: https://github.com/bnoordhuis/netif/pull/10
Stop allowing clippy::derive-partial-eq-without-eq and fix warnings
about deriving PartialEq without also deriving Eq.
In one case I removed the PartialEq because it a) wasn't necessary,
and b) sketchy because it was comparing floating point numbers.
IMO, that's a good argument for enforcing the lint rule, because it
would most likely have been caught during review if it had been enabled.
This commit allows the Node compatibility layer to skip
environment variable permission checks when --unstable
is passed and the variable name is one that Node uses.
Fixes: https://github.com/denoland/deno/issues/15890
This commit splits `Deno.upgradeHttp` into two different APIs, because
the same API is currently overloaded with two different functions. Flash
requests upgrade immediately, with no need to return a `Response`
object. Instead you have to manually write the response to the socket.
Hyper requests only upgrade once a `Response` object has been sent.
These two behaviours are now split into `Deno.upgradeHttp` and
`Deno.upgradeHttpRaw`. The latter is flash only. The former only
supports hyper requests at the moment, but can be updated to support
flash in the future.
Additionally this removes `void | Promise<void>` as valid return types
for the handler function. If one wants to use `Deno.upgradeHttpRaw`,
they will have to type cast the handler signature - the signature is
meant for the 99.99%, and should not be complicated for the 0.01% that
use `Deno.upgradeHttpRaw()`.
Adds error event dispatching for queueMicrotask(). Consequently unhandled errors are now reported with Deno.core.terminate(), which is immune to the existing quirk with plainly thrown errors (#14158).
- Merge "Deno.serve()" and "Deno.serveTls()" API
- Remove first argument and use "fetch" field options instead
- Update type declarations
- Add more documentation
Welcome to better optimised op calls! Currently opSync is called with parameters of every type and count. This most definitely makes the call megamorphic. Additionally, it seems that spread params leads to V8 not being able to optimise the calls quite as well (apparently Fast Calls cannot be used with spread params).
Monomorphising op calls should lead to some improved performance. Now that unwrapping of sync ops results is done on Rust side, this is pretty simple:
```
opSync("op_foo", param1, param2);
// -> turns to
ops.op_foo(param1, param2);
```
This means sync op calls are now just directly calling the native binding function. When V8 Fast API Calls are enabled, this will enable those to be called on the optimised path.
Monomorphising async ops likely requires using callbacks and is left as an exercise to the reader.
This commit adds "ext/node" extension that implementes CommonJS module system.
In the future this extension might be extended to actually contain implementation of
Node compatibility layer in favor of "deno_std/node".
Currently this functionality is not publicly exposed, it is available via "Deno[Deno.internal].require"
namespace and is meant to be used by other functionality to be landed soon.
This is a minimal first pass, things that still don't work:
support for dynamic imports in CJS
conditional exports
- "SpawnOutput" extends "ChildStatus" instead of composing it
- "SpawnOutput::stdout", "SpawnOutput::stderr", "Child::stdin",
"Child::stdout" and "Child::stderr" are no longer optional, instead
made them getters that throw at runtime if that stream wasn't set
to "piped".
- Remove the complicated "<T extends SpawnOptions = SpawnOptions>"
which we currently need to give proper type hints for the availability of
these fields. Their typings for these would get increasingly complex
if it became dependent on more options (e.g. "SpawnOptions::pty"
which if set should make the stdio streams unavailable)
Relanding #12994
This commit adds support for "unhandledrejection" event.
This event will trigger event listeners registered using:
"globalThis.addEventListener("unhandledrejection")
"globalThis.onunhandledrejection"
This is done by registering a default handler using
"Deno.core.setPromiseRejectCallback" that allows to
handle rejected promises in JavaScript instead of Rust.
This commit will make it possible to polyfill
"process.on("unhandledRejection")" in the Node compat
layer.
Co-authored-by: Colin Ihrig <cjihrig@gmail.com>
This commit adds support for "unhandledrejection" event.
This event will trigger event listeners registered using:
"globalThis.addEventListener("unhandledrejection")
"globalThis.onunhandledrejection"
This is done by registering a default handler using
"Deno.core.setPromiseRejectCallback" that allows to
handle rejected promises in JavaScript instead of Rust.
This commit will make it possible to polyfill
"process.on("unhandledRejection")" in the Node compat
layer.
Co-authored-by: Colin Ihrig <cjihrig@gmail.com>
This commit adds support for unstable FFI
callbacks. A callback is registered using
the `Deno.UnsafeCallback` API.
The backing memory for the callback can
be disposed of using `Deno.UnsafeCallback#close`.
It is not safe to pass the callback after calling
close.
Callbacks from other than the isolate thread
are not supported.
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Bert Belder <bertbelder@gmail.com>
