1. Rewrites the tests to be more back and forth rather than getting the
output all at once (which I believe was causing the hangs on linux and
maybe mac)
2. Runs the pty tests on the linux ci.
3. Fixes a bunch of tests that were just wrong.
4. Adds timeouts on the pty tests.
This commit changes how data required to bootstrap main and worker
runtime is serialized.
Instead of relying on serde_v8 and using JSON object,
we're doing manual serialization to a "v8::Array". This limits number
of V8 strings that need to be serialized by 16.
It also made it clear that some data could be obtained during
snapshotting instead of during bootstrap.
Since we can preserve ops in the snapshot these days, we no longer
need to have "Deno[Deno.internal].nodeUnstable" namespace.
Instead, various built-in Node.js modules can use appropriate APIs
directly.
This commit adds unstable "Deno.openKv()" API that allows to open
a key-value database at a specified path.
---------
Co-authored-by: Luca Casonato <hello@lcas.dev>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This PR _**temporarily**_ removes WebGPU (which has behind the
`--unstable` flag in Deno), due to performance complications due to its
presence.
It will be brought back in the future; as a point of reference, Chrome
will ship WebGPU to stable on 26/04/2023.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit changes how "unload" event is handled - before
this commit an event listener was added unconditionally in
the runtime bootstrapping function, which for some reason was
very expensive (0.3ms). Instead of adding an event listener,
a check was added to "dispatchEvent" function that performs
the same action (so it's only called if there's an event dispatched).
This commit renames "deno_core::InternalModuleLoader" to
"ExtModuleLoader" and changes the specifiers used by the
modules loaded from this loader to "ext:".
"internal:" scheme was really ambiguous and it's more characters than
"ext:", which should result in slightly smaller snapshot size.
Closes https://github.com/denoland/deno/issues/18020
This commit moves some of the code from "99_main.js" to
be executed during the snapshot time instead of on each
worker bootstrap. These should minimally help with startup
time benchmark.
This API is required by several extensions like "ext/node", "ext/ffi"
and also FS APIs that we want to move to a separate crate. Because
of that "pathFromURL" API was moved to "deno_web" extension so
other extension crates can rely on it.
We use information about build in several extension crates like
"ext/node" or "runtime/". In an effort to move "fs" APIs to a separate
crate it is a prerequisite to have this information available outside
of the "runtime/" crate.
This commit moves definition of "build" object to "Deno.core" that is
later forwarded to "Deno.build".
This commit merges "runtime/js/40_spawn.js" into
"runtime/js/40_process.js", and "runtime::ops::spawn"
into "runtime::ops::process".
It makes little sense to have them separated given that we want to
factor out these APIs into a separate extension crate.
JavaScript APIs from "runtime/js/40_files.js" combined abstractions
for stdio streams ("Stdout", "Stderr", "Stdin") and file system file
("File", "FsFile"). APIs from "runtime/js/40_read_file.js" and
"runtime/js/40_write_file.js" were implemented using ops from
"runtime/ops/fs.rs".
This file was removed and relevant APIs were moved to "deno_io/12_io.js"
and "runtime/js/30_fs.js".
This work is meant to enable factoring out "deno_fs" crate.
This file was not used anywhere - even though it exposed an unstable
"Deno.DiagnosticCategory" API. Seems like a left over from when we
had "Deno.emit" API.
This commit changes "include_js_files!" macro from "deno_core"
in a way that "dir" option doesn't cause specifiers to be rewritten
to include it.
Example:
```
include_js_files! {
dir "js",
"hello.js",
}
```
The above definition required embedders to use:
`import ... from "internal:<ext_name>/js/hello.js"`.
But with this change, the "js" directory in which the files are stored
is an implementation detail, which for embedders results in:
`import ... from "internal:<ext_name>/hello.js"`.
The directory the files are stored in, is an implementation detail and
in some cases might result in a significant size difference for the
snapshot. As an example, in "deno_node" extension, we store the
source code in "polyfills" directory; which resulted in each specifier
to look like "internal:deno_node/polyfills/<module_name>", but with
this change it's "internal:deno_node/<module_name>".
Given that "deno_node" has over 100 files, many of them having
several import specifiers to the same extension, this change removes
10 characters from each import specifier.
Runtime generation of async op wrappers contributed to increased startup
time and core became unusable with
`--disallow-code-generation-from-strings` flag. The optimization only
affects very small microbenchmarks so this revert will not cause any
regressions.
This commit further improves startup time by:
- no relying on "JsRuntime::execute_script" for runtime bootstrapping,
this is instead done using V8 APIs directly
- registering error classes during the snapshot time, instead of on
startup
Further improvements can be made, mainly around removing
"core.initializeAsyncOps()" which takes around 2ms.
This commit should result in ~1ms startup time improvement.
Adds two test files: "cli/tests/unit_node/process_test.ts" and
"cli/tests/unit_node/child_process_test.ts"
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This commit moves "deno_std/node" in "ext/node" crate. The code is
transpiled and snapshotted during the build process.
During the first pass a minimal amount of work was done to create the
snapshot, a lot of code in "ext/node" depends on presence of "Deno"
global. This code will be gradually fixed in the follow up PRs to migrate
it to import relevant APIs from "internal:" modules.
Currently the code from snapshot is not used in any way, and all
Node/npm compatibility still uses code from
"https://deno.land/std/node" (or from the location specified by
"DENO_NODE_COMPAT_URL"). This will also be handled in a follow
up PRs.
---------
Co-authored-by: crowlkats <crowlkats@toaxl.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This is a proof of concept for being able to snapshot TypeScript files.
Currently only a single runtime file is authored in TypeScript -
"runtime/js/01_version.ts".
Not needed infrastructure was removed from "core/snapshot_util.rs".
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This PR refactors all internal js files (except core) to be written as
ES modules.
`__bootstrap`has been mostly replaced with static imports in form in
`internal:[path to file from repo root]`.
To specify if files are ESM, an `esm` method has been added to
`Extension`, similar to the `js` method.
A new ModuleLoader called `InternalModuleLoader` has been added to
enable the loading of internal specifiers, which is used in all
situations except when a snapshot is only loaded, and not a new one is
created from it.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds sync versions of async APIs to "Deno.permissions"
namespace.
Following APIs were added:
- "Deno.permissions.querySync"
- "Deno.permissions.requestSync"
- "Deno.permissions.revokeSync"
This commit removes "Deno.core" namespace. It is strictly private API
that has no stability guarantees, we were supposed to remove it long time ago.
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
Updated third_party dlint to v0.37.0 for GitHub Actions. This PR
includes following changes:
* fix(prefer-primordials): Stop using array pattern assignments
* fix(prefer-primordials): Stop using global intrinsics except for
`SharedArrayBuffer`
* feat(guard-for-in): Apply new guard-for-in rule
Previously, `Deno.permissions.[revoke|request]()` wouldn't correctly
process the `path: URL` when `name` was `ffi`. This change fixes that
behaviour and adds a new function, `formDescriptor()`, to ensure `URL`
arguments are consistently handled across
`Deno.permissions.[query|revoke|request]()`.
This commit changes implementation of "Deno.memoryUsage()" to return
correct value for "rss" field. To do that we implement a specialized function
per os to retrieve this information.
This commit removes three unstable Deno APIs:
- "Deno.spawn()"
- "Deno.spawnSync()"
- "Deno.spawnChild()"
These APIs were replaced by a unified "Deno.Command" API.
Refactors the `Deno.Command` class to not handle any state, but only being an intermediary to calling its methods, and as such any methods and properties besides `output`, `outputSync` & `spawn` have been removed. Interracting with a `spawn`ed subprocess now works by using the methods and properties on the returned class of the `spawn` method.
**This patch**
```
benchmark time (avg) (min … max) p75 p99 p995
------------------------------------------------- -----------------------------
echo deno 23.99 ms/iter (22.51 ms … 33.61 ms) 23.97 ms 33.61 ms 33.61 ms
cat 16kb 24.27 ms/iter (22.5 ms … 35.21 ms) 24.2 ms 35.21 ms 35.21 ms
cat 1mb 25.88 ms/iter (25.04 ms … 30.28 ms) 26.12 ms 30.28 ms 30.28 ms
cat 15mb 38.41 ms/iter (35.7 ms … 50 ms) 38.31 ms 50 ms 50 ms
```
**main**
```
benchmark time (avg) (min … max) p75 p99 p995
------------------------------------------------- -----------------------------
echo deno 35.66 ms/iter (34.53 ms … 41.84 ms) 35.79 ms 41.84 ms 41.84 ms
cat 16kb 35.99 ms/iter (34.52 ms … 44.94 ms) 36.05 ms 44.94 ms 44.94 ms
cat 1mb 38.68 ms/iter (36.67 ms … 50.44 ms) 37.95 ms 50.44 ms 50.44 ms
cat 15mb 48.4 ms/iter (46.19 ms … 58.41 ms) 49.16 ms 58.41 ms 58.41 ms
```
This PR adds copies of several unstable APIs that are available
in "Deno[Deno.internal].nodeUnstable" namespace.
These copies do not perform unstable check (ie. don't require
"--unstable" flag to be present). Otherwise they work exactly
the same, including permission checks.
These APIs are not meant to be used by users directly and
can change at any time.
Copies of following APIs are available in that namespace:
- Deno.spawnChild
- Deno.spawn
- Deno.spawnSync
- Deno.serve
- Deno.upgradeHttpRaw
- Deno.listenDatagram
V8's JIT can do a better job knowing the argument count and also enable
fast call path (in future).
This also lets us call async ops without `opAsync`:
```js
const { ops } = Deno.core;
await ops.op_void_async();
```
this patch: 4405286 ops/sec
main: 3508771 ops/sec
This commit stabilizes "Deno.consoleSize()" API.
There is one change compared to previous unstable API,
in that the API doesn't accept any arguments. Console size
is established by querying syscalls for stdio streams at fd
0, 1 and 2.
Previously `op_net_listen`, `op_net_accept`, and various other ops in
ext/net where variadic on the transport. This created a lot of code
bloat. This commit updates the code to instead have separate ops for
each transport.
This change adds `windowsRawArguments` to `SpawnOptions`. The option enables
skipping the default quoting and escaping while creating the command on
windows.
The option works in a similar way as `windowsVerbatimArguments` in
child_process.spawn options in Node.js, and is necessary for simulating
it in `std/node`.
closes #8852
This commit adds a new op_write_all to core that allows writing an
entire chunk in a single async op call. Internally this calls
`Resource::write_all`.
The `writableStreamForRid` has been moved to `06_streams.js` now, and
uses this new op. Various other code paths now also use this new op.
Closes #16227
We can use Resource::read_return & op_read instead. This allows HTTP
request bodies to participate in FastStream.
To make this work, `readableStreamForRid` required a change to allow non
auto-closing resources to be handled. This required some minor changes
in our FastStream paths in ext/http and ext/flash.
This commit splits `Deno.upgradeHttp` into two different APIs, because
the same API is currently overloaded with two different functions. Flash
requests upgrade immediately, with no need to return a `Response`
object. Instead you have to manually write the response to the socket.
Hyper requests only upgrade once a `Response` object has been sent.
These two behaviours are now split into `Deno.upgradeHttp` and
`Deno.upgradeHttpRaw`. The latter is flash only. The former only
supports hyper requests at the moment, but can be updated to support
flash in the future.
Additionally this removes `void | Promise<void>` as valid return types
for the handler function. If one wants to use `Deno.upgradeHttpRaw`,
they will have to type cast the handler signature - the signature is
meant for the 99.99%, and should not be complicated for the 0.01% that
use `Deno.upgradeHttpRaw()`.
Adds error event dispatching for queueMicrotask(). Consequently unhandled errors are now reported with Deno.core.terminate(), which is immune to the existing quirk with plainly thrown errors (#14158).
- Merge "Deno.serve()" and "Deno.serveTls()" API
- Remove first argument and use "fetch" field options instead
- Update type declarations
- Add more documentation
Welcome to better optimised op calls! Currently opSync is called with parameters of every type and count. This most definitely makes the call megamorphic. Additionally, it seems that spread params leads to V8 not being able to optimise the calls quite as well (apparently Fast Calls cannot be used with spread params).
Monomorphising op calls should lead to some improved performance. Now that unwrapping of sync ops results is done on Rust side, this is pretty simple:
```
opSync("op_foo", param1, param2);
// -> turns to
ops.op_foo(param1, param2);
```
This means sync op calls are now just directly calling the native binding function. When V8 Fast API Calls are enabled, this will enable those to be called on the optimised path.
Monomorphising async ops likely requires using callbacks and is left as an exercise to the reader.
- "SpawnOutput" extends "ChildStatus" instead of composing it
- "SpawnOutput::stdout", "SpawnOutput::stderr", "Child::stdin",
"Child::stdout" and "Child::stderr" are no longer optional, instead
made them getters that throw at runtime if that stream wasn't set
to "piped".
- Remove the complicated "<T extends SpawnOptions = SpawnOptions>"
which we currently need to give proper type hints for the availability of
these fields. Their typings for these would get increasingly complex
if it became dependent on more options (e.g. "SpawnOptions::pty"
which if set should make the stdio streams unavailable)
Relanding #12994
This commit adds support for "unhandledrejection" event.
This event will trigger event listeners registered using:
"globalThis.addEventListener("unhandledrejection")
"globalThis.onunhandledrejection"
This is done by registering a default handler using
"Deno.core.setPromiseRejectCallback" that allows to
handle rejected promises in JavaScript instead of Rust.
This commit will make it possible to polyfill
"process.on("unhandledRejection")" in the Node compat
layer.
Co-authored-by: Colin Ihrig <cjihrig@gmail.com>
This commit adds support for "unhandledrejection" event.
This event will trigger event listeners registered using:
"globalThis.addEventListener("unhandledrejection")
"globalThis.onunhandledrejection"
This is done by registering a default handler using
"Deno.core.setPromiseRejectCallback" that allows to
handle rejected promises in JavaScript instead of Rust.
This commit will make it possible to polyfill
"process.on("unhandledRejection")" in the Node compat
layer.
Co-authored-by: Colin Ihrig <cjihrig@gmail.com>
This commit adds support for unstable FFI
callbacks. A callback is registered using
the `Deno.UnsafeCallback` API.
The backing memory for the callback can
be disposed of using `Deno.UnsafeCallback#close`.
It is not safe to pass the callback after calling
close.
Callbacks from other than the isolate thread
are not supported.
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Bert Belder <bertbelder@gmail.com>
This commit adds "Deno.core.setFormatExceptionCallback" which
can be used to provide custom formatting for errors. It is useful
in cases when user throws something that is non-Error (eg.
a string, plain object, etc).
This commit improves the permission descriptor validation by
explicitly checking for object types and using optional
chaining when creating error messages in case the descriptor
is not an object.
Fixes: https://github.com/denoland/deno/issues/14675
This commit removes "WorkerOptions.deno" option as a boolean,
as well as "WorkerOptions.deno.namespace" settings. Starting
with this commit all workers have access to "Deno" namespace
by default.
This commit moves "op_format_location" to "core/ops_builtin.rs"
and removes "Deno.core.createPrepareStackTrace" in favor of
"Deno.core.prepareStackTrace".
Co-authored-by: Aaron O'Mullan <aaron.omullan@gmail.com>
This commit:
- removes "fmt_errors::PrettyJsError" in favor of "format_js_error" fn
- removes "deno_core::JsError::create" and
"deno_core::RuntimeOptions::js_error_create_fn"
- adds new option to "deno_runtime::ops::worker_host::init"
This commit changes "deno bench" subcommand, by updating
the "Deno.bench" API as follows:
- remove "Deno.BenchDefinition.n"
- remove "Deno.BenchDefintion.warmup"
- add "Deno.BenchDefinition.group"
- add "Deno.BenchDefintion.baseline"
This is done because bench cases are no longer run fixed amount
of iterations, but instead they are run until there is difference between
subsequent runs that is statistically insiginificant.
Additionally, console reporter was rewritten completely, to looks
similar to "hyperfine" reporter.
This commit rewrites test runner to send structured error data from JavaScript
to Rust instead of passing strings. This will allow to customize display of errors
in test report (which will be addressed in follow up commits).
The following transformations gradually faced by "JsError" have all been
moved up front to "JsError::from_v8_exception()":
- finding the first non-"deno:" source line;
- moving "JsError::script_resource_name" etc. into the first error stack
in case of syntax errors;
- source mapping "JsError::script_resource_name" etc. when wrapping
the error even though the frame locations are source mapped earlier;
- removing "JsError::{script_resource_name,line_number,start_column,end_column}"
entirely in favour of "js_error.frames.get(0)".
We also no longer pass a js-side callback to "core/02_error.js" from cli.
I avoided doing this on previous occasions because the source map lookups
were in an awkward place.
This commit changes "deno test" to better denote user output coming
from test cases.
This is done by printing "---- output ----" and "---- output end ----"
markers if an output is produced. The output from "console" and
"Deno.core.print" is captured, as well as direct writes to "Deno.stdout"
and "Deno.stderr".
To achieve that new APIs were added to "deno_core" crate, that allow
to replace an existing resource with a different one (while keeping resource
ids intact). Resources for stdout and stderr are replaced by pipes.
Co-authored-by: David Sherret <dsherret@gmail.com>
This commit adds following fields to "Deno.TestContext" interface:
- name
- origin
- parent
These are prerequisites for supporting snapshot functionality in
"std/testing".
This commit adds "Deno.upgradeHttp" API, which
allows to "hijack" connection and switch protocols, to eg.
implement WebSocket required for Node compat.
Co-authored-by: crowlkats <crowlkats@toaxl.com>
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds "deno bench" subcommand and "Deno.bench()"
API that allows to register bench cases.
The API is modelled after "Deno.test()" and "deno test" subcommand.
Currently the output is rudimentary and bench cases and not
subject to "ops" and "resource" sanitizers.
Co-authored-by: evan <github@evan.lol>
This commit adds "--trace-ops" flag to "deno test" subcommand.
This flag enables saving of stack traces for async ops, that before were always
saved. While the feature proved to be very useful it comes with a significant performance
hit, it's caused by excessive source mapping of stack frames.
This commit improves the error messages for the `deno test` async op
sanitizer. It does this in two ways:
- it uses handwritten error messages for each op that could be leaking
- it includes traces showing where each op was started
This "async op tracing" functionality is a new feature in deno_core.
It likely has a significant performance impact, which is why it is only
enabled in tests.
This commit adds `readable` and `writable` properties to `Deno.File` and
`Deno.Conn`. This makes it very simple to use files and network sockets
with fetch or the native HTTP server.
Adds another callback to WebWorkerOptions that allows to execute
some modules before actual worker code executes. This allows to set up Node
global using std/node.
This commit makes the errors produced from the resource sanitizer much
more human readable. It does this by using real words rather than our
"resource names" when referring to resources, and by giving helpful
hints on how to clean up each of the resources.
Although not easy to replicate in the wild, the `deno test` op sanitizer
can fail when there are intervals that started before a test runs, since
the op sanitizer can end up running in the time between the timer op for
an interval's run resolves and the op for the next run starts.
This change fixes that by adding a new macrotask callback that will run
after the timer macrotask queue has drained. This ensures that there is
a timer op if there are any timers which are unresolved by the time the
op sanitizer runs.
Although not easy to replicate in the wild, the `deno test` op sanitizer
can fail when there are intervals that started before a test runs, since
the op sanitizer can end up running in the time between the timer op for
an interval's run resolves and the op for the next run starts.
This change fixes that by adding a new macrotask callback that will run
after the timer macrotask queue has drained. This ensures that there is
a timer op if there are any timers which are unresolved by the time the
op sanitizer runs.
Set the exit code to use if none is provided to Deno.exit(), or when
Deno exits naturally.
Needed for process.exitCode Node compat. Paves the way for #12888.
This commit adds an ability to "ref" or "unref" pending ops.
Up to this point Deno had a notion of "async ops" and "unref async ops";
the former keep event loop alive, while the latter do not block event loop
from finishing. It was not possible to change between op types after
dispatching, one had to decide which type to use before dispatch.
Instead of storing ops in two separate "FuturesUnordered" collections,
now ops are stored in a single collection, with supplemental "HashSet"
storing ids of promises that were "unrefed".
Two APIs were added to "Deno.core":
"Deno.core.refOp(promiseId)" which allows to mark promise id
to be "refed" and keep event loop alive (the default behavior)
"Deno.core.unrefOp(promiseId)" which allows to mark promise
id as "unrefed" which won't block event loop from exiting
In tests, the function to format errors would assume that any error with
a property `errors` would be an `AggregateError`, and therefore the
property `errors` would contain an error. This is not necessarily the
case.
This allows resources to be "streams" by implementing read/write/shutdown. These streams are implicit since their nature (read/write/duplex) isn't known until called, but we could easily add another method to explicitly tag resources as streams.
`op_read/op_write/op_shutdown` are now builtin ops provided by `deno_core`
Note: this current implementation is simple & straightforward but it results in an additional alloc per read/write call
Closes #12556
`Window`'s `self` property and `DedicatedWorkerGlobalScope`'s `name`
property are defined as Web IDL read-only attributes with the
`[Replaceable]` extended attribute, meaning that their setter will
redefine the property as a data property with the set value, rather than
changing some internal state. Deno currently defines them as read-only
data properties instead.
Given that Web IDL requires all attributes to be accessor properties
rather than data properties, but Deno exposes almost all of those
properties as either read-only or writable data properties, it makes
sense to expose `[Replaceable]` properties as writable as well – as is
already the case with `WindowOrWorkerGlobalScope`'s `performance`
property.
* perf(runtime/fs): optimize readFile by using a single large buffer
* handle extended/truncated files during read
Allocate an extra byte in our read buffer to detect "overflow" then fallback to unsized readAll for remainder of extended file, this is a slowpath that should rarely happen in practice
A comment in `runtime.js` reads that `console` seems to be "the only one
that should be writable and non-enumerable", which explains why it is
declared with `util.writable` but then has its property descriptor's
`enumerable` key changed to false.
But it is not in fact true that `console` is the only global property
for which this holds, and it wasn't even when this behavior was
introduced in denoland#9013. All WebIDL interfaces are also writable and
non-enumerable – the only difference here being that `console` is a
namespace rather than an interface.
Since WebIDL interfaces are defined with `util.nonEnumerable`, and
`console` uses the same descriptor keys, this PR changes the definition
of `console` to use `util.nonEnumerable` as well.
This adds support for the URLPattern API.
The API is added in --unstable only, as it has not yet shipped in any
browser. It is targeted for shipping in Chrome 95.
Spec: https://wicg.github.io/urlpattern/
Co-authored-by: crowlKats < crowlkats@toaxl.com >
When `worker.terminate()` is called, the spec requires that the
corresponding port message queue is emptied, so no messages can be
received after the call, even if they were sent from the worker before
it was terminated.
The spec doesn't require this of `self.close()`, and since Deno uses
different channels to send messages and to notify that the worker was
closed, messages might still arrive after the worker is known to be
closed, which are currently being dropped. This change fixes that.
The fix involves two parts: one on the JS side and one on the Rust side.
The JS side was using the `#terminated` flag to keep track of whether
the worker is known to be closed, without distinguishing whether further
messages should be dropped or not. This PR changes that flag to an
enum `#state`, which can be one of `"RUNNING"`, `"CLOSED"` or
`"TERMINATED"`.
The Rust side was removing the `WorkerThread` struct from the workers
table when a close control was received, regardless of whether there
were any messages left to read, which made any subsequent calls to
`op_host_recv_message` to return `Ok(None)`, as if there were no more
mesasges. This change instead waits for both a close control and for
the message channel's sender to be closed before the worker thread is
removed from the table.
