Calling `worker.terminate()` used to kill the worker's isolate and
then block until the worker's thread finished. This blocks the calling
thread if the worker's event loop was blocked in a sync op (as with
`Deno.sleepSync`), which wasn't realized at the time, but since the
worker's isolate was killed at that moment, it would not block the
calling thread if the worker was in a JS endless loop.
However, in #12831, in order to work around a V8 bug, worker
termination was changed to first set a signal to let the worker event
loop know that termination has been requested, and only kill the
isolate if the event loop has not finished after 2 seconds. However,
this change kept the blocking, which meant that JS endless loops in
the worker now blocked the parent for 2 seconds.
As it turns out, after #12831 it is fine to signal termination and
even kill the worker's isolate without waiting for the thread to
finish, so this change does that. However, that might leave the async
ops that receive messages and control data from the worker pending
after `worker.terminate()`, which leads to odd results from the op
sanitizer. Therefore, we set up a `CancelHandler` to cancel those ops
when the worker is terminated.
This commit:
- removes "fmt_errors::PrettyJsError" in favor of "format_js_error" fn
- removes "deno_core::JsError::create" and
"deno_core::RuntimeOptions::js_error_create_fn"
- adds new option to "deno_runtime::ops::worker_host::init"
This commit changes "deno bench" subcommand, by updating
the "Deno.bench" API as follows:
- remove "Deno.BenchDefinition.n"
- remove "Deno.BenchDefintion.warmup"
- add "Deno.BenchDefinition.group"
- add "Deno.BenchDefintion.baseline"
This is done because bench cases are no longer run fixed amount
of iterations, but instead they are run until there is difference between
subsequent runs that is statistically insiginificant.
Additionally, console reporter was rewritten completely, to looks
similar to "hyperfine" reporter.
This commit changes "deno test" to filter out stack frames if it is beneficial to the user.
This is the case when there are stack frames coming from "internal" code
below frames coming from user code.
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
This commit fixes and edge case, where testing/benching code could pledge new
permission set before restoring the previous pledge.
Appropriate panics were added and tests that assert that process is killed
in case of "recursive pledge".
This commit rewrites test runner to send structured error data from JavaScript
to Rust instead of passing strings. This will allow to customize display of errors
in test report (which will be addressed in follow up commits).
This commit adds "aggregated" field to "deno_core::JsError" that stores
instances of "JsError" recursively to properly handle "AggregateError"
formatting. Appropriate logics was added to "PrettyJsError" and
"console" API to format AggregateErrors.
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
The following transformations gradually faced by "JsError" have all been
moved up front to "JsError::from_v8_exception()":
- finding the first non-"deno:" source line;
- moving "JsError::script_resource_name" etc. into the first error stack
in case of syntax errors;
- source mapping "JsError::script_resource_name" etc. when wrapping
the error even though the frame locations are source mapped earlier;
- removing "JsError::{script_resource_name,line_number,start_column,end_column}"
entirely in favour of "js_error.frames.get(0)".
We also no longer pass a js-side callback to "core/02_error.js" from cli.
I avoided doing this on previous occasions because the source map lookups
were in an awkward place.
This commit changes "deno test" to better denote user output coming
from test cases.
This is done by printing "---- output ----" and "---- output end ----"
markers if an output is produced. The output from "console" and
"Deno.core.print" is captured, as well as direct writes to "Deno.stdout"
and "Deno.stderr".
To achieve that new APIs were added to "deno_core" crate, that allow
to replace an existing resource with a different one (while keeping resource
ids intact). Resources for stdout and stderr are replaced by pipes.
Co-authored-by: David Sherret <dsherret@gmail.com>
Following changes were done in this commit:
- remove "test" prefix before each test
- use gray color for "running N tests from ..." prompt
- use relative path or remote URL instead of full URL in "running N tests from ..." prompt
- in "failures" section, add file path/remote URL before the test name
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This commit adds new "deno check" subcommand.
Currently it is an alias for "deno cache" with the difference that remote
modules don't emit TS diagnostics by default.
Prints warning for "deno run" subcommand if "--check" flag is not present
and there's no "--no-check" flag. Adds "DENO_FUTURE_CHECK" env
variable that allows to opt into new behavior now.
`handleWasmStreaming` is the function that provides the binding with
the `fetch` API needed for `WebAssembly.instantiateStreaming()` and
`WebAssembly.compileStreaming()`. When I implemented it in #11200, I
thought V8 was calling these functions with the argument of the
`WebAssembly` streaming functions, without doing any resolving, and so
`handleWasmStreaming` awaits for the parameter to resolve. However,
as discovered in
https://github.com/denoland/deno/issues/13917#issuecomment-1065805565,
V8 does in fact resolve the parameter if it's a promise (and handles
rejections arising from that).
This change removes the `async` IIFE inside `handleWasmStreaming`,
letting initial errors be handled synchronously (which will however
not throw synchronously from the `WebAssembly` namespace functions).
Awaiting is still necessary for reading the bytes of the response,
though, and so there is an `async` IIFE for that.
When an exception is thrown during the processing of streaming WebAssembly,
`op_wasm_streaming_abort` is called. This op calls into V8, which synchronously
rejects the promise and calls into the promise rejection handler, if applicable.
But calling an op borrows the isolate's `JsRuntimeState` for the duration of the
op, which means it is borrowed when V8 calls into `promise_reject_callback`,
which tries to borrow it again, panicking.
This change changes `op_wasm_streaming_abort` from an op to a binding
(`Deno.core.abortWasmStreaming`). Although that binding must borrow the
`JsRuntimeState` in order to access the `WasmStreamingResource` stored in the
`OpTable`, it also takes ownership of that `WasmStreamingResource` instance,
which means it can drop any borrows of the `JsRuntimeState` before calling into
V8.
This commit adds "deno bench" subcommand and "Deno.bench()"
API that allows to register bench cases.
The API is modelled after "Deno.test()" and "deno test" subcommand.
Currently the output is rudimentary and bench cases and not
subject to "ops" and "resource" sanitizers.
Co-authored-by: evan <github@evan.lol>
This commit fixes CJS/ESM interop in compat mode for dynamically
imported modules.
"ProcState::prepare_module_load" was changed to accept a list
of "graph roots" without associated "module kind". That module kind
was always hardcoded to "ESM" which is not true for CJS/ESM interop -
a CommonJs module might be imported using "import()" function. In
such case the root of the graph should have "CommonJs" module kind
instead of "ESM".
This commit adds CJS/ESM interoperability when running in --compat mode.
Before executing files, they are analyzed and all CommonJS modules are
transformed on the fly to a ES modules. This is done by utilizing analyze_cjs()
functionality from deno_ast. After discovering exports and reexports, an ES
module is rendered and saved in memory for later use.
There's a caveat that all files ending with ".js" extension are considered as
CommonJS modules (unless there's a related "package.json" with "type": "module").
This commit adds "--trace-ops" flag to "deno test" subcommand.
This flag enables saving of stack traces for async ops, that before were always
saved. While the feature proved to be very useful it comes with a significant performance
hit, it's caused by excessive source mapping of stack frames.
This commit improves the error messages for the `deno test` async op
sanitizer. It does this in two ways:
- it uses handwritten error messages for each op that could be leaking
- it includes traces showing where each op was started
This "async op tracing" functionality is a new feature in deno_core.
It likely has a significant performance impact, which is why it is only
enabled in tests.
Adds another callback to WebWorkerOptions that allows to execute
some modules before actual worker code executes. This allows to set up Node
global using std/node.
This commit makes the errors produced from the resource sanitizer much
more human readable. It does this by using real words rather than our
"resource names" when referring to resources, and by giving helpful
hints on how to clean up each of the resources.
This commit fixes an error when user deletes "window" global JS
variable. Instead of relying on "window" or "globalThis" to dispatch
"load" and "unload" events, we are default to global scope of the
worker.
Deno's module loader currently strips a shebang if a module file
starts with one. However, this is no longer necessary, since there is
a stage-3 TC39 that adds support for shebangs (or "hashbangs") to the
language (https://github.com/tc39/proposal-hashbang), and V8, `tsc`
and `swc` all support it.
Furthermore, stripping shebangs causes a correctness bug with JSON
modules, since a JSON file with a shebang should not parse as a JSON
module, yet it does with this stripping. This change fixes this.
This commit adds lint and fmt ignore directives to bundled
code as well as a comment stating that the code was bundled
and shouldn't be edited manually.
Covered ranges were not merged and thus it appeared that some lines
might be uncovered. To fix this I used "v8-coverage" that takes care
of merging the ranges properly. With this change, coverage collected
from a file by multiple entrypoints is now correctly calculated.
I ended up forking https://github.com/demurgos/v8-coverage and adding
"cli/tools/coverage/merge.rs" and "cli/tools/coverage/range_tree.rs".
This commit fixes inspector integration with "deno test" subcommand
by waiting for inspector sessions to connect if "--inspect-brk" flag
is passed.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds proper support for import assertions and JSON modules.
Implementation of "core/modules.rs" was changed to account for multiple possible
module types, instead of always assuming that the code is an "ES module". In
effect "ModuleMap" now has knowledge about each modules' type (stored via
"ModuleType" enum). Module loading pipeline now stores information about
expected module type for each request and validates that expected type matches
discovered module type based on file's "MediaType".
Relevant tests were added to "core/modules.rs" and integration tests,
additionally multiple WPT tests were enabled.
There are still some rough edges in the implementation and not all WPT were
enabled, due to:
a) unclear BOM handling in source code by "FileFetcher"
b) design limitation of Deno's "FileFetcher" that doesn't download the same
module multiple times in a single run
Co-authored-by: Kitson Kelly <me@kitsonkelly.com>
Fixes "op_set_exit_code" by sharing a single "Arc" between
all workers (via "op state") instead of having a "global" value stored in
"deno_runtime" crate. As a consequence setting an exit code is always
scoped to a tree of workers, instead of being overridable if there are
multiple worker tree (like in "deno test --jobs" subcommand).
Refactored "cli/main.rs" functions to return "Result<i32, AnyError>" instead
of "Result<(), AnyError>" so they can return exit code.
This change also makes the timers implementation closer to the spec, and
sets up the stage to implement AbortSignal.timeout() (whatwg/dom#1032).
Fixes #8965
Fixes #10974
Fixes #11398
Although not easy to replicate in the wild, the `deno test` op sanitizer
can fail when there are intervals that started before a test runs, since
the op sanitizer can end up running in the time between the timer op for
an interval's run resolves and the op for the next run starts.
This change fixes that by adding a new macrotask callback that will run
after the timer macrotask queue has drained. This ensures that there is
a timer op if there are any timers which are unresolved by the time the
op sanitizer runs.
Due to a bug in V8, terminating an isolate while a module with top-level
await is being evaluated would crash the process. This change makes it
so calling `worker.terminate()` will signal the worker to terminate at
the next iteration of the event loop, and it schedules a proper
termination of the worker's isolate after 2 seconds.
Although not easy to replicate in the wild, the `deno test` op sanitizer
can fail when there are intervals that started before a test runs, since
the op sanitizer can end up running in the time between the timer op for
an interval's run resolves and the op for the next run starts.
This change fixes that by adding a new macrotask callback that will run
after the timer macrotask queue has drained. This ensures that there is
a timer op if there are any timers which are unresolved by the time the
op sanitizer runs.
Set the exit code to use if none is provided to Deno.exit(), or when
Deno exits naturally.
Needed for process.exitCode Node compat. Paves the way for #12888.
Reduce the number of iterations from 1,024 to 128.
On my big bruiser of a desktop machine it already takes up close to a
minute to complete when nothing else is running so no way it's going to
finish in the allotted time on the CI.
The fact that the test used to pass may be indicative of a performance
regression somewhere but it's not clear to me when or where that would
have been introduced.
Fixes #12887.
This commit introduces "ProcState::maybe_resolver" field, which
stores a single instance of resolver for the whole lifetime of the
process, instead of creating these resolvers for each creation
of module graph. As a result, this resolver can be used in fallback
case where graph is not constructed (REPL, loading modules using
"require") unifying resolution logic.
This commit changes formatting of elapsed time in test
runner output.
Instead of "XXXms", reporter outputs one of:
- "XXXms" for <1000ms
- "XXs" for <60s
- "XXXmYYs" for >=60s
This commit integrates import map and "classic" resolutions in
the "--compat" mode when using ES modules; in effect
"http:", "https:" and "blob:" imports now work in compat mode.
The algorithm works as follows:
1. If there's an import map, try to resolve using it and if succeeded
return the specifier
2. Try to resolve using "Node ESM resolution", and if succeeded return
the specifier
3. Fall back to regular ESM resolution
Closes #11826
**BREAKING CHANGE** this behaviour was disable when introduced in Deno 1.14/TypeScript 4.4. It will highlight code that unsafely handles variables that are caught, and will cause type errors in unsafe code.
Closes #11882
BREAKING CHANGE: Previously when `--location` was set, the unique storage key was derived from the the URL of the location instead of just the origin. This change correctly uses just the origin. This may cause previously persisted storage to change its key and data to not be available with the same location as before.
This commit adds CJS and ESM Node resolvers to the "--compat" mode.
The functionality is spread across "cli/compat" module and Node compatibility
layer in "deno_std/node"; this stems from the fact that ES module resolution
can only be implemented in Rust as it needs to directly integrated with
"deno_core"; however "deno_std/node" already provided CJS module resolution.
Currently this resolution is only active when running a files using
"deno run --compat --unstable <filename>", and is not available in other
subcommands, which will be changed in follow up commits.
Returns empty values in case of errors, source lines are non-essential anyway. These errors can happen e.g. when source files change at runtime. A warning is also printed to help us track when it happens in unexpected cases besides this.
This commit fixes problem with LSP where diagnostics coming
from "deno lint" don't respect configuration file.
LSP was changed to store "Option<ConfigFile>", "Option<LintConfig>"
and "Option<FmtConfig>" on "Inner"; as well as storing "Option<LintConfig>"
and "Option<FmtConfig>" on "StateSnapshot".
Co-authored-by: Kitson Kelly <me@kitsonkelly.com>
`Window`'s `self` property and `DedicatedWorkerGlobalScope`'s `name`
property are defined as Web IDL read-only attributes with the
`[Replaceable]` extended attribute, meaning that their setter will
redefine the property as a data property with the set value, rather than
changing some internal state. Deno currently defines them as read-only
data properties instead.
Given that Web IDL requires all attributes to be accessor properties
rather than data properties, but Deno exposes almost all of those
properties as either read-only or writable data properties, it makes
sense to expose `[Replaceable]` properties as writable as well – as is
already the case with `WindowOrWorkerGlobalScope`'s `performance`
property.
WebAssembly modules compiled through `WebAssembly.compile()` and similar
non-streaming APIs don't have a URL associated to them, because they
have been compiled from a buffer source. In stack traces, V8 will use
a URL such as `wasm://wasm/d1c677ea`, with a hash of the module.
However, wasm modules compiled through streaming APIs, like
`WebAssembly.compileStreaming()`, do have a known URL, which can be
obtained from the `Response` object passed into the streaming APIs. And
as per the developer-facing display conventions in the WebAssembly
Web API spec, this URL should be used in stack traces. This change
implements that.
This commit adds automatic injection of Node globals when "--compat" flag
is present.
This is done by executing "https://deno.land/std/node/global.ts" as a "side module",
before main module is executed.
This commit makes "--compat" required to be used with "--unstable" flag, as some
of Node globals require unstable Deno APIs.
This commit adds "--compat" flag. When the flag is passed a set of mappings for
built-in Node modules is injected into the import map. If user doesn't
explicitly provide an import map (using "--import-map" flag) then a map is
created on the fly. If there are already existing mappings in import map that
would clash with built-in Node modules a set of diagnostics is printed to the
terminal with suggestions how to proceed.
This panic could happen in the following cases:
- A non-fatal error being thrown from a worker, that doesn't terminate
the worker's execution, but propagates to the main thread without
being handled, and makes the main thread terminate.
- A nested worker being alive while its parent worker gets terminated.
- A race condition if the main event loop terminates the worker as part
of its last task, but the worker doesn't fully terminate before the
main event loop stops running.
This panic happens because a worker's event loop should have pending ops
as long as the worker isn't closed or terminated – but if an event loop
finishes running while it has living workers, its associated
`WorkerThread` structs will be dropped, closing the channels that keep
those ops pending.
This change adds a `Drop` implementation to `WorkerThread`, which
terminates the worker without waiting for a response. This fixes the
panic, and makes it so nested workers are automatically terminated once
any of their ancestors is closed or terminated.
This change also refactors a worker's termination code into a
`WorkerThread::terminate()` method.
Closes #11342.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds support for configuration file for "deno fmt"
subcommand. It is also respected by LSP when formatting
files.
Example configuration:
{
"fmt": {
"files": {
"include": ["src/"],
"exclude": ["src/testdata/"]
},
"options": {
"useTabs": true,
"lineWidth": 80,
"indentWidth": 4,
"singleQuote": true,
"textWrap": "preserve"
}
}
}
When `worker.terminate()` is called, the spec requires that the
corresponding port message queue is emptied, so no messages can be
received after the call, even if they were sent from the worker before
it was terminated.
The spec doesn't require this of `self.close()`, and since Deno uses
different channels to send messages and to notify that the worker was
closed, messages might still arrive after the worker is known to be
closed, which are currently being dropped. This change fixes that.
The fix involves two parts: one on the JS side and one on the Rust side.
The JS side was using the `#terminated` flag to keep track of whether
the worker is known to be closed, without distinguishing whether further
messages should be dropped or not. This PR changes that flag to an
enum `#state`, which can be one of `"RUNNING"`, `"CLOSED"` or
`"TERMINATED"`.
The Rust side was removing the `WorkerThread` struct from the workers
table when a close control was received, regardless of whether there
were any messages left to read, which made any subsequent calls to
`op_host_recv_message` to return `Ok(None)`, as if there were no more
mesasges. This change instead waits for both a close control and for
the message channel's sender to be closed before the worker thread is
removed from the table.
This commit adds support for following flags in deno lint subcommand:
--config - allows to load configuration file and parses "lint" object
--rules-tags=<tags> - allows specifying which set of tagged rules should be run
--rules-include=<rules> - allow specifying which rules should be run
--rules-exclude=<rules> - allow specifying which rules should not be run
This commit merges the two vectors of specifiers into a single one introducing
the concept of a "TestMode" which is a tri-state enum specifying how a specifier
is to be tested (as documentation, as an executable module or as both).
This is determined during the collection phase and determines how a specifier
will be executed based on how the specifier was collected (directly or not) and
if it has an eligible media_type when fetched.
For example "deno test README.md" is marked as documentation because, while it
is a direct inclusion it is not an executable media type therefore will only
have the fenced code blocks that can be parsed from it tested.
Classic worker scripts are now executed in the context of a Tokio
runtime. This does mean we can not spawn more tokio runtimes in
"op_worker_sync_fetch". We instead spawn a new thread there, that can
create a new Tokio runtime that we can use to block the worker thread.
