In #9118, TLS streams were split into a "read half" and a "write half"
using tokio::io::split() to allow concurrent Conn#read() and
Conn#write() calls without one blocking the other. However, this
introduced a bug: outgoing data gets discarded when the TLS stream is
gracefully closed, because the read half is closed too early, before all
TLS control data has been received.
Fixes: #9692
Fixes: #10049
Fixes: #10296
Fixes: denoland/deno_std#750
Currently file passed to --config file is parsed using TsConfig structure
that does multiple things when loading the file. Instead of relying on that
structure I've introduced ConfigFile structure that can be updated to
sniff out more fields from the config file in the future.
This commit implements file watching for deno test.
When a file is changed, only the test modules which
use it as a dependency are rerun.
This is accomplished by reworking the file watching infrastructure
to pass the paths which have changed to the resolver, and then
constructing a module graph for each test module to check if it
contains any changed files.
`Deno.core.*` is unstable and not fit for public consumption, although this is a somewhat internal bench some people may use it as reference code and start using `Deno.core.encode()` in their own code
Makes the codebase more searchable and helps distinguish op functions from helper functions
Besides tests/examples/benches this pattern appears to be used everywhere else in the codebase
Fixes a pesky bug in the fetch implementation where if the init part is
specified in `fetch` instead of the `Request` constructor, the
fillHeaders function receives two references to the same object, causing
it to append to the same list being iterated over.
This commit adds support for running test in parallel.
Entire test runner functionality has been rewritten
from JavaScript to Rust and a set of ops was added to support reporting in Rust.
A new "--jobs" flag was added to "deno test" that allows to configure
how many threads will be used. When given no value it defaults to 2.
Extensions allow declarative extensions to "JsRuntime" (ops, state, JS or middleware).
This allows for:
- `op_crates` to be plug-and-play & self-contained, reducing complexity leaked to consumers
- op middleware (like metrics_op) to be opt-in and for new middleware (unstable, tracing,...)
- `MainWorker` and `WebWorker` to be composable, allowing users to extend workers with their ops whilst benefiting from the other infrastructure (inspector, etc...)
In short extensions improve deno's modularity, reducing complexity and leaky abstractions for embedders and the internal codebase.
General cleanup of module loading code, tried to reduce indentation in various methods
on "JsRuntime" to improve readability.
Added "JsRuntime::handle_scope" helper function, which returns a "v8::HandleScope".
This was done to reduce a code pattern that happens all over the "deno_core".
Additionally if event loop hangs during loading of dynamic modules a list of
currently pending dynamic imports is printed.
`InvalidDNSNameError` is thrown when a string is not a valid hostname,
e.g. it contains invalid characters, or starts with a numeric digit. It
does not involve a (failed) DNS lookup.
denort is an optimization to "deno compile" to produce slightly smaller
output. It's a decent idea, but causes a lot of negative side-effects:
- Deno's link time is a source of constant agony both locally and in CI,
denort doubles link time.
- The release process is a long and arduous undertaking with many manual
steps. denort necessitates an additional manual zip + upload from M1
apple computers.
- The "deno compile" interface is complicated with the "--lite" option.
This is confusing for uses ("why wouldn't you want lite?").
The benefits of this feature do not outweigh the negatives. We must find
a different approach to optimizing "deno compile" output.
This commits adds adds "permissions" option to the test definitions
which allows tests to run with different permission sets than
the process's permission.
The change will only be in effect within the test function, once the
test has completed the original process permission set is restored.
Test permissions cannot exceed the process's permission.
You can only narrow or drop permissions, failure to acquire a
permission results in an error being thrown and the test case will fail.
Even if bootstrapping the JS runtime is low level, it's an abstraction leak of
core to require users to call `Deno.core.ops()` in JS space.
So instead we're introducing a `JsRuntime::sync_ops_cache()` method,
once we have runtime extensions a new runtime will ensure the ops
cache is setup (for the provided extensions) and then loading/unloading
plugins should be the only operations that require op cache syncs
- register builtin v8 errors in core.js so consumers don't have to
- remove complexity of error args handling (consumers must provide a
constructor with custom args, core simply provides msg arg)
This commit aligns the `fetch` API and the `Request` / `Response`
classes belonging to it to the spec. This commit enables all the
relevant `fetch` WPT tests. Spec compliance is now at around 90%.
Performance is essentially identical now (within 1% of 1.9.0).
This commit fixes the URL returned from `request.url` in the HTTP server
to be fully qualified. This previously existed, but was removed and
accidentially not readded during optimizations of the HTTP ops.
Returning a non fully qualified URL from `Request#url` is not spec
compliant.
The panic was caused by the lack of an error class mapping for
futures::channel::TrySendError, but it shouldn't have been throwing an error in
the first place - when a worker has terminated, postMessage should just return.
The issue was that the termination message hadn't yet been recieved, so it was
carrying on with trying to send the message. This adds another check on the Rust
side for if the channel is closed, and if it is the worker is treated as
terminated.
This commit aligns `Headers` to spec. It also removes the now unused
03_dom_iterable.js file. We now pass all relevant `Headers` WPT. We do
not implement any sort of header filtering, as we are a server side
runtime.
This is likely not the most efficient implementation of `Headers` yet.
It is however spec compliant. Once all the APIs in the `HTTP` hot loop
are correct we can start optimizing them. It is likely that this commit
reduces bench throughput temporarily.
Raise the soft limit to the hard limit when possible. This is similar
to what Node.js does to avoid running into "out of file descriptors"
errors too quickly.
On most Linux systems, raises the limit from 1,024 to 1,048,576.
On most macOS systems, raises the limit from 256 to 10,240.
Fixes #10148.
This stabilizes Deno.ftruncate and Deno.ftruncateSync.
This is a well known system call and the interface is
not going to change. Implicitly requires write permissions
as the file has to be opened with write to be truncated.
This commit adds allowlist support to `--allow-run` flag.
Additionally `Deno.permissions.query()` allows to query for specific
programs within allowlist.
This commit adds blob URL support. Blob URLs are stored in a process
global storage, that can be accessed from all workers, and the module
loader. Blob URLs can be created using `URL.createObjectURL` and revoked
using `URL.revokeObjectURL`.
This commit does not add support for `fetch`ing blob URLs. This will be
added in a follow up commit.
This commit marks the `Deno.iter` and `Deno.iterSync` utils as
deprecated, and schedules them for removal in Deno 2.0. These
utilities are implemented in pure JS, so should not be part of the
Deno namespace.
These utilities are now available in std/io/util:
denoland/deno_std#843.
This commit marks the `Deno.Buffer` / `Deno.readAll` /
`Deno.readAllSync` / `Deno.writeAll` / `Deno.writeAllSync` utils as
deprecated, and schedules them for removal in Deno 2.0. These
utilities are implemented in pure JS, so should not be part of the
Deno namespace.
These utilities are now available in std/io/buffer and std/io/util:
https://github.com/denoland/deno_std/pull/808.
This additionallty removes all internal dependance on Deno.Buffer.
- Improves op performance.
- Handle op-metadata (errors, promise IDs) explicitly in the op-layer vs
per op-encoding (aka: out-of-payload).
- Remove shared queue & custom "asyncHandlers", all async values are
returned in batches via js_recv_cb.
- The op-layer should be thought of as simple function calls with little
indirection or translation besides the conceptually straightforward
serde_v8 bijections.
- Preserve concepts of json/bin/min as semantic groups of their
inputs/outputs instead of their op-encoding strategy, preserving these
groups will also facilitate partial transitions over to v8 Fast API for the
"min" and "bin" groups
This commit moves implementation of bin ops to "deno_core" crates
as well as unifying logic between bin ops and json ops to reuse
as much code as possible (both in Rust and JavaScript).
This patch doesn't actually fix the bug I was hoping to fix, which is
that `update_diagnostics()` sometimes gets called even when there are
more updates that should be processed first. I did eventually figure out
that this issue is caused by Tokio's cooperative yielding, which
currently can't be disabled.
However overall it makes the debounce code somewhat more readable IMO,
which is why I'm suggesting to land it anyway.
Change `Console.#printFunc` to pass a log level as the second argument
(0 = debug, 3 = error), instead of a boolean for `isErr`. This does not
change the Deno runtime behaviour at all.
This commit rewrites "dispatch_minimal" into "dispatch_buffer".
It's part of an effort to unify JS interface for ops for both json
and minimal (buffer) ops.
Before this commit "minimal ops" could be either sync or async
depending on the return type from the op, but this commit changes
it to have separate signatures for sync and async ops (just like
in case of json ops).
This commit starts splitting out the deno_web op crate into multiple
smaller crates. This commit splits out WebIDL and URL API, but in the
future I want to split out each spec into its own crate. That means we
will have (in rough order of loading): `webidl`, `dom`, `streams`,
`console`, `encoding`, `url`, `file`, `fetch`, `websocket`, and
`webgpu` crates.
This commit rewrites implementation of "JsRuntime::mod_evaluate".
Event loop is no longer polled automatically and users must manually
drive event loop forward after calling "mod_evaluate".
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit updates implementation of import maps to
align it to current revision of the spec.
Existing tests were removed in favor of using suite from
WPT.
This commit changes formatting of JS errors; by not showing
source lines for internal code. Where possible, instead using
the top stack frame associated with user code i.e. the first
location that is colourful and not a "deno:" URL.
This commit makes "Deno.link" and "Deno.linkSync" stable.
The permission required has been changed to read-write to
ensure one cannot escape the sandbox.
This commit adds support for type definitions in "deno doc";
with this change "deno doc" is able to leverage the same directives
as TS compiler.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds a new subcommand called "coverage"
which can generate code coverage reports to stdout in
multiple formats from code coverage profiles collected to disk.
Currently this supports outputting a pretty printed diff and
the lcov format for interoperability with third-party services and tools.
Code coverage is still collected via other subcommands
that run and collect code coverage such as
"deno test --coverage=<directory>" but that command no
longer prints a pretty printed report at the end of a test
run with coverage collection enabled.
The restrictions on which files that can be reported on has
also been relaxed and are fully controllable with the include
and exclude regular expression flags on the coverage subcommand.
Co-authored-by: Luca Casonato <lucacasonato@yahoo.com>
This commit removes redundant "--reload" args because "util::deno_cmd"
recreates "DENO_DIR".
This commit also fixes ta_reload in integration tests to actually test reload.
This adds an exit sanitizer to ensure that code being tested or
dependencies of that code can't accidentally call "Deno.exit"
leading to partial test runs and false results.
This commit rewrites "JsRuntime::poll" function to fix a corner case that
might caused "overflown_response" to be overwritten by other overflown response.
The logic has been changed to allow returning multiple overflown response
alongside responses from shared queue.
This PR deprecates the "--ts"/"-T" flag of "deno eval" (which will later be removed in 2.0)
and introduces "--ext" which is used by "deno fmt" for content type selection.
This is to ensure we have a single flag that can be used across subcommands
to select the language (JS/TS).
Currently if WebSocket is closed without code, it will error
while on Chrome it would close with code 1005 instead.
Co-authored-by: crowlKats <13135287+crowlKats@users.noreply.github.com>
This commit adds support for formatting JSON and JSONC
in "deno fmt".
New values "json" and "jsonc" are added to "--ext" flag for
standard input processing.
This commit adds support for loading import maps from URLs,
both remote and local.
This feature is supported in CLI flag as well as in runtime
compiler API.
This PR makes json_op_sync/async generic to all Deserialize/Serialize types
instead of the loosely-typed serde_json::Value. Since serde_json::Value
implements Deserialize/Serialize, very little existing code needs to be updated,
however as json_op_sync/async are now generic, type inference is broken in some
cases (see cli/build.rs:146). I've found this reduces a good bit of boilerplate,
as seen in the updated deno_core examples.
This change may also reduce serialization and deserialization overhead as serde
has a better idea of what types it is working with. I am currently working on
benchmarks to confirm this and I will update this PR with my findings.
When we migrated away from all the locks, there was a regression that
was not caught immediately. The tsc::get_asset() would attempt to modify
the snapshot, but the problem was that the snapshot was a .clone() of
the inner language server's assets, which meant that modifications to
that where lost. When we then attempted to do a hover on those assets,
the inner language servers assets didn't have the retrieved asset, and
therefore would throw an error.
Commit 2828690fc ("fix(lsp): fix deadlocks, use one big mutex") from
last month introduced a regression in asset cache lookups where results
of lazy caching were lost due to operating on a copy of the asset cache.
This commit fixes that by copying the asset from the copy to the real
cache.
The mutex was used to hide the fact that the Sources object mutates
itself when it's queried. Be honest about that and mark everything that
directly or indirectly mutates it as `mut`.
This is a follow-up to commit 2828690fc7
from last month ("fix(lsp): fix deadlocks, use one big mutex (#9271)")
Include the lower-level error message in the generic error message.
No test because I can't actually make it fail by passing it bad PEM.
I checked and `reqwest::Certificate::from_pem()` always returns `Ok()`.
Fixes #9364.
This removes the std folder from the tree.
Various parts of the tests are pretty tightly dependent
on std (47 direct imports and 75 indirect imports, not
counting the cli tests that use them as fixtures) so I've
added std as a submodule for now.
This commit reorganises cli/tests/integration_tests.rs.
All integration tests had been moved into integration module,
which allows to run only integration tests by "cargo test integration".
Additionally some tests were further grouped under nested modules
like "inspector", "file_watcher" or "repl".
This commits makes use of source maps and the original source
when printing lacking line coverage in the pretty printer.
Only the executable lines are checked as before (as non-executable
lines will always be ignored anyways). The lines then mapped to the
appropriate source line when a source map is present.
The LSP code had numerous places where competing threads could take out
out locks in different orders, making it very prone to deadlocks.
This commit sidesteps the entire issue by switching to a single lock.
The above is a little white lie: the Sources struct still uses a mutex
internally to avoid having to boil the ocean (because being honest about
what it does involves changing all its methods to `&mut self` but that
ripples out extensively...) I'll save that one for another day.
This commit adds support for formatting markdown files with "deno fmt".
Additionally "--ext={js|jsx|ts|tsx|md}" flag was added to "deno fmt"
that allows to specify file type when providing contents over stdio.
This commit adds --target and --lite flags to deno compile subcommand.
--target allows to cross-compile binary to different target architectures by
fetching appropriate binary from remote server on first run. All downloaded
binaries are stored in "$DENO_DIR/dl".
--lite allows to use lite version of the runtime (ie. the one that doesn't contain
built-in tooling like formatter or linter).
Previously, calling `Process#kill()` after the process had exited would
sometimes throw a `TypeError` on Windows. After this patch, it will
throw `NotFound` instead, just like other platforms.
This patch also fixes flakiness of the `runKillAfterStatus` test on
Windows.
* fix: align DOMException API to the spec
* test: fix test case 070_location
* test(DOMException): disable "does not inherit from Error: class-side"
test of WPT
* test: remove test cases in deno codebase
* docs: add note about skipped test
This commit fixes hang in web workers occuring when sending
"undefined" as message value. It is a temporary band-aid
until proper structured close is implemented.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds new binary target called "denort".
It is a "lite" version of "deno" binary that can only execute
code embedded inside the binary itself.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
When we were doing single process in-memory reports getting the source
from the runtime was practical, but now that we're writing to disk this
conflicts with the format tools taking raw v8 coverage dumps expect.
This commit adds new option to "Worker" Web API that allows to
configure permissions.
New "Worker.deno.permissions" option can be used to define limited
permissions to the worker thread by either:
- inherit set of parent thread permissions
- use limited subset of parent thread permissions
- revoke all permissions (full sandbox)
In order to achieve this functionality "CliModuleLoader"
was modified to accept "initial permissions", which are used
for top module loading (ie. uses parent thread permission set
to load top level module of a worker).
This commit removes "js" module from "cli".
It contained stuff related to TypeScript compiler (snapshot,
declaration files) and thus it was moved to "tsc" module.
This commit rewrites initialisation of the "shared queue" and
in effect prevents from double execution of "core/core.js" and
"core/error.js".
Previously both of these files were executed every time a "JsRuntime"
was created. That lead to a situation where one copy of each script
was included in the snapshot and then another copy would be
executed after loading the snapshot.
Effectively "JsRuntime::shared_init" was removed; instead execution
of those scripts and actual initialisation of shared queue
was split into two helper functions: "JsRuntime::js_init" and
"JsRuntime::share_queue_init".
Additionally stale TODO comments were removed.
Merging multiple runs isn't quite right because we
rely on a 0 count to signal that a block hasn't been called.
Other tools like c8 expect this to be true as-well so we
need to do our best to merge coverage files rather
than duplicating them.
The child process kept running and printing "hello" to stdout.
This commit also removes the dependency on reqwest and instead
switches to the re-export from the fetch crate.
Brings back commit 1a2e7741c3.
This commit adds back "/json/list" endpoint to
inspector server which was erroneously removed
during server rewrite.
Co-authored-by: Ben Noordhuis <info@bnoordhuis.nl>
Fix bug in regular expression and make the regular expression more
strict.
In a string passed to new RegExp(), '[\t\s]' is identical to '[ts]' and
not `/[\t\s]/`. For that, the backslash needs to be escaped in the
string. Futhermore, `\t` is the tab character and is included in the
special regexp value `\s` so is unnecessary.
That would reduce the RegExp to new RegExp(`^${value}\\s*;?`) but
there's no point in matching 0 or more space characters followed by 0 or
one semi-colons as that will match no matter what follows `value`.
To make it more strict, require one of space, semicolon, or
end-of-string after value.
Allowlist checking already uses hosts but for some reason
requests, revokes and the runtime permissions API use URLs.
- BREAKING(lib.deno.unstable.d.ts): Change
NetPermissionDescriptor::url to NetPermissionDescriptor::host
- fix(runtime/permissions): Don't add whole URLs to the
allowlist on request
- fix(runtime/permissions): Harden strength semantics:
({ name: "net", host: "127.0.0.1" } is stronger than
{ name: "net", host: "127.0.0.1:8000" }) for blocklisting
- refactor(runtime/permissions): Use tuples for hosts, make
the host optional in Permissions::{query_net, request_net, revoke_net}()
This PR refactors "cli/flags.rs" and "runtime/permissions.rs" so
that "allow_read", "allow_write" and "allow_net" themselves
have allowlists, instead of storing them in additional fields.
This commit rewrites "chown_test.ts" to use the GNU "id" command
instead of python. This won't work on Windows, but these tests aren't
currently run on Windows anyway.
Implementors of `deno_core::JsRuntime` might want to do additional actions
during each turn of event loop, eg. `deno_runtime::Worker` polls inspector,
`deno_runtime::WebWorker` receives/dispatches messages from/to worker host.
Previously `JsRuntime::mod_evaluate` was implemented in such fashion that it
only polled `JsRuntime`'s event loop. This behavior turned out to be wrong
in the example of `WebWorker` which couldn't receive/dispatch messages because
its implementation of event loop was never called.
This commit rewrites "mod_evaluate" to return a handle to receiver that resolves
when module's promise resolves. It is now implementors responsibility to poll
event loop after calling `mod_evaluate`.
This commit migrates all ops to use new resource table
and "AsyncRefCell".
Old implementation of resource table was completely
removed and all code referencing it was updated to use
new system.
This commit moves Deno JS runtime, ops, permissions and
inspector implementation to new "deno_runtime" crate located
in "runtime/" directory.
Details in "runtime/README.md".
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
This commit does major refactor of "Worker" and "WebWorker",
in order to decouple them from "ProgramState" and "Flags".
The main points of interest are "create_main_worker()" and
"create_web_worker_callback()" functions which are responsible
for creating "Worker" and "WebWorker" in CLI context.
As a result it is now possible to factor out common "runtime"
functionality into a separate crate.
This commit adds "Deno.core.createPrepareStackTrace". This function
was moved from "cli/rt/40_error_stack.js" to unify handling of stack frames in core
(before this PR there was implicit dependency on logic in "core/error.rs::JsError").
Unfortunately formatting logic must still be duplicated in "cli/error.js::PrettyJsError"
to provide coloring, but currently there's no solution to this problem.
"createPrepareStackTrace" can accept a single argument; a function that takes
a location and provides source mapped location back.
