This commit introduces "ProcState::maybe_resolver" field, which
stores a single instance of resolver for the whole lifetime of the
process, instead of creating these resolvers for each creation
of module graph. As a result, this resolver can be used in fallback
case where graph is not constructed (REPL, loading modules using
"require") unifying resolution logic.
This commit speeds up compat tests by using local copy of "deno_std"
instead of downloading it from https://deno.land for each test.
Additionally type checking is skipped.
This allows resources to be "streams" by implementing read/write/shutdown. These streams are implicit since their nature (read/write/duplex) isn't known until called, but we could easily add another method to explicitly tag resources as streams.
`op_read/op_write/op_shutdown` are now builtin ops provided by `deno_core`
Note: this current implementation is simple & straightforward but it results in an additional alloc per read/write call
Closes #12556
This commit changes formatting of elapsed time in test
runner output.
Instead of "XXXms", reporter outputs one of:
- "XXXms" for <1000ms
- "XXs" for <60s
- "XXXmYYs" for >=60s
This commit integrates import map and "classic" resolutions in
the "--compat" mode when using ES modules; in effect
"http:", "https:" and "blob:" imports now work in compat mode.
The algorithm works as follows:
1. If there's an import map, try to resolve using it and if succeeded
return the specifier
2. Try to resolve using "Node ESM resolution", and if succeeded return
the specifier
3. Fall back to regular ESM resolution
Closes #11826
**BREAKING CHANGE** this behaviour was disable when introduced in Deno 1.14/TypeScript 4.4. It will highlight code that unsafely handles variables that are caught, and will cause type errors in unsafe code.
Closes #11882
BREAKING CHANGE: Previously when `--location` was set, the unique storage key was derived from the the URL of the location instead of just the origin. This change correctly uses just the origin. This may cause previously persisted storage to change its key and data to not be available with the same location as before.
A `handshake()` method was added that returns when the TLS handshake is
complete. The `TlsListener` and `TlsConn` interfaces were added to
accomodate this new method.
Closes: #11759.
`fetch()` and client-side websocket used to support HTTP/2, but this
regressed in #11491. This patch reenables it by explicitly adding `h2`
and `http/1.1` to the list of ALPN protocols on the HTTP and websocket
clients.
A bug was fixed that could cause a hang when a method was
called on a TlsConn object that had thrown an exception earlier.
Additionally, a bug was fixed that caused TlsConn.write() to not
completely flush large buffers (>64kB) to the socket.
The public `TlsConn.handshake()` API is scheduled for inclusion in the
next minor release. See https://github.com/denoland/deno/pull/12467.
This commit adds CJS and ESM Node resolvers to the "--compat" mode.
The functionality is spread across "cli/compat" module and Node compatibility
layer in "deno_std/node"; this stems from the fact that ES module resolution
can only be implemented in Rust as it needs to directly integrated with
"deno_core"; however "deno_std/node" already provided CJS module resolution.
Currently this resolution is only active when running a files using
"deno run --compat --unstable <filename>", and is not available in other
subcommands, which will be changed in follow up commits.
Returns empty values in case of errors, source lines are non-essential anyway. These errors can happen e.g. when source files change at runtime. A warning is also printed to help us track when it happens in unexpected cases besides this.
Currently all async ops are polled lazily, which means that op
initialization code is postponed until control is yielded to the event
loop. This has some weird consequences, e.g.
```js
let listener = Deno.listen(...);
let conn_promise = listener.accept();
listener.close();
// `BadResource` is thrown. A reasonable error would be `Interrupted`.
let conn = await conn_promise;
```
JavaScript promises are expected to be eagerly evaluated. This patch
makes ops actually do that.
On OS X, the watcher sometimes witnesses the creation of it's own root
directory. Creating that directory using a sync op instead of an async
op sidesteps the issue.
This commit fixes problem with LSP where diagnostics coming
from "deno lint" don't respect configuration file.
LSP was changed to store "Option<ConfigFile>", "Option<LintConfig>"
and "Option<FmtConfig>" on "Inner"; as well as storing "Option<LintConfig>"
and "Option<FmtConfig>" on "StateSnapshot".
Co-authored-by: Kitson Kelly <me@kitsonkelly.com>
`Window`'s `self` property and `DedicatedWorkerGlobalScope`'s `name`
property are defined as Web IDL read-only attributes with the
`[Replaceable]` extended attribute, meaning that their setter will
redefine the property as a data property with the set value, rather than
changing some internal state. Deno currently defines them as read-only
data properties instead.
Given that Web IDL requires all attributes to be accessor properties
rather than data properties, but Deno exposes almost all of those
properties as either read-only or writable data properties, it makes
sense to expose `[Replaceable]` properties as writable as well – as is
already the case with `WindowOrWorkerGlobalScope`'s `performance`
property.
GET/HEAD requests can't have bodies according to `fetch` spec. This
commit changes the HTTP server to hide request bodies for requests with
GET or HEAD methods.
This commit annotates errors returned from FS Deno APIs to include
paths that were passed to the API calls.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
WebAssembly modules compiled through `WebAssembly.compile()` and similar
non-streaming APIs don't have a URL associated to them, because they
have been compiled from a buffer source. In stack traces, V8 will use
a URL such as `wasm://wasm/d1c677ea`, with a hash of the module.
However, wasm modules compiled through streaming APIs, like
`WebAssembly.compileStreaming()`, do have a known URL, which can be
obtained from the `Response` object passed into the streaming APIs. And
as per the developer-facing display conventions in the WebAssembly
Web API spec, this URL should be used in stack traces. This change
implements that.
This commit adds automatic injection of Node globals when "--compat" flag
is present.
This is done by executing "https://deno.land/std/node/global.ts" as a "side module",
before main module is executed.
This commit makes "--compat" required to be used with "--unstable" flag, as some
of Node globals require unstable Deno APIs.
This commit adds "--compat" flag. When the flag is passed a set of mappings for
built-in Node modules is injected into the import map. If user doesn't
explicitly provide an import map (using "--import-map" flag) then a map is
created on the fly. If there are already existing mappings in import map that
would clash with built-in Node modules a set of diagnostics is printed to the
terminal with suggestions how to proceed.
This adds support for using in memory CA certificates for
`Deno.startTLS`, `Deno.connectTLS` and `Deno.createHttpClient`.
`certFile` is deprecated in `startTls` and `connectTls`, and removed
from `Deno.createHttpClient`.
Our oneshot receiver in `HyperService::call` would unwrap and panic, the `.await` on the oneshot receiver happens when the sender is dropped.
The sender is dropped in `op_http_response` because:
1. We take `ResponseSenderResource`
2. Then get `ConnResource` and early exit on failure (conn already closed)
3. The taken sender then gets dropped in this early exit before any response is sent over the channel
Fallbacking to returning a dummy response to hyper seems to be a fine quickfix
This panic could happen in the following cases:
- A non-fatal error being thrown from a worker, that doesn't terminate
the worker's execution, but propagates to the main thread without
being handled, and makes the main thread terminate.
- A nested worker being alive while its parent worker gets terminated.
- A race condition if the main event loop terminates the worker as part
of its last task, but the worker doesn't fully terminate before the
main event loop stops running.
This panic happens because a worker's event loop should have pending ops
as long as the worker isn't closed or terminated – but if an event loop
finishes running while it has living workers, its associated
`WorkerThread` structs will be dropped, closing the channels that keep
those ops pending.
This change adds a `Drop` implementation to `WorkerThread`, which
terminates the worker without waiting for a response. This fixes the
panic, and makes it so nested workers are automatically terminated once
any of their ancestors is closed or terminated.
This change also refactors a worker's termination code into a
`WorkerThread::terminate()` method.
Closes #11342.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds support for configuration file for "deno fmt"
subcommand. It is also respected by LSP when formatting
files.
Example configuration:
{
"fmt": {
"files": {
"include": ["src/"],
"exclude": ["src/testdata/"]
},
"options": {
"useTabs": true,
"lineWidth": 80,
"indentWidth": 4,
"singleQuote": true,
"textWrap": "preserve"
}
}
}
This adds support for the URLPattern API.
The API is added in --unstable only, as it has not yet shipped in any
browser. It is targeted for shipping in Chrome 95.
Spec: https://wicg.github.io/urlpattern/
Co-authored-by: crowlKats < crowlkats@toaxl.com >
When `worker.terminate()` is called, the spec requires that the
corresponding port message queue is emptied, so no messages can be
received after the call, even if they were sent from the worker before
it was terminated.
The spec doesn't require this of `self.close()`, and since Deno uses
different channels to send messages and to notify that the worker was
closed, messages might still arrive after the worker is known to be
closed, which are currently being dropped. This change fixes that.
The fix involves two parts: one on the JS side and one on the Rust side.
The JS side was using the `#terminated` flag to keep track of whether
the worker is known to be closed, without distinguishing whether further
messages should be dropped or not. This PR changes that flag to an
enum `#state`, which can be one of `"RUNNING"`, `"CLOSED"` or
`"TERMINATED"`.
The Rust side was removing the `WorkerThread` struct from the workers
table when a close control was received, regardless of whether there
were any messages left to read, which made any subsequent calls to
`op_host_recv_message` to return `Ok(None)`, as if there were no more
mesasges. This change instead waits for both a close control and for
the message channel's sender to be closed before the worker thread is
removed from the table.
This commit adds support for following flags in deno lint subcommand:
--config - allows to load configuration file and parses "lint" object
--rules-tags=<tags> - allows specifying which set of tagged rules should be run
--rules-include=<rules> - allow specifying which rules should be run
--rules-exclude=<rules> - allow specifying which rules should not be run
This commit merges the two vectors of specifiers into a single one introducing
the concept of a "TestMode" which is a tri-state enum specifying how a specifier
is to be tested (as documentation, as an executable module or as both).
This is determined during the collection phase and determines how a specifier
will be executed based on how the specifier was collected (directly or not) and
if it has an eligible media_type when fetched.
For example "deno test README.md" is marked as documentation because, while it
is a direct inclusion it is not an executable media type therefore will only
have the fenced code blocks that can be parsed from it tested.
Classic worker scripts are now executed in the context of a Tokio
runtime. This does mean we can not spawn more tokio runtimes in
"op_worker_sync_fetch". We instead spawn a new thread there, that can
create a new Tokio runtime that we can use to block the worker thread.
The streaming WASM support code inspects the Response object's
Content-Type header but if that was missing, it failed with a fairly
inscrutable "String.prototype.toLowerCase called on null or undefined"
exception. Now it raises a more legible "Invalid WebAssembly content
type" exception.
This commit adds a test case for "Http: connection closed before
message completed" error as well as fixing an edge with resource
leak when the error is raised.
This commit adds "--unsafely-treat-insecure-origin-as-secure" flag
that allows to disable SSL verification for all domains, or specific
domains if they were passed as an argument to the flag.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds "DENO_TLS_CA_STORE" env variable to support
optionally loading certificates from the users local certificate store.
This will allow them to successfully connect via tls with corporate
and self signed certs provided they have them installed in their keystore.
It also allows them to deal with revoked certs by simply updating
their keystore without having to upgrade Deno.
Currently supported values are "mozilla", "system" or empty value.
This commit removes implementation of "native plugins" and replaces
it with FFI API.
Effectively "Deno.openPlugin" API was replaced with "Deno.dlopen" API.
This commit adds support for specifying threshold in the "--fail-fast"
flag for "deno test" subcommand. Previously using "--fail-fast" stopped
running the test suite after first failure and with this change users
may specify number of failed tests that will cause the suite to be interrupted.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds "--terse" flag to "deno test" that makes testing reporter
output one character per test case instead of one line per case.
This is an unstable feature.
The WebAssembly streaming APIs used to be enabled, but used to take
buffer sources as their first argument (see #6154 and #7259). This
change re-enables them, requiring a Promise<Response> instead, as well as
enabling asynchronous compilation of WebAssembly modules.
This commits moves implementation of net related APIs available on "Deno"
namespace to "deno_net" extension.
Following APIs were moved:
- Deno.listen()
- Deno.connect()
- Deno.listenTls()
- Deno.serveHttp()
- Deno.shutdown()
- Deno.resolveDns()
- Deno.listenDatagram()
- Deno.startTls()
- Deno.Conn
- Deno.Listener
- Deno.DatagramConn
This commit adds support for piping console messages to inspector.
This is done by "wrapping" Deno's console implementation with default
console provided by V8 by the means of "Deno.core.callConsole" binding.
Effectively each call to "console.*" methods calls a method on Deno's
console and V8's console.
This PR refactors the usages of delay utility in js unit testing. The same
utiliy is defined in several places with different names. This PR replaces those
usages with the one provided in std/async/delay.ts to improve the readability
and consistency of test code.
This commit adds new options to unstable "Deno.createHttpClient" API.
"proxy" and "basicAuth" options were added that allow to use custom proxy
when client instance is passed to "fetch" API.
This commit renames "JsRuntime::execute" to "JsRuntime::execute_script". Additionally
same renames were applied to methods on "deno_runtime::Worker" and
"deno_runtime::WebWorker".
A new macro was added to "deno_core" called "located_script_name" which
returns the name of Rust file alongside line no and col no of that call site.
This macro is useful in combination with "JsRuntime::execute_script"
and allows to provide accurate place where "one-off" JavaScript scripts
are executed for internal runtime functions.
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
This commit changes module loading implementation in "deno_core"
to call "ModuleLoader::prepare" hook only once per entry point.
This is done to avoid multiple type checking of the same code
in case of duplicated dynamic imports.
Relevant code in "cli/module_graph.rs" was updated as well.
This commit updates type declarations for Worker to accept specifiers
as either strings or URL, bringing it in line with TypeScript
declarations and browser behavior.
This commit changes "op_http_response_write" to first send response chunk
and then poll the underlying HTTP connection.
Previously after writing a chunk of response HTTP connection wasn't polled
and thus data wasn't written to the socket until after next op interacting
with the connection.
Waiting on next request in Deno.serveHttp() API hanged
when responses were using ReadableStream. This was caused
by op_http_request_next op that was never woken after
response was fully written. This commit adds waker field to
DenoService which is called after response is finished.
This commit removes all JS based text encoding / text decoding. Instead
encoding now happens in Rust via encoding_rs (already in tree). This
implementation retains stream support, but adds the last missing
encodings. We are incredibly close to 100% WPT on text encoding now.
This should reduce our baseline heap by quite a bit.
This commit adds "CancelHandle" to "ConnResource" and changes
"op_http_next_request" to await for the cancel signal. In turn
when async iterating over "Deno.HttpConn" the iterator breaks
upon closing of the resource.
Replaces the file-backed provider by an in-memory one because proper
file locking is a hard problem that detracts from the proof of concept.
Teach the WPT runner how to extract tests from .html files because all
the relevant tests in test_util/wpt/webmessaging/broadcastchannel are
inside basics.html and interface.html.
In #9118, TLS streams were split into a "read half" and a "write half"
using tokio::io::split() to allow concurrent Conn#read() and
Conn#write() calls without one blocking the other. However, this
introduced a bug: outgoing data gets discarded when the TLS stream is
gracefully closed, because the read half is closed too early, before all
TLS control data has been received.
Fixes: #9692
Fixes: #10049
Fixes: #10296
Fixes: denoland/deno_std#750
This commit implements file watching for deno test.
When a file is changed, only the test modules which
use it as a dependency are rerun.
This is accomplished by reworking the file watching infrastructure
to pass the paths which have changed to the resolver, and then
constructing a module graph for each test module to check if it
contains any changed files.
Fixes a pesky bug in the fetch implementation where if the init part is
specified in `fetch` instead of the `Request` constructor, the
fillHeaders function receives two references to the same object, causing
it to append to the same list being iterated over.
This commit adds support for running test in parallel.
Entire test runner functionality has been rewritten
from JavaScript to Rust and a set of ops was added to support reporting in Rust.
A new "--jobs" flag was added to "deno test" that allows to configure
how many threads will be used. When given no value it defaults to 2.
Extensions allow declarative extensions to "JsRuntime" (ops, state, JS or middleware).
This allows for:
- `op_crates` to be plug-and-play & self-contained, reducing complexity leaked to consumers
- op middleware (like metrics_op) to be opt-in and for new middleware (unstable, tracing,...)
- `MainWorker` and `WebWorker` to be composable, allowing users to extend workers with their ops whilst benefiting from the other infrastructure (inspector, etc...)
In short extensions improve deno's modularity, reducing complexity and leaky abstractions for embedders and the internal codebase.
General cleanup of module loading code, tried to reduce indentation in various methods
on "JsRuntime" to improve readability.
Added "JsRuntime::handle_scope" helper function, which returns a "v8::HandleScope".
This was done to reduce a code pattern that happens all over the "deno_core".
Additionally if event loop hangs during loading of dynamic modules a list of
currently pending dynamic imports is printed.
`InvalidDNSNameError` is thrown when a string is not a valid hostname,
e.g. it contains invalid characters, or starts with a numeric digit. It
does not involve a (failed) DNS lookup.
denort is an optimization to "deno compile" to produce slightly smaller
output. It's a decent idea, but causes a lot of negative side-effects:
- Deno's link time is a source of constant agony both locally and in CI,
denort doubles link time.
- The release process is a long and arduous undertaking with many manual
steps. denort necessitates an additional manual zip + upload from M1
apple computers.
- The "deno compile" interface is complicated with the "--lite" option.
This is confusing for uses ("why wouldn't you want lite?").
The benefits of this feature do not outweigh the negatives. We must find
a different approach to optimizing "deno compile" output.
This commits adds adds "permissions" option to the test definitions
which allows tests to run with different permission sets than
the process's permission.
The change will only be in effect within the test function, once the
test has completed the original process permission set is restored.
Test permissions cannot exceed the process's permission.
You can only narrow or drop permissions, failure to acquire a
permission results in an error being thrown and the test case will fail.
This commit aligns the `fetch` API and the `Request` / `Response`
classes belonging to it to the spec. This commit enables all the
relevant `fetch` WPT tests. Spec compliance is now at around 90%.
Performance is essentially identical now (within 1% of 1.9.0).
This commit fixes the URL returned from `request.url` in the HTTP server
to be fully qualified. This previously existed, but was removed and
accidentially not readded during optimizations of the HTTP ops.
Returning a non fully qualified URL from `Request#url` is not spec
compliant.
The panic was caused by the lack of an error class mapping for
futures::channel::TrySendError, but it shouldn't have been throwing an error in
the first place - when a worker has terminated, postMessage should just return.
The issue was that the termination message hadn't yet been recieved, so it was
carrying on with trying to send the message. This adds another check on the Rust
side for if the channel is closed, and if it is the worker is treated as
terminated.
This commit aligns `Headers` to spec. It also removes the now unused
03_dom_iterable.js file. We now pass all relevant `Headers` WPT. We do
not implement any sort of header filtering, as we are a server side
runtime.
This is likely not the most efficient implementation of `Headers` yet.
It is however spec compliant. Once all the APIs in the `HTTP` hot loop
are correct we can start optimizing them. It is likely that this commit
reduces bench throughput temporarily.
This commit adds allowlist support to `--allow-run` flag.
Additionally `Deno.permissions.query()` allows to query for specific
programs within allowlist.
This commit adds blob URL support. Blob URLs are stored in a process
global storage, that can be accessed from all workers, and the module
loader. Blob URLs can be created using `URL.createObjectURL` and revoked
using `URL.revokeObjectURL`.
This commit does not add support for `fetch`ing blob URLs. This will be
added in a follow up commit.
This commit marks the `Deno.Buffer` / `Deno.readAll` /
`Deno.readAllSync` / `Deno.writeAll` / `Deno.writeAllSync` utils as
deprecated, and schedules them for removal in Deno 2.0. These
utilities are implemented in pure JS, so should not be part of the
Deno namespace.
These utilities are now available in std/io/buffer and std/io/util:
https://github.com/denoland/deno_std/pull/808.
This additionallty removes all internal dependance on Deno.Buffer.
- Improves op performance.
- Handle op-metadata (errors, promise IDs) explicitly in the op-layer vs
per op-encoding (aka: out-of-payload).
- Remove shared queue & custom "asyncHandlers", all async values are
returned in batches via js_recv_cb.
- The op-layer should be thought of as simple function calls with little
indirection or translation besides the conceptually straightforward
serde_v8 bijections.
- Preserve concepts of json/bin/min as semantic groups of their
inputs/outputs instead of their op-encoding strategy, preserving these
groups will also facilitate partial transitions over to v8 Fast API for the
"min" and "bin" groups
This commit moves implementation of bin ops to "deno_core" crates
as well as unifying logic between bin ops and json ops to reuse
as much code as possible (both in Rust and JavaScript).
Change `Console.#printFunc` to pass a log level as the second argument
(0 = debug, 3 = error), instead of a boolean for `isErr`. This does not
change the Deno runtime behaviour at all.
This commit rewrites "dispatch_minimal" into "dispatch_buffer".
It's part of an effort to unify JS interface for ops for both json
and minimal (buffer) ops.
Before this commit "minimal ops" could be either sync or async
depending on the return type from the op, but this commit changes
it to have separate signatures for sync and async ops (just like
in case of json ops).
This commit rewrites implementation of "JsRuntime::mod_evaluate".
Event loop is no longer polled automatically and users must manually
drive event loop forward after calling "mod_evaluate".
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit changes formatting of JS errors; by not showing
source lines for internal code. Where possible, instead using
the top stack frame associated with user code i.e. the first
location that is colourful and not a "deno:" URL.
This commit adds support for type definitions in "deno doc";
with this change "deno doc" is able to leverage the same directives
as TS compiler.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds a new subcommand called "coverage"
which can generate code coverage reports to stdout in
multiple formats from code coverage profiles collected to disk.
Currently this supports outputting a pretty printed diff and
the lcov format for interoperability with third-party services and tools.
Code coverage is still collected via other subcommands
that run and collect code coverage such as
"deno test --coverage=<directory>" but that command no
longer prints a pretty printed report at the end of a test
run with coverage collection enabled.
The restrictions on which files that can be reported on has
also been relaxed and are fully controllable with the include
and exclude regular expression flags on the coverage subcommand.
Co-authored-by: Luca Casonato <lucacasonato@yahoo.com>
This commit removes redundant "--reload" args because "util::deno_cmd"
recreates "DENO_DIR".
This commit also fixes ta_reload in integration tests to actually test reload.
This adds an exit sanitizer to ensure that code being tested or
dependencies of that code can't accidentally call "Deno.exit"
leading to partial test runs and false results.
This commit rewrites "JsRuntime::poll" function to fix a corner case that
might caused "overflown_response" to be overwritten by other overflown response.
The logic has been changed to allow returning multiple overflown response
alongside responses from shared queue.
