This commit fixes the URL returned from `request.url` in the HTTP server
to be fully qualified. This previously existed, but was removed and
accidentially not readded during optimizations of the HTTP ops.
Returning a non fully qualified URL from `Request#url` is not spec
compliant.
The panic was caused by the lack of an error class mapping for
futures::channel::TrySendError, but it shouldn't have been throwing an error in
the first place - when a worker has terminated, postMessage should just return.
The issue was that the termination message hadn't yet been recieved, so it was
carrying on with trying to send the message. This adds another check on the Rust
side for if the channel is closed, and if it is the worker is treated as
terminated.
This commit aligns `Headers` to spec. It also removes the now unused
03_dom_iterable.js file. We now pass all relevant `Headers` WPT. We do
not implement any sort of header filtering, as we are a server side
runtime.
This is likely not the most efficient implementation of `Headers` yet.
It is however spec compliant. Once all the APIs in the `HTTP` hot loop
are correct we can start optimizing them. It is likely that this commit
reduces bench throughput temporarily.
Raise the soft limit to the hard limit when possible. This is similar
to what Node.js does to avoid running into "out of file descriptors"
errors too quickly.
On most Linux systems, raises the limit from 1,024 to 1,048,576.
On most macOS systems, raises the limit from 256 to 10,240.
Fixes #10148.
This stabilizes Deno.ftruncate and Deno.ftruncateSync.
This is a well known system call and the interface is
not going to change. Implicitly requires write permissions
as the file has to be opened with write to be truncated.
This commit adds allowlist support to `--allow-run` flag.
Additionally `Deno.permissions.query()` allows to query for specific
programs within allowlist.
This commit adds blob URL support. Blob URLs are stored in a process
global storage, that can be accessed from all workers, and the module
loader. Blob URLs can be created using `URL.createObjectURL` and revoked
using `URL.revokeObjectURL`.
This commit does not add support for `fetch`ing blob URLs. This will be
added in a follow up commit.
This commit marks the `Deno.iter` and `Deno.iterSync` utils as
deprecated, and schedules them for removal in Deno 2.0. These
utilities are implemented in pure JS, so should not be part of the
Deno namespace.
These utilities are now available in std/io/util:
denoland/deno_std#843.
This commit marks the `Deno.Buffer` / `Deno.readAll` /
`Deno.readAllSync` / `Deno.writeAll` / `Deno.writeAllSync` utils as
deprecated, and schedules them for removal in Deno 2.0. These
utilities are implemented in pure JS, so should not be part of the
Deno namespace.
These utilities are now available in std/io/buffer and std/io/util:
https://github.com/denoland/deno_std/pull/808.
This additionallty removes all internal dependance on Deno.Buffer.
- Improves op performance.
- Handle op-metadata (errors, promise IDs) explicitly in the op-layer vs
per op-encoding (aka: out-of-payload).
- Remove shared queue & custom "asyncHandlers", all async values are
returned in batches via js_recv_cb.
- The op-layer should be thought of as simple function calls with little
indirection or translation besides the conceptually straightforward
serde_v8 bijections.
- Preserve concepts of json/bin/min as semantic groups of their
inputs/outputs instead of their op-encoding strategy, preserving these
groups will also facilitate partial transitions over to v8 Fast API for the
"min" and "bin" groups
This commit moves implementation of bin ops to "deno_core" crates
as well as unifying logic between bin ops and json ops to reuse
as much code as possible (both in Rust and JavaScript).
This patch doesn't actually fix the bug I was hoping to fix, which is
that `update_diagnostics()` sometimes gets called even when there are
more updates that should be processed first. I did eventually figure out
that this issue is caused by Tokio's cooperative yielding, which
currently can't be disabled.
However overall it makes the debounce code somewhat more readable IMO,
which is why I'm suggesting to land it anyway.
Change `Console.#printFunc` to pass a log level as the second argument
(0 = debug, 3 = error), instead of a boolean for `isErr`. This does not
change the Deno runtime behaviour at all.
This commit rewrites "dispatch_minimal" into "dispatch_buffer".
It's part of an effort to unify JS interface for ops for both json
and minimal (buffer) ops.
Before this commit "minimal ops" could be either sync or async
depending on the return type from the op, but this commit changes
it to have separate signatures for sync and async ops (just like
in case of json ops).
This commit starts splitting out the deno_web op crate into multiple
smaller crates. This commit splits out WebIDL and URL API, but in the
future I want to split out each spec into its own crate. That means we
will have (in rough order of loading): `webidl`, `dom`, `streams`,
`console`, `encoding`, `url`, `file`, `fetch`, `websocket`, and
`webgpu` crates.
This commit rewrites implementation of "JsRuntime::mod_evaluate".
Event loop is no longer polled automatically and users must manually
drive event loop forward after calling "mod_evaluate".
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit updates implementation of import maps to
align it to current revision of the spec.
Existing tests were removed in favor of using suite from
WPT.
This commit changes formatting of JS errors; by not showing
source lines for internal code. Where possible, instead using
the top stack frame associated with user code i.e. the first
location that is colourful and not a "deno:" URL.
This commit makes "Deno.link" and "Deno.linkSync" stable.
The permission required has been changed to read-write to
ensure one cannot escape the sandbox.
This commit adds support for type definitions in "deno doc";
with this change "deno doc" is able to leverage the same directives
as TS compiler.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds a new subcommand called "coverage"
which can generate code coverage reports to stdout in
multiple formats from code coverage profiles collected to disk.
Currently this supports outputting a pretty printed diff and
the lcov format for interoperability with third-party services and tools.
Code coverage is still collected via other subcommands
that run and collect code coverage such as
"deno test --coverage=<directory>" but that command no
longer prints a pretty printed report at the end of a test
run with coverage collection enabled.
The restrictions on which files that can be reported on has
also been relaxed and are fully controllable with the include
and exclude regular expression flags on the coverage subcommand.
Co-authored-by: Luca Casonato <lucacasonato@yahoo.com>
This commit removes redundant "--reload" args because "util::deno_cmd"
recreates "DENO_DIR".
This commit also fixes ta_reload in integration tests to actually test reload.
This adds an exit sanitizer to ensure that code being tested or
dependencies of that code can't accidentally call "Deno.exit"
leading to partial test runs and false results.
This commit rewrites "JsRuntime::poll" function to fix a corner case that
might caused "overflown_response" to be overwritten by other overflown response.
The logic has been changed to allow returning multiple overflown response
alongside responses from shared queue.
This PR deprecates the "--ts"/"-T" flag of "deno eval" (which will later be removed in 2.0)
and introduces "--ext" which is used by "deno fmt" for content type selection.
This is to ensure we have a single flag that can be used across subcommands
to select the language (JS/TS).
Currently if WebSocket is closed without code, it will error
while on Chrome it would close with code 1005 instead.
Co-authored-by: crowlKats <13135287+crowlKats@users.noreply.github.com>
This commit adds support for formatting JSON and JSONC
in "deno fmt".
New values "json" and "jsonc" are added to "--ext" flag for
standard input processing.
This commit adds support for loading import maps from URLs,
both remote and local.
This feature is supported in CLI flag as well as in runtime
compiler API.
This PR makes json_op_sync/async generic to all Deserialize/Serialize types
instead of the loosely-typed serde_json::Value. Since serde_json::Value
implements Deserialize/Serialize, very little existing code needs to be updated,
however as json_op_sync/async are now generic, type inference is broken in some
cases (see cli/build.rs:146). I've found this reduces a good bit of boilerplate,
as seen in the updated deno_core examples.
This change may also reduce serialization and deserialization overhead as serde
has a better idea of what types it is working with. I am currently working on
benchmarks to confirm this and I will update this PR with my findings.
When we migrated away from all the locks, there was a regression that
was not caught immediately. The tsc::get_asset() would attempt to modify
the snapshot, but the problem was that the snapshot was a .clone() of
the inner language server's assets, which meant that modifications to
that where lost. When we then attempted to do a hover on those assets,
the inner language servers assets didn't have the retrieved asset, and
therefore would throw an error.
Commit 2828690fc ("fix(lsp): fix deadlocks, use one big mutex") from
last month introduced a regression in asset cache lookups where results
of lazy caching were lost due to operating on a copy of the asset cache.
This commit fixes that by copying the asset from the copy to the real
cache.
The mutex was used to hide the fact that the Sources object mutates
itself when it's queried. Be honest about that and mark everything that
directly or indirectly mutates it as `mut`.
This is a follow-up to commit 2828690fc7
from last month ("fix(lsp): fix deadlocks, use one big mutex (#9271)")
Include the lower-level error message in the generic error message.
No test because I can't actually make it fail by passing it bad PEM.
I checked and `reqwest::Certificate::from_pem()` always returns `Ok()`.
Fixes #9364.
This removes the std folder from the tree.
Various parts of the tests are pretty tightly dependent
on std (47 direct imports and 75 indirect imports, not
counting the cli tests that use them as fixtures) so I've
added std as a submodule for now.
This commit reorganises cli/tests/integration_tests.rs.
All integration tests had been moved into integration module,
which allows to run only integration tests by "cargo test integration".
Additionally some tests were further grouped under nested modules
like "inspector", "file_watcher" or "repl".
This commits makes use of source maps and the original source
when printing lacking line coverage in the pretty printer.
Only the executable lines are checked as before (as non-executable
lines will always be ignored anyways). The lines then mapped to the
appropriate source line when a source map is present.
The LSP code had numerous places where competing threads could take out
out locks in different orders, making it very prone to deadlocks.
This commit sidesteps the entire issue by switching to a single lock.
The above is a little white lie: the Sources struct still uses a mutex
internally to avoid having to boil the ocean (because being honest about
what it does involves changing all its methods to `&mut self` but that
ripples out extensively...) I'll save that one for another day.
This commit adds support for formatting markdown files with "deno fmt".
Additionally "--ext={js|jsx|ts|tsx|md}" flag was added to "deno fmt"
that allows to specify file type when providing contents over stdio.
This commit adds --target and --lite flags to deno compile subcommand.
--target allows to cross-compile binary to different target architectures by
fetching appropriate binary from remote server on first run. All downloaded
binaries are stored in "$DENO_DIR/dl".
--lite allows to use lite version of the runtime (ie. the one that doesn't contain
built-in tooling like formatter or linter).
Previously, calling `Process#kill()` after the process had exited would
sometimes throw a `TypeError` on Windows. After this patch, it will
throw `NotFound` instead, just like other platforms.
This patch also fixes flakiness of the `runKillAfterStatus` test on
Windows.
* fix: align DOMException API to the spec
* test: fix test case 070_location
* test(DOMException): disable "does not inherit from Error: class-side"
test of WPT
* test: remove test cases in deno codebase
* docs: add note about skipped test
This commit fixes hang in web workers occuring when sending
"undefined" as message value. It is a temporary band-aid
until proper structured close is implemented.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds new binary target called "denort".
It is a "lite" version of "deno" binary that can only execute
code embedded inside the binary itself.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
When we were doing single process in-memory reports getting the source
from the runtime was practical, but now that we're writing to disk this
conflicts with the format tools taking raw v8 coverage dumps expect.
This commit adds new option to "Worker" Web API that allows to
configure permissions.
New "Worker.deno.permissions" option can be used to define limited
permissions to the worker thread by either:
- inherit set of parent thread permissions
- use limited subset of parent thread permissions
- revoke all permissions (full sandbox)
In order to achieve this functionality "CliModuleLoader"
was modified to accept "initial permissions", which are used
for top module loading (ie. uses parent thread permission set
to load top level module of a worker).
This commit removes "js" module from "cli".
It contained stuff related to TypeScript compiler (snapshot,
declaration files) and thus it was moved to "tsc" module.
This commit rewrites initialisation of the "shared queue" and
in effect prevents from double execution of "core/core.js" and
"core/error.js".
Previously both of these files were executed every time a "JsRuntime"
was created. That lead to a situation where one copy of each script
was included in the snapshot and then another copy would be
executed after loading the snapshot.
Effectively "JsRuntime::shared_init" was removed; instead execution
of those scripts and actual initialisation of shared queue
was split into two helper functions: "JsRuntime::js_init" and
"JsRuntime::share_queue_init".
Additionally stale TODO comments were removed.
Merging multiple runs isn't quite right because we
rely on a 0 count to signal that a block hasn't been called.
Other tools like c8 expect this to be true as-well so we
need to do our best to merge coverage files rather
than duplicating them.
The child process kept running and printing "hello" to stdout.
This commit also removes the dependency on reqwest and instead
switches to the re-export from the fetch crate.
Brings back commit 1a2e7741c3.
This commit adds back "/json/list" endpoint to
inspector server which was erroneously removed
during server rewrite.
Co-authored-by: Ben Noordhuis <info@bnoordhuis.nl>
Fix bug in regular expression and make the regular expression more
strict.
In a string passed to new RegExp(), '[\t\s]' is identical to '[ts]' and
not `/[\t\s]/`. For that, the backslash needs to be escaped in the
string. Futhermore, `\t` is the tab character and is included in the
special regexp value `\s` so is unnecessary.
That would reduce the RegExp to new RegExp(`^${value}\\s*;?`) but
there's no point in matching 0 or more space characters followed by 0 or
one semi-colons as that will match no matter what follows `value`.
To make it more strict, require one of space, semicolon, or
end-of-string after value.
Allowlist checking already uses hosts but for some reason
requests, revokes and the runtime permissions API use URLs.
- BREAKING(lib.deno.unstable.d.ts): Change
NetPermissionDescriptor::url to NetPermissionDescriptor::host
- fix(runtime/permissions): Don't add whole URLs to the
allowlist on request
- fix(runtime/permissions): Harden strength semantics:
({ name: "net", host: "127.0.0.1" } is stronger than
{ name: "net", host: "127.0.0.1:8000" }) for blocklisting
- refactor(runtime/permissions): Use tuples for hosts, make
the host optional in Permissions::{query_net, request_net, revoke_net}()
This PR refactors "cli/flags.rs" and "runtime/permissions.rs" so
that "allow_read", "allow_write" and "allow_net" themselves
have allowlists, instead of storing them in additional fields.
This commit rewrites "chown_test.ts" to use the GNU "id" command
instead of python. This won't work on Windows, but these tests aren't
currently run on Windows anyway.
Implementors of `deno_core::JsRuntime` might want to do additional actions
during each turn of event loop, eg. `deno_runtime::Worker` polls inspector,
`deno_runtime::WebWorker` receives/dispatches messages from/to worker host.
Previously `JsRuntime::mod_evaluate` was implemented in such fashion that it
only polled `JsRuntime`'s event loop. This behavior turned out to be wrong
in the example of `WebWorker` which couldn't receive/dispatch messages because
its implementation of event loop was never called.
This commit rewrites "mod_evaluate" to return a handle to receiver that resolves
when module's promise resolves. It is now implementors responsibility to poll
event loop after calling `mod_evaluate`.
This commit migrates all ops to use new resource table
and "AsyncRefCell".
Old implementation of resource table was completely
removed and all code referencing it was updated to use
new system.
This commit moves Deno JS runtime, ops, permissions and
inspector implementation to new "deno_runtime" crate located
in "runtime/" directory.
Details in "runtime/README.md".
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
This commit does major refactor of "Worker" and "WebWorker",
in order to decouple them from "ProgramState" and "Flags".
The main points of interest are "create_main_worker()" and
"create_web_worker_callback()" functions which are responsible
for creating "Worker" and "WebWorker" in CLI context.
As a result it is now possible to factor out common "runtime"
functionality into a separate crate.
This commit adds "Deno.core.createPrepareStackTrace". This function
was moved from "cli/rt/40_error_stack.js" to unify handling of stack frames in core
(before this PR there was implicit dependency on logic in "core/error.rs::JsError").
Unfortunately formatting logic must still be duplicated in "cli/error.js::PrettyJsError"
to provide coloring, but currently there's no solution to this problem.
"createPrepareStackTrace" can accept a single argument; a function that takes
a location and provides source mapped location back.
This commit disables source mapping of errors
for standalone binaries. Since applying source
maps relies on using file fetcher infrastructure
it's not feasible to use it for standalone binaries
that are not supposed to use that infrastructure.
This commit makes the file watcher continue to work even if module
resolution fails at the initial attempt, allowing us to execute `run`
or `bundle` subcommand when a script has invalid syntax. In such
cases, the watcher observes a single file that is specified as an
command line argument.
Fixes panic occurring in worker when "self.close()" is called
at the top level, ie. worker shuts down while
module evaluation promise hasn't yet resolved.
This commit fixes order of events for "onload" event.
Previously handler attached using "window.onload" was
always fired before handlers added using "addEventListener".
Factored out "init_v8_flags", "init_logger" and
"get_subcommand" from "main" function.
Also "Worker" was removed in favor of moving
logic to "MainWorker" and "WebWorker" respectively.
This commit removes unnecessary conversion into BTreeMap. The value
that gets converted into BTreeMap is _originally_ BTreeMap, so this
conversion is just superfluous.
Additionally, a test is added so that we can make sure the keys in the
emitted lockfile are sorted alphabetically.
This commit adds support for "--watch" flag for "bundle"
and "fmt" subcommands.
In addition to this, it refactors "run --watch" command so that
module resolution will occur every time the file watcher detects
file addition/deletion, which allows the watcher to observe a file
that is newly added to the dependency as well.
This commit does major overhaul of "upgrade" subcommand,
reducing complexity & giving more sensible console output.
Removes gz support for archives.
Uses last part of url instead of scraping to get latest version.
This commit simplifies code responsible for serialization
of web worker events. Instead of using "PrettyJsError"
it's enough to downcast to "deno_core::JsError", making
the code less specific to cli/ implementation.
This commit renames "fmt_errors::JsError" to "PrettyJsError"
to avoid confusion with "deno_core::JsError".
Consequently "CoreJsError" aliases to "deno_core::JsError"
were removed.
Additionally source mapping step has been removed from
"PrettyJsError::create" to better separate domains.
This commit moves following tools into a single "tools"
module located at "cli/tools/mod.rs":
- formatter
- linter
- test runner
- coverage collector
- installer
- binary upgrader
- repl
This commit renames "fs" module in "cli/" to "fs_util". This is purely
cosmetic change; there were a few places which aliased "crate::fs"
to "deno_fs" which was very confusing with "fs" module in ops.
This commit merges implementations of "collect_files" and "files_in_subtree",
leaving only the former. Additionally it was ensured that directories are not yielded
from this function.
This commit reworks "collect_files" utility to accept "ignore" parameter
which allows to filter out files in a single iteration instead of walking
file tree second time to excude "ignored" files.
This commit rewrites scripts in "tools/" directory
to use Deno instead of Python. In return it allows
to remove huge number of Python packages in "third_party/".
This commit removes ProgramState::permissions field.
Having permissions parsed from CLI flags stored on globally
accessible state object made it easy to mistakenly use these
permissions in situations which required "runtime" permissions.
This commit rewrites file watcher used with --watch flag.
Instead of creating new watcher after each restart, only a single
watcher is created for whole process. Additionally debouncing
mechanism has been added to prevent infinite restart loops
if multiple files were changed in quick succession.
Co-authored-by: bartossh <lenart.consulting@gmail.com>
Previously, entering a single ']' would cause repl to forever accepting
new lines, due to that `ValidationResult::Invalid` would actually be
consumed by the editor itself while continue building the lines. Instead
we should mark it as `Valid` and send the bad input for evaluation to
get the proper error from V8.
Before:
```
> ]
(you can keep entering new line here, and it will never consume input
until you Ctrl-C)
```
After:
```
> ]
Uncaught SyntaxError: Unexpected token ']'
>
```
This commit renames two assertion functions to better align with JS API:
- assertStringContains -> assertStringIncludes
- assertArrayContains -> assertArrayIncludes
This changes the behavior of keyboard interrupts (ctrl+c) to continue,
clearing the current line instead of exiting.
Exit can still be done with ctrl+d or by calling close().
This writes all evaluaton results to stdout regardless if the result is
an error or not.
This matches the behavior of other read-eval-print-loops like Node.
This commit fixes test runner by awaitning "Deno.runTests()" call,
which ensures proper error is returned when there's an unresolved
promise that's being awaited.
This extracts prelude injection into a helper function and moves the
prelude string literal into a top level static string to help trim some
of the fat out of the run function.
This commit changes implementation of top-level-await in "deno_core".
Previously promise returned from module evaluation was not awaited,
leading to out-of-order execution of modules that have TLA. It's been
fixed by changing "JsRuntime::mod_evaluate" to be an async function
that resolves when the promise returned from module evaluation also
resolves. When waiting for promise resolution event loop is polled
repeatedly, until there are no more dynamic imports or pending
ops.
This commit fixes the inspection of functions. The current
implementation gets the name of the type of the function
from "f.__proto__.constructor.name", and it throws when
the prototype is set to null.
This commit checks the prototype before accessing its
constructor name and uses the generic name 'Function'
if the prototype is not available.
* Revert "refactor: Worker is not a Future (#7895)"
This reverts commit f4357f0ff9.
* Revert "refactor(core): JsRuntime is not a Future (#7855)"
This reverts commit d8879feb8c.
* Revert "fix(core): module execution with top level await (#7672)"
This reverts commit c7c7677825.
This commit rewrites deno::Worker to not implement Future
trait.
Instead there are two separate methods:
- Worker::poll_event_loop() - does single tick of event loop
- Worker::run_event_loop() - runs event loop to completion
Additionally some cleanup to Worker's field visibility was done.
This commit rewrites deno_core::JsRuntime to not implement Future
trait.
Instead there are two separate methods:
- JsRuntime::poll_event_loop() - does single tick of event loop
- JsRuntime::run_event_loop() - runs event loop to completion
This commit renames occurrences of "isolate" variable name
to "js_runtime". This was outstanding debt after renaming
deno_core::CoreIsolate to JsRuntime.
This changes the signature of InspectorSession.post_message to take a
&str rather than a String avoiding the need call str.to_string at each
call site.
This commit fixes implementation of top level await in "deno_core".
Previously promise returned from module execution was ignored causing to execute
modules out-of-order.
With this commit promise returned from module execution is stored on "JsRuntime"
and event loop is polled until the promise resolves.
