This commit adds blob URL support. Blob URLs are stored in a process
global storage, that can be accessed from all workers, and the module
loader. Blob URLs can be created using `URL.createObjectURL` and revoked
using `URL.revokeObjectURL`.
This commit does not add support for `fetch`ing blob URLs. This will be
added in a follow up commit.
This commit marks the `Deno.Buffer` / `Deno.readAll` /
`Deno.readAllSync` / `Deno.writeAll` / `Deno.writeAllSync` utils as
deprecated, and schedules them for removal in Deno 2.0. These
utilities are implemented in pure JS, so should not be part of the
Deno namespace.
These utilities are now available in std/io/buffer and std/io/util:
https://github.com/denoland/deno_std/pull/808.
This additionallty removes all internal dependance on Deno.Buffer.
- Improves op performance.
- Handle op-metadata (errors, promise IDs) explicitly in the op-layer vs
per op-encoding (aka: out-of-payload).
- Remove shared queue & custom "asyncHandlers", all async values are
returned in batches via js_recv_cb.
- The op-layer should be thought of as simple function calls with little
indirection or translation besides the conceptually straightforward
serde_v8 bijections.
- Preserve concepts of json/bin/min as semantic groups of their
inputs/outputs instead of their op-encoding strategy, preserving these
groups will also facilitate partial transitions over to v8 Fast API for the
"min" and "bin" groups
This commit moves implementation of bin ops to "deno_core" crates
as well as unifying logic between bin ops and json ops to reuse
as much code as possible (both in Rust and JavaScript).
Change `Console.#printFunc` to pass a log level as the second argument
(0 = debug, 3 = error), instead of a boolean for `isErr`. This does not
change the Deno runtime behaviour at all.
This commit rewrites "dispatch_minimal" into "dispatch_buffer".
It's part of an effort to unify JS interface for ops for both json
and minimal (buffer) ops.
Before this commit "minimal ops" could be either sync or async
depending on the return type from the op, but this commit changes
it to have separate signatures for sync and async ops (just like
in case of json ops).
This commit rewrites implementation of "JsRuntime::mod_evaluate".
Event loop is no longer polled automatically and users must manually
drive event loop forward after calling "mod_evaluate".
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit changes formatting of JS errors; by not showing
source lines for internal code. Where possible, instead using
the top stack frame associated with user code i.e. the first
location that is colourful and not a "deno:" URL.
This commit adds support for type definitions in "deno doc";
with this change "deno doc" is able to leverage the same directives
as TS compiler.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds a new subcommand called "coverage"
which can generate code coverage reports to stdout in
multiple formats from code coverage profiles collected to disk.
Currently this supports outputting a pretty printed diff and
the lcov format for interoperability with third-party services and tools.
Code coverage is still collected via other subcommands
that run and collect code coverage such as
"deno test --coverage=<directory>" but that command no
longer prints a pretty printed report at the end of a test
run with coverage collection enabled.
The restrictions on which files that can be reported on has
also been relaxed and are fully controllable with the include
and exclude regular expression flags on the coverage subcommand.
Co-authored-by: Luca Casonato <lucacasonato@yahoo.com>
This commit removes redundant "--reload" args because "util::deno_cmd"
recreates "DENO_DIR".
This commit also fixes ta_reload in integration tests to actually test reload.
This adds an exit sanitizer to ensure that code being tested or
dependencies of that code can't accidentally call "Deno.exit"
leading to partial test runs and false results.
This commit rewrites "JsRuntime::poll" function to fix a corner case that
might caused "overflown_response" to be overwritten by other overflown response.
The logic has been changed to allow returning multiple overflown response
alongside responses from shared queue.
This PR deprecates the "--ts"/"-T" flag of "deno eval" (which will later be removed in 2.0)
and introduces "--ext" which is used by "deno fmt" for content type selection.
This is to ensure we have a single flag that can be used across subcommands
to select the language (JS/TS).
Currently if WebSocket is closed without code, it will error
while on Chrome it would close with code 1005 instead.
Co-authored-by: crowlKats <13135287+crowlKats@users.noreply.github.com>
This commit adds support for formatting JSON and JSONC
in "deno fmt".
New values "json" and "jsonc" are added to "--ext" flag for
standard input processing.
This commit adds support for loading import maps from URLs,
both remote and local.
This feature is supported in CLI flag as well as in runtime
compiler API.
This PR makes json_op_sync/async generic to all Deserialize/Serialize types
instead of the loosely-typed serde_json::Value. Since serde_json::Value
implements Deserialize/Serialize, very little existing code needs to be updated,
however as json_op_sync/async are now generic, type inference is broken in some
cases (see cli/build.rs:146). I've found this reduces a good bit of boilerplate,
as seen in the updated deno_core examples.
This change may also reduce serialization and deserialization overhead as serde
has a better idea of what types it is working with. I am currently working on
benchmarks to confirm this and I will update this PR with my findings.
When we migrated away from all the locks, there was a regression that
was not caught immediately. The tsc::get_asset() would attempt to modify
the snapshot, but the problem was that the snapshot was a .clone() of
the inner language server's assets, which meant that modifications to
that where lost. When we then attempted to do a hover on those assets,
the inner language servers assets didn't have the retrieved asset, and
therefore would throw an error.
This removes the std folder from the tree.
Various parts of the tests are pretty tightly dependent
on std (47 direct imports and 75 indirect imports, not
counting the cli tests that use them as fixtures) so I've
added std as a submodule for now.
This commit reorganises cli/tests/integration_tests.rs.
All integration tests had been moved into integration module,
which allows to run only integration tests by "cargo test integration".
Additionally some tests were further grouped under nested modules
like "inspector", "file_watcher" or "repl".
This commits makes use of source maps and the original source
when printing lacking line coverage in the pretty printer.
Only the executable lines are checked as before (as non-executable
lines will always be ignored anyways). The lines then mapped to the
appropriate source line when a source map is present.
This commit adds support for formatting markdown files with "deno fmt".
Additionally "--ext={js|jsx|ts|tsx|md}" flag was added to "deno fmt"
that allows to specify file type when providing contents over stdio.
Previously, calling `Process#kill()` after the process had exited would
sometimes throw a `TypeError` on Windows. After this patch, it will
throw `NotFound` instead, just like other platforms.
This patch also fixes flakiness of the `runKillAfterStatus` test on
Windows.
* fix: align DOMException API to the spec
* test: fix test case 070_location
* test(DOMException): disable "does not inherit from Error: class-side"
test of WPT
* test: remove test cases in deno codebase
* docs: add note about skipped test
This commit fixes hang in web workers occuring when sending
"undefined" as message value. It is a temporary band-aid
until proper structured close is implemented.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds new binary target called "denort".
It is a "lite" version of "deno" binary that can only execute
code embedded inside the binary itself.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds new option to "Worker" Web API that allows to
configure permissions.
New "Worker.deno.permissions" option can be used to define limited
permissions to the worker thread by either:
- inherit set of parent thread permissions
- use limited subset of parent thread permissions
- revoke all permissions (full sandbox)
In order to achieve this functionality "CliModuleLoader"
was modified to accept "initial permissions", which are used
for top module loading (ie. uses parent thread permission set
to load top level module of a worker).
Merging multiple runs isn't quite right because we
rely on a 0 count to signal that a block hasn't been called.
Other tools like c8 expect this to be true as-well so we
need to do our best to merge coverage files rather
than duplicating them.
The child process kept running and printing "hello" to stdout.
This commit also removes the dependency on reqwest and instead
switches to the re-export from the fetch crate.
Brings back commit 1a2e7741c3.
This commit adds back "/json/list" endpoint to
inspector server which was erroneously removed
during server rewrite.
Co-authored-by: Ben Noordhuis <info@bnoordhuis.nl>
Fix bug in regular expression and make the regular expression more
strict.
In a string passed to new RegExp(), '[\t\s]' is identical to '[ts]' and
not `/[\t\s]/`. For that, the backslash needs to be escaped in the
string. Futhermore, `\t` is the tab character and is included in the
special regexp value `\s` so is unnecessary.
That would reduce the RegExp to new RegExp(`^${value}\\s*;?`) but
there's no point in matching 0 or more space characters followed by 0 or
one semi-colons as that will match no matter what follows `value`.
To make it more strict, require one of space, semicolon, or
end-of-string after value.
Allowlist checking already uses hosts but for some reason
requests, revokes and the runtime permissions API use URLs.
- BREAKING(lib.deno.unstable.d.ts): Change
NetPermissionDescriptor::url to NetPermissionDescriptor::host
- fix(runtime/permissions): Don't add whole URLs to the
allowlist on request
- fix(runtime/permissions): Harden strength semantics:
({ name: "net", host: "127.0.0.1" } is stronger than
{ name: "net", host: "127.0.0.1:8000" }) for blocklisting
- refactor(runtime/permissions): Use tuples for hosts, make
the host optional in Permissions::{query_net, request_net, revoke_net}()
This commit rewrites "chown_test.ts" to use the GNU "id" command
instead of python. This won't work on Windows, but these tests aren't
currently run on Windows anyway.
Implementors of `deno_core::JsRuntime` might want to do additional actions
during each turn of event loop, eg. `deno_runtime::Worker` polls inspector,
`deno_runtime::WebWorker` receives/dispatches messages from/to worker host.
Previously `JsRuntime::mod_evaluate` was implemented in such fashion that it
only polled `JsRuntime`'s event loop. This behavior turned out to be wrong
in the example of `WebWorker` which couldn't receive/dispatch messages because
its implementation of event loop was never called.
This commit rewrites "mod_evaluate" to return a handle to receiver that resolves
when module's promise resolves. It is now implementors responsibility to poll
event loop after calling `mod_evaluate`.
This commit migrates all ops to use new resource table
and "AsyncRefCell".
Old implementation of resource table was completely
removed and all code referencing it was updated to use
new system.
This commit does major refactor of "Worker" and "WebWorker",
in order to decouple them from "ProgramState" and "Flags".
The main points of interest are "create_main_worker()" and
"create_web_worker_callback()" functions which are responsible
for creating "Worker" and "WebWorker" in CLI context.
As a result it is now possible to factor out common "runtime"
functionality into a separate crate.
This commit adds "Deno.core.createPrepareStackTrace". This function
was moved from "cli/rt/40_error_stack.js" to unify handling of stack frames in core
(before this PR there was implicit dependency on logic in "core/error.rs::JsError").
Unfortunately formatting logic must still be duplicated in "cli/error.js::PrettyJsError"
to provide coloring, but currently there's no solution to this problem.
"createPrepareStackTrace" can accept a single argument; a function that takes
a location and provides source mapped location back.
This commit disables source mapping of errors
for standalone binaries. Since applying source
maps relies on using file fetcher infrastructure
it's not feasible to use it for standalone binaries
that are not supposed to use that infrastructure.
This commit makes the file watcher continue to work even if module
resolution fails at the initial attempt, allowing us to execute `run`
or `bundle` subcommand when a script has invalid syntax. In such
cases, the watcher observes a single file that is specified as an
command line argument.
Fixes panic occurring in worker when "self.close()" is called
at the top level, ie. worker shuts down while
module evaluation promise hasn't yet resolved.
This commit fixes order of events for "onload" event.
Previously handler attached using "window.onload" was
always fired before handlers added using "addEventListener".
This commit adds support for "--watch" flag for "bundle"
and "fmt" subcommands.
In addition to this, it refactors "run --watch" command so that
module resolution will occur every time the file watcher detects
file addition/deletion, which allows the watcher to observe a file
that is newly added to the dependency as well.
This commit rewrites scripts in "tools/" directory
to use Deno instead of Python. In return it allows
to remove huge number of Python packages in "third_party/".
This commit rewrites file watcher used with --watch flag.
Instead of creating new watcher after each restart, only a single
watcher is created for whole process. Additionally debouncing
mechanism has been added to prevent infinite restart loops
if multiple files were changed in quick succession.
Co-authored-by: bartossh <lenart.consulting@gmail.com>
Previously, entering a single ']' would cause repl to forever accepting
new lines, due to that `ValidationResult::Invalid` would actually be
consumed by the editor itself while continue building the lines. Instead
we should mark it as `Valid` and send the bad input for evaluation to
get the proper error from V8.
Before:
```
> ]
(you can keep entering new line here, and it will never consume input
until you Ctrl-C)
```
After:
```
> ]
Uncaught SyntaxError: Unexpected token ']'
>
```
This commit renames two assertion functions to better align with JS API:
- assertStringContains -> assertStringIncludes
- assertArrayContains -> assertArrayIncludes
This writes all evaluaton results to stdout regardless if the result is
an error or not.
This matches the behavior of other read-eval-print-loops like Node.
This commit fixes test runner by awaitning "Deno.runTests()" call,
which ensures proper error is returned when there's an unresolved
promise that's being awaited.
This commit changes implementation of top-level-await in "deno_core".
Previously promise returned from module evaluation was not awaited,
leading to out-of-order execution of modules that have TLA. It's been
fixed by changing "JsRuntime::mod_evaluate" to be an async function
that resolves when the promise returned from module evaluation also
resolves. When waiting for promise resolution event loop is polled
repeatedly, until there are no more dynamic imports or pending
ops.
This commit fixes the inspection of functions. The current
implementation gets the name of the type of the function
from "f.__proto__.constructor.name", and it throws when
the prototype is set to null.
This commit checks the prototype before accessing its
constructor name and uses the generic name 'Function'
if the prototype is not available.
* Revert "refactor: Worker is not a Future (#7895)"
This reverts commit f4357f0ff9.
* Revert "refactor(core): JsRuntime is not a Future (#7855)"
This reverts commit d8879feb8c.
* Revert "fix(core): module execution with top level await (#7672)"
This reverts commit c7c7677825.
This commit fixes implementation of top level await in "deno_core".
Previously promise returned from module execution was ignored causing to execute
modules out-of-order.
With this commit promise returned from module execution is stored on "JsRuntime"
and event loop is polled until the promise resolves.
This makes use of a default referrer when its empty in repl mode so that
dynamic imports work in the global evaluation context.
Co-authored-by: Bartek Iwanczuk <biwanczuk@gmail.com>
This ports the REPL over to Rust and makes use of an inspector session to run a REPL on top of any isolate which lets make full use of rustylines various things like validators and completors without having to introduce a bunch of hard to test internal ops and glue code.
An accidental but good side effect of this is that the multiple line input we previously had is now an editable multi-line input prompt that is correctly stored in the history as a single entry.
This commits removes "CompilerWorker" in favor of
using "JsRuntime".
"cli/ops/compiler.rs" has been removed in favor of inline
registration of ops in "cli/tsc.rs"
This commit removes cli/tsc/40_error_stack.js as it is not
needed in TSC host. All errors originating in TSC are terminal
and don't require source mapping hence we can rely on default
stack traces provided by deno_core.
Additionally tsc/06_util.js was removed and its code moved
to tsc/99_main_compiler.js
Moves op_close and op_resources to deno_core::ops and exports them.
Adds serde dependency to deno_core and reexports it.
Moves JS implementation of those ops to Deno.core and reexports them in Deno.
This commit adds basic support for collecting coverage
data using "deno test".
Currently the report is only a text added to the end
of output from "deno test".
This commit fixes panic caused by "unimplemented!()"
calls for some variants of "nix::errno::Errno".
Catch-all variant now returns "Error" class name
instead of panicking.
Co-authored-by: Bert Belder <bertbelder@gmail.com>
This commit changes "deno info" subcommand logic.
- Modules are no longer loaded into V8 isolate - analysis
is done using ModuleGraph.
- Removed deno_core::Deps structure.
- Modules are no longer type-checked and transpiled -
"compiled" file is shown only if it is already available.
- Added number of unique dependencies for root module.
- Changed tree output:
- file size is shown next to the dependency
- repeated dependencies are marked with "*"
- used less spaces in prefix to save terminal width
All benchmarks are done in Rust and can be invoked with
`cargo bench`.
Currently this has it's own "harness" that behaves like
`./tools/benchmark.py` did.
Because of this tests inside `cli/bench` are currently not run.
This should be switched to the language provided harness
once the `#[bench]` attribute has been stabilized.
This ensures that all spaces are set to be "+" in the string rather than
just the first and brings deno into line with how browsers handle spaces
in URLSearchParams, see #7001.
- Fix protocol regex.
- Truncate repeated leading slashes in file paths.
- Make drive letter support platform-independent.
- Drop the hostname if a drive letter is parsed.
- Fix drive letter normalization and basing.
- Allow basing over the host.
- Fix same-protocol basing.
- Remove Windows UNC path support.
- Reverts #6418. This is non-standard. Wouldn't be too much of a problem but it
makes other parts of the spec hard to realize.
This PR is intentionally ugly. It duplicates all of the code in cli/js2/ into
cli/tsc/ ... because it's very important that we all understand that this code
is unnecessarily duplicated in our binary. I hope this ugliness provides the
motivation to clean it up.
The typescript git submodule is removed, because it's a very large repo and
contains all sorts of stuff we don't need. Instead the necessary files are
copied directly into the deno repo. Hence +200k lines.
COMPILER_SNAPSHOT.bin size
```
master 3448139
this branch 3320972
```
Fixes #6812
This commit adds a fallback mechanism for absent compiled source file.
Because imported type declaration files are not emitted by TS compiler
and their imports are not elided users often hit "No such file or directory"
error. With this commit in such situation an empty source file will be
provided to V8 with a warning to the user suggesting using "import type"/
"export type" syntax instead.
This commit adds a "--no-check" option to following subcommands:
- "deno cache"
- "deno info"
- "deno run"
- "deno test"
The "--no-check" options allows to skip type checking step and instead
directly transpiles TS sources to JS sources.
This solution uses `ts.transpileModule()` API and is just an interim
solution before implementing it fully in Rust.
Currently, the documentation makes it sound like the test subcommand's filter
flag could accept some kind of pattern matching value like a glob or a regex,
although the function "createFilterFn" accepts a regex as an argument, there's
no way to pass an actual regex value from the CLI.
This commit makes it possible to pass a string that could be cast as regex
when string matches "^/.*/$".
With this change, a user can use the filter flag as follow:
deno test --filter "/test-.+/"
Also tested that `\` get escaped properly, on MacOS at least, and this is
also a valid flag:
deno test --filter "/test-\d+/"
This commit adds incremental compilation capabilities to internal TS compiler.
Instead of using "ts.createProgram()" API for compilation step (during deno
startup), "ts.createIncrementalProgram()" API is used instead.
Thanks to TS' ".tsbuildinfo" file that already stores all necessary metadata
for compilation I was able to remove our own invention that is ".graph" file.
".tsbuildinfo" file is stored alongside compiled source and is used to
cache-bust outdated dependencies, facilitated by the "version" field.
The value for "version" field is computed in Rust during loading of module
graph and is basically a hash of the file contents.
Please keep in mind that incremental compilation is only used for initial
compilation (or dynamic imports compilation) - bundling and runtime compiler
APIs haven't been changed at all.
Due to problems with source map I changed compilation settings to inline
source map (inlineSourceMap instead of sourceMap).
The following used to fail in Deno despite working in the browser:
```javascript
new Request('http://localhost/', {method: 'POST', body: new URLSearchParams({hello: 'world'})}).text().then(console.log)
```
* refactor "compile" and "runtimeCompile" in "compiler.ts" and factor out
separate methods for "compile" and "bundle" operations
* remove noisy debug output from "compiler.ts"
* provide "Serialize" implementations for enums in "msg.rs"
* rename "analyze_dependencies_and_references" to "pre_process_file" and
move it to "tsc.rs"
* refactor ModuleGraph to use more concrete types and properly annotate
locations where errors occur
* remove dead code from "file_fetcher.rs" - "SourceFile.types_url" is no
longer needed, as type reference parsing is done in "ModuleGraph"
* remove unneeded field "source_path" from ".meta" files stored for
compiled source file (towards #6080)
This commit provides a "system_loader_es5.js" bundle loader which will be added
to the bundle when the target is < ES2017, which is the minimum target syntax
required for "system_loader.js".
Supports #5913 (via Deno.bundle()) with a couple caveats:
* Allowing "deno bundle" to take a different target is not supported, as we
specifically ignore "target" when passed in a TypeScript config file. This is
because deno bundle is really intended to generate bundles that work in Deno.
It is an unintentional side effect that some bundles are loadable in browsers.
* While a target of "es3" will be accepted, the module loader will still only be
compatible with ES5 or later. Realistically no one should be expecting bundles
generated by Deno to be used on IE8 and prior, and there is just too much
"baggage" to support that at this point.
This is a minor variation of 75bb9d, which exposed some sort of internal V8 bug.
Ref #6358
This is 100% authored by Kitson Kelly. Github might change the author when landing
so I'm leaving this in:
Co-authored-by: Kitson Kelly <me@kitsonkelly.com>
This commit provides a "system_loader_es5.js" bundle loader which will be added
to the bundle when the target is < ES2017, which is the minimum target syntax
required for "system_loader.js".
Supports #5913 (via Deno.bundle()) with a couple caveats:
* Allowing "deno bundle" to take a different target is not supported, as we
specifically ignore "target" when passed in a TypeScript config file. This is
because deno bundle is really intended to generate bundles that work in Deno.
It is an unintentional side effect that some bundles are loadable in browsers.
* While a target of "es3" will be accepted, the module loader will still only be
compatible with ES5 or later. Realistically no one should be expecting bundles
generated by Deno to be used on IE8 and prior, and there is just too much
"baggage" to support that at this point.
The tests for testing that `Deno.truncateSync` and `Deno.truncate`
require write permissions seem to not call the functions they are
testing *at all* and are calling `Deno.mkdir` and `Deno.mkdirSync`
instead.
This commit replaces those calls with calls to `Deno.truncateSync`
and `Deno.truncate` respectively.
Currently WebAssembly runtime errors don't propagate up to the user as
they use urls to denote where the error occurred which get caught by the source-map
pipeline which doesn't support the wasm scheme.
This commit:
* added default file globs so "deno lint" can be run
without arguments (just like "deno fmt")
* added test for globs in "deno lint"
* upgrade "deno_lint" crate to v0.1.9
This commit fixes several regressions in TS compiler:
* double compilation of same module during same process run
* compilation of JavaScript entry point with non-JS imports
* unexpected skip of emit during compilation
Additional checks were added to ensure "allowJs" setting is
used in TS compiler if JavaScript has non-JS dependencies.
Currently sync operations on stdin are failing because tokio::Stdin
cannot be converted to a std::File.
This commit replaces tokio::stdin with a raw file descriptor
wrapped in a std::fs::File which can be converted to a
tokio::File and back again making the synchronous version
of op_read actually work.
This reverts commit c4c6a8dae4
There is some controversy about this change because vscode doesn't interpret the fragments correctly. Needs more discussion before landing.
This commit fixes regression that caused TS dependencies
not being compiled.
Check was added that ensures TS compiler is run if
any of dependencies in module graph is TS/TSX/JSX.
