This commit removes all JS based text encoding / text decoding. Instead
encoding now happens in Rust via encoding_rs (already in tree). This
implementation retains stream support, but adds the last missing
encodings. We are incredibly close to 100% WPT on text encoding now.
This should reduce our baseline heap by quite a bit.
This commit adds "CancelHandle" to "ConnResource" and changes
"op_http_next_request" to await for the cancel signal. In turn
when async iterating over "Deno.HttpConn" the iterator breaks
upon closing of the resource.
Replaces the file-backed provider by an in-memory one because proper
file locking is a hard problem that detracts from the proof of concept.
Teach the WPT runner how to extract tests from .html files because all
the relevant tests in test_util/wpt/webmessaging/broadcastchannel are
inside basics.html and interface.html.
In #9118, TLS streams were split into a "read half" and a "write half"
using tokio::io::split() to allow concurrent Conn#read() and
Conn#write() calls without one blocking the other. However, this
introduced a bug: outgoing data gets discarded when the TLS stream is
gracefully closed, because the read half is closed too early, before all
TLS control data has been received.
Fixes: #9692
Fixes: #10049
Fixes: #10296
Fixes: denoland/deno_std#750
This commit implements file watching for deno test.
When a file is changed, only the test modules which
use it as a dependency are rerun.
This is accomplished by reworking the file watching infrastructure
to pass the paths which have changed to the resolver, and then
constructing a module graph for each test module to check if it
contains any changed files.
Fixes a pesky bug in the fetch implementation where if the init part is
specified in `fetch` instead of the `Request` constructor, the
fillHeaders function receives two references to the same object, causing
it to append to the same list being iterated over.
This commit adds support for running test in parallel.
Entire test runner functionality has been rewritten
from JavaScript to Rust and a set of ops was added to support reporting in Rust.
A new "--jobs" flag was added to "deno test" that allows to configure
how many threads will be used. When given no value it defaults to 2.
Extensions allow declarative extensions to "JsRuntime" (ops, state, JS or middleware).
This allows for:
- `op_crates` to be plug-and-play & self-contained, reducing complexity leaked to consumers
- op middleware (like metrics_op) to be opt-in and for new middleware (unstable, tracing,...)
- `MainWorker` and `WebWorker` to be composable, allowing users to extend workers with their ops whilst benefiting from the other infrastructure (inspector, etc...)
In short extensions improve deno's modularity, reducing complexity and leaky abstractions for embedders and the internal codebase.
General cleanup of module loading code, tried to reduce indentation in various methods
on "JsRuntime" to improve readability.
Added "JsRuntime::handle_scope" helper function, which returns a "v8::HandleScope".
This was done to reduce a code pattern that happens all over the "deno_core".
Additionally if event loop hangs during loading of dynamic modules a list of
currently pending dynamic imports is printed.
`InvalidDNSNameError` is thrown when a string is not a valid hostname,
e.g. it contains invalid characters, or starts with a numeric digit. It
does not involve a (failed) DNS lookup.
denort is an optimization to "deno compile" to produce slightly smaller
output. It's a decent idea, but causes a lot of negative side-effects:
- Deno's link time is a source of constant agony both locally and in CI,
denort doubles link time.
- The release process is a long and arduous undertaking with many manual
steps. denort necessitates an additional manual zip + upload from M1
apple computers.
- The "deno compile" interface is complicated with the "--lite" option.
This is confusing for uses ("why wouldn't you want lite?").
The benefits of this feature do not outweigh the negatives. We must find
a different approach to optimizing "deno compile" output.
This commits adds adds "permissions" option to the test definitions
which allows tests to run with different permission sets than
the process's permission.
The change will only be in effect within the test function, once the
test has completed the original process permission set is restored.
Test permissions cannot exceed the process's permission.
You can only narrow or drop permissions, failure to acquire a
permission results in an error being thrown and the test case will fail.
This commit aligns the `fetch` API and the `Request` / `Response`
classes belonging to it to the spec. This commit enables all the
relevant `fetch` WPT tests. Spec compliance is now at around 90%.
Performance is essentially identical now (within 1% of 1.9.0).
This commit fixes the URL returned from `request.url` in the HTTP server
to be fully qualified. This previously existed, but was removed and
accidentially not readded during optimizations of the HTTP ops.
Returning a non fully qualified URL from `Request#url` is not spec
compliant.
The panic was caused by the lack of an error class mapping for
futures::channel::TrySendError, but it shouldn't have been throwing an error in
the first place - when a worker has terminated, postMessage should just return.
The issue was that the termination message hadn't yet been recieved, so it was
carrying on with trying to send the message. This adds another check on the Rust
side for if the channel is closed, and if it is the worker is treated as
terminated.
This commit aligns `Headers` to spec. It also removes the now unused
03_dom_iterable.js file. We now pass all relevant `Headers` WPT. We do
not implement any sort of header filtering, as we are a server side
runtime.
This is likely not the most efficient implementation of `Headers` yet.
It is however spec compliant. Once all the APIs in the `HTTP` hot loop
are correct we can start optimizing them. It is likely that this commit
reduces bench throughput temporarily.
This commit adds allowlist support to `--allow-run` flag.
Additionally `Deno.permissions.query()` allows to query for specific
programs within allowlist.
This commit adds blob URL support. Blob URLs are stored in a process
global storage, that can be accessed from all workers, and the module
loader. Blob URLs can be created using `URL.createObjectURL` and revoked
using `URL.revokeObjectURL`.
This commit does not add support for `fetch`ing blob URLs. This will be
added in a follow up commit.
This commit marks the `Deno.Buffer` / `Deno.readAll` /
`Deno.readAllSync` / `Deno.writeAll` / `Deno.writeAllSync` utils as
deprecated, and schedules them for removal in Deno 2.0. These
utilities are implemented in pure JS, so should not be part of the
Deno namespace.
These utilities are now available in std/io/buffer and std/io/util:
https://github.com/denoland/deno_std/pull/808.
This additionallty removes all internal dependance on Deno.Buffer.
- Improves op performance.
- Handle op-metadata (errors, promise IDs) explicitly in the op-layer vs
per op-encoding (aka: out-of-payload).
- Remove shared queue & custom "asyncHandlers", all async values are
returned in batches via js_recv_cb.
- The op-layer should be thought of as simple function calls with little
indirection or translation besides the conceptually straightforward
serde_v8 bijections.
- Preserve concepts of json/bin/min as semantic groups of their
inputs/outputs instead of their op-encoding strategy, preserving these
groups will also facilitate partial transitions over to v8 Fast API for the
"min" and "bin" groups
This commit moves implementation of bin ops to "deno_core" crates
as well as unifying logic between bin ops and json ops to reuse
as much code as possible (both in Rust and JavaScript).
Change `Console.#printFunc` to pass a log level as the second argument
(0 = debug, 3 = error), instead of a boolean for `isErr`. This does not
change the Deno runtime behaviour at all.
This commit rewrites "dispatch_minimal" into "dispatch_buffer".
It's part of an effort to unify JS interface for ops for both json
and minimal (buffer) ops.
Before this commit "minimal ops" could be either sync or async
depending on the return type from the op, but this commit changes
it to have separate signatures for sync and async ops (just like
in case of json ops).
This commit rewrites implementation of "JsRuntime::mod_evaluate".
Event loop is no longer polled automatically and users must manually
drive event loop forward after calling "mod_evaluate".
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit changes formatting of JS errors; by not showing
source lines for internal code. Where possible, instead using
the top stack frame associated with user code i.e. the first
location that is colourful and not a "deno:" URL.
This commit adds support for type definitions in "deno doc";
with this change "deno doc" is able to leverage the same directives
as TS compiler.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds a new subcommand called "coverage"
which can generate code coverage reports to stdout in
multiple formats from code coverage profiles collected to disk.
Currently this supports outputting a pretty printed diff and
the lcov format for interoperability with third-party services and tools.
Code coverage is still collected via other subcommands
that run and collect code coverage such as
"deno test --coverage=<directory>" but that command no
longer prints a pretty printed report at the end of a test
run with coverage collection enabled.
The restrictions on which files that can be reported on has
also been relaxed and are fully controllable with the include
and exclude regular expression flags on the coverage subcommand.
Co-authored-by: Luca Casonato <lucacasonato@yahoo.com>
This commit removes redundant "--reload" args because "util::deno_cmd"
recreates "DENO_DIR".
This commit also fixes ta_reload in integration tests to actually test reload.
This adds an exit sanitizer to ensure that code being tested or
dependencies of that code can't accidentally call "Deno.exit"
leading to partial test runs and false results.
This commit rewrites "JsRuntime::poll" function to fix a corner case that
might caused "overflown_response" to be overwritten by other overflown response.
The logic has been changed to allow returning multiple overflown response
alongside responses from shared queue.
This PR deprecates the "--ts"/"-T" flag of "deno eval" (which will later be removed in 2.0)
and introduces "--ext" which is used by "deno fmt" for content type selection.
This is to ensure we have a single flag that can be used across subcommands
to select the language (JS/TS).
Currently if WebSocket is closed without code, it will error
while on Chrome it would close with code 1005 instead.
Co-authored-by: crowlKats <13135287+crowlKats@users.noreply.github.com>
This commit adds support for formatting JSON and JSONC
in "deno fmt".
New values "json" and "jsonc" are added to "--ext" flag for
standard input processing.
This commit adds support for loading import maps from URLs,
both remote and local.
This feature is supported in CLI flag as well as in runtime
compiler API.
This PR makes json_op_sync/async generic to all Deserialize/Serialize types
instead of the loosely-typed serde_json::Value. Since serde_json::Value
implements Deserialize/Serialize, very little existing code needs to be updated,
however as json_op_sync/async are now generic, type inference is broken in some
cases (see cli/build.rs:146). I've found this reduces a good bit of boilerplate,
as seen in the updated deno_core examples.
This change may also reduce serialization and deserialization overhead as serde
has a better idea of what types it is working with. I am currently working on
benchmarks to confirm this and I will update this PR with my findings.
When we migrated away from all the locks, there was a regression that
was not caught immediately. The tsc::get_asset() would attempt to modify
the snapshot, but the problem was that the snapshot was a .clone() of
the inner language server's assets, which meant that modifications to
that where lost. When we then attempted to do a hover on those assets,
the inner language servers assets didn't have the retrieved asset, and
therefore would throw an error.
This removes the std folder from the tree.
Various parts of the tests are pretty tightly dependent
on std (47 direct imports and 75 indirect imports, not
counting the cli tests that use them as fixtures) so I've
added std as a submodule for now.
This commit reorganises cli/tests/integration_tests.rs.
All integration tests had been moved into integration module,
which allows to run only integration tests by "cargo test integration".
Additionally some tests were further grouped under nested modules
like "inspector", "file_watcher" or "repl".
This commits makes use of source maps and the original source
when printing lacking line coverage in the pretty printer.
Only the executable lines are checked as before (as non-executable
lines will always be ignored anyways). The lines then mapped to the
appropriate source line when a source map is present.
This commit adds support for formatting markdown files with "deno fmt".
Additionally "--ext={js|jsx|ts|tsx|md}" flag was added to "deno fmt"
that allows to specify file type when providing contents over stdio.
Previously, calling `Process#kill()` after the process had exited would
sometimes throw a `TypeError` on Windows. After this patch, it will
throw `NotFound` instead, just like other platforms.
This patch also fixes flakiness of the `runKillAfterStatus` test on
Windows.
* fix: align DOMException API to the spec
* test: fix test case 070_location
* test(DOMException): disable "does not inherit from Error: class-side"
test of WPT
* test: remove test cases in deno codebase
* docs: add note about skipped test
This commit fixes hang in web workers occuring when sending
"undefined" as message value. It is a temporary band-aid
until proper structured close is implemented.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds new binary target called "denort".
It is a "lite" version of "deno" binary that can only execute
code embedded inside the binary itself.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds new option to "Worker" Web API that allows to
configure permissions.
New "Worker.deno.permissions" option can be used to define limited
permissions to the worker thread by either:
- inherit set of parent thread permissions
- use limited subset of parent thread permissions
- revoke all permissions (full sandbox)
In order to achieve this functionality "CliModuleLoader"
was modified to accept "initial permissions", which are used
for top module loading (ie. uses parent thread permission set
to load top level module of a worker).
Merging multiple runs isn't quite right because we
rely on a 0 count to signal that a block hasn't been called.
Other tools like c8 expect this to be true as-well so we
need to do our best to merge coverage files rather
than duplicating them.
The child process kept running and printing "hello" to stdout.
This commit also removes the dependency on reqwest and instead
switches to the re-export from the fetch crate.
Brings back commit 1a2e7741c3.
This commit adds back "/json/list" endpoint to
inspector server which was erroneously removed
during server rewrite.
Co-authored-by: Ben Noordhuis <info@bnoordhuis.nl>
Fix bug in regular expression and make the regular expression more
strict.
In a string passed to new RegExp(), '[\t\s]' is identical to '[ts]' and
not `/[\t\s]/`. For that, the backslash needs to be escaped in the
string. Futhermore, `\t` is the tab character and is included in the
special regexp value `\s` so is unnecessary.
That would reduce the RegExp to new RegExp(`^${value}\\s*;?`) but
there's no point in matching 0 or more space characters followed by 0 or
one semi-colons as that will match no matter what follows `value`.
To make it more strict, require one of space, semicolon, or
end-of-string after value.
Allowlist checking already uses hosts but for some reason
requests, revokes and the runtime permissions API use URLs.
- BREAKING(lib.deno.unstable.d.ts): Change
NetPermissionDescriptor::url to NetPermissionDescriptor::host
- fix(runtime/permissions): Don't add whole URLs to the
allowlist on request
- fix(runtime/permissions): Harden strength semantics:
({ name: "net", host: "127.0.0.1" } is stronger than
{ name: "net", host: "127.0.0.1:8000" }) for blocklisting
- refactor(runtime/permissions): Use tuples for hosts, make
the host optional in Permissions::{query_net, request_net, revoke_net}()
This commit rewrites "chown_test.ts" to use the GNU "id" command
instead of python. This won't work on Windows, but these tests aren't
currently run on Windows anyway.
Implementors of `deno_core::JsRuntime` might want to do additional actions
during each turn of event loop, eg. `deno_runtime::Worker` polls inspector,
`deno_runtime::WebWorker` receives/dispatches messages from/to worker host.
Previously `JsRuntime::mod_evaluate` was implemented in such fashion that it
only polled `JsRuntime`'s event loop. This behavior turned out to be wrong
in the example of `WebWorker` which couldn't receive/dispatch messages because
its implementation of event loop was never called.
This commit rewrites "mod_evaluate" to return a handle to receiver that resolves
when module's promise resolves. It is now implementors responsibility to poll
event loop after calling `mod_evaluate`.
This commit migrates all ops to use new resource table
and "AsyncRefCell".
Old implementation of resource table was completely
removed and all code referencing it was updated to use
new system.
This commit does major refactor of "Worker" and "WebWorker",
in order to decouple them from "ProgramState" and "Flags".
The main points of interest are "create_main_worker()" and
"create_web_worker_callback()" functions which are responsible
for creating "Worker" and "WebWorker" in CLI context.
As a result it is now possible to factor out common "runtime"
functionality into a separate crate.
This commit adds "Deno.core.createPrepareStackTrace". This function
was moved from "cli/rt/40_error_stack.js" to unify handling of stack frames in core
(before this PR there was implicit dependency on logic in "core/error.rs::JsError").
Unfortunately formatting logic must still be duplicated in "cli/error.js::PrettyJsError"
to provide coloring, but currently there's no solution to this problem.
"createPrepareStackTrace" can accept a single argument; a function that takes
a location and provides source mapped location back.
This commit disables source mapping of errors
for standalone binaries. Since applying source
maps relies on using file fetcher infrastructure
it's not feasible to use it for standalone binaries
that are not supposed to use that infrastructure.
This commit makes the file watcher continue to work even if module
resolution fails at the initial attempt, allowing us to execute `run`
or `bundle` subcommand when a script has invalid syntax. In such
cases, the watcher observes a single file that is specified as an
command line argument.
Fixes panic occurring in worker when "self.close()" is called
at the top level, ie. worker shuts down while
module evaluation promise hasn't yet resolved.
This commit fixes order of events for "onload" event.
Previously handler attached using "window.onload" was
always fired before handlers added using "addEventListener".
This commit adds support for "--watch" flag for "bundle"
and "fmt" subcommands.
In addition to this, it refactors "run --watch" command so that
module resolution will occur every time the file watcher detects
file addition/deletion, which allows the watcher to observe a file
that is newly added to the dependency as well.
This commit rewrites scripts in "tools/" directory
to use Deno instead of Python. In return it allows
to remove huge number of Python packages in "third_party/".
This commit rewrites file watcher used with --watch flag.
Instead of creating new watcher after each restart, only a single
watcher is created for whole process. Additionally debouncing
mechanism has been added to prevent infinite restart loops
if multiple files were changed in quick succession.
Co-authored-by: bartossh <lenart.consulting@gmail.com>
Previously, entering a single ']' would cause repl to forever accepting
new lines, due to that `ValidationResult::Invalid` would actually be
consumed by the editor itself while continue building the lines. Instead
we should mark it as `Valid` and send the bad input for evaluation to
get the proper error from V8.
Before:
```
> ]
(you can keep entering new line here, and it will never consume input
until you Ctrl-C)
```
After:
```
> ]
Uncaught SyntaxError: Unexpected token ']'
>
```
This commit renames two assertion functions to better align with JS API:
- assertStringContains -> assertStringIncludes
- assertArrayContains -> assertArrayIncludes
This writes all evaluaton results to stdout regardless if the result is
an error or not.
This matches the behavior of other read-eval-print-loops like Node.
This commit fixes test runner by awaitning "Deno.runTests()" call,
which ensures proper error is returned when there's an unresolved
promise that's being awaited.
This commit changes implementation of top-level-await in "deno_core".
Previously promise returned from module evaluation was not awaited,
leading to out-of-order execution of modules that have TLA. It's been
fixed by changing "JsRuntime::mod_evaluate" to be an async function
that resolves when the promise returned from module evaluation also
resolves. When waiting for promise resolution event loop is polled
repeatedly, until there are no more dynamic imports or pending
ops.
This commit fixes the inspection of functions. The current
implementation gets the name of the type of the function
from "f.__proto__.constructor.name", and it throws when
the prototype is set to null.
This commit checks the prototype before accessing its
constructor name and uses the generic name 'Function'
if the prototype is not available.
* Revert "refactor: Worker is not a Future (#7895)"
This reverts commit f4357f0ff9.
* Revert "refactor(core): JsRuntime is not a Future (#7855)"
This reverts commit d8879feb8c.
* Revert "fix(core): module execution with top level await (#7672)"
This reverts commit c7c7677825.
This commit fixes implementation of top level await in "deno_core".
Previously promise returned from module execution was ignored causing to execute
modules out-of-order.
With this commit promise returned from module execution is stored on "JsRuntime"
and event loop is polled until the promise resolves.
This makes use of a default referrer when its empty in repl mode so that
dynamic imports work in the global evaluation context.
Co-authored-by: Bartek Iwanczuk <biwanczuk@gmail.com>
This ports the REPL over to Rust and makes use of an inspector session to run a REPL on top of any isolate which lets make full use of rustylines various things like validators and completors without having to introduce a bunch of hard to test internal ops and glue code.
An accidental but good side effect of this is that the multiple line input we previously had is now an editable multi-line input prompt that is correctly stored in the history as a single entry.
This commits removes "CompilerWorker" in favor of
using "JsRuntime".
"cli/ops/compiler.rs" has been removed in favor of inline
registration of ops in "cli/tsc.rs"
This commit removes cli/tsc/40_error_stack.js as it is not
needed in TSC host. All errors originating in TSC are terminal
and don't require source mapping hence we can rely on default
stack traces provided by deno_core.
Additionally tsc/06_util.js was removed and its code moved
to tsc/99_main_compiler.js
Moves op_close and op_resources to deno_core::ops and exports them.
Adds serde dependency to deno_core and reexports it.
Moves JS implementation of those ops to Deno.core and reexports them in Deno.
This commit adds basic support for collecting coverage
data using "deno test".
Currently the report is only a text added to the end
of output from "deno test".
This commit fixes panic caused by "unimplemented!()"
calls for some variants of "nix::errno::Errno".
Catch-all variant now returns "Error" class name
instead of panicking.
Co-authored-by: Bert Belder <bertbelder@gmail.com>
This commit changes "deno info" subcommand logic.
- Modules are no longer loaded into V8 isolate - analysis
is done using ModuleGraph.
- Removed deno_core::Deps structure.
- Modules are no longer type-checked and transpiled -
"compiled" file is shown only if it is already available.
- Added number of unique dependencies for root module.
- Changed tree output:
- file size is shown next to the dependency
- repeated dependencies are marked with "*"
- used less spaces in prefix to save terminal width
All benchmarks are done in Rust and can be invoked with
`cargo bench`.
Currently this has it's own "harness" that behaves like
`./tools/benchmark.py` did.
Because of this tests inside `cli/bench` are currently not run.
This should be switched to the language provided harness
once the `#[bench]` attribute has been stabilized.
This ensures that all spaces are set to be "+" in the string rather than
just the first and brings deno into line with how browsers handle spaces
in URLSearchParams, see #7001.
- Fix protocol regex.
- Truncate repeated leading slashes in file paths.
- Make drive letter support platform-independent.
- Drop the hostname if a drive letter is parsed.
- Fix drive letter normalization and basing.
- Allow basing over the host.
- Fix same-protocol basing.
- Remove Windows UNC path support.
- Reverts #6418. This is non-standard. Wouldn't be too much of a problem but it
makes other parts of the spec hard to realize.
This PR is intentionally ugly. It duplicates all of the code in cli/js2/ into
cli/tsc/ ... because it's very important that we all understand that this code
is unnecessarily duplicated in our binary. I hope this ugliness provides the
motivation to clean it up.
The typescript git submodule is removed, because it's a very large repo and
contains all sorts of stuff we don't need. Instead the necessary files are
copied directly into the deno repo. Hence +200k lines.
COMPILER_SNAPSHOT.bin size
```
master 3448139
this branch 3320972
```
Fixes #6812
This commit adds a fallback mechanism for absent compiled source file.
Because imported type declaration files are not emitted by TS compiler
and their imports are not elided users often hit "No such file or directory"
error. With this commit in such situation an empty source file will be
provided to V8 with a warning to the user suggesting using "import type"/
"export type" syntax instead.
This commit adds a "--no-check" option to following subcommands:
- "deno cache"
- "deno info"
- "deno run"
- "deno test"
The "--no-check" options allows to skip type checking step and instead
directly transpiles TS sources to JS sources.
This solution uses `ts.transpileModule()` API and is just an interim
solution before implementing it fully in Rust.
Currently, the documentation makes it sound like the test subcommand's filter
flag could accept some kind of pattern matching value like a glob or a regex,
although the function "createFilterFn" accepts a regex as an argument, there's
no way to pass an actual regex value from the CLI.
This commit makes it possible to pass a string that could be cast as regex
when string matches "^/.*/$".
With this change, a user can use the filter flag as follow:
deno test --filter "/test-.+/"
Also tested that `\` get escaped properly, on MacOS at least, and this is
also a valid flag:
deno test --filter "/test-\d+/"
This commit adds incremental compilation capabilities to internal TS compiler.
Instead of using "ts.createProgram()" API for compilation step (during deno
startup), "ts.createIncrementalProgram()" API is used instead.
Thanks to TS' ".tsbuildinfo" file that already stores all necessary metadata
for compilation I was able to remove our own invention that is ".graph" file.
".tsbuildinfo" file is stored alongside compiled source and is used to
cache-bust outdated dependencies, facilitated by the "version" field.
The value for "version" field is computed in Rust during loading of module
graph and is basically a hash of the file contents.
Please keep in mind that incremental compilation is only used for initial
compilation (or dynamic imports compilation) - bundling and runtime compiler
APIs haven't been changed at all.
Due to problems with source map I changed compilation settings to inline
source map (inlineSourceMap instead of sourceMap).
The following used to fail in Deno despite working in the browser:
```javascript
new Request('http://localhost/', {method: 'POST', body: new URLSearchParams({hello: 'world'})}).text().then(console.log)
```
* refactor "compile" and "runtimeCompile" in "compiler.ts" and factor out
separate methods for "compile" and "bundle" operations
* remove noisy debug output from "compiler.ts"
* provide "Serialize" implementations for enums in "msg.rs"
* rename "analyze_dependencies_and_references" to "pre_process_file" and
move it to "tsc.rs"
* refactor ModuleGraph to use more concrete types and properly annotate
locations where errors occur
* remove dead code from "file_fetcher.rs" - "SourceFile.types_url" is no
longer needed, as type reference parsing is done in "ModuleGraph"
* remove unneeded field "source_path" from ".meta" files stored for
compiled source file (towards #6080)
This commit provides a "system_loader_es5.js" bundle loader which will be added
to the bundle when the target is < ES2017, which is the minimum target syntax
required for "system_loader.js".
Supports #5913 (via Deno.bundle()) with a couple caveats:
* Allowing "deno bundle" to take a different target is not supported, as we
specifically ignore "target" when passed in a TypeScript config file. This is
because deno bundle is really intended to generate bundles that work in Deno.
It is an unintentional side effect that some bundles are loadable in browsers.
* While a target of "es3" will be accepted, the module loader will still only be
compatible with ES5 or later. Realistically no one should be expecting bundles
generated by Deno to be used on IE8 and prior, and there is just too much
"baggage" to support that at this point.
This is a minor variation of 75bb9d, which exposed some sort of internal V8 bug.
Ref #6358
This is 100% authored by Kitson Kelly. Github might change the author when landing
so I'm leaving this in:
Co-authored-by: Kitson Kelly <me@kitsonkelly.com>
This commit provides a "system_loader_es5.js" bundle loader which will be added
to the bundle when the target is < ES2017, which is the minimum target syntax
required for "system_loader.js".
Supports #5913 (via Deno.bundle()) with a couple caveats:
* Allowing "deno bundle" to take a different target is not supported, as we
specifically ignore "target" when passed in a TypeScript config file. This is
because deno bundle is really intended to generate bundles that work in Deno.
It is an unintentional side effect that some bundles are loadable in browsers.
* While a target of "es3" will be accepted, the module loader will still only be
compatible with ES5 or later. Realistically no one should be expecting bundles
generated by Deno to be used on IE8 and prior, and there is just too much
"baggage" to support that at this point.
The tests for testing that `Deno.truncateSync` and `Deno.truncate`
require write permissions seem to not call the functions they are
testing *at all* and are calling `Deno.mkdir` and `Deno.mkdirSync`
instead.
This commit replaces those calls with calls to `Deno.truncateSync`
and `Deno.truncate` respectively.
Currently WebAssembly runtime errors don't propagate up to the user as
they use urls to denote where the error occurred which get caught by the source-map
pipeline which doesn't support the wasm scheme.
This commit:
* added default file globs so "deno lint" can be run
without arguments (just like "deno fmt")
* added test for globs in "deno lint"
* upgrade "deno_lint" crate to v0.1.9
This commit fixes several regressions in TS compiler:
* double compilation of same module during same process run
* compilation of JavaScript entry point with non-JS imports
* unexpected skip of emit during compilation
Additional checks were added to ensure "allowJs" setting is
used in TS compiler if JavaScript has non-JS dependencies.
Currently sync operations on stdin are failing because tokio::Stdin
cannot be converted to a std::File.
This commit replaces tokio::stdin with a raw file descriptor
wrapped in a std::fs::File which can be converted to a
tokio::File and back again making the synchronous version
of op_read actually work.
This reverts commit c4c6a8dae4
There is some controversy about this change because vscode doesn't interpret the fragments correctly. Needs more discussion before landing.
This commit fixes regression that caused TS dependencies
not being compiled.
Check was added that ensures TS compiler is run if
any of dependencies in module graph is TS/TSX/JSX.
This PR addresses many problems with module graph loading
introduced in #5029, as well as many long standing issues.
"ModuleGraphLoader" has been wired to "ModuleLoader" implemented
on "State" - that means that dependency analysis and fetching is done
before spinning up TS compiler worker.
Basic dependency tracking for TS compilation has been implemented.
Errors caused by import statements are now annotated with import
location.
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
Since everything that Deno loads is treated as an ES Module,
it means that all code is treated as "use strict" except for
when using the REPL. This PR changes that so code in the
REPL is also always evaluated with "use strict". There are
also a couple other places where we load code as scripts
which should also use "use strict" just in case.
This commit changes how error occurring in SWC are handled.
Changed lexer settings to properly handle TS decorators.
Changed output of SWC error to annotate with position in file.
This commit fixes a bug introduced in #5029 that caused bad
handling of redirects during module analysis.
Also ensured that duplicate modules are not downloaded.
This commit completely overhauls how module analysis is
performed in TS compiler by moving the logic to Rust.
In the current setup module analysis is performed using
"ts.preProcessFile" API in a special TS compiler worker
running on a separate thread.
"ts.preProcessFile" allowed us to build a lot of functionality
in CLI including X-TypeScript-Types header support
and @deno-types directive support. Unfortunately at the
same time complexity of the ops required to perform
supporting tasks exploded and caused some hidden
permission escapes.
This PR introduces "ModuleGraphLoader" which can parse
source and load recursively all dependent source files; as
well as declaration files. All dependencies used in TS
compiler and now fetched and collected upfront in Rust
before spinning up TS compiler.
To achieve feature parity with existing APIs this commit
includes a lot of changes:
* add "ModuleGraphLoader"
- can fetch local and remote sources
- parses source code using SWC and extracts imports, exports, file references, special
headers
- this struct inherited all of the hidden complexity and cruft from TS version and requires
several follow up PRs
* rewrite cli/tsc.rs to perform module analysis upfront and send all required source code to
TS worker in one message
* remove op_resolve_modules and op_fetch_source_files from cli/ops/compiler.rs
* run TS worker on the same thread
This commit removes "check_stderr" setting from itest! macro used
to generate integration tests. Without this setting on tests discarded
output of stderr making it very hard to debug the problem in test.
Numerous tests were changed by adding "--quiet" flag to not display
"Compile"/"Download" prompts.
This commit fixes problems with source maps in Chrome Devtools
by substituting source map URL generated by TS compiler with
actual file URL pointing to DENO_DIR.
Dummy value of "source_map_url" has been removed from
"ScriptOrigin".
Also fixes lock file which used compiled source code to generate
lock hash; it now uses source code of the file that is
being compiled.
This PR hot-fixes permission escapes in dynamic imports, workers
and runtime compiler APIs.
"permissions" parameter was added to public APIs of SourceFileFetcher
and appropriate permission checks are performed during loading of
local and remote files.
Importing .wasm files is non-standardized therefore deciding to
support current functionality past 1.0 release is risky.
Besides that .wasm import posed many challenges in our codebase
due to complex interactions with TS compiler which spawned
thread for each encountered .wasm import.
This commit removes:
- cli/compilers/wasm.rs
- cli/compilers/wasm_wrap.js
- two integration tests related to .wasm imports
This commit updates "deno_typescript" crate to properly map
bundle entrypoint file to internal specifier.
All import specifiers were remapped from "file:///a/b/c.ts" to
"$deno$/a/b/c.ts", but that was not the case for entrypoint file
"main.ts" and "compiler.ts".
Because of that internal stack traces were inconsistent; showing
"file:///some/random/path/on/ci/machine.ts" URL in frames that
originate from "main.ts" or "compiler.ts" and "$deno$/file.ts"
for all other imports.
This PR removes op_cache and refactors how Deno interacts with TS compiler.
Ultimate goal is to completely sandbox TS compiler worker; it should operate on
simple request -> response basis. With this commit TS compiler no longer
caches compiled sources as they are generated but rather collects all sources
and sends them back to Rust when compilation is done.
Additionally "Diagnostic" and its children got refactored to use "Deserialize" trait
instead of manually implementing JSON deserialization.
The issue is solved by proxying websocket messages over a pair of
`futures::mpsc::unbounded` channels. As these are are implemented in
the 'futures' crate, they can't participate in Tokio's cooperative
task yielding.
This PR removes the hack in CLI that allows to run scripts with shorthand: deno script.ts.
Removing this functionality because it hacks around short-comings of clap our CLI parser. We agree that this shorthand syntax is desirable, but it needs to be rethinked and reimplemented. For 1.0 we should go with conservative approach that is correct.
Fixes #4000 and fixes #4476. Now always tries to fetch reference types
for JS files. Does not throw if it fails, since Typescript compiler will
complain if the file is not there(it will try to fetch it again first)
and people who just use JS should not be bothered by this error.
Not sure about my test, it passes and catches the bug but maybe there is
a better way to express it.
1. Array elements are now grouped the same as in Node.js
2. Limit to 100 (Node.js default) elements to display in iterable
3. Print each element in new line if excessing max line length (same as in Node.js)
4. Print length of the TypedArray
5. Print information about empty items in Array
This is a first pass implementation which is still missing several important
features:
- support for --inspect-brk (#4503)
- support for source maps (#4501)
- support for piping console.log to devtools console (#4502)
Co-authored-by: Bert Belder <bertbelder@gmail.com>
Co-authored-by: Matt Harrison <mt.harrison86@gmail.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
* Remove DENO_BUILD_MODE and DENO_BUILD_PATH
Also remove outdated docs related to ninja/gn.
* fix
* remove parameter to build_mode()
* remove arg parsing from benchmark.py
* add tests for "Deno.core.encode" and "Deno.core.decode" for empty inputs
* use "Deno.core.encode" in "TextEncoder"
* use "Deno.core.decode" in "TextDecoder"
* remove "core_decode" and "core_encode" benchmarks
After splitting "failFast" and "exitOnFail" arguments, there was a situation where failing tests did not exit with code 1.
* fixed argument value passed to Deno.runTests() in deno test
* fixed argument value passed to Deno.runTests() in std/testing/runner.ts
* added integration tests for deno test to ensure failFast and exitOnFail work as expected
* don't write test file to file system, but keep it in memory
This PR brings assertOps and assertResources sanitizers to Deno.test() API.
assertOps checks that test doesn't leak async ops, ie. there are no unresolved
promises originating from Deno APIs. Enabled by default, can be disabled using
Deno.TestDefinition.disableOpSanitizer.
assertResources checks that test doesn't leak resources, ie. all resources used
in test are closed. For example; if a file is opened during a test case it must be
explicitly closed before test case finishes. It's most useful for asynchronous
generators. Enabled by default, can be disabled using
Deno.TestDefinition.disableResourceSanitizer.
We've used those sanitizers in internal runtime tests and it proved very useful in
surfacing incorrect tests which resulted in interference between the tests.
All tests have been sanitized.
Closes #4208
Fixes #4101
Previously, we would just provide the raw JSON to the TypeScript
compiler worker, but TypeScript does not transform JSON. This caused
a problem when emitting a bundle, that the JSON would just be "inlined"
into the output, instead of being transformed into a module.
This fixes this problem by providing the compiled JSON to the TypeScript
compiler, so TypeScript just sees JSON as a "normal" TypeScript module.
This commit changes output of default test reporter to resemble output from Rust test runner;
first the name of running test is printed with "...", then after test has run result is printed on the same line.
* Split "Deno.TestEvent.Result" into "TestStart" and "TestEnd";
* changes TestReporter interface to support both events;
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
This commits add two new methods to "Deno.core" namespace: "encode" and "decode".
Those methods are bound in Rust to provide a) fast b) generally available of encoding and decoding UTF-8 strings.
Both methods are now used in "cli/js/dispatch_json.ts".
* drop server guard before unit test result check
To prevent cascading test failures when js_unit_test http server
guard is dropped before asserting that tests were successful.
This is really a band-aid and doesn't solve underlying issue with
http server.
* Update CLI for unit_test_runner.ts
* Change cli/js/tests/unit_test_runner.ts command line interface to work in 3
modes:
- "one-off" - run tests that match permissions of currently running
process
- "master" - run tests for all possible permission combinations, by
spawning subprocesses running in "worker" mode and communicating via
TCP socket; requires elevated permissions
- "worker" - run tests for set of permissions provided by CLI arg;
requires elevated permissions to setup TCP connection to "master";
after initial setup process drops permissions to given set
* Support filtering of tests by string passed after "--" CLI arg
* Update cli/js/tests/README.md
Rewrites "cli/js/unit_test_runner.ts" to communicate with spawned subprocesses
using TCP socket.
* Rewrite "Deno.runTests()" by factoring out testing logic to private "TestApi"
class. "TestApi" implements "AsyncIterator" that yields "TestEvent"s,
which is an interface for different types of event occuring during running
tests.
* Add "reporter" argument to "Deno.runTests()" to allow users to provide custom
reporting mechanism for tests. It's represented by "TestReporter" interface,
that implements hook functions for each type of "TestEvent". If "reporter"
is not provided then default console reporting is used (via
"ConsoleReporter").
* Change how "unit_test_runner" communicates with spawned suprocesses. Instead
of parsing text data from child's stdout, a TCP socket is created and used
for communication. "unit_test_runner" can run in either "master" or "worker"
mode. Former is responsible for test discovery and establishing needed
permission combinations; while latter (that is spawned by "master") executes
tests that match given permission set.
* Use "SocketReporter" that implements "TestReporter" interface to send output
of tests to "master" process. Data is sent as stringified JSON and then
parsed by "master" as structured data. "master" applies it's own reporting
logic to output tests to console (by reusing default "ConsoleReporter").
- moves compiler implementation to "cli/js/compiler/" directory
- moves more APIs to "cli/js/web":
* "console.ts"
* "console_table.ts"
* "performance.ts"
* "timers.ts"
* "workers.ts"
- removes some dead code from "cli/js/"
All Deno runtime test files were moved to cli/js/tests/ directory.
It makes a clear distinction that cli/js/tests/ contains code
that is run under Deno runtime as opposed to code in cli/js/ which
is used to create bundle and snapshot with "deno_typescript".
Following JS ops were moved to separate files in cli/js/ops directory:
- compiler
- dispatch_json
- dispatch_minimal
- errors
- fetch
- fs_events
- os
- random
- repl
- resources
- runtime_compiler
- runtime
- tty
* add "assertOps" test assertion which makes sure test case
is not "leaking" ops - ie. after test finishes there are no
pending async ops
* apply "assertOps" to all tests in "cli/js/"
* fix numerous tests leaking ops
* document problem with edge case in "clearInterval"
and "clearTimeout" implementation where they
may leak async ops
* move "cli/js/worker_test.ts" to "cli/tests/worker_test.ts" and
run as integration test; workers leak ops because of missing
"terminate" implementation
Fixes #4602
We turned off `allowJs` by default, to keep the compiler from grabbing
a bunch of files that it wouldn't actually do anything useful with. On
the other hand, this caused problems with bundles, where the compiler
needs to gather all the dependencies, including JavaScript ones. This
fixes this so that when we are bundling, we analyse JavaScript imports
in the compiler.
Previously, bundles always utilised top level await, even if the bundled
modules didn't require top level await. Now, analysis of the bundle is
done and if none of the bundled modules are asynchronously executed,
then the bundle as a whole will be synchronously executed.
Fixes #4055
Fixes #4123
This PR fixes an issue where we recursively analysed imports on plain JS files
in the compiler irrespective of "checkJs" being true. This caused problems
where when analysing the imports of those files, we would mistake some
import like structures (AMD/CommonJS) as dependencies and try to resolve
the "modules" even though the compiler would not actually look at those files.
Fixes #4031
When a bundle contains a single module, we were incorrectly determining
the module name, resulting in a non-functional bundle. This PR corrects
that determination.
Moves to using a minimal System loader for bundles generated by Deno.
TypeScript in 3.8 will be able to output TLA for modules, and the loader
is written to take advantage of that as soon as we update Deno to TS
3.8.
System also allows us to support `import.meta` and provide more ESM
aligned assignment of exports, as well as there is better handling of
circular imports.
The loader is also very terse versus to try to save overhead.
Also, fixed an issue where abstract classes were not being re-exported.
Fixes #2553
Fixes #3559
Fixes #3751
Fixes #3825
Refs #3301
* Use PathBuf for DenoSubcommand::Bundle's out_file
* Use PathBuf for DenoSubcommand::Format's files
* Use PathBuf for DenoSubcommand::Install's dir
* Use PathBuf for read/write whitelists
* establish basic event loop for workers
* make "self.close()" inside worker
* remove "runWorkerMessageLoop() - instead manually call global function
in Rust when message arrives. This is done in preparation for structured clone
* refactor "WorkerChannel" and use distinct structs for internal
and external channels; "WorkerChannelsInternal" and "WorkerHandle"
* move "State.worker_channels_internal" to "Worker.internal_channels"
* add "WorkerEvent" enum for child->host communication;
currently "Message(Buf)" and "Error(ErrBox)" variants are supported
* add tests for nested workers
* add tests for worker throwing error on startup
This change simplifies how we execute V8. Previously V8 Isolates jumped
around threads every time they were woken up. This was overly complex and
potentially hurting performance in a myriad ways. Now isolates run on
their own dedicated thread and never move.
- blocking_json spawns a thread and does not use a thread pool
- op_host_poll_worker and op_host_resume_worker are non-operational
- removes Worker::get_message and Worker::post_message
- ThreadSafeState::workers table contains WorkerChannel entries instead
of actual Worker instances.
- MainWorker and CompilerWorker are no longer Futures.
- The multi-threaded version of deno_core_http_bench was removed.
- AyncOps no longer need to be Send + Sync
This PR is very large and several tests were disabled to speed
integration:
- installer_test_local_module_run
- installer_test_remote_module_run
- _015_duplicate_parallel_import
- _026_workers
This flag was added to evaluate performance relative to tokio's threaded
runtime. Although it's faster in the HTTP benchmark, it's clear the runtime
is not the only perf problem.
Removing this flag will simplify further refactors, in particular
adopting the #[tokio::main] macro. This will be done in a follow up.
Ultimately we expect to move to the current thread runtime with Isolates
pinned to specific threads, but that will be a much larger refactor. The
--current-thread just complicates that effort.
Before:
```
▶ target/debug/deno https://deno.land/std/examples/echo_server.ts
error: Uncaught PermissionDenied: run again with the --allow-net flag
► $deno$/dispatch_json.ts:40:11
at DenoError ($deno$/errors.ts:20:5)
at unwrapResponse ($deno$/dispatch_json.ts:40:11)
at sendSync ($deno$/dispatch_json.ts:67:10)
at listen ($deno$/net.ts:170:15)
at https://deno.land/std/examples/echo_server.ts:4:23
```
```
▶ target/debug/deno --allow-read=/usr https://deno.land/std/examples/cat.ts /etc/passwd
error: Uncaught PermissionDenied: run again with the --allow-read flag
► $deno$/dispatch_json.ts:40:11
at DenoError ($deno$/errors.ts:20:5)
at unwrapResponse ($deno$/dispatch_json.ts:40:11)
at sendAsync ($deno$/dispatch_json.ts:91:10)
```
After:
```
▶ target/debug/deno https://deno.land/std/examples/echo_server.ts
error: Uncaught PermissionDenied: network access to "0.0.0.0:8080", run again with the --allow-net flag
► $deno$/dispatch_json.ts:40:11
at DenoError ($deno$/errors.ts:20:5)
at unwrapResponse ($deno$/dispatch_json.ts:40:11)
at sendSync ($deno$/dispatch_json.ts:67:10)
at listen ($deno$/net.ts:170:15)
at https://deno.land/std/examples/echo_server.ts:4:23
```
```
▶ target/debug/deno --allow-read=/usr https://deno.land/std/examples/cat.ts /etc/passwd
error: Uncaught PermissionDenied: read access to "/etc/passwd", run again with the --allow-read flag
► $deno$/dispatch_json.ts:40:11
at DenoError ($deno$/errors.ts:20:5)
at unwrapResponse ($deno$/dispatch_json.ts:40:11)
at sendAsync ($deno$/dispatch_json.ts:91:10)
```
* move is_dyn_import argument from Loader::resolve to Loader::load - it was always kind of strange that resolve() checks permissions.
* change argument type from &str to &ModuleSpecifier where applicable
* split ops/worker.rs into ops/worker_host.rs and ops/web_worker.rs
* refactor js/workers.ts and factor out js/worker_main.ts - entry point for WebWorker runtime
* BREAKING CHANGE: remove support for blob: URL in Worker
* BREAKING CHANGE: remove Deno namespace support and noDenoNamespace option in Worker constructor
* introduce WebWorker struct which is a stripped down version of cli::Worker
- Remove ability to specify run arguments like `--allow-net` after the
script argument. It's too hacky to make work with clap.
- Remove `--v8-options`, instead use `--v8-flags=--help`
- Give more descriptive names to unit tests in flags.rs
- Assume argv and subcommand into DenoFlags struct so the output of
flags module is only DenoFlags rather than the tuple (subcommand, flags,
argv).
- Improve CLI help text
- Make `deno run` specific args like `--allow-net` only show up in 'deno
help run' instead of as global flags in `deno help`.
- Removes `deno version` to simplify our implementation and be closer to
clap defaults. `deno -V` now only shows Deno's version and not V8's nor
TypeScript. `Deno.versions` can be used to see that information.
- Prevent clap from auto-detecting terminal width and attempting to wrap
text.
- Bundles are fully standalone. They now include the shared loader with
`deno_typescript`.
- Refactor of the loader in `deno_typescript` to perform module
instantiation in a more
- Change of behaviour when an output file is not specified on the CLI.
Previously a default name was determined and the bundle written to that
file, now the bundle will be sent to `stdout`.
- Refactors in the TypeScript compiler to be able to support the concept
of a request type. This provides a cleaner abstraction and makes it
easier to support things like single module transpiles to the userland.
- Remove a "dangerous" circular dependency between `os.ts` and `deno.ts`,
and define `pid` and `noColor` in a better way.
- Don't bind early to `console` in `repl.ts`.
- Add an integration test for generating a bundle.
Basically this does pre-processing of TypeScript files and gathers all the
dependencies asynchronously. Only then after all the dependencies are gathered,
does it do a compile, which at that point all the dependencies are cached in
memory in the compiler, so with the exception of the hard coded assets, there
are no ops during the compilation.
Because op_fetch_source_files is now handled asynchronously in the runtime, we
can eliminate the tokio_util::block_on() which was causing the increase in
threads. Benchmarking on my machine has shown about a 5% improvement in speed
when dealing with compiling TypeScript. Still a long way to go, but an
improvement.
In theory the module name resolution and the fetching of the source files could
be broken out as two different ops. This would prevent situations of sending the
full source file all the time when actually the module is the same module
referenced by multiple modules, but that could be done subsequently to this.
This ensures the deno executable is properly created before running the integration tests.
Also allows deno_cli to be used as a lib. Docs are now properly generated: https://docs.rs/deno_cli/0.18.4/deno_cli/
Towards #2933
Prep for #2955
