Classic worker scripts are now executed in the context of a Tokio
runtime. This does mean we can not spawn more tokio runtimes in
"op_worker_sync_fetch". We instead spawn a new thread there, that can
create a new Tokio runtime that we can use to block the worker thread.
This commit implements classic workers, but only when the `--enable-testing-features-do-not-use` flag is provided. This change is not user facing. Classic workers are used extensively in WPT tests. The classic workers do not support loading from disk, and do not support TypeScript.
Co-authored-by: Luca Casonato <hello@lcas.dev>
Currently, calling the `abort()` method on a `FileReader` object aborts
any current read operation, but it also prevents any read operation
started at some later point from starting. The File API instead
specifies that calling `abort()` should reset the `FileReader`'s state
and result, as well as removing any queued tasks from the current
operation that haven't yet run.
Currently, a WPT test is considered failed if its status code is
anything other than 0, regardless of whether the test suite completed
running or not, and any subtests that haven't finished running are not
considered to be failures.
But a test can exit with a zero status code before it has completed
running, if the event loop has run out of tasks because of a bug in one
of the ops, leading to false positives. This change fixes that.
The WebAssembly streaming APIs used to be enabled, but used to take
buffer sources as their first argument (see #6154 and #7259). This
change re-enables them, requiring a Promise<Response> instead, as well as
enabling asynchronous compilation of WebAssembly modules.
Additionally, if the existing `Request`'s body is disturbed, the Request creation
should fail.
This change also updates the step numbers in the Request constructor to match
whatwg/fetch#1249.
This adds a daily scheduled CI pipeline that runs WPT tests against
the most recent epochs/daily every night. Results are uploaded to
wpt.fyi.
WPTs are run on all supported platforms, on both stable and canary.
These reports can be consumed by tools like `wptreport` or
https://wpt.fyi. The old style report could be removed in a future PR
when wpt.deno.land is updated.
This commit removes all JS based text encoding / text decoding. Instead
encoding now happens in Rust via encoding_rs (already in tree). This
implementation retains stream support, but adds the last missing
encodings. We are incredibly close to 100% WPT on text encoding now.
This should reduce our baseline heap by quite a bit.
Replaces the file-backed provider by an in-memory one because proper
file locking is a hard problem that detracts from the proof of concept.
Teach the WPT runner how to extract tests from .html files because all
the relevant tests in test_util/wpt/webmessaging/broadcastchannel are
inside basics.html and interface.html.
This commit aligns the `fetch` API and the `Request` / `Response`
classes belonging to it to the spec. This commit enables all the
relevant `fetch` WPT tests. Spec compliance is now at around 90%.
Performance is essentially identical now (within 1% of 1.9.0).
This commit aligns `Headers` to spec. It also removes the now unused
03_dom_iterable.js file. We now pass all relevant `Headers` WPT. We do
not implement any sort of header filtering, as we are a server side
runtime.
This is likely not the most efficient implementation of `Headers` yet.
It is however spec compliant. Once all the APIs in the `HTTP` hot loop
are correct we can start optimizing them. It is likely that this commit
reduces bench throughput temporarily.
- Improves op performance.
- Handle op-metadata (errors, promise IDs) explicitly in the op-layer vs
per op-encoding (aka: out-of-payload).
- Remove shared queue & custom "asyncHandlers", all async values are
returned in batches via js_recv_cb.
- The op-layer should be thought of as simple function calls with little
indirection or translation besides the conceptually straightforward
serde_v8 bijections.
- Preserve concepts of json/bin/min as semantic groups of their
inputs/outputs instead of their op-encoding strategy, preserving these
groups will also facilitate partial transitions over to v8 Fast API for the
"min" and "bin" groups
This commit rewrites scripts in "tools/" directory
to use Deno instead of Python. In return it allows
to remove huge number of Python packages in "third_party/".
All benchmarks are done in Rust and can be invoked with
`cargo bench`.
Currently this has it's own "harness" that behaves like
`./tools/benchmark.py` did.
Because of this tests inside `cli/bench` are currently not run.
This should be switched to the language provided harness
once the `#[bench]` attribute has been stabilized.
This PR is intentionally ugly. It duplicates all of the code in cli/js2/ into
cli/tsc/ ... because it's very important that we all understand that this code
is unnecessarily duplicated in our binary. I hope this ugliness provides the
motivation to clean it up.
The typescript git submodule is removed, because it's a very large repo and
contains all sorts of stuff we don't need. Instead the necessary files are
copied directly into the deno repo. Hence +200k lines.
COMPILER_SNAPSHOT.bin size
```
master 3448139
this branch 3320972
```
Fixes #6812
This commit adds a "--no-check" option to following subcommands:
- "deno cache"
- "deno info"
- "deno run"
- "deno test"
The "--no-check" options allows to skip type checking step and instead
directly transpiles TS sources to JS sources.
This solution uses `ts.transpileModule()` API and is just an interim
solution before implementing it fully in Rust.
This commit changes how error occurring in SWC are handled.
Changed lexer settings to properly handle TS decorators.
Changed output of SWC error to annotate with position in file.
This commit fixes a bug introduced in #5029 that caused bad
handling of redirects during module analysis.
Also ensured that duplicate modules are not downloaded.
This PR removes the hack in CLI that allows to run scripts with shorthand: deno script.ts.
Removing this functionality because it hacks around short-comings of clap our CLI parser. We agree that this shorthand syntax is desirable, but it needs to be rethinked and reimplemented. For 1.0 we should go with conservative approach that is correct.
- Removes the __fetch namespace from `deno types`
- Response.redirect should be a static.
- Response.body should not be AsyncIterable.
- Disables the deno_proxy benchmark
- Makes std/examples/curl.ts buffer the body before printing to stdout
* Remove DENO_BUILD_MODE and DENO_BUILD_PATH
Also remove outdated docs related to ninja/gn.
* fix
* remove parameter to build_mode()
* remove arg parsing from benchmark.py
* add tests for "Deno.core.encode" and "Deno.core.decode" for empty inputs
* use "Deno.core.encode" in "TextEncoder"
* use "Deno.core.decode" in "TextDecoder"
* remove "core_decode" and "core_encode" benchmarks
This commits add two new methods to "Deno.core" namespace: "encode" and "decode".
Those methods are bound in Rust to provide a) fast b) generally available of encoding and decoding UTF-8 strings.
Both methods are now used in "cli/js/dispatch_json.ts".
Rewrites "cli/js/unit_test_runner.ts" to communicate with spawned subprocesses
using TCP socket.
* Rewrite "Deno.runTests()" by factoring out testing logic to private "TestApi"
class. "TestApi" implements "AsyncIterator" that yields "TestEvent"s,
which is an interface for different types of event occuring during running
tests.
* Add "reporter" argument to "Deno.runTests()" to allow users to provide custom
reporting mechanism for tests. It's represented by "TestReporter" interface,
that implements hook functions for each type of "TestEvent". If "reporter"
is not provided then default console reporting is used (via
"ConsoleReporter").
* Change how "unit_test_runner" communicates with spawned suprocesses. Instead
of parsing text data from child's stdout, a TCP socket is created and used
for communication. "unit_test_runner" can run in either "master" or "worker"
mode. Former is responsible for test discovery and establishing needed
permission combinations; while latter (that is spawned by "master") executes
tests that match given permission set.
* Use "SocketReporter" that implements "TestReporter" interface to send output
of tests to "master" process. Data is sent as stringified JSON and then
parsed by "master" as structured data. "master" applies it's own reporting
logic to output tests to console (by reusing default "ConsoleReporter").
This change simplifies how we execute V8. Previously V8 Isolates jumped
around threads every time they were woken up. This was overly complex and
potentially hurting performance in a myriad ways. Now isolates run on
their own dedicated thread and never move.
- blocking_json spawns a thread and does not use a thread pool
- op_host_poll_worker and op_host_resume_worker are non-operational
- removes Worker::get_message and Worker::post_message
- ThreadSafeState::workers table contains WorkerChannel entries instead
of actual Worker instances.
- MainWorker and CompilerWorker are no longer Futures.
- The multi-threaded version of deno_core_http_bench was removed.
- AyncOps no longer need to be Send + Sync
This PR is very large and several tests were disabled to speed
integration:
- installer_test_local_module_run
- installer_test_remote_module_run
- _015_duplicate_parallel_import
- _026_workers
This flag was added to evaluate performance relative to tokio's threaded
runtime. Although it's faster in the HTTP benchmark, it's clear the runtime
is not the only perf problem.
Removing this flag will simplify further refactors, in particular
adopting the #[tokio::main] macro. This will be done in a follow up.
Ultimately we expect to move to the current thread runtime with Isolates
pinned to specific threads, but that will be a much larger refactor. The
--current-thread just complicates that effort.
- Bundles are fully standalone. They now include the shared loader with
`deno_typescript`.
- Refactor of the loader in `deno_typescript` to perform module
instantiation in a more
- Change of behaviour when an output file is not specified on the CLI.
Previously a default name was determined and the bundle written to that
file, now the bundle will be sent to `stdout`.
- Refactors in the TypeScript compiler to be able to support the concept
of a request type. This provides a cleaner abstraction and makes it
easier to support things like single module transpiles to the userland.
- Remove a "dangerous" circular dependency between `os.ts` and `deno.ts`,
and define `pid` and `noColor` in a better way.
- Don't bind early to `console` in `repl.ts`.
- Add an integration test for generating a bundle.
The yarn.lock file is moved from //third_party to the root of
the main repository. This is where other package metadata
files (e.g. Cargo.lock and package.json) are also located.
- Fixes cargo publish on deno_typescript, deno_cli_snapshots, and
deno_cli.
- Combines cli_snapshots and js into one directory.
- Extracts TS version at compile time rather than runtime
- Bumps version awkwardly - it was necessary to test end-to-end
publishing. Sorry.
- Adds git submodule deno_typescript/typescript
* Remove reference to removed dir 'third_party/rust_crates'.
* Remove reference to unused environment variable 'DENO_NINJA_PATH'.
* Remove helper functions 'root()' and 'tp()'.
* Move definition of 'third_party_path' to build.py.
* Move definition of 'gn_exe()' to setup.py.
* Move 'download_sccache()' and 'download_hyperfine()' from prebuilt.py
to third_party.py, and delete prebuilt.py.
* Add helper function 'get_platform_dir_name()' to locate the
platform-specific 'v8/buildtools/<platform>' and
'prebuilt/<platform>' directories.
* Add helper function 'get_prebuilt_tool_path()' that returns the full
path to a platform-specific executable in //prebuilt.
* Cosmetic improvements.
//build was updated and our patch rebased:
https://github.com/denoland/chromium_build/tree/20190910_win_clang_depfile
V8 has removed the binaries third_party/v8/buildtools/*/gn.sha1 and started
using CIPD. In this patch, I've hacked in the original .sha1 files and continue to
use download_from_google_storage instead of the new CIPD. This will be fixed
in later work.
Instead of using core/snapshot_creator.rs, instead two crates are
introduced which allow building the snapshot during build.rs.
Rollup is removed and replaced with our own bundler. This removes
the Node build dependency. Modules in //js now use Deno-style imports
with file extensions, rather than Node style extensionless imports.
This improves incremental build time when changes are made to //js files
by about 40 seconds.
* rename `ModuleMetaData` to `SourceFile` and remove TS specific
functionality
* add `TsCompiler` struct encapsulating processing of TypeScript files
* move `SourceMapGetter` trait implementation to `//cli/compiler.rs`
* add low-level `DiskCache` API for general purpose caches and use it in
`DenoDir` and `TsCompiler` for filesystem access
* don't use hash-like filenames for compiled modules, instead use
metadata file for storing compilation hash
* add `SourceFileCache` for in-process caching of loaded files for fast
subsequent access
* define `SourceFileFetcher` trait encapsulating loading of local and
remote files and implement it for `DenoDir`
* define `use_cache` and `no_fetch` flags on `DenoDir` instead of using
in fetch methods
Don't mix every http request in with the tests output.
Don't print that the file servers are starting unless
-vv flag is passed.
Capture the output of run with run_output which returns
stdout, stderr and exit_code. Test against this rather
than relying on sys.exit.
* use subclass of unittest.TestCase for all test cases
* allow to run single test file (eg. python tools/integration_tests.py)
* test filtering (via --pattern/-p CLI flag)
* use common CLI parser for all tests:
usage: test.py [-h] [--failfast] [--verbose] [--executable EXECUTABLE]
[--release] [--pattern PATTERN] [--build-dir BUILD_DIR]
optional arguments:
-h, --help show this help message and exit
--failfast, -f Stop on first failure
--verbose, -v Verbose output
--executable EXECUTABLE
Use external executable of Deno
--release Test against release executable
--pattern PATTERN, -p PATTERN
Run tests that match provided pattern
--build-dir BUILD_DIR
Deno build directory
* respect NO_COLOR variable
Move every test to a method on DenoTestCase.
test.py is a single TestSuite of every TestCase.
Add a Spawn context manager for http_server,
this is explicitly used where it's needed.
Each python test file can now be run independently
without needing to manually run http_server.
Add --help and consistent flags using argparse for
each python test, including --failfast.
Use ColorTextTestRunner so that '... ok' is green.
This is to ensure a more fair test. Also we were already downloading
from the internet since we changed the URL to use std@v0.5.0. This
change exposes an OOM bug, which is then fixed in the upcoming compiler
refactor by changing checkJs compiler option to false.
The following tests were commented out in order to get this to go green :
- bodyMultipartFormData
- bodyURLEncodedFormData
- fetchRequestInitStringBody
- netConcurrentAccept
- netListenAsyncIterator
They were silently broken in 780e72 due to flag reordering. This commit
also includes a new assert that would avoid that kind of failure in the
future.
Fixes some sed errors introduced in c43cfe.
Unfortunately moving libdeno required splitting build.rs into two parts,
one for cli and one for core.
I've also removed the arm64 build - it's complicating things at this
re-org and we're not even testing it. I need to swing back to it and get
tools/test.py running for it.
This disables a few tests which are broken still:
- tests/error_004_missing_module.test
- tests/error_005_missing_dynamic_import.test
- tests/error_006_import_ext_failure.test
- repl_test test_set_timeout
- repl_test test_async_op
- repl_test test_set_timeout_interlaced
- all of permission_prompt_test
We always pass around Box<[u8]>, and adding this generic is an
unnecessary complication.
Add deno_core_http_bench_test to test.py
sharedQueue works on deno_core_http_bench
Previously run_node.py would always attempt to remove and then re-create
the 'target/xx/node_modules' symlink. This causes sporadic build errors
on windows when multiple build targets that use run_node.py are being
built concurrently.
This fixes an issue on Windows that causes build to fails when
fix_symlinks() is called concurrently with another build step.
It is also no longer necessary, since recent versions of git know how to
properly create symbolic links on checkout.
Resolves #1705
This PR adds the Deno APIs as a global namespace named `Deno`. For backwards
compatibility, the ability to `import * from "deno"` is preserved. I have tried
to convert every test and internal code the references the module to use the
namespace instead, but because I didn't break compatibility I am not sure.
On the REPL, `deno` no longer exists, replaced only with `Deno` to align with
the regular runtime.
The runtime type library includes both the namespace and module. This means it
duplicates the whole type information. When we remove the functionality from the
runtime, it will be a one line change to the library generator to remove the
module definition from the type library.
I marked a `TODO` in a couple places where to remove the `"deno"` module, but
there are additional places I know I didn't mark.
This is more optimial and this fixes a problem where occasionally
remove_and_symlink will error with:
gen/compiler_bundle/main.js --sourcemapFile /Users/rld/src/deno --silent
Traceback (most recent call last):
File "../../tools/run_node.py", line 18, in <module>
util.remove_and_symlink(target_rel, "node_modules", True)
File "/Users/rld/src/deno/tools/util.py", line 103, in remove_and_symlink
symlink(target, name, target_is_dir)
File "/Users/rld/src/deno/tools/util.py", line 141, in symlink
os.symlink(target, name)
OSError: [Errno 17] File exists
This still retains the ability to build V8 from scratch, with an number
of configurations. The prebuilt binaries were created using
DENO_BUILD_MODE=release ./tools/build.py v8
- Based on code from @qti3e and @piscisaureus in #724 and #1125
respectively.
- TODO The DENO_BUILD_PATH env var must be supplied and must be an
absolute path, this restriction should be removed in future work.
(Use C:\deno instead of c:\deno in appveyor config because it's cloned to c:\ by clone_folder variable in .appveyor.yml. On the other hand, build directory is pointed to C:\ by $(APPVEYOR_BUILD_FOLDER) so that test targets are placed on separated partitions.)
Checks the output more carefully. The first line of output from
js/unit_tests.ts should be something like "running 96 tests"
And the last line should be something like
"test result: ok. 96 passed; 0 failed; 0 ignored; 0 measured; 36
filtered out"
This parses those strings and make sure they align.
This will catch silent death bugs.
* Changed tools/lint.py to lint the entire js and tests directorys and sub directories, currently it was pointing at tsconfig and would only lint files that were part of js/main.ts or node_modules/typescript/lib/lib.esnext.d.ts and their dependencies
* Broke the typescript linting out into separate steps for the main typescript programing and tests.
* Fixed linting issues in ts tests.
