This change simplifies how we execute V8. Previously V8 Isolates jumped
around threads every time they were woken up. This was overly complex and
potentially hurting performance in a myriad ways. Now isolates run on
their own dedicated thread and never move.
- blocking_json spawns a thread and does not use a thread pool
- op_host_poll_worker and op_host_resume_worker are non-operational
- removes Worker::get_message and Worker::post_message
- ThreadSafeState::workers table contains WorkerChannel entries instead
of actual Worker instances.
- MainWorker and CompilerWorker are no longer Futures.
- The multi-threaded version of deno_core_http_bench was removed.
- AyncOps no longer need to be Send + Sync
This PR is very large and several tests were disabled to speed
integration:
- installer_test_local_module_run
- installer_test_remote_module_run
- _015_duplicate_parallel_import
- _026_workers
For some reason, the unit tests for Deno.remove() were not being imported to
unit_tests.ts and, consequently, not being executed. Thus, I imported them,
refactored some existent ones and wrote new ones for the symlink removal case.
Since the creation of a symlink is not implemented for Windows yet, assertions
that consider this state were added when the tests are executed in this OS.
This flag was added to evaluate performance relative to tokio's threaded
runtime. Although it's faster in the HTTP benchmark, it's clear the runtime
is not the only perf problem.
Removing this flag will simplify further refactors, in particular
adopting the #[tokio::main] macro. This will be done in a follow up.
Ultimately we expect to move to the current thread runtime with Isolates
pinned to specific threads, but that will be a much larger refactor. The
--current-thread just complicates that effort.
Before:
```
▶ target/debug/deno https://deno.land/std/examples/echo_server.ts
error: Uncaught PermissionDenied: run again with the --allow-net flag
► $deno$/dispatch_json.ts:40:11
at DenoError ($deno$/errors.ts:20:5)
at unwrapResponse ($deno$/dispatch_json.ts:40:11)
at sendSync ($deno$/dispatch_json.ts:67:10)
at listen ($deno$/net.ts:170:15)
at https://deno.land/std/examples/echo_server.ts:4:23
```
```
▶ target/debug/deno --allow-read=/usr https://deno.land/std/examples/cat.ts /etc/passwd
error: Uncaught PermissionDenied: run again with the --allow-read flag
► $deno$/dispatch_json.ts:40:11
at DenoError ($deno$/errors.ts:20:5)
at unwrapResponse ($deno$/dispatch_json.ts:40:11)
at sendAsync ($deno$/dispatch_json.ts:91:10)
```
After:
```
▶ target/debug/deno https://deno.land/std/examples/echo_server.ts
error: Uncaught PermissionDenied: network access to "0.0.0.0:8080", run again with the --allow-net flag
► $deno$/dispatch_json.ts:40:11
at DenoError ($deno$/errors.ts:20:5)
at unwrapResponse ($deno$/dispatch_json.ts:40:11)
at sendSync ($deno$/dispatch_json.ts:67:10)
at listen ($deno$/net.ts:170:15)
at https://deno.land/std/examples/echo_server.ts:4:23
```
```
▶ target/debug/deno --allow-read=/usr https://deno.land/std/examples/cat.ts /etc/passwd
error: Uncaught PermissionDenied: read access to "/etc/passwd", run again with the --allow-read flag
► $deno$/dispatch_json.ts:40:11
at DenoError ($deno$/errors.ts:20:5)
at unwrapResponse ($deno$/dispatch_json.ts:40:11)
at sendAsync ($deno$/dispatch_json.ts:91:10)
```
* move is_dyn_import argument from Loader::resolve to Loader::load - it was always kind of strange that resolve() checks permissions.
* change argument type from &str to &ModuleSpecifier where applicable
Ref #3712. This change allowed the deno_typescript crate to reference
cli/js/lib.deno_runtime.d.ts which breaks "cargo package". We intend to
reintroduce a revised version of this patch later once "cargo
package" is working and tested.
This reverts commit 737ab94ea1.
* split ops/worker.rs into ops/worker_host.rs and ops/web_worker.rs
* refactor js/workers.ts and factor out js/worker_main.ts - entry point for WebWorker runtime
* BREAKING CHANGE: remove support for blob: URL in Worker
* BREAKING CHANGE: remove Deno namespace support and noDenoNamespace option in Worker constructor
* introduce WebWorker struct which is a stripped down version of cli::Worker
* refactored RecursiveLoad - it was renamed to RecursiveModuleLoad, it does not take ownership of isolate anymore - a struct implementing Stream that yields SourceCodeInfo
* untangled module loading logic between RecursiveLoad and isolate - that logic is encapsulated in EsIsolate and RecursiveModuleLoad, where isolate just consumes modules as they become available - does not require to pass Arc<Mutex<Isolate>> around anymore
* removed EsIsolate.mods_ in favor of Modules and moved them inside EsIsolate
* EsIsolate now requires "loader" argument during construction - struct that implements Loader trait
* rewrite first methods on isolate as async
tokio_util::run and tokio::run_on_current_thread should accept Future<Output=()> instead of Future<Output=Result<(), ()>>. Currently, all the passed futures have to add Ok(()) or futures::future::ok(()) unnecessarily to call this method.
- Remove ability to specify run arguments like `--allow-net` after the
script argument. It's too hacky to make work with clap.
- Remove `--v8-options`, instead use `--v8-flags=--help`
- Give more descriptive names to unit tests in flags.rs
- Assume argv and subcommand into DenoFlags struct so the output of
flags module is only DenoFlags rather than the tuple (subcommand, flags,
argv).
- Improve CLI help text
- Make `deno run` specific args like `--allow-net` only show up in 'deno
help run' instead of as global flags in `deno help`.
- Removes `deno version` to simplify our implementation and be closer to
clap defaults. `deno -V` now only shows Deno's version and not V8's nor
TypeScript. `Deno.versions` can be used to see that information.
- Prevent clap from auto-detecting terminal width and attempting to wrap
text.
This PR removes tokio_util::block_on - refactored compiler and file
fetcher slightly so that we can safely block there - that's because
only blocking path consist of only synchronous operations.
Additionally I removed excessive use of tokio_util::panic_on_error
and tokio_util::run_in_task and moved both functions to cli/worker.rs,
to tests module.
Closes #2960
After landing #3358 the benchmarks exploded indicating problems with workers and deno_core_http_bench.
This PR dramatically fixes thread/syscall count that showed up on benchmarks. Thread count is not back to previous levels but difference went from hundreds/thousands to about ~50.
- removes global `RESOURCE_TABLE` - resource tables are now created per `Worker`
in `State`
- renames `CliResource` to `StreamResource` and moves all logic related
to it to `cli/ops/io.rs`
- removes `cli/resources.rs`
- adds `state` argument to `op_read` and `op_write` and consequently adds
`stateful_minimal_op` to `State`
- IMPORTANT NOTE: workers don't have access to process stdio - this is
caused by fact that dropping worker would close stdout for process
(because it's constructed from raw handle, which closes underlying file
descriptor on drop)
This patch does not work with the recent bundler changes (#3325).
Unfortunately I didn't merge master before landing this patch. It has
something to do with console.log not working inside the compiler worker.
This reverts commit fd62379eaf.
- removes global `RESOURCE_TABLE` - resource tables are now created per `Worker`
in `State`
- renames `CliResource` to `StreamResource` and moves all logic related
to it to `cli/ops/io.rs`
- removes `cli/resources.rs`
- adds `state` argument to `op_read` and `op_write` and consequently adds
`stateful_minimal_op` to `State`
- IMPORTANT NOTE: workers don't have access to process stdio - this is
caused by fact that dropping worker would close stdout for process
(because it's constructed from raw handle, which closes underlying file
descriptor on drop)
- Bundles are fully standalone. They now include the shared loader with
`deno_typescript`.
- Refactor of the loader in `deno_typescript` to perform module
instantiation in a more
- Change of behaviour when an output file is not specified on the CLI.
Previously a default name was determined and the bundle written to that
file, now the bundle will be sent to `stdout`.
- Refactors in the TypeScript compiler to be able to support the concept
of a request type. This provides a cleaner abstraction and makes it
easier to support things like single module transpiles to the userland.
- Remove a "dangerous" circular dependency between `os.ts` and `deno.ts`,
and define `pid` and `noColor` in a better way.
- Don't bind early to `console` in `repl.ts`.
- Add an integration test for generating a bundle.
Towards simplifying (or better removing entirely) the CoreResource
trait. Resources should be any bit of privileged heap allocated memory
that needs to be referenced from JS, not very specific trait
implementations. Therefore CoreResource should be pushed towards being
as general as possible.
* Split ThreadSafeState into State and GlobalState. State is a "local"
state belonging to "Worker" while "GlobalState" is state shared by
whole program.
* Update "Worker" and ops to use "GlobalState" where applicable
* Move and refactor "WorkerChannels" resource
This avoids a crash when the Deno process has been running for
2**32 ms (about 50 days). Additionaly, time complexity of finding which
timer is due to fire next is reduced from from O(n) to O(log n).
Worth noting due to implementation of the Headers class the contents of headersMap have lowercase keys, although this matches the specification as header keys are case agnostic it does seem to not match behaviour of other implementations in other languages I have seen, would require some rewriting of Headers.ts
Basically this does pre-processing of TypeScript files and gathers all the
dependencies asynchronously. Only then after all the dependencies are gathered,
does it do a compile, which at that point all the dependencies are cached in
memory in the compiler, so with the exception of the hard coded assets, there
are no ops during the compilation.
Because op_fetch_source_files is now handled asynchronously in the runtime, we
can eliminate the tokio_util::block_on() which was causing the increase in
threads. Benchmarking on my machine has shown about a 5% improvement in speed
when dealing with compiling TypeScript. Still a long way to go, but an
improvement.
In theory the module name resolution and the fetching of the source files could
be broken out as two different ops. This would prevent situations of sending the
full source file all the time when actually the module is the same module
referenced by multiple modules, but that could be done subsequently to this.
When the global timer fires earlier than expected, which apparently
happens sometimes on server editions of Windows, we didn't call any
setTimeout callbacks, but we *also* didn't reschedule the global timer
to fire again later.
When this situation occurred it would make deno exit abruptly if there
were no other asynchronous ops running on the event loop. It could also
lead to application hangs if the upcoming setTimeout callback was
critical for the application to make progress.
This ensures the deno executable is properly created before running the integration tests.
Also allows deno_cli to be used as a lib. Docs are now properly generated: https://docs.rs/deno_cli/0.18.4/deno_cli/
Towards #2933
Prep for #2955
- Fixes cargo publish on deno_typescript, deno_cli_snapshots, and
deno_cli.
- Combines cli_snapshots and js into one directory.
- Extracts TS version at compile time rather than runtime
- Bumps version awkwardly - it was necessary to test end-to-end
publishing. Sorry.
- Adds git submodule deno_typescript/typescript
Instead of using core/snapshot_creator.rs, instead two crates are
introduced which allow building the snapshot during build.rs.
Rollup is removed and replaced with our own bundler. This removes
the Node build dependency. Modules in //js now use Deno-style imports
with file extensions, rather than Node style extensionless imports.
This improves incremental build time when changes are made to //js files
by about 40 seconds.
This removes dispatch_flatbuffers as it is now unused. There are still a
few places where msg_generated is used: ErrorKind and MediaType. These
will be dealt with later.
* Revert "port more ops to JSON (#2809)"
This reverts commit 137f33733d.
* Revert "port ops to JSON: compiler, errors, fetch, files (#2804)"
This reverts commit 79f82cf10e.
* Revert "Port rest of os ops to JSON (#2802)"
This reverts commit 5b2baa5c99.
* remove fetch_source_file_and_maybe_compile_async and
replace it with State.fetch_compiled_module
* remove SourceFile.js_source()
* introduce CompiledModule which is basically the same as
deno::SourceInfo and represents arbitrary file that has been
compiled to JS module
* introduce //cli/compilers module containing all compilers
* introduce JsCompiler which is a no-op compiler
- output is the same as input, no compilation takes place
- it is used for MediaType::JavaScript and MediaType::Unknown
* introduce JsonCompiler that wraps JSON in default export
* support JS-to-JS compilation using checkJs
* merge SourceFileFetcher trait and FileFetcher struct
* move logic related to source file fetching to //cli/file_fetcher.rs
* use Result when creating new ThreadSafeState
* rename `ModuleMetaData` to `SourceFile` and remove TS specific
functionality
* add `TsCompiler` struct encapsulating processing of TypeScript files
* move `SourceMapGetter` trait implementation to `//cli/compiler.rs`
* add low-level `DiskCache` API for general purpose caches and use it in
`DenoDir` and `TsCompiler` for filesystem access
* don't use hash-like filenames for compiled modules, instead use
metadata file for storing compilation hash
* add `SourceFileCache` for in-process caching of loaded files for fast
subsequent access
* define `SourceFileFetcher` trait encapsulating loading of local and
remote files and implement it for `DenoDir`
* define `use_cache` and `no_fetch` flags on `DenoDir` instead of using
in fetch methods
The rules are now as follows:
* In `import` statements, as mandated by the WHATWG specification,
the import specifier is always treated as a URL.
If it is a relative URL, it must start with either / or ./ or ../
* A script name passed to deno as a command line argument may be either
an absolute URL or a local path.
- If the name starts with a valid URI scheme followed by a colon, e.g.
'http:', 'https:', 'file:', 'foo+bar:', it always interpreted as a
URL (even if Deno doesn't support the indicated protocol).
- Otherwise, the script name is interpreted as a local path. The local
path may be relative, and operating system semantics determine how
it is resolved. Prefixing a relative path with ./ is not required.
A lot of its functionality is unused still, but the goal it to slowly
migrate logging functionality to it. There is also a useful progress bar
which can be ported over later - it depends on this module.
4c1fa54d10/src/cargo/util/progress.rs
1. Separate Snapshot and Script StartupData functions based on cfg "no-snapshot-init"
2. Replace deprecated Once::ONCE_INIT with Once::new (https://github.com/rust-lang/rust/pull/61757)
3. Elide lifetime
4. Fix typos
