This commit changes current "deno_core::resolve_url_or_path" API to
"resolve_url_or_path_deprecated" and adds new "resolve_url_or_path"
API that requires to explicitly pass the directory from which paths
should be resolved to.
Some of the call sites were updated to use the new API, the reminder
of them will be updated in a follow up PR.
Towards landing https://github.com/denoland/deno/pull/15454
This commit removes "deno_core::RuntimeOptions::extensions_with_js".
Now it's embedders' responsibility to properly register extensions
that will not contains JavaScript sources when running from an existing
snapshot.
Prerequisite for https://github.com/denoland/deno/pull/18080
This commit changes "ModuleMap" initialization to over-allocate by 16
for vectors storing module information and module V8 handles. In 99%
cases there's gonna be at least one additional module loaded, so it's
very wasteful to have to reallocate when the module is executed (IIRC
Rust will double the size of the vector) and move all of the elements.
```
Benchmark 1: deno run -A ../empty.js
Time (mean ± σ): 20.5 ms ± 0.5 ms [User: 13.4 ms, System: 5.1 ms]
Range (min … max): 19.8 ms … 24.0 ms 119 runs
Benchmark 2: target/release/deno run -A ../empty.js
Time (mean ± σ): 18.8 ms ± 0.3 ms [User: 13.0 ms, System: 4.9 ms]
Range (min … max): 18.3 ms … 19.9 ms 129 runs
```
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit renames "deno_core::InternalModuleLoader" to
"ExtModuleLoader" and changes the specifiers used by the
modules loaded from this loader to "ext:".
"internal:" scheme was really ambiguous and it's more characters than
"ext:", which should result in slightly smaller snapshot size.
Closes https://github.com/denoland/deno/issues/18020
There's no point for this API to expect result. If something fails it should
result in a panic during build time to signal to embedder that setup is
wrong.
This API is required by several extensions like "ext/node", "ext/ffi"
and also FS APIs that we want to move to a separate crate. Because
of that "pathFromURL" API was moved to "deno_web" extension so
other extension crates can rely on it.
This commit changes "InternalModuleLoader" from "deno_core" to
store a list of used modules during snapshotting. If a module was not
used during snapshotting "InternalModuleLoader" will panic in its "Drop"
handler signaling to the embedder that they made a mistake somewhere.
We use information about build in several extension crates like
"ext/node" or "runtime/". In an effort to move "fs" APIs to a separate
crate it is a prerequisite to have this information available outside
of the "runtime/" crate.
This commit moves definition of "build" object to "Deno.core" that is
later forwarded to "Deno.build".
This commit changes "include_js_files!" macro from "deno_core"
in a way that "dir" option doesn't cause specifiers to be rewritten
to include it.
Example:
```
include_js_files! {
dir "js",
"hello.js",
}
```
The above definition required embedders to use:
`import ... from "internal:<ext_name>/js/hello.js"`.
But with this change, the "js" directory in which the files are stored
is an implementation detail, which for embedders results in:
`import ... from "internal:<ext_name>/hello.js"`.
The directory the files are stored in, is an implementation detail and
in some cases might result in a significant size difference for the
snapshot. As an example, in "deno_node" extension, we store the
source code in "polyfills" directory; which resulted in each specifier
to look like "internal:deno_node/polyfills/<module_name>", but with
this change it's "internal:deno_node/<module_name>".
Given that "deno_node" has over 100 files, many of them having
several import specifiers to the same extension, this change removes
10 characters from each import specifier.
Runtime generation of async op wrappers contributed to increased startup
time and core became unusable with
`--disallow-code-generation-from-strings` flag. The optimization only
affects very small microbenchmarks so this revert will not cause any
regressions.
This commit further improves startup time by:
- no relying on "JsRuntime::execute_script" for runtime bootstrapping,
this is instead done using V8 APIs directly
- registering error classes during the snapshot time, instead of on
startup
Further improvements can be made, mainly around removing
"core.initializeAsyncOps()" which takes around 2ms.
This commit should result in ~1ms startup time improvement.
Instead of relying on "serde_v8" which is very inefficient in
serializing enums, I'm hand rolling serde for "ModuleMap" data
that is stored in the V8 snapshot to make ES modules
snapshottable.
```
// this branch
Benchmark #2: ./target/release/deno run empty.js
Time (mean ± σ): 21.4 ms ± 0.9 ms [User: 15.6 ms, System: 6.4 ms]
Range (min … max): 20.2 ms … 24.4 ms
// main branch
Benchmark #2: ./target/release/deno run empty.js
Time (mean ± σ): 23.1 ms ± 1.2 ms [User: 17.0 ms, System: 6.2 ms]
Range (min … max): 21.0 ms … 26.0 ms
```
This allows to not include source code into the binary (because
it will already be included in the V8 snapshot).
Nothing changes for the embedders - everything should still build the
same.
This commit brings the binary size from 87Mb to 82Mb on M1.
Alternative to https://github.com/denoland/deno/pull/17820 and
https://github.com/denoland/deno/pull/17653
---------
Co-authored-by: Leo Kettmeir <crowlkats@toaxl.com>
This commit changes definition of "ExtensionFileSource", by changing
"code" field to being "ExtensionFileSourceCode" enum. Currently the enum
has only a single variant "IncludedInBinary". It is done in preparation
to allow embedders to decide if they want to include the source code in the
binary when snapshotting (in most cases they shouldn't do that).
In the follow up commit we'll add more variants to
"ExtensionFileSourceCode".
"include_js_files_dir!" macro was removed in favor "include_js_files!"
macro which can now accept "dir" option.
Adds two test files: "cli/tests/unit_node/process_test.ts" and
"cli/tests/unit_node/child_process_test.ts"
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This PR changes Node.js/npm compatibility layer to use polyfills for
built-in Node.js
embedded in the snapshot (that are coming from "ext/node" extension).
As a result loading `std/node`, either from
"https://deno.land/std@<latest>/" or
from "DENO_NODE_COMPAT_URL" env variable were removed. All code that is
imported via "npm:" specifiers now uses code embedded in the snapshot.
Several fixes were applied to various modules in "ext/node" to make
tests pass.
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Compressing the TSC snapshot in debug build took
~45s on M1 MacBook Pro; without compression it took ~1s.
Thus we're not not using compressed snapshot, trading off
a lot of build time for some startup time in debug build.
This commit adds "ExtensionBuilder::esm_entry_point()" function that
allows to specify which of the extension files should be treated as an
entry point. If the entry point is not provided all modules are loaded
and evaluated, but if it is provided then only the entry point is explicitly
loaded and evaluated.
Co-authored-by: Leo Kettmeir <crowlkats@toaxl.com>
This is a proof of concept for being able to snapshot TypeScript files.
Currently only a single runtime file is authored in TypeScript -
"runtime/js/01_version.ts".
Not needed infrastructure was removed from "core/snapshot_util.rs".
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This PR refactors all internal js files (except core) to be written as
ES modules.
`__bootstrap`has been mostly replaced with static imports in form in
`internal:[path to file from repo root]`.
To specify if files are ESM, an `esm` method has been added to
`Extension`, similar to the `js` method.
A new ModuleLoader called `InternalModuleLoader` has been added to
enable the loading of internal specifiers, which is used in all
situations except when a snapshot is only loaded, and not a new one is
created from it.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
The commit derives Eq, PartialEq, and Debug traits for the
`ResolutionKind` enum to make it possible for external
implementors to assert ResolutionKind.
- changes module id to be usize & 0 based instead of 1 based
- merges `ids_by_handle` & `handles_by_id` to be a single `handles`
vector
- removes `next_module_id`, as vector is used
- turns `info` into a vector
This commit adds support for snapshotting ES modules. This is done by
adding an ability to serialize and deserialize a "ModuleMap" and attach
it
to the snapshot, using "add_context_data" API.
This has been tested with 400 modules and seems to not have a limit on
the number of modules that might be snapshotted.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
The `JsRuntimeState` struct stores a number of JS callbacks that are
used either in the event loop or when interacting with V8. Some of these
callback fields are vectors of callbacks, and therefore could plausibly
store at least one callback per realm. However, some of those fields are
`Option<v8::Global<v8::Function>>`, which would make the callbacks set
by a realm override the one that might have been set by a different
realm.
As it turns out, all of the current such optional callbacks
(`js_promise_reject_cb`, `js_format_exception_cb` and
`js_wasm_streaming_cb`) are only used from inside a realm, and therefore
this change makes it so such callbacks can only be set from inside a
realm, and will only affect that realm.
This is a reland of #15599.
Towards #13239.
Currently realms are supported on `deno_core`, but there was no support
for async ops anywhere other than the main realm. The main issue is that
the `js_recv_cb` callback, which resolves promises corresponding to
async ops, was only set for the main realm, so async ops in other realms
would never resolve. Furthermore, promise ID's are specific to each
realm, which meant that async ops from other realms would result in a
wrong promise from the main realm being resolved.
This change takes the `ContextState` struct added in #17050, and adds to
it a `js_recv_cb` callback for each realm. Combined with the fact that
that same PR also added a list of known realms to `JsRuntimeState`, and
that #17174 made `OpCtx` instances realm-specific and had them include
an index into that list of known realms, this makes it possible to know
the current realm in the `queue_async_op` and `queue_fast_async_op`
methods, and therefore to send the results of promises for each realm to
that realm, and prevent the ID's from getting mixed up.
Additionally, since promise ID's are no longer unique to the isolate,
having a single set of unrefed ops doesn't work. This change therefore
also moves `unrefed_ops` from `JsRuntimeState` to `ContextState`, and
adds the lengths of the unrefed op sets for all known realms to get the
total number of unrefed ops to compare in the event loop.
This PR is a reland of #14734 after it was reverted in #16366, except
that `ContextState` and `JsRuntimeState::known_realms` were previously
relanded in #17050. Another significant difference with the original PR
is passing around an index into `JsRuntimeState::known_realms` instead
of a `v8::Global<v8::Context>` to identify the realm, because async op
queuing in fast calls cannot call into V8, and therefore cannot have
access to V8 globals. This also simplified the implementation of
`resolve_async_ops`.
Co-authored-by: Luis Malheiro <luismalheiro@gmail.com>
This commit changes signature of "deno_core::ModuleLoader::resolve" to pass
an enum indicating whether or not we're resolving a specifier for dynamic import.
Additionally "CliModuleLoader" was changes to store both "parent permissions" (or
"root permissions") as well as "dynamic permissions" that allow to check for permissions
in top-level module load an dynamic imports.
Then all code paths that have anything to do with Node/npm compat are now checking
for permissions which are passed from module loader instance associated with given
worker.
This commit fixes handling of rejected promises in dynamic imports
evaluation.
Previously we were running callbacks for next ticks and macrotasks
_before_ polling
dynamic imports and checked for unhandled rejections immediately after.
This is wrong,
as `unhandledrejection` event is dispatched and its callbacks are run as
macrotasks.
This commit changes order of actions performed by the event loop to
following:
- poll async ops
- poll dynamic imports
- run next tick callbacks
- run macrotask callbacks
- check for unhandled promise rejections
Found this while debugging
https://github.com/denoland/deno/issues/16280.
Before:
```
TypeError: Could not resolve 'file:///Users/ib/dev/test_rollup/mocha/rollup.config.js' from 'file:///Users/ib/Library/Caches/deno/npm/registry.npmjs.org/rollup/3.7.3/dist/shared/loadConfigFile.js'.
at async getConfigFileExport (file:///Users/ib/Library/Caches/deno/npm/registry.npmjs.org/rollup/3.7.3/dist/shared/loadConfigFile.js:432:17)
at async Object.loadConfigFile (file:///Users/ib/Library/Caches/deno/npm/registry.npmjs.org/rollup/3.7.3/dist/shared/loadConfigFile.js:391:59)
at async getConfigs (file:///Users/ib/Library/Caches/deno/npm/registry.npmjs.org/rollup/3.7.3/dist/bin/rollup:1679:39)
at async runRollup (file:///Users/ib/Library/Caches/deno/npm/registry.npmjs.org/rollup/3.7.3/dist/bin/rollup:1656:43)
```
After:
```
TypeError: Could not resolve 'file:///Users/ib/dev/test_rollup/mocha/rollup.config.js' from 'file:///Users/ib/Library/Caches/deno/npm/registry.npmjs.org/rollup/3.7.3/dist/shared/loadConfigFile.js'.
Caused by:
Reading /Users/ib/dev/test_rollup/mocha/package.json is not allowed
at async getConfigFileExport (file:///Users/ib/Library/Caches/deno/npm/registry.npmjs.org/rollup/3.7.3/dist/shared/loadConfigFile.js:432:17)
at async Object.loadConfigFile (file:///Users/ib/Library/Caches/deno/npm/registry.npmjs.org/rollup/3.7.3/dist/shared/loadConfigFile.js:391:59)
at async getConfigs (file:///Users/ib/Library/Caches/deno/npm/registry.npmjs.org/rollup/3.7.3/dist/bin/rollup:1679:39)
at async runRollup (file:///Users/ib/Library/Caches/deno/npm/registry.npmjs.org/rollup/3.7.3/dist/bin/rollup:1656:43)
```
This commit fixes formatting of JSError with "errors" property. Before this
commit all instances of "Error" were treated as if they were "AggregateError"
if they had "errors" property. After this commit only actual instances of
"AggregateError" are formatted in such a way, while instances of "Error"
that have "errors" property are formatted without showing details of "errors".
Although PR #16366 did not fully revert `deno_core`'s support for
realms, since `JsRealm` still existed after that, it did remove the
`ContextState` struct, originally introduced in #14734. This change made
`js_build_custom_error_cb`, among other properties, a per-runtime
callback, rather than per-realm, which cause a bug where errors thrown
from an op would always be constructed in the main realm, rather than in
the current one.
This change adds back `ContextState` to fix this bug, adds back the
`known_realms` field of `JsRuntimeState` (needed to be able to drop the
callback when snapshotting), and also relands #14750, which adds the
`js_realm_sync_ops` test for this bug that was removed in #16366.
This commit changes implementation of "Deno.memoryUsage()" to return
correct value for "rss" field. To do that we implement a specialized function
per os to retrieve this information.
- `JsRuntime::built_from_snapshot` was removed because it was redundant
with `JsRuntime::snapshot_options`.
- Updates to stale documentation of `JsRuntime::create_realm`.
- `JsRuntime::create_realm` now calls `JsRuntime::init_extension_js`
unconditionally, since if the runtime was built from a snapshot,
`init_extension_js` will be a no-op.
- Typo fix in the documentation for `JsRealm`.
This commit adds new "--inspect-wait" flag which works similarly
to "--inspect-brk" in that it waits for inspector session to be
established before running code. However it doesn't break on the first
statement of user code, but instead runs it as soon as a session
is established.
Currently runtime exception are only displayed at the program end in
terminal, which makes it only a partial fix, as a full fix requires
https://github.com/denoland/rusty_v8/pull/1149 which adds new bindings
to the inspector that allows to notify it about thrown exceptions.
This will be handled in a follow up commit.
