This commit splits "<ext_name>::init" functions into "init_ops" and
"init_ops_and_esm". That way we don't have to construct list of
ESM sources on each startup if we're running with a snapshot.
In a follow up commit "deno_core" will be changed to not have a split
between "extensions" and "extensions_with_js" - it will be embedders'
responsibility to pass appropriately configured extensions.
Prerequisite for https://github.com/denoland/deno/pull/18080
Since we are snapshotting extension source at build time, there's no
need to define list of sources for the extension at runtime.
This commit changes "deno_node" extension by removing "init_polyfill"
function in favor of "init_polyfill_ops_and_esm()" and "init_polyfill_ops()".
The former is used during snapshot and when "deno_runtime" is compiled
with "dont_create_runtime_snapshot" cargo feature flag. The latter is used
when running a worker from an existing snapshot.
This is a start of a bigger refactor to all extensions - thanks to this
change, we don't have to iterate over all defined source files for extension at
runtime, and because of that we don't have to create a filepath for each of the
source files. It's not a big deal, but we are iterating over 300 files on each start,
and concatenating 3 strings before creating a "PathBuf" for ~200 of them.
This is already visible on the startup flamegraphs and should be avoided.
There's no point for this API to expect result. If something fails it should
result in a panic during build time to signal to embedder that setup is
wrong.
JavaScript APIs from "runtime/js/40_files.js" combined abstractions
for stdio streams ("Stdout", "Stderr", "Stdin") and file system file
("File", "FsFile"). APIs from "runtime/js/40_read_file.js" and
"runtime/js/40_write_file.js" were implemented using ops from
"runtime/ops/fs.rs".
This file was removed and relevant APIs were moved to "deno_io/12_io.js"
and "runtime/js/30_fs.js".
This work is meant to enable factoring out "deno_fs" crate.
This allows to not include source code into the binary (because
it will already be included in the V8 snapshot).
Nothing changes for the embedders - everything should still build the
same.
This commit brings the binary size from 87Mb to 82Mb on M1.
Alternative to https://github.com/denoland/deno/pull/17820 and
https://github.com/denoland/deno/pull/17653
---------
Co-authored-by: Leo Kettmeir <crowlkats@toaxl.com>
This commit changes definition of "ExtensionFileSource", by changing
"code" field to being "ExtensionFileSourceCode" enum. Currently the enum
has only a single variant "IncludedInBinary". It is done in preparation
to allow embedders to decide if they want to include the source code in the
binary when snapshotting (in most cases they shouldn't do that).
In the follow up commit we'll add more variants to
"ExtensionFileSourceCode".
"include_js_files_dir!" macro was removed in favor "include_js_files!"
macro which can now accept "dir" option.
This commit moves "deno_std/node" in "ext/node" crate. The code is
transpiled and snapshotted during the build process.
During the first pass a minimal amount of work was done to create the
snapshot, a lot of code in "ext/node" depends on presence of "Deno"
global. This code will be gradually fixed in the follow up PRs to migrate
it to import relevant APIs from "internal:" modules.
Currently the code from snapshot is not used in any way, and all
Node/npm compatibility still uses code from
"https://deno.land/std/node" (or from the location specified by
"DENO_NODE_COMPAT_URL"). This will also be handled in a follow
up PRs.
---------
Co-authored-by: crowlkats <crowlkats@toaxl.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This commit stabilizes Node-API, the "--unstable" flag is no longer
required to load native extensions. "--allow-ffi" permission is still
required to load them.
Compressing the TSC snapshot in debug build took
~45s on M1 MacBook Pro; without compression it took ~1s.
Thus we're not not using compressed snapshot, trading off
a lot of build time for some startup time in debug build.
This is a proof of concept for being able to snapshot TypeScript files.
Currently only a single runtime file is authored in TypeScript -
"runtime/js/01_version.ts".
Not needed infrastructure was removed from "core/snapshot_util.rs".
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This PR refactors all internal js files (except core) to be written as
ES modules.
`__bootstrap`has been mostly replaced with static imports in form in
`internal:[path to file from repo root]`.
To specify if files are ESM, an `esm` method has been added to
`Extension`, similar to the `js` method.
A new ModuleLoader called `InternalModuleLoader` has been added to
enable the loading of internal specifiers, which is used in all
situations except when a snapshot is only loaded, and not a new one is
created from it.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Turns out we were cloning permissions which after prompting were discarded,
so the state of permissions was never preserved. To handle that we need to store
all permissions behind "Arc<Mutex<>>" (because there are situations where we
need to send them to other thread).
Testing and benching code still uses "Permissions" in most places - it's undesirable
to share the same permission set between various test/bench files - otherwise
granting or revoking permissions in one file would influence behavior of other test
files.
This commit allows to execute more JS code from extensions when
creating a snapshot from an existing snapshot.
"deno_core::RuntimeOptions::extensions_with_js" field was added
that is used to pass a list of extensions whose both "ops" and
associated JS source should be executed upon start.
Co-authored-by: crowlkats <crowlkats@toaxl.com>
This PR fixes a regression that caused deno binaries produced by the CI
release workflows to be larger than expected.
**The problem:** The build script will determine whether the linker
supports the `--export-dynamic-symbol-list` flag by looking at the glibc
version installed on the system. Ubuntu 20.04 ships with glibc 2.31,
which does not support this flag. Upon investigation, I discovered that
the CI pipeline does not use the gcc compiler provided by the
`build-essential` package, and instead uses *clang-14*, which does
support the new flag.
**The solution:** Whenever a custom C Compiler is configured, the build
script now assumes the compiler supports the
`--export-dynamic-symbol-list` flag. This is not always going to be the
case (you could use clang-8, for example), but it puts the onus on the
user making the override to ensure the compiler has support.
This will return deno builds for Linux to their previous size of ~100MB,
and also allow builds under older glibc/gcc versions to succeed. If a
user is compiling deno with a custom compiler that does not support this
new flag, however, their build will fail. I expect this is a rare
scenario, however, and suggest we cross that bridge if and when we come
to it.
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
Fixes the error reported in #16304.
> = note:
/usr/bin/ld:/home/abotella/Projects/deno/cli/generated_symbol_exports_list_linux.def:1:
syntax error in dynamic list
collect2: error: ld returned 1 exit status
This was caused by the format of the symbols list on Linux being
malformed (as the error implies).
The format is documented in ld's
[VERSION](https://sourceware.org/binutils/docs/ld/VERSION.html) as well
as:
> --export-dynamic-symbol-list=file
Specify a --export-dynamic-symbol for each pattern in the
file. The format of the file is the same as the version node
without scope and node name. See VERSION for more
information.
Previously, the format for the Linux symbols list was simply a list of
symbols, now it follows the format:
```
{ symbol_name_a; ...; symbol_name_z };
```
Follow-up to #16208.
- Refactors build.rs behaviour to use `-exported_symbols_list` /
`--export-dynamic-symbol-list`
- Since all build systems now rely on a symbols list file, I have added
`generate_exported_symbols_list`, which derives the symbol list file
depending on the platform, which makes `tools/napi/generate_link_win.js`
redundant.
- Fixes a missed instance of `i8` being used instead of `c_char`
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Changes introduced in #13633 have broken the ability to compile for
linux/aarch64 - specifically the use of a `i8` as a char type, which is
an `u8` on linux/aarch64.
This PR:
- Replaces instances of `i8` with the architecture-aware wrapper type
`c_char`
- Skips the use of `--export-dynamic-symbol` on linux-aarch64, because
the target environments often rely on older libc/binutils versions
The current location was causing failures during v1.26.1 publication.
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
Currently, we use `-rdynamic` for exporting Node API symbols to the
symbol table. `-rdynamic` will export *all* symbols, that means
previously unused functions will not be optimized away introducing a lot
of binary bloat.
This patch uses `-exported_symbol` and `--export-dynamic-symbol` link
flags (not as universal as `-rdynamic`) to only mark Node API symbols to
be put in the dynamic symbol table.
This PR implements the NAPI for loading native modules into Deno.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: DjDeveloper <43033058+DjDeveloperr@users.noreply.github.com>
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
