Follow-up to #18210:
* we are passing the generated `cfg` object into the state function
rather than passing individual config fields
* reduce cloning dramatically by making the state_fn `FnOnce`
* `take` for `ExtensionBuilder` to avoid more unnecessary copies
* renamed `config` to `options`
This implements two macros to simplify extension registration and centralize a lot of the boilerplate as a base for future improvements:
* `deno_core::ops!` registers a block of `#[op]`s, optionally with type
parameters, useful for places where we share lists of ops
* `deno_core::extension!` is used to register an extension, and creates
two methods that can be used at runtime/snapshot generation time:
`init_ops` and `init_ops_and_esm`.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit removes "deno_core::RuntimeOptions::extensions_with_js".
Now it's embedders' responsibility to properly register extensions
that will not contains JavaScript sources when running from an existing
snapshot.
Prerequisite for https://github.com/denoland/deno/pull/18080
This commit splits "<ext_name>::init" functions into "init_ops" and
"init_ops_and_esm". That way we don't have to construct list of
ESM sources on each startup if we're running with a snapshot.
In a follow up commit "deno_core" will be changed to not have a split
between "extensions" and "extensions_with_js" - it will be embedders'
responsibility to pass appropriately configured extensions.
Prerequisite for https://github.com/denoland/deno/pull/18080
This commit renames "deno_core::InternalModuleLoader" to
"ExtModuleLoader" and changes the specifiers used by the
modules loaded from this loader to "ext:".
"internal:" scheme was really ambiguous and it's more characters than
"ext:", which should result in slightly smaller snapshot size.
Closes https://github.com/denoland/deno/issues/18020
There's no point for this API to expect result. If something fails it should
result in a panic during build time to signal to embedder that setup is
wrong.
This API is required by several extensions like "ext/node", "ext/ffi"
and also FS APIs that we want to move to a separate crate. Because
of that "pathFromURL" API was moved to "deno_web" extension so
other extension crates can rely on it.
Fixes denoland#16922.
The error messages in the `ffi` module are somewhat cryptic when passing
functions that have invalid `parameters` or `result` type strings. While
the generated serializer for the `ForeignFunction` struct correctly
outputs a correct and verbose message, the user sees a far less helpful
`data did not match any variant` message instead.
The underlying cause appears to be the fallback message in the
auto-derived deserializer for untagged enums [1] generated as a result
of `ForeignSymbol` being marked as `#[serde(untagged)]` [2]. Passing an
unexpected value for `NativeType` causes it to error out while
attempting to deserialize both enum variants -- once because it's not a
match for the `ForeignStatic` variant, and once because the
`ForeignFunction` deserializer rejects the invalid type for the
parameters/return type. This is currently open as [serde
#773](https://github.com/serde-rs/serde/issues/773), and not a trivial
exercise to fix generically.
[1]
https://github.com/serde-rs/serde/blob/v0.9.7/serde_derive/src/de.rs#L730
[2] https://github.com/denoland/deno/blob/main/ext/ffi/dlfcn.rs#L102
[3] https://github.com/serde-rs/serde/issues/773
Note that the auto-generated deserializer for untagged enums uses a
private API to buffer deserializer content that we don't have access to.
Instead, we can make use of the `serde_value` crate to buffer the
values. This can likely be removed once the official buffering API lands
(see [4] and [5]). In addition, this crate pulls in `serde_json` as a
cheap way to test that the deserializer works properly.
[4] https://github.com/serde-rs/serde/issues/741
[5] https://github.com/serde-rs/serde/pull/2348
This PR refactors all internal js files (except core) to be written as
ES modules.
`__bootstrap`has been mostly replaced with static imports in form in
`internal:[path to file from repo root]`.
To specify if files are ESM, an `esm` method has been added to
`Extension`, similar to the `js` method.
A new ModuleLoader called `InternalModuleLoader` has been added to
enable the loading of internal specifiers, which is used in all
situations except when a snapshot is only loaded, and not a new one is
created from it.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This patch makes `NativeType` to `libffi::middle::Type` conversion
failliable and w.t disallows struct with empty fields. libffi does not
handle "empty" struct because they don't exist in C (or Rust).
Fixes #17481
Updated third_party dlint to v0.37.0 for GitHub Actions. This PR
includes following changes:
* fix(prefer-primordials): Stop using array pattern assignments
* fix(prefer-primordials): Stop using global intrinsics except for
`SharedArrayBuffer`
* feat(guard-for-in): Apply new guard-for-in rule
Adds support for passing and returning structs as buffers to FFI. This does not implement fastapi support for structs. Needed for certain system APIs such as AppKit on macOS.
- [x] `dlfcn.rs` - `dlopen()`-related code.
- [x] `turbocall.rs` - Call trampoline JIT compiler.
- [x] `repr.rs` - Pointer representation. Home of the UnsafePointerView
ops.
- [x] `symbol.rs` - Function symbol related code.
- [x] `callback.rs` - Home of `Deno.UnsafeCallback` ops.
- [x] `ir.rs` - Intermediate representation for values. Home of the
`NativeValue` type.
- [x] `call.rs` - Generic call ops. Home to everything related to
calling FFI symbols.
- [x] `static.rs` - static symbol support
I find easier to work with this setup, I eventually want to expand
TurboCall to unroll type conversion loop in generic calls, generate code
for individual symbols (lazy function pointers), etc.
Fixes https://github.com/denoland/deno/issues/16934
Example compiler error:
```
error: mutable opstate is not supported in async ops
--> core/ops_builtin.rs:122:1
|
122 | #[op]
| ^^^^^
|
= note: this error originates in the attribute macro `op` (in Nightly builds, run with -Z macro-backtrace for more info)
```
Currently, slow call path will always create a dangling pointer to
replace a null pointer when called with eg. a `new Uint8Array()`
parameter, which V8 initialises as a null pointer backed buffer.
However, the fast call path will never change the pointer value and will
thus expose a null pointer. Thus, it's possible that the pointer value
that a native call sees coming from Deno changes between two sequential
invocations of the same function with the exact same parameters.
Since null pointers can be quite important, and `Uint8Array` is the
chosen fast path for Deno FFI `"buffer"` parameters, I think it is
fairly important that the null pointer be properly exposed to the native
code. Thus this PR.
### `*mut c_void`
While here, I also changed the type of our pointer values to `*mut
c_void`. This is mainly due to JS buffers always being `*mut`, and
because we offer a way to turn a pointer into a JS `ArrayBuffer`
(`op_ffi_get_buf`) which is read-write. I'm not exactly sure which way
we should really go here, we have pointers that are definitely mut but
we also cannot assume all of our pointers are. So, do we go with the
maxima or the minima?
### `optimisedCall(new Uint8Array())`
V8 seems to have a bug where calling an optimised function with a newly
created empty `Uint8Array` (no argument or 0) will not see the data
pointer being null but instead it's some stable pointer, perhaps
pointing to some internal null-backing-store. The pointer value is also
an odd (not even) number, so it might specifically be a tagged pointer.
This will probably be an issue for some users, if they try to use eg.
`method(cstr("something"), new Uint8Array())` as a way to do a fast call
to `method` with a null pointer as the second parameter.
If instead of a `new Uint8Array()` the user instead uses some `const
NULL = new Uint8Array()` where the `NULL` buffer has been passed to a
slow call previously, then the fast call will properly see a null
pointer.
I'll take this up with some V8 engineers to see if this couldn't be
fixed.
