When using the `eval` option on Node's `worker_threads` the code is
passed as a `data:` URL. But we didn't encode the actual code for that,
which lead to syntax errors when including characters not allowed in an
URL.
Fixes a part of https://github.com/denoland/deno/issues/27167
Fixes #22995. Fixes #23000.
There were a handful of bugs here causing the hang (each with a
corresponding minimized test):
- We were canceling recv futures when `receiveMessageOnPort` was called,
but this caused the "receive loop" in the message port to exit. This was
due to the fact that `CancelHandle`s are never reset (i.e., once you
`cancel` a `CancelHandle`, it remains cancelled). That meant that after
`receieveMessageOnPort` was called, the subsequent calls to
`op_message_port_recv_message` would throw `Interrupted` exceptions, and
we would exit the loop.
The cancellation, however, isn't actually necessary.
`op_message_port_recv_message` only borrows the underlying port for long
enough to poll the receiver, so the borrow there could never overlap
with `op_message_port_recv_message_sync`.
- Calling `MessagePort.unref()` caused the "receive loop" in the message
port to exit. This was because we were setting
`messageEventListenerCount` to 0 on unref. Not only does that break the
counter when multiple `MessagePort`s are present in the same thread, but
we also exited the "receive loop" whenever the listener count was 0. I
assume this was to prevent the recv promise from keeping the event loop
open.
Instead of this, I chose to just unref the recv promise as needed to
control the event loop.
- The last bug causing the hang (which was a doozy to debug) ended up
being an unfortunate interaction between how we implement our
messageport "receive loop" and a pattern found in `npm:piscina` (which
angular uses). The gist of it is that piscina uses an atomic wait loop
along with `receiveMessageOnPort` in its worker threads, and as the
worker is getting started, the following incredibly convoluted series of
events occurs:
1. Parent sends a MessagePort `p` to worker
2. Parent sends a message `m` to the port `p`
3. Parent notifies the worker with `Atomics.notify` that a new message
is available
4. Worker receives message, adds "message" listener to port `p`
5. Adding the listener triggers `MessagePort.start()` on `p`
6. Receive loop in MessagePort.start receives the message `m`, but then
hits an await point and yields (before dispatching the "message" event)
7. Worker continues execution, starts the atomic wait loop, and
immediately receives the existing notification from the parent that a
message is available
8. Worker attempts to receive the new message `m` with
`receiveMessageOnPort`, but this returns `undefined` because the receive
loop already took the message in 6
9. Atomic wait loop continues to next iteration, waiting for the next
message with `Atomic.wait`
10. `Atomic.wait` blocks the worker thread, which prevents the receive
loop from continuing and dispatching the "message" event for the
received message
11. The parent waits for the worker to respond to the first message, and
waits
12. The thread can't make any more progress, and the whole process hangs
The fix I've chosen here (which I don't particularly love, but it works)
is to just delay the `MessagePort.start` call until the end of the event
loop turn, so that the atomic wait loop receives the message first. This
prevents the hang.
---
Those were the main issues causing the hang. There ended up being a few
other small bugs as well, namely `exit` being emitted multiple times,
and not patching up the message port when it's received by
`receiveMessageOnPort`.
Fixes #23281. Part of #20613.
We were emitting the `online` event in the constructor, so the caller
could never receive it (since there was no time for them to add a
listener). Instead, emit the event where it's intended – after the
worker is initialized.
---
After this parcel no longer freezes, but still will fail due to other
bugs (which will be fixed in other PRs)
Support `MessagePort.once` in Node mode and enable relevant
`worker_threads` test. Noticed that another Node test was passing as
well, so I enabled that too.
<!--
Before submitting a PR, please read
https://docs.deno.com/runtime/manual/references/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
Fixes https://github.com/denoland/deno/issues/23801
---------
Signed-off-by: Marvin Hagemeister <marvinhagemeister50@gmail.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Most common argument to `env` option for `worker_threads.Worker` will be
`process.env`.
In Deno `process.env` is a `Proxy` which can't be cloned using
structured clone algorithm.
So to be safe, I'm creating a copy of actual object before it's sent to
the worker thread.
Ref #23522
Closes https://github.com/denoland/deno/issues/23362
Previously we were panicking if there was a pending read on a
port and `receiveMessageOnPort` was called. This is now fixed
by cancelling the pending read, trying to read a message and
resuming reading in a loop.
MessagePort if directly assigned to workerData property instead of
embedding it in an object then it is not patched to a NodeMessagePort.
This commit fixes the bug.
This commit fixes race condition in "node:worker_threads" module were
the first message did a setup of "threadId", "workerData" and
"environmentData".
Now this data is passed explicitly during workers creation and is set up
before any user code is executed.
Closes https://github.com/denoland/deno/issues/22783
Closes https://github.com/denoland/deno/issues/22672
---------
Co-authored-by: Satya Rohith <me@satyarohith.com>
This commit changes how we figure out if we're running on main
thread in `node:worker_threads` module. Instead of relying on quirky
"magic variable" for a name to check if we're on main thread, we are
now explicitly passing this information during bootstrapping of the
runtime. As a side effect, `WorkerOptions.name` is more useful
and matches what Node.js does more closely (though not fully).
Towards https://github.com/denoland/deno/issues/22783
Follow up to #22157.
This leaves us with 4 usages of `ensureFastOps()` in `deno` itself.
There's also about 150 usages of `Deno.core.ops.<op_name>` left as well.
This PR ensures that node's `worker_threads` module exports
`MessageChannel`, `MessagePort` and the `BroadcastChannel` API. Fixing
these won't make `esbuild` work, but brings us one step closer 🎉
Fixes #19028 .
This commit renames "deno_core::InternalModuleLoader" to
"ExtModuleLoader" and changes the specifiers used by the
modules loaded from this loader to "ext:".
"internal:" scheme was really ambiguous and it's more characters than
"ext:", which should result in slightly smaller snapshot size.
Closes https://github.com/denoland/deno/issues/18020
This commit changes "include_js_files!" macro from "deno_core"
in a way that "dir" option doesn't cause specifiers to be rewritten
to include it.
Example:
```
include_js_files! {
dir "js",
"hello.js",
}
```
The above definition required embedders to use:
`import ... from "internal:<ext_name>/js/hello.js"`.
But with this change, the "js" directory in which the files are stored
is an implementation detail, which for embedders results in:
`import ... from "internal:<ext_name>/hello.js"`.
The directory the files are stored in, is an implementation detail and
in some cases might result in a significant size difference for the
snapshot. As an example, in "deno_node" extension, we store the
source code in "polyfills" directory; which resulted in each specifier
to look like "internal:deno_node/polyfills/<module_name>", but with
this change it's "internal:deno_node/<module_name>".
Given that "deno_node" has over 100 files, many of them having
several import specifiers to the same extension, this change removes
10 characters from each import specifier.
This commit moves "deno_std/node" in "ext/node" crate. The code is
transpiled and snapshotted during the build process.
During the first pass a minimal amount of work was done to create the
snapshot, a lot of code in "ext/node" depends on presence of "Deno"
global. This code will be gradually fixed in the follow up PRs to migrate
it to import relevant APIs from "internal:" modules.
Currently the code from snapshot is not used in any way, and all
Node/npm compatibility still uses code from
"https://deno.land/std/node" (or from the location specified by
"DENO_NODE_COMPAT_URL"). This will also be handled in a follow
up PRs.
---------
Co-authored-by: crowlkats <crowlkats@toaxl.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
