1
0
Fork 0
mirror of https://github.com/denoland/deno.git synced 2024-11-24 15:19:26 -05:00
denoland-deno/ext/node/polyfills
Nathan Whitaker dd8cbf5e29
fix(node): fix worker_threads issues blocking Angular support (#26024)
Fixes #22995. Fixes #23000.

There were a handful of bugs here causing the hang (each with a
corresponding minimized test):

- We were canceling recv futures when `receiveMessageOnPort` was called,
but this caused the "receive loop" in the message port to exit. This was
due to the fact that `CancelHandle`s are never reset (i.e., once you
`cancel` a `CancelHandle`, it remains cancelled). That meant that after
`receieveMessageOnPort` was called, the subsequent calls to
`op_message_port_recv_message` would throw `Interrupted` exceptions, and
we would exit the loop.

The cancellation, however, isn't actually necessary.
`op_message_port_recv_message` only borrows the underlying port for long
enough to poll the receiver, so the borrow there could never overlap
with `op_message_port_recv_message_sync`.

- Calling `MessagePort.unref()` caused the "receive loop" in the message
port to exit. This was because we were setting
`messageEventListenerCount` to 0 on unref. Not only does that break the
counter when multiple `MessagePort`s are present in the same thread, but
we also exited the "receive loop" whenever the listener count was 0. I
assume this was to prevent the recv promise from keeping the event loop
open.

Instead of this, I chose to just unref the recv promise as needed to
control the event loop.

- The last bug causing the hang (which was a doozy to debug) ended up
being an unfortunate interaction between how we implement our
messageport "receive loop" and a pattern found in `npm:piscina` (which
angular uses). The gist of it is that piscina uses an atomic wait loop
along with `receiveMessageOnPort` in its worker threads, and as the
worker is getting started, the following incredibly convoluted series of
events occurs:
   1. Parent sends a MessagePort `p` to worker
   2. Parent sends a message `m` to the port `p`
3. Parent notifies the worker with `Atomics.notify` that a new message
is available
   4. Worker receives message, adds "message" listener to port `p`
   5. Adding the listener triggers `MessagePort.start()` on `p`
6. Receive loop in MessagePort.start receives the message `m`, but then
hits an await point and yields (before dispatching the "message" event)
7. Worker continues execution, starts the atomic wait loop, and
immediately receives the existing notification from the parent that a
message is available
8. Worker attempts to receive the new message `m` with
`receiveMessageOnPort`, but this returns `undefined` because the receive
loop already took the message in 6
9. Atomic wait loop continues to next iteration, waiting for the next
message with `Atomic.wait`
10. `Atomic.wait` blocks the worker thread, which prevents the receive
loop from continuing and dispatching the "message" event for the
received message
11. The parent waits for the worker to respond to the first message, and
waits
12. The thread can't make any more progress, and the whole process hangs

The fix I've chosen here (which I don't particularly love, but it works)
is to just delay the `MessagePort.start` call until the end of the event
loop turn, so that the atomic wait loop receives the message first. This
prevents the hang.

---

Those were the main issues causing the hang. There ended up being a few
other small bugs as well, namely `exit` being emitted multiple times,
and not patching up the message port when it's received by
`receiveMessageOnPort`.
2024-10-04 09:26:32 -07:00
..
_fs Revert "feat(fmt): sort type-only named import/exports last" (#25705) 2024-09-18 13:54:52 +00:00
_process feat(cli): use NotCapable error for permission errors (#25431) 2024-09-10 11:12:24 -07:00
_util fix(runtime): use more null proto objects again (#25040) 2024-09-06 12:52:59 +02:00
assert chore: update copyright to 2024 (#21753) 2024-01-01 19:58:21 +00:00
dns chore: update copyright to 2024 (#21753) 2024-01-01 19:58:21 +00:00
fs fix(node): add cp to fs/promises (#22263) 2024-02-06 12:32:58 +05:30
internal feat(ext/node): buffer.transcode() (#25972) 2024-10-02 08:23:14 +00:00
internal_binding fix(ext/node): avoid showing UNKNOWN error from TCP handle (#25550) 2024-09-11 19:19:02 +09:00
path fix(ext/node): Add missing node:path exports (#25567) 2024-09-11 00:00:08 +00:00
readline chore: update copyright to 2024 (#21753) 2024-01-01 19:58:21 +00:00
stream feat(ext/node): export more symbols from streams and timers/promises (#25582) 2024-09-12 01:02:50 +02:00
timers feat(ext/node): export 'promises' symbol from 'node:timers' (#25589) 2024-09-12 20:30:49 +00:00
util chore: update copyright to 2024 (#21753) 2024-01-01 19:58:21 +00:00
00_globals.js chore: update copyright to 2024 (#21753) 2024-01-01 19:58:21 +00:00
01_require.js fix: cjs resolution cases (#25739) 2024-09-19 21:10:34 -07:00
02_init.js fix(ext/node): attach console stream properties (#25617) 2024-09-13 17:28:35 -07:00
_brotli.js fix(runtime): use more null proto objects again (#25040) 2024-09-06 12:52:59 +02:00
_events.d.ts chore: update copyright to 2024 (#21753) 2024-01-01 19:58:21 +00:00
_events.mjs feat(ext/node): add abort helpers, process & streams fix (#25262) 2024-09-05 09:22:52 +02:00
_global.d.ts feat(ext/node): rewrite crypto keys (#24463) 2024-08-07 08:43:58 +02:00
_http_agent.mjs chore: update the copyright of c, mjs, and jsx (#25005) 2024-08-12 12:41:32 -04:00
_http_common.ts feat(ext/node): expose ES modules for _ modules (#25588) 2024-09-12 00:52:08 +00:00
_http_outgoing.ts feat(ext/node): expose ES modules for _ modules (#25588) 2024-09-12 00:52:08 +00:00
_http_server.ts feat(ext/node): expose ES modules for _ modules (#25588) 2024-09-12 00:52:08 +00:00
_next_tick.ts fix: reland async context (#25140) 2024-08-29 02:25:38 +00:00
_readline.d.ts chore: update copyright to 2024 (#21753) 2024-01-01 19:58:21 +00:00
_readline.mjs chore: update the copyright of c, mjs, and jsx (#25005) 2024-08-12 12:41:32 -04:00
_readline_shared_types.d.ts chore: update copyright to 2024 (#21753) 2024-01-01 19:58:21 +00:00
_stream.d.ts chore: update copyright to 2024 (#21753) 2024-01-01 19:58:21 +00:00
_stream.mjs feat(ext/node): export more symbols from streams and timers/promises (#25582) 2024-09-12 01:02:50 +02:00
_tls_common.ts chore: update copyright to 2024 (#21753) 2024-01-01 19:58:21 +00:00
_tls_wrap.ts feat(ext/node): expose ES modules for _ modules (#25588) 2024-09-12 00:52:08 +00:00
_utils.ts chore: enable no-console dlint rule (#25113) 2024-08-20 15:14:37 -04:00
_zlib.mjs chore: update the copyright of c, mjs, and jsx (#25005) 2024-08-12 12:41:32 -04:00
_zlib_binding.mjs chore: update the copyright of c, mjs, and jsx (#25005) 2024-08-12 12:41:32 -04:00
assert.ts feat: vm rewrite (#24596) 2024-08-06 12:52:53 +00:00
assertion_error.ts feat: Deno.{stdin,stdout,stderr}.isTerminal(), deprecate Deno.isatty() (#22011) 2024-01-24 00:01:56 +01:00
async_hooks.ts feat(ext/node): export missing symbols from domain, puncode, repl, tls (#25585) 2024-09-12 02:25:46 +02:00
buffer.ts feat(ext/node): buffer.transcode() (#25972) 2024-10-02 08:23:14 +00:00
child_process.ts fix(node): Pass NPM_PROCESS_STATE to subprocesses via temp file instead of env var (#25896) 2024-09-27 12:35:37 -07:00
cluster.ts fix(node/cluster): improve stubs to make log4js work (#25146) 2024-08-26 11:19:58 +02:00
console.ts fix(ext/node): use primordials in ext/node/polyfills/console.ts (#25572) 2024-09-11 17:55:58 +10:00
constants.ts fix(ext/node): Add missing node:fs and node:constants exports (#25568) 2024-09-11 00:37:23 +00:00
crypto.ts fix(ext/node): add crypto.subtle (#23027) 2024-03-22 18:56:36 +05:30
dgram.ts fix(ext/node/net): emit error before close when connection is refused (#24656) 2024-07-24 20:33:45 +09:00
diagnostics_channel.js fix(ext/node): better support for node:diagnostics_channel module (#24088) 2024-06-15 00:07:02 +02:00
dns.ts fix(ext/node): Match punycode module behavior to node (#22847) 2024-03-11 15:49:43 -07:00
domain.ts feat(ext/node): export missing symbols from domain, puncode, repl, tls (#25585) 2024-09-12 02:25:46 +02:00
events.ts feat(ext/node): add abort helpers, process & streams fix (#25262) 2024-09-05 09:22:52 +02:00
fs.ts fix(node): Implement fs.lchown (and process.getegid) (#24418) 2024-07-05 18:32:51 +00:00
http.ts feat(ext/node): expose ES modules for _ modules (#25588) 2024-09-12 00:52:08 +00:00
http2.ts fix(ext/node): export request and response clases from http2 module (#25592) 2024-09-12 02:03:57 +00:00
https.ts Revert "feat(fmt): sort type-only named import/exports last" (#25705) 2024-09-18 13:54:52 +00:00
inspector.ts fix(node/inspector): Session constructor should not throw (#25041) 2024-08-14 13:34:24 +00:00
net.ts fix(ext/node): fix process.stdin.pause() (#25864) 2024-09-26 08:47:29 +05:30
os.ts refactor(ext): throw new error instead of throw error (#25272) 2024-08-28 22:40:37 +02:00
path.ts chore: update copyright to 2024 (#21753) 2024-01-01 19:58:21 +00:00
perf_hooks.ts fix(ext/node): don't throw when calling PerformanceObserver.observe (#25036) 2024-08-14 13:03:53 +02:00
process.ts fix(ext/node): don't throw error for unsupported signal binding on windows (#25699) 2024-09-19 12:22:01 +09:00
punycode.ts feat(ext/node): export missing symbols from domain, puncode, repl, tls (#25585) 2024-09-12 02:25:46 +02:00
querystring.js fix(ext/node): querystring stringify without encode callback (#21740) 2024-01-02 04:54:11 +00:00
readline.ts chore: update copyright to 2024 (#21753) 2024-01-01 19:58:21 +00:00
README.md chore: move tools/node_compat to tests/node_compat/runner (#23025) 2024-04-03 09:24:55 +11:00
repl.ts feat(ext/node): export missing symbols from domain, puncode, repl, tls (#25585) 2024-09-12 02:25:46 +02:00
stream.ts feat(ext/node): export more symbols from streams and timers/promises (#25582) 2024-09-12 01:02:50 +02:00
string_decoder.ts fix(runtime): use more null proto objects again (#25040) 2024-09-06 12:52:59 +02:00
sys.ts chore: update copyright to 2024 (#21753) 2024-01-01 19:58:21 +00:00
testing.ts chore: enable no-console dlint rule (#25113) 2024-08-20 15:14:37 -04:00
timers.ts feat(ext/node): export 'promises' symbol from 'node:timers' (#25589) 2024-09-12 20:30:49 +00:00
tls.ts feat(ext/node): add rootCertificates to node:tls (#25707) 2024-09-18 21:14:26 +02:00
trace_events.ts fix(ext/node): add stubs for node:trace_events (#25628) 2024-09-15 08:15:09 +05:30
tty.js fix(node/tty): fix tty.WriteStream.hasColor with different args (#25094) 2024-08-19 17:13:09 +02:00
url.ts chore: sync up Node.js test files for v20.11.1 (#24066) 2024-06-11 11:41:44 +00:00
util.ts fix(runtime): use more null proto objects again (#25040) 2024-09-06 12:52:59 +02:00
v8.ts fix(ext/node): remove unimplemented promiseHook stubs (#25979) 2024-10-02 12:52:05 +05:30
vm.js fix(ext/node): add vm.constants (#25630) 2024-09-15 08:15:28 +05:30
wasi.ts fix(ext/node): use primordials in ext/node/polyfills/wasi.ts (#25608) 2024-09-13 22:31:07 +05:30
worker_threads.ts fix(node): fix worker_threads issues blocking Angular support (#26024) 2024-10-04 09:26:32 -07:00
zlib.ts feat(ext/node): export missing constants from 'zlib' module (#25584) 2024-09-12 01:12:10 +02:00

Deno Node.js compatibility

This module is meant to have a compatibility layer for the Node.js standard library.

Warning: Any function of this module should not be referred anywhere in the Deno standard library as it's a compatibility module.

Supported modules

  • assert
  • assert/strict partly
  • async_hooks partly
  • buffer
  • child_process partly
  • cluster partly
  • console partly
  • constants partly
  • crypto partly
  • dgram partly
  • diagnostics_channel partly
  • dns partly
  • events
  • fs partly
  • fs/promises partly
  • http partly
  • http2
  • https partly
  • inspector partly
  • module
  • net
  • os partly
  • path
  • path/posix
  • path/win32
  • perf_hooks
  • process partly
  • punycode
  • querystring
  • readline
  • repl partly
  • stream
  • stream/promises
  • stream/web partly
  • string_decoder
  • sys
  • timers
  • timers/promises
  • tls
  • trace_events
  • tty partly
  • url
  • util partly
  • util/types partly
  • v8
  • vm partly
  • wasi
  • webcrypto
  • worker_threads
  • zlib
  • node globals partly

Deprecated

These modules are deprecated in Node.js and will probably not be polyfilled:

  • domain
  • freelist

Experimental

These modules are experimental in Node.js and will not be polyfilled until they are stable:

  • diagnostics_channel
  • async_hooks
  • policies
  • trace_events
  • wasi
  • webcrypto

CommonJS modules loading

createRequire(...) is provided to create a require function for loading CJS modules. It also sets supported globals.

import { createRequire } from "node:module";

const require = createRequire(import.meta.url);
// Loads native module polyfill.
const path = require("path");
// Loads extensionless module.
const cjsModule = require("./my_mod");
// Visits node_modules.
const leftPad = require("left-pad");

Contributing

Setting up the test runner and running tests

See tests/node_compat/runner/README.md.

Best practices

When converting from promise-based to callback-based APIs, the most obvious way is like this:

promise.then((value) => callback(null, value)).catch(callback);

This has a subtle bug - if the callback throws an error, the catch statement will also catch that error, and the callback will be called twice. The correct way to do it is like this:

promise.then((value) => callback(null, value), callback);

The second parameter of then can also be used to catch errors, but only errors from the existing promise, not the new one created by the callback.

If the Deno equivalent is actually synchronous, there's a similar problem with try/catch statements:

try {
  const value = process();
  callback(null, value);
} catch (err) {
  callback(err);
}

Since the callback is called within the try block, any errors from it will be caught and call the callback again.

The correct way to do it is like this:

let err, value;
try {
  value = process();
} catch (e) {
  err = e;
}
if (err) {
  callback(err); // Make sure arguments.length === 1
} else {
  callback(null, value);
}

It's not as clean, but prevents the callback being called twice.

Remaining Tests

Node compatibility can be measured by how many native Node tests pass. If you'd like to know what you can work on, check out the list of Node tests remaining here.