1
0
Fork 0
mirror of https://github.com/denoland/deno.git synced 2025-01-07 06:46:59 -05:00
denoland-deno/ext/node/polyfills
Aapo Alasuutari 9d6584c16f
perf(ext/node): Optimise Buffer string operations (#20158)
Extracted from https://github.com/denoland/deno/pull/17815

Optimise Buffer's string operations, most significantly when dealing
with ASCII and UTF-16. Base64 and HEX encodings are affected to much
lesser degrees.

## Performance

### String length 15
With very small strings we're at break-even or sometimes even lose a tad
bit of performance from creating a `DataView` that ends up not paying
for itself.

**This PR:**
```
benchmark                                                     time (avg)        iter/s             (min … max)       p75       p99      p995
-------------------------------------------------------------------------------------------------------------- -----------------------------
Buffer.from ascii string                                       1.15 µs/iter     871,388.6   (728.78 ns … 1.56 µs)   1.23 µs   1.56 µs   1.56 µs
Buffer.from base64 string                                      1.63 µs/iter     612,790.9     (1.31 µs … 1.96 µs)   1.77 µs   1.96 µs   1.96 µs
Buffer.from utf16 string                                       1.41 µs/iter     707,396.3   (915.24 ns … 1.93 µs)   1.61 µs   1.93 µs   1.93 µs
Buffer.from hex string                                         1.87 µs/iter     535,357.9     (1.56 µs … 2.19 µs)      2 µs   2.19 µs   2.19 µs
Buffer.toString ascii string                                 154.58 ns/iter   6,469,162.8    (149.69 ns … 198 ns) 154.51 ns 182.89 ns 191.91 ns
Buffer.toString base64 string                                161.65 ns/iter   6,186,189.6 (150.91 ns … 181.15 ns) 165.18 ns 171.87 ns 174.94 ns
Buffer.toString utf16 string                                 292.74 ns/iter   3,415,959.8 (285.43 ns … 312.47 ns) 295.25 ns 310.47 ns 312.47 ns
Buffer.toString hex string                                    89.61 ns/iter  11,159,315.6  (81.09 ns … 123.77 ns)  91.09 ns 113.62 ns 119.28 ns
```

**Main:**
```
benchmark                                                     time (avg)        iter/s             (min … max)       p75       p99      p995
-------------------------------------------------------------------------------------------------------------- -----------------------------
Buffer.from ascii string                                       1.26 µs/iter     794,875.8     (1.07 µs … 1.46 µs)   1.31 µs   1.46 µs   1.46 µs
Buffer.from base64 string                                      1.65 µs/iter     607,853.3     (1.38 µs … 2.01 µs)   1.69 µs   2.01 µs   2.01 µs
Buffer.from utf16 string                                       1.34 µs/iter     744,894.6     (1.09 µs … 1.55 µs)   1.45 µs   1.55 µs   1.55 µs
Buffer.from hex string                                         2.01 µs/iter     496,345.8      (1.54 µs … 2.6 µs)   2.26 µs    2.6 µs    2.6 µs
Buffer.toString ascii string                                 150.16 ns/iter   6,659,630.5 (144.99 ns … 166.68 ns)  152.4 ns 157.26 ns 159.14 ns
Buffer.toString base64 string                                164.73 ns/iter   6,070,692.0 (158.77 ns … 185.63 ns) 168.48 ns 175.74 ns 176.68 ns
Buffer.toString utf16 string                                 150.61 ns/iter   6,639,864.0  (148.2 ns … 168.29 ns) 150.93 ns 157.21 ns 168.15 ns
Buffer.toString hex string                                    94.21 ns/iter  10,614,972.9   (86.21 ns … 98.75 ns)  95.43 ns  97.99 ns  98.21 ns
```

### String length 1500
With moderate lengths we already see great upsides for `Buffer.from()`
with ASCII and UTF-16.

**This PR:**
```
benchmark                                                     time (avg)        iter/s             (min … max)       p75       p99      p995
-------------------------------------------------------------------------------------------------------------- -----------------------------
Buffer.from ascii string                                       5.79 µs/iter     172,562.6     (4.72 µs … 4.71 ms)   5.04 µs   10.3 µs  11.67 µs
Buffer.from base64 string                                      5.08 µs/iter     196,678.9     (4.97 µs … 5.76 µs)   5.08 µs   5.76 µs   5.76 µs
Buffer.from utf16 string                                       9.68 µs/iter     103,316.5     (7.14 µs … 3.44 ms)  10.32 µs  13.42 µs  15.21 µs
Buffer.from hex string                                         53.7 µs/iter      18,620.2     (49.37 µs … 2.2 ms)  54.74 µs   72.2 µs  81.07 µs
Buffer.toString ascii string                                   6.63 µs/iter     150,761.3     (5.59 µs … 1.11 ms)   6.08 µs  15.68 µs  24.77 µs
Buffer.toString base64 string                                460.57 ns/iter   2,171,224.4 (448.33 ns … 511.73 ns) 465.05 ns 495.54 ns 511.73 ns
Buffer.toString utf16 string                                   6.52 µs/iter     153,287.0     (6.47 µs … 6.66 µs)   6.53 µs   6.66 µs   6.66 µs
Buffer.toString hex string                                     3.68 µs/iter     271,965.4     (3.64 µs … 3.82 µs)   3.68 µs   3.82 µs   3.82 µs
```

**Main:**
```
benchmark                                                     time (avg)        iter/s             (min … max)       p75       p99      p995
-------------------------------------------------------------------------------------------------------------- -----------------------------
Buffer.from ascii string                                      11.46 µs/iter      87,298.1    (8.53 µs … 834.1 µs)   9.61 µs  83.31 µs   87.3 µs
Buffer.from base64 string                                       5.4 µs/iter     185,027.8     (5.07 µs … 7.49 µs)   5.44 µs   7.49 µs   7.49 µs
Buffer.from utf16 string                                       20.3 µs/iter      49,270.8  (13.55 µs … 649.11 µs)   18.8 µs 113.93 µs 125.17 µs
Buffer.from hex string                                        52.03 µs/iter      19,218.9    (48.74 µs … 2.59 ms)  52.84 µs  67.05 µs  73.56 µs
Buffer.toString ascii string                                   6.46 µs/iter     154,822.5     (6.32 µs … 6.69 µs)   6.52 µs   6.69 µs   6.69 µs
Buffer.toString base64 string                                440.19 ns/iter   2,271,764.6    (427 ns … 490.77 ns) 444.74 ns 484.64 ns 490.77 ns
Buffer.toString utf16 string                                   6.89 µs/iter     145,106.7     (6.81 µs … 7.24 µs)   6.91 µs   7.24 µs   7.24 µs
Buffer.toString hex string                                     3.66 µs/iter     273,456.5      (3.6 µs … 4.02 µs)   3.64 µs   4.02 µs   4.02 µs
```

### String length 2^20
With massive lengths we the difference in ASCII and UTF-16 parsing
performance is enormous.

**This PR:**
```
benchmark                                                           time (avg)        iter/s             (min … max)       p75       p99      p995
-------------------------------------------------------------------------------------------------------------------- -----------------------------
Buffer.from ascii string                                              4.1 ms/iter         243.7     (2.64 ms … 6.74 ms)   4.43 ms   6.26 ms   6.74 ms
Buffer.from base64 string                                            3.74 ms/iter         267.6     (2.91 ms … 4.92 ms)   3.96 ms   4.31 ms   4.92 ms
Buffer.from utf16 string                                             7.72 ms/iter         129.5    (5.91 ms … 11.03 ms)   7.97 ms  11.03 ms  11.03 ms
Buffer.from hex string                                              35.72 ms/iter          28.0   (34.71 ms … 38.42 ms)  35.93 ms  38.42 ms  38.42 ms
Buffer.toString ascii string                                        78.92 ms/iter          12.7   (42.72 ms … 94.13 ms)  91.64 ms  94.13 ms  94.13 ms
Buffer.toString base64 string                                      833.62 µs/iter       1,199.6   (638.05 µs … 5.97 ms) 826.86 µs   2.45 ms   2.48 ms
Buffer.toString utf16 string                                        79.35 ms/iter          12.6    (69.72 ms … 88.9 ms)  86.66 ms   88.9 ms   88.9 ms
Buffer.toString hex string                                          31.04 ms/iter          32.2      (4.3 ms … 46.9 ms)  37.21 ms   46.9 ms   46.9 ms
```

**Main:**
```
benchmark                                                           time (avg)        iter/s             (min … max)       p75       p99      p995
-------------------------------------------------------------------------------------------------------------------- -----------------------------
Buffer.from ascii string                                            18.66 ms/iter          53.6   (15.61 ms … 23.26 ms)  20.62 ms  23.26 ms  23.26 ms
Buffer.from base64 string                                             4.7 ms/iter         212.9     (2.94 ms … 9.07 ms)   4.65 ms   9.06 ms   9.07 ms
Buffer.from utf16 string                                            33.49 ms/iter          29.9   (31.24 ms … 35.67 ms)  34.08 ms  35.67 ms  35.67 ms
Buffer.from hex string                                              39.38 ms/iter          25.4   (38.66 ms … 42.36 ms)  39.58 ms  42.36 ms  42.36 ms
Buffer.toString ascii string                                        77.68 ms/iter          12.9   (67.46 ms … 95.68 ms)  84.71 ms  95.68 ms  95.68 ms
Buffer.toString base64 string                                      825.53 µs/iter       1,211.3   (655.38 µs … 6.69 ms) 816.62 µs   3.07 ms   3.13 ms
Buffer.toString utf16 string                                        76.54 ms/iter          13.1    (66.9 ms … 85.26 ms)  83.63 ms  85.26 ms  85.26 ms
Buffer.toString hex string                                          38.56 ms/iter          25.9   (33.83 ms … 46.56 ms)  45.33 ms  46.56 ms  46.56 ms
```
2023-09-07 14:41:16 -06:00
..
_fs chore: upgrade to dprint 0.39 (#19768) 2023-07-08 18:34:08 +00:00
_process fix(node): Bump hardcoded version to latest (#20366) 2023-09-04 12:33:15 +02:00
_util feat: support import attributes (#20342) 2023-09-07 09:09:16 -04:00
assert refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
dns refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
fs refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
internal perf(ext/node): Optimise Buffer string operations (#20158) 2023-09-07 14:41:16 -06:00
internal_binding perf(ext/node): Optimise Buffer string operations (#20158) 2023-09-07 14:41:16 -06:00
path chore(ext/node): disable prefer-primordials on a per-file basis (#19553) 2023-06-27 15:18:22 +09:00
readline chore(ext/node): disable prefer-primordials on a per-file basis (#19553) 2023-06-27 15:18:22 +09:00
stream fix(ext/node): fix stream/promises export (#19820) 2023-07-17 22:10:34 +09:00
timers refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
util refactor: rename InternalModuleLoader to ExtModuleLoader, use ext: scheme for snapshotted modules (#18041) 2023-03-08 12:44:54 +01:00
00_globals.js feat(ext/node): properly segregate node globals (#19307) 2023-07-19 10:30:04 +02:00
01_require.js fix(require): use canonicalized path for loading content (#20133) 2023-08-15 09:10:54 +02:00
02_init.js feat(ext/node): eagerly bootstrap node (#20153) 2023-08-16 04:36:36 +09:00
_brotli.js refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
_events.d.ts feat(ext/node): embed std/node into the snapshot (#17724) 2023-02-14 17:38:45 +01:00
_events.mjs chore(ext/node): disable prefer-primordials on a per-file basis (#19553) 2023-06-27 15:18:22 +09:00
_global.d.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
_http_agent.mjs refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
_http_common.ts chore(ext/node): disable prefer-primordials on a per-file basis (#19553) 2023-06-27 15:18:22 +09:00
_http_outgoing.ts feat: support import attributes (#20342) 2023-09-07 09:09:16 -04:00
_next_tick.ts chore(ext/node): disable prefer-primordials on a per-file basis (#19553) 2023-06-27 15:18:22 +09:00
_readline.d.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
_readline.mjs refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
_readline_shared_types.d.ts refactor: rename InternalModuleLoader to ExtModuleLoader, use ext: scheme for snapshotted modules (#18041) 2023-03-08 12:44:54 +01:00
_stream.d.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
_stream.mjs refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
_tls_common.ts feat(ext/node): embed std/node into the snapshot (#17724) 2023-02-14 17:38:45 +01:00
_tls_wrap.ts fix(node): implement TLSSocket._start (#20120) 2023-08-11 11:57:41 +00:00
_utils.ts chore(ext/node): disable prefer-primordials on a per-file basis (#19553) 2023-06-27 15:18:22 +09:00
_zlib.mjs fix(ext/node): support dictionary option in zlib init (#20035) 2023-08-11 11:42:35 +00:00
_zlib_binding.mjs fix(ext/node): support dictionary option in zlib init (#20035) 2023-08-11 11:42:35 +00:00
assert.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
assertion_error.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
async_hooks.ts fix(node/async_hooks): don't pop async context frame if stack if empty (#20077) 2023-08-10 09:00:25 +05:30
buffer.ts refactor: rename InternalModuleLoader to ExtModuleLoader, use ext: scheme for snapshotted modules (#18041) 2023-03-08 12:44:54 +01:00
child_process.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
cluster.ts refactor: rename InternalModuleLoader to ExtModuleLoader, use ext: scheme for snapshotted modules (#18041) 2023-03-08 12:44:54 +01:00
console.ts chore(ext/node): disable prefer-primordials on a per-file basis (#19553) 2023-06-27 15:18:22 +09:00
constants.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
crypto.ts chore(ext/node): disable prefer-primordials on a per-file basis (#19553) 2023-06-27 15:18:22 +09:00
dgram.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
diagnostics_channel.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
dns.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
domain.ts refactor: rename InternalModuleLoader to ExtModuleLoader, use ext: scheme for snapshotted modules (#18041) 2023-03-08 12:44:54 +01:00
events.ts refactor: rename InternalModuleLoader to ExtModuleLoader, use ext: scheme for snapshotted modules (#18041) 2023-03-08 12:44:54 +01:00
fs.ts refactor: rename InternalModuleLoader to ExtModuleLoader, use ext: scheme for snapshotted modules (#18041) 2023-03-08 12:44:54 +01:00
http.ts fix(node/http): don't leak resources on destroyed request (#20040) 2023-08-29 12:13:58 +00:00
http2.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
https.ts fix(node/http): add encrypted field to FakeSocket (#19886) 2023-07-21 02:18:07 +02:00
inspector.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
net.ts fix(node/net): Server connection callback include socket value (#19779) 2023-07-19 00:33:43 +02:00
os.ts fix(ext/node): allow for the reassignment of userInfo() on Windows (#20165) 2023-08-16 11:28:49 +02:00
path.ts fix(path): Remove non node symbols (#18630) 2023-04-18 10:44:25 +02:00
perf_hooks.ts fix(ext/node): Define performance.timeOrigin as getter property (#19714) 2023-07-04 20:19:18 +03:00
process.ts fix(ext/node): fix argv[1] in Worker (#20305) 2023-08-29 12:18:25 +09:00
punycode.ts refactor: rename InternalModuleLoader to ExtModuleLoader, use ext: scheme for snapshotted modules (#18041) 2023-03-08 12:44:54 +01:00
querystring.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
readline.ts refactor: rename InternalModuleLoader to ExtModuleLoader, use ext: scheme for snapshotted modules (#18041) 2023-03-08 12:44:54 +01:00
README.md chore: update commonjs loading docs (#19904) 2023-07-22 05:48:06 +02:00
repl.ts fix(node): repl._builtinLibs (#20046) 2023-08-04 14:30:48 +02:00
stream.ts refactor: rename InternalModuleLoader to ExtModuleLoader, use ext: scheme for snapshotted modules (#18041) 2023-03-08 12:44:54 +01:00
string_decoder.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
sys.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
testing.ts fix(node): node:test reports correct location (#20025) 2023-08-02 17:11:04 +02:00
timers.ts fix(ext/node): ignore cancelled timer when node timer refresh (#19637) 2023-07-02 19:11:34 +00:00
tls.ts chore(ext/node): disable prefer-primordials on a per-file basis (#19553) 2023-06-27 15:18:22 +09:00
tty.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
url.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
util.ts fix(fmt): do not insert expr stmt leading semi-colon in do while stmt body (#20093) 2023-08-08 09:15:19 -04:00
v8.ts chore(ext/node): disable prefer-primordials on a per-file basis (#19553) 2023-06-27 15:18:22 +09:00
vm.ts chore(ext/node): disable prefer-primordials on a per-file basis (#19553) 2023-06-27 15:18:22 +09:00
wasi.ts chore(ext/node): disable prefer-primordials on a per-file basis (#19553) 2023-06-27 15:18:22 +09:00
worker_threads.ts refactor: rename built-in node modules from ext:deno_node/ to node: (#19680) 2023-07-02 20:19:30 +02:00
zlib.ts fix(ext/node): support brotli APIs (#19223) 2023-06-24 19:42:08 +05:30

Deno Node.js compatibility

This module is meant to have a compatibility layer for the Node.js standard library.

Warning: Any function of this module should not be referred anywhere in the Deno standard library as it's a compatibility module.

Supported modules

  • assert
  • assert/strict partly
  • async_hooks partly
  • buffer
  • child_process partly
  • cluster partly
  • console partly
  • constants partly
  • crypto partly
  • dgram partly
  • diagnostics_channel partly
  • dns partly
  • events
  • fs partly
  • fs/promises partly
  • http partly
  • http2
  • https partly
  • inspector partly
  • module
  • net
  • os partly
  • path
  • path/posix
  • path/win32
  • perf_hooks
  • process partly
  • punycode
  • querystring
  • readline
  • repl partly
  • stream
  • stream/promises
  • stream/web partly
  • string_decoder
  • sys
  • timers
  • timers/promises
  • tls
  • trace_events
  • tty partly
  • url
  • util partly
  • util/types partly
  • v8
  • vm partly
  • wasi
  • webcrypto
  • worker_threads
  • zlib
  • node globals partly

Deprecated

These modules are deprecated in Node.js and will probably not be polyfilled:

  • domain
  • freelist

Experimental

These modules are experimental in Node.js and will not be polyfilled until they are stable:

  • diagnostics_channel
  • async_hooks
  • policies
  • trace_events
  • wasi
  • webcrypto

CommonJS modules loading

createRequire(...) is provided to create a require function for loading CJS modules. It also sets supported globals.

import { createRequire } from "node:module";

const require = createRequire(import.meta.url);
// Loads native module polyfill.
const path = require("path");
// Loads extensionless module.
const cjsModule = require("./my_mod");
// Visits node_modules.
const leftPad = require("left-pad");

Contributing

Setting up the test runner and running tests

See tools/node_compat/README.md.

Best practices

When converting from promise-based to callback-based APIs, the most obvious way is like this:

promise.then((value) => callback(null, value)).catch(callback);

This has a subtle bug - if the callback throws an error, the catch statement will also catch that error, and the callback will be called twice. The correct way to do it is like this:

promise.then((value) => callback(null, value), callback);

The second parameter of then can also be used to catch errors, but only errors from the existing promise, not the new one created by the callback.

If the Deno equivalent is actually synchronous, there's a similar problem with try/catch statements:

try {
  const value = process();
  callback(null, value);
} catch (err) {
  callback(err);
}

Since the callback is called within the try block, any errors from it will be caught and call the callback again.

The correct way to do it is like this:

let err, value;
try {
  value = process();
} catch (e) {
  err = e;
}
if (err) {
  callback(err); // Make sure arguments.length === 1
} else {
  callback(null, value);
}

It's not as clean, but prevents the callback being called twice.

Remaining Tests

Node compatibility can be measured by how many native Node tests pass. If you'd like to know what you can work on, check out the list of Node tests remaining here.