Currently, importing Node-Addons modules in a standalone binary results
in a `missing symbol called` error
(https://github.com/denoland/deno/issues/24614). Because the NAPI
symbols are not exported in this mode. This PR should fix the issue.
Regression from
04f9db5b22
Originally I thought to fix the issue in the PR we needed to explicitly
pass through the `node-modules-dir` flag, but after applying the correct
fix that david pointed out (setting `NPM_PROCESS_STATE`) that wasn't
necessary (or correct).
We had a test for deno task with BYONM, but it only tested with
`"unstable": ["byonm"]` in deno.json, so it didn't catch this.
Originally landed in
f6fd6619e7.
Reverted in https://github.com/denoland/deno/pull/24574.
This reland contains a fix that sends "Accept: */*" header for calls made
from "FileFetcher". Absence of this header made downloading source code
from JSR broken. This is tested by ensuring this header is present in the
test server that servers JSR packages.
---------
Co-authored-by: Sean McArthur <sean@seanmonstar.com>
This makes bare specifiers for npm packages work when inside a
workspace, which emulates the same behaviour as when there's a
node_modules directory. The bare specifier can be overwritten by
specifying an import map entry or package.json dependency entry.
* https://github.com/denoland/deno_config/pull/88
Closes #24605
This makes it easier to tell what kind of error something is (even for
deeply nested errors) and will help in the future once we add error
codes to the JS errors this returns.
This commit re-implements `ext/fetch` and all dependent crates
using `hyper` and `hyper-util`, instead of `reqwest`.
The reasoning is that we want to have greater control and access
to low level `hyper` APIs when implementing `fetch` API as well
as `node:http` module.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
NPM inserts a default install script when a package has a `binding.gyp`
file.
It's possible, however, for the package to exclude the `binding.gyp`
file when they publish, and in this case the install script will never
succeed for a user of the package.
This happens with `fsevents`, for instance. They don't include the
`binding.gyp` file in their published tarball, but the default install
script appears in the manifest served by `npm`.
This causes us to warn that `fsevents` has an install script, but when
you try to run it it fails due to `binding.gyp` not existing.
Previously when we printed out the packages that skipped install
scripts, we didn't prefix them with `npm:`. When you pass
`--allow-scripts` though, we require `npm:`, which means you can't just
copy paste the package name from the warning message.
This commit fixes memory leak described in
https://github.com/denoland/deno/issues/24380.
This is done by upgrading following crates:
- deno_ast
- deno_graph
- eszip
- dprint-plugin-typescript
- deno_lint
- deno_doc
- deno_emit
Fixes #24241
* Support "statfs", "username", "getPriority" and "setPriority" kinds
for `--allow-sys`.
* Check individual permissions in `node:os.userInfo()` instead of a
single "userInfo" permission.
* Check for "uid" permission in `node:process.geteuid()` instead of
"geteuid".
* Add missing "homedir" to `SysPermissionDescriptor.kind` union
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Adds support for running npm package lifecycle scripts, opted into via a
new `--allow-scripts` flag.
With this PR, when running `deno cache` (or `DENO_FUTURE=1 deno
install`) you can specify the `--allow-scripts=pkg1,pkg2` flag to run
lifecycle scripts attached to the given packages.
Note at the moment this only works when `nodeModulesDir` is true (using
the local resolver).
When a package with un-run lifecycle scripts is encountered, we emit a
warning suggesting things may not work and to try running lifecycle
scripts. Additionally, if a package script implicitly requires
`node-gyp` and it's not found on the system, we emit a warning.
Extra things in this PR:
- Extracted out bits of `task.rs` into a separate module for reuse
- Added a couple fields to `process.config` in order to support
`node-gyp` (it relies on a few variables being there)
- Drive by fix to downloading new npm packages to test registry
---
TODO:
- [x] validation for allow-scripts args (make sure it looks like an npm
package)
- [x] make allow-scripts matching smarter
- [ ] figure out what issues this closes
---
Review notes:
- This adds a bunch of deps to our test registry due to using
`node-gyp`, so it's pretty noisy
This commit deprecates `deno vendor` subcommand in favor
of using `--vendor` flag or `"vendor": true` setting in the config file.
The subcommand is still available (until Deno 2) but is hidden from
the help output.
Closes #20584
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This adds object config for the workspace config:
```json
{
"workspace": {
"members": ["./member-1", "./member-2"]
}
}
```
This is a more verbose version of `"workspace": ["./member-1",
"./member-2"]`. Although we don't need it at the moment, it makes the
naming of `"workspace"` more clear and leaves the object open for more
config in the future.
Closes https://github.com/denoland/deno/issues/24456
The `jest` test runner popularized putting tests into a `__tests__`
folder. Whilst many have switched to going with a `.test` suffix in the
file name these days, there are still many jest projects that have
`__tests__`. By adding this to the default test detection logic it makes
`deno test` discover those out of the box.
Also don't panic on invalid domain names and addresses.
Extracted with cleanups up from #24080
Co-authored-by: Yazan AbdAl-Rahman <yazan.abdalrahman@exalt.ps>
Closes: https://github.com/denoland/deno/issues/22633
This commit adds support for `confirm` and `prompt` APIs,
that instead of reading from stdin are using notebook frontend
to show modal boxes and wait for answers.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Adds much better support for the unstable Deno workspaces as well as
support for npm workspaces. npm workspaces is still lacking in that we
only install packages into the root node_modules folder. We'll make it
smarter over time in order for it to figure out when to add node_modules
folders within packages.
This includes a breaking change in config file resolution where we stop
searching for config files on the first found package.json unless it's
in a workspace. For the previous behaviour, the root deno.json needs to
be updated to be a workspace by adding `"workspace":
["./path-to-pkg-json-folder-goes-here"]`. See details in
https://github.com/denoland/deno_config/pull/66
Closes #24340
Closes #24159
Closes #24161
Closes #22020
Closes #18546
Closes #16106
Closes #24160
Part of #18218
- Adds `fs.lutimes` and `fs.lutimesSync` to our node polyfills. To do
this I added methods to the `FileSystem` trait + ops to expose the
functionality to JS.
- Exports `fs._toUnixTimestamp`. Node exposes an internal util
`toUnixTimestamp` from the fs module to be used by unit tests (so we
need it for the unit test to pass unmodified). It's weird because it's
only supposed to be used internally but it's still publicly accessible
- Matches up error handling and timestamp handling for fs.futimes and
fs.utimes with node
- Enables the node_compat utimes test - this exercises futimes, lutimes,
and utimes.
Prevent panic when enabling a feature that is already enabled by
removing duplicate features.
Closes #22015
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Closes #18296.
Adds a `--frozen` (alias `--frozen-lockfile`) flag that errors out if
the lockfile is out of date. This is useful for running in CI (where an
out of date lockfile is usually a mistake) or to prevent accidental
changes in dependencies.
As suggested in
https://github.com/denoland/deno/pull/24355#discussion_r1657875422.
I wasn't able to hide the mutex stuff as much as I'd like (ended up just
adding an escape hatch `inner()` method that locks the inner mutex),
because you can't return references to the inner fields through a mutex.
This is mostly motivated by the frozen lockfile changes
This commit adds discovery of `.npmrc` files in user's homedir.
This is not a perfect fix as it doesn't merge multiple `.npmrc` files
together as per https://github.com/denoland/deno/issues/23954
but allows to fallback if no `.npmrc` file is discovered in the project
root.
- add fallback impls of external string apis which always copy. after
upstream changes to rusty_v8 we can support non-copying api as well.
- `napi_get_buffer_data` needs to work on all TypedArray instances.
- Fixes: https://github.com/denoland/deno/issues/24209
- `target_defaults.default_configuration` is used by some modules to
find the corresponding node file from node-gyp
- `node_api_get_module_filename` expects the filename to be a `file:`
url.
This is the release commit being forwarded back to main for 1.44.3
Co-authored-by: bartlomieju <bartlomieju@users.noreply.github.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Fixes #23493.
What was happening here was that napi-rs was freeing the napi reference
([here](19e3488efc/crates/napi/src/bindgen_runtime/mod.rs (L62)))
during its finalize callback (which we call
[here](fb31eaa9ca/cli/napi/js_native_api.rs (L132))).
We then were [reading the `ownership`
field](fb31eaa9ca/cli/napi/js_native_api.rs (L136))
of that freed reference.
For some reason on arm macs the freed memory gets zeroed, so the value
of `ownership` was `0` when we read it (i.e. it was
`ReferenceOwnership::Runtime`). We then freed it again (since we thought
we owned it), causing the segfault.
This commit updates Deno to use `reqwest` at 0.12.4
and `rustls` at 0.22. Other related crates were updated
as well to match versions accepted by `reqwest` and `rustls`.
Note: we are not using the latest available `rustls` yet,
but this upgrade was non-trivial already, so a bump to
0.23 for `rustls` will be done in a separate commit.
Closes #23370
---------
Signed-off-by: Ryan Dahl <ry@tinyclouds.org>
Signed-off-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Fixes a regression introduced in
https://github.com/denoland/deno/pull/24170, where we wouldn't actually
set up the node modules dir on `deno install` if there was an up to date
deno lockfile present.
Previously we were relying on the fact that resolving pending module
resolution called `cache_packages` (which sets up the node modules dir).
When pending resolutions were removed, and the `resolve_pending`
function with it, we also removed the `cache_packages` call needed to
set up node modules.
Fixes #22050.
It seems very unlikely that a user would be intending to enable deno's
internal debug logs by setting the DEBUG env var. If they really want
that, they can set `RUST_LOG=debug` instead.
Previously various reads of files in `node_modules` would error on
invalid UTF-8. These were cases involving:
- reading package.json from Rust
- reading package.json from JS
- reading CommonJS files from JS
- reading CommonJS files from Rust (for ESM translation)
- reading ESM files from Rust
Also removes permissions being passed in for node resolution. It was
completely useless because we only checked it for reading package.json
files, but Deno reading package.json files for resolution is perfectly
fine.
My guess is this is also a perf improvement because Deno is doing less
work.
This is the release commit being forwarded back to main for 1.44.1
Co-authored-by: devsnek <devsnek@users.noreply.github.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Introduces a `SyncReadAsyncWriteLock` to make it harder to write to the
npm resolution without first waiting async in a queue. For the npm
resolution, reading synchronously is fine, but when updating, someone
should wait async, clone the data, then write the data at the end back.
Factoring out `dlint` upgrade from
https://github.com/denoland/deno/pull/24034 as it
requires us to change the lint step on mac to use ARM runners.
---------
Co-authored-by: Luca Casonato <hello@lcas.dev>
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
The PEM file specified by DENO_CERT can contain multiple certificates
13924fdb1b/cli/args/mod.rs (L722-L742)
Signed-off-by: Andreas Kohn <andreas.kohn@gmail.com>
Fixes #24012.
In the case of multiple packages providing a binary with a same name, we
were basically leaving the results undefined (since we set up things in
parallel, and whichever got set up first won). In addition, we were
warning about these cases, even though it's a situation that's expected
to occur.
Instead, in the case of a collision in the binary names, we prefer the
binary provided by the package with the least depth in the dependency
tree.
While I was at it, I also took moved more code to `bin_entries.rs` since
it was starting to get a bit cluttered.
This commits adds the ability to set a would-be exit code
for the Deno process without forcing an immediate exit,
through the new `Deno.exitCode` API.
- **Implements `Deno.exitCode` getter and setter**: Adds support for
setting
and retrieving a would-be exit code via `Deno.exitCode`.
This allows for asynchronous cleanup before process termination
without immediately exiting.
- **Ensures type safety**: The setter for `Deno.exitCode` validates that
the provided value is a number, throwing a TypeError if not, to ensure
that
only valid exit codes are set.
Closes to #23605
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Brings in:
* More fully typed structures (for when we get to implementing more)
* `with_metadata`, `with_buffers`, etc. from
https://github.com/runtimed/runtimed/pull/99
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
In https://github.com/denoland/deno/pull/23955 we changed the sqlite db
journal mode to WAL. This causes issues when someone is running an old
version of Deno using TRUNCATE and a new version because the two fight
against each other.
The same issue in two different places - doing blocking FS work in an
async task, limiting the amount of work that happens concurrently.
- When setting up node_modules, where we try to set up entries
concurrently but were blocking other tasks from actually running.
- When loading package info from the npm registry file cache, loading
and deserializing is expensive and prevents concurrency. This was
especially noticeable when loading an npm resolution snapshot from a
lockfile (`snapshot_from_lockfile` in `deno_npm`).
Installing deps in `deno-docs`:
```
❯ hyperfine -i -p 'rm -rf node_modules/' '../d7/deno-main i' '../d7/target/release/deno i'
Benchmark 1: ../d7/deno-main i
Time (mean ± σ): 2.193 s ± 0.027 s [User: 0.589 s, System: 1.033 s]
Range (min … max): 2.151 s … 2.242 s 10 runs
Benchmark 2: ../d7/target/release/deno i
Time (mean ± σ): 1.597 s ± 0.021 s [User: 0.977 s, System: 1.337 s]
Range (min … max): 1.550 s … 1.627 s 10 runs
Summary
../d7/target/release/deno i ran
1.37 ± 0.02 times faster than ../d7/deno-main i
```
Caching `npm:@11ty/eleventy`:
```
❯ hyperfine -i -p 'rm -rf node_modules/' --warmup 5 '../../d7/deno-main cache npm:@11ty/eleventy' '../../d7/target/release/deno cache npm:@11ty/eleventy'
Benchmark 1: ../../d7/deno-main cache npm:@11ty/eleventy
Time (mean ± σ): 129.9 ms ± 2.2 ms [User: 27.5 ms, System: 101.3 ms]
Range (min … max): 127.5 ms … 135.8 ms 10 runs
Benchmark 2: ../../d7/target/release/deno cache npm:@11ty/eleventy
Time (mean ± σ): 100.6 ms ± 1.3 ms [User: 38.8 ms, System: 233.8 ms]
Range (min … max): 99.3 ms … 103.2 ms 10 runs
Summary
../../d7/target/release/deno cache npm:@11ty/eleventy ran
1.29 ± 0.03 times faster than ../../d7/deno-main cache npm:@11ty/eleventy
```
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
Hard linking (`linkat`) is ridiculously slow on mac. `copyfile` is
better, but what's even faster is `clonefile`. It doesn't have the space
savings that comes with hardlinking, but the performance difference is
worth it imo.
```
❯ hyperfine -i -p 'rm -rf node_modules/' '../../d7/target/release/deno cache npm:@11ty/eleventy' 'deno cache npm:@11ty/eleventy'
Benchmark 1: ../../d7/target/release/deno cache npm:@11ty/eleventy
Time (mean ± σ): 115.4 ms ± 1.2 ms [User: 27.2 ms, System: 87.3 ms]
Range (min … max): 113.7 ms … 117.5 ms 10 runs
Benchmark 2: deno cache npm:@11ty/eleventy
Time (mean ± σ): 619.3 ms ± 6.4 ms [User: 34.3 ms, System: 575.6 ms]
Range (min … max): 612.2 ms … 633.3 ms 10 runs
Summary
../../d7/target/release/deno cache npm:@11ty/eleventy ran
5.37 ± 0.08 times faster than deno cache npm:@11ty/eleventy
```
# Summary
This PR resolves about the issue.
fixes #10810
And the formerly context is in the PR.
#22582
Here is an expected behaviour example with this change.
- 🦕.test.ts
```ts
import { assertEquals } from "https://deno.land/std@0.215.0/assert/mod.ts";
Deno.test("example test", () => {
assertEquals("🍋", "🦕");
});
```
The mixed `number | bigint` representation was useful optimization for
pointers. Now, pointers are represented as V8 externals. As part of the
FFI stabilization effort we want to make `bigint` the only
representation for `u64` and `i64`.
BigInt representation performance is almost on par with mixed
representation with the added benefit that its less confusing and users
don't need manual checks and conversions for doing operations on the
value.
```
cpu: AMD Ryzen 5 7530U with Radeon Graphics
runtime: deno 1.43.6+92a8d09 (x86_64-unknown-linux-gnu)
file:///home/divy/gh/ffi/main.ts
benchmark time (avg) iter/s (min … max) p75 p99 p995
-------------------------------------------------------------------------- -----------------------------
nop 4.01 ns/iter 249,533,690.5 (3.97 ns … 10.8 ns) 3.97 ns 4.36 ns 9.03 ns
ret bigint 7.74 ns/iter 129,127,186.8 (7.72 ns … 10.46 ns) 7.72 ns 8.11 ns 8.82 ns
ret i32 7.81 ns/iter 128,087,100.5 (7.77 ns … 12.72 ns) 7.78 ns 8.57 ns 9.75 ns
ret bigint (add op) 15.02 ns/iter 66,588,253.2 (14.64 ns … 24.99 ns) 14.76 ns 19.13 ns 19.44 ns
ret i32 (add op) 12.02 ns/iter 83,209,131.8 (11.95 ns … 18.18 ns) 11.98 ns 13.11 ns 14.5 ns
```
Enhanced warning message for --env flag with run and eval subcommands.
The commit is specifically made to address issue #23674 by improving the
warning messages that appear when using the --env flag with run or eval
subcommands in the following scenarios:
1. Missing environment file.
2. Incorrect syntax in the environment file content.
**Changes made**
- Distinguishes between cases of missing environment file and wrong
syntax in the environment file content.
- Shows a concise warning message to convey the case/issue occurred.
**Code changes & enhancements**
- Implemented a match statement to handle different types of errors
received while getting and parsing the file content to display a concise
warning message, rather than simple error check and then displaying the
same warning message for whatever the type of error is.
- Updated the related existing tests to reflect the new warning
messages.
- Added two test cases to cover the wrong environment file content
syntax with both run and eval subcommands.
**Impact**
The use of --env flag with both run/eval would be more user-friendly as
it gives a precise description of what is not right when using
incorrectly.
If you could give it a look, @dsherret , I appreciate your feedback on
these changes.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds initial support for ".npmrc" files.
Currently we only discover ".npmrc" files next to "package.json" files
and discovering these files in user home dir is left for a follow up.
This pass supports "_authToken" and "_auth" configuration
for providing authentication.
LSP support has been left for a follow up PR.
Towards https://github.com/denoland/deno/issues/16105
Fixes #23571.
Previously, we required a `deno.json` to be present (or the `--lock`
flag) in order for us to resolve a `deno.lock` file. This meant that if
you were using deno in an npm-first project deno wouldn't use a
lockfile.
Additionally, while I was fixing that, I discovered there were a couple
bugs keeping the future `install` command from using a lockfile.
With this PR, `install` will actually resolve the lockfile (or create
one if not present), and update it if it's not up-to-date. This also
speeds up `deno install`, as we can use the lockfile to skip work during
npm resolution.
This PR removes the use of the custom `utc_now` function in favor of the
`chrono` implementation. It resolves #22864.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
While investigating poor cold start performance on my GCP VM (32 cores,
130GB SSD), I found that writing to the various sqlite databases in
DENO_DIR was quite slow. The slowness seems to primarily be caused by
excessive latency from a number of `fsync()` calls.
The performance difference is best demonstrated by deleting the sqlite
databases from DENO_DIR while leaving the downloaded sources in place.
The benchmark (see notes below):
```
piscisaureus@bert-us:~/erofs/source$ export DENO_DIR=./.deno
piscisaureus@bert-us:~/erofs/source$ hyperfine --warmup 3 \
--prepare "rm -rf .deno/*_v1*" \
"deno run -A --cached-only demo.ts" \
"eatmydata deno run -A --cached-only demo.ts" \
"~/deno/target/release/deno run -A --cached-only demo.ts"
Benchmark 1: deno run -A --cached-only demo.ts
Time (mean ± σ): 1.174 s ± 0.037 s [User: 0.153 s, System: 0.184 s]
Range (min … max): 1.104 s … 1.212 s 10 runs
Benchmark 2: eatmydata deno run -A --cached-only demo.ts
Time (mean ± σ): 265.5 ms ± 3.6 ms [User: 138.5 ms, System: 135.1 ms]
Range (min … max): 260.6 ms … 271.2 ms 11 runs
Benchmark 3: ~/deno/target/release/deno run -A --cached-only demo.ts
Time (mean ± σ): 226.2 ms ± 9.2 ms [User: 136.7 ms, System: 93.3 ms]
Range (min … max): 218.8 ms … 247.1 ms 13 runs
Summary
~/deno/target/release/deno run -A --cached-only demo.ts ran
1.17 ± 0.05 times faster than eatmydata deno run -A --cached-only demo.ts
5.19 ± 0.27 times faster than deno run -A --cached-only demo.ts
```
Notes:
* Benchmark 1: unmodified Deno 1.43.6
* Benchmark 2: unmodified Deno 1.43.6 wrapped with `eatmydata` (which is
a tool to neuter `fsync()` calls)
* Benchmark 3: this PR applied on top of Deno 1.43.6
The script that got benchmarked:
```typescript
// demo.ts
import * as express from "npm:express@4.16.3";
import * as postgres from "https://deno.land/x/postgres/mod.ts";
let _dummy = [express, postgres]; // Force use of imports.
console.log("hello world");
```
This is a primordialization effort to improve resistance against users
tampering with the global `Object` prototype.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
By default, uses a 60 second timeout, backing off 2x each time (can be
overridden using the hidden `DENO_SLOW_TEST_TIMEOUT` which we implement
only really for spec testing.
```
Deno.test(async function test() {
await new Promise(r => setTimeout(r, 130_000));
});
```
```
$ target/debug/deno test /tmp/test_slow.ts
Check file:///tmp/test_slow.ts
running 1 test from ../../../../../../tmp/test_slow.ts
test ...'test' is running very slowly (1m0s)
'test' is running very slowly (2m0s)
ok (2m10s)
ok | 1 passed | 0 failed (2m10s)
```
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This brings in [`runtimelib`](https://github.com/runtimed/runtimed) to
use:
## Fully typed structs for Jupyter Messages
```rust
let msg = connection.read().await?;
self
.send_iopub(
runtimelib::Status::busy().as_child_of(msg),
)
.await?;
```
## Jupyter paths
Jupyter paths are implemented in Rust, allowing the Deno kernel to be
installed completely via Deno without a requirement on Python or
Jupyter. Deno users will be able to install and use the kernel with just
VS Code or other editors that support Jupyter.
```rust
pub fn status() -> Result<(), AnyError> {
let user_data_dir = user_data_dir()?;
let kernel_spec_dir_path = user_data_dir.join("kernels").join("deno");
let kernel_spec_path = kernel_spec_dir_path.join("kernel.json");
if kernel_spec_path.exists() {
log::info!("✅ Deno kernel already installed");
Ok(())
} else {
log::warn!("ℹ️ Deno kernel is not yet installed, run `deno jupyter --install` to set it up");
Ok(())
}
}
```
Closes https://github.com/denoland/deno/issues/21619
