For CommonJS packages we were not trying different extensions for files
specified as subpath of the package ([package_name]/[subpath]).
This commit fixes that.
If "--lock-write" flag was present we never passed instance of the lockfile to
the npm resolver, which made it skip adding discovered npm packages to
the lockfile. This commit fixes that, by always passing lockfile to the npm
resolver and only regenerating resolver snapshot is "--lock-write" is not
present.
Closes https://github.com/denoland/deno/issues/16666
Supports package names that aren't all lowercase.
This stores the package with a leading underscore (since that's not
allowed in npm's registry and no package exists with a leading
underscore) then base32 encoded (A-Z0-9) so it can be lowercased and
avoid collisions.
Global cache dir:
```
$DENO_DIR/npm/registry.npmjs.org/_{base32_encode(package_name).to_lowercase()}/{version}
```
node_modules dir `.deno` folder:
```
node_modules/.deno/_{base32_encode(package_name).to_lowercase()}@{version}/node_modules/<package-name>
```
Within node_modules folder:
```
node_modules/<package-name>
```
So, direct childs of the node_modules folder can have collisions between
packages like `JSON` vs `json`, but this is already something npm itself
doesn't handle well. Plus, Deno doesn't actually ever resolve to the
`node_modules/<package-name>` folder, but just has that for
compatibility. Additionally, packages in the `.deno` dir could have
collissions if they have multiple dependencies that only differ in
casing or a dependency that has different casing, but if someone is
doing that then they're already going to have trouble with npm and they
are asking for trouble in general.
If resolving types for an npm package, we didn't find "types" entry in
the conditional exports declaration we were falling-through to regular
resolution, instead of short-circuiting and giving up on resolving
types, which might lead to unwarranted errors.
Closes https://github.com/denoland/deno/issues/16649
**This patch**
```
benchmark time (avg) (min … max) p75 p99 p995
------------------------------------------------- -----------------------------
echo deno 23.99 ms/iter (22.51 ms … 33.61 ms) 23.97 ms 33.61 ms 33.61 ms
cat 16kb 24.27 ms/iter (22.5 ms … 35.21 ms) 24.2 ms 35.21 ms 35.21 ms
cat 1mb 25.88 ms/iter (25.04 ms … 30.28 ms) 26.12 ms 30.28 ms 30.28 ms
cat 15mb 38.41 ms/iter (35.7 ms … 50 ms) 38.31 ms 50 ms 50 ms
```
**main**
```
benchmark time (avg) (min … max) p75 p99 p995
------------------------------------------------- -----------------------------
echo deno 35.66 ms/iter (34.53 ms … 41.84 ms) 35.79 ms 41.84 ms 41.84 ms
cat 16kb 35.99 ms/iter (34.52 ms … 44.94 ms) 36.05 ms 44.94 ms 44.94 ms
cat 1mb 38.68 ms/iter (36.67 ms … 50.44 ms) 37.95 ms 50.44 ms 50.44 ms
cat 15mb 48.4 ms/iter (46.19 ms … 58.41 ms) 49.16 ms 58.41 ms 58.41 ms
```
Supports npm specifiers for `deno install`. This will by default always
use a lockfile (which is generated on first run) unless `--no-lock` is
specified.
Peer dependency resolution wasn't handling a peer dependency being
resolved without a dep higher in the tree and then with one being found
higher in the tree.
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
1. There was a lot of cloning going on with `NpmPackageInfo`. This is
now stored in an `Arc<NpmPackageInfo>` and cloning only happens on the
individual version.
2. The package cache is now cleared from memory after resolution.
3. This surfaced a bug in `deno cache` and I noticed it can be more
efficient if we have multiple root specifiers if we provide all the
specifiers as roots.
This PR adds copies of several unstable APIs that are available
in "Deno[Deno.internal].nodeUnstable" namespace.
These copies do not perform unstable check (ie. don't require
"--unstable" flag to be present). Otherwise they work exactly
the same, including permission checks.
These APIs are not meant to be used by users directly and
can change at any time.
Copies of following APIs are available in that namespace:
- Deno.spawnChild
- Deno.spawn
- Deno.spawnSync
- Deno.serve
- Deno.upgradeHttpRaw
- Deno.listenDatagram
This commit makes "npm:" specifiers not require "--unstable" flag.
At the moment some APIs used by Node polyfills still require
"--unstable" which will be addressed in follow up PRs.
This PR makes it possible for applications to create workers from custom
snapshots to improve runtime performance (without having to fork/copy
`runtime/workers.rs`).
This adds support for peer dependencies in npm packages.
1. If not found higher in the tree (ancestor and ancestor siblings),
peer dependencies are resolved like a dependency similar to npm 7.
2. Optional peer dependencies are only resolved if found higher in the
tree.
3. This creates "copy packages" or duplicates of a package when a
package has different resolution due to peer dependency resolution—see
https://pnpm.io/how-peers-are-resolved. Unlike pnpm though, duplicates
of packages will have `_1`, `_2`, etc. added to the end of the package
version in the directory in order to minimize the chance of hitting the
max file path limit on Windows. This is done for both the local
"node_modules" directory and also the global npm cache. The files are
hard linked in this case to reduce hard drive space.
This is a first pass and the code is definitely more inefficient than it
could be.
Closes #15823
This commit fixes CJS resolution when there's a local "node_modules/"
directory.
Before this commit relative imports from CJS files where resolved
relative to
root directory of the package instead of relative to referrer file.
When streaming a resource in ext/http, with compression enabled, we
didn't flush individual chunks. This became very problematic when we
enabled `req.body` from `fetch` for FastStream recently.
This commit now correctly flushes each resource chunk after compression.
This commit adds autodiscovery of lockfile.
This only happens if Deno discovers the configuration file (either
"deno.json" or "deno.jsonc"). In such case Deno tries to load
"deno.lock"
file that sits next to the configuration file, or creates one for user
if
the lockfile doesn't exist yet.
As a consequence, "--lock" and "--lock-write" flags had been updated.
"--lock" no longer requires a value, if one is not provided, it defaults
to "./deno.lock" resolved from the current working directory.
"--lock-write"
description was updated to say that it forces to overwrite a lockfile.
Autodiscovery is currently not handled by the LSP.
This commit changes lockfile to be "additive" - ie. integrity check only fails if
file/package is already specified in the lockfile, but its integrity doesn't match.
If file/package is not present in the lockfile, it will be added to the lockfile and
the lockfile will be written to disk.
In order for test cases to pass regardless of each individual's environment,
this commit adds calls to `slice` method when printing the filenames so
we can avoid getting `console.log` to truncate them.
Fixes #16305
This API needs `--allow-sys` permissions nowadays, but the docs still
mention `--allow-env` permissions.
```
deno run .\file.ts
⚠️ ┌ Deno requests sys access to "osRelease".
├ Requested by `Deno.osRelease()` API
├ Run again with --allow-sys to bypass this prompt.
└ Allow? [y/n] (y = yes, allow; n = no, deny) >
```
This test has hung a lot recently on macOS. I am not sure if this is
because of a bug in the test or because of the macOS runner that is extremely
slow and flaky in general.
The "proposed" feature that we depend upon in tower-lsp, turns on the
"proposed" feature in lsp-types which has breaking changes in patch
releases because it's explicitly unstable. We need to pin it to prevent
it breaking cargo publish.
V8's JIT can do a better job knowing the argument count and also enable
fast call path (in future).
This also lets us call async ops without `opAsync`:
```js
const { ops } = Deno.core;
await ops.op_void_async();
```
this patch: 4405286 ops/sec
main: 3508771 ops/sec
This commit stabilizes "Deno.consoleSize()" API.
There is one change compared to previous unstable API,
in that the API doesn't accept any arguments. Console size
is established by querying syscalls for stdio streams at fd
0, 1 and 2.
This commit adds a `reuseAddress` option for UDP sockets. When this
option is enabled, one can listen on an address even though it is
already being listened on from a different process or thread. The new
socket will steal the address from the existing socket.
On Windows and Linux this uses the `SO_REUSEADDR` option, while on other
Unixes this is done with `SO_REUSEPORT`.
This behavior aligns with what libuv does.
TCP sockets still unconditionally set the `SO_REUSEADDR` flag - this
behavior matches Node.js and Go. This PR does not change this behaviour.
Co-authored-by: Luca Casonato <hello@lcas.dev>
When listening on a UNIX socket path, Deno currently tries to unlink
this path prior to actually listening. The implementation of this
behaviour is VERY racy, involves 2 additional syscalls, and does not
match the behaviour of any other runtime (Node.js, Go, Rust, etc).
This commit removes this behaviour. If a user wants to listen on an
existing socket, they must now unlink the file themselves prior to
listening.
This change in behaviour only impacts --unstable APIs, so it is not
a breaking change.
This PR fixes a regression that caused deno binaries produced by the CI
release workflows to be larger than expected.
**The problem:** The build script will determine whether the linker
supports the `--export-dynamic-symbol-list` flag by looking at the glibc
version installed on the system. Ubuntu 20.04 ships with glibc 2.31,
which does not support this flag. Upon investigation, I discovered that
the CI pipeline does not use the gcc compiler provided by the
`build-essential` package, and instead uses *clang-14*, which does
support the new flag.
**The solution:** Whenever a custom C Compiler is configured, the build
script now assumes the compiler supports the
`--export-dynamic-symbol-list` flag. This is not always going to be the
case (you could use clang-8, for example), but it puts the onus on the
user making the override to ensure the compiler has support.
This will return deno builds for Linux to their previous size of ~100MB,
and also allow builds under older glibc/gcc versions to succeed. If a
user is compiling deno with a custom compiler that does not support this
new flag, however, their build will fail. I expect this is a rare
scenario, however, and suggest we cross that bridge if and when we come
to it.
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
There were still remaining bigint usages for pointers. This now finally
fixes all of them, there is only the one `type PointerValue = number |
bigint;` line that references `bigint` in the unstable type definition
file.
Potential fix for type-code mismatch in FFI buffer types. The code
supports ArrayBuffers, but types only reflect TypedArray support.
There's also an existing type for this sort of stuff: `BufferSource`.
(Although, it uses `ArrayBufferView` which doesn't actually connect with
the TypedArray interfaces specifically, but it's just a type inheritance
difference and nothing more.)
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
Makes `op_ffi_ptr_of` fast. One of the tests changed from printing
`false` to `true` as the fast `&[u8]` slice path creates the slice with
a null pointer. Thus the `op_ffi_ptr_of` will now return a null pointer
value whereas previously it returned a dangling pointer value.
This PR makes pointer read methods of `Deno.UnsafePointerView` Fast API
compliant, with the exception of `getCString` which cannot be made fast
with current V8 Fast API.
Introduces a new lockfile format that will be used to support locking
"npm" dependencies.
Currently the format looks as follows:
```
// This file is automatically generated by Deno, do not edit its contents
// manually. This file should be commited to your repository.
{
"version": "2",
"remote": {
"https://deno.land/std@0.160.0/http/server.ts": "asdwetsw44523asdfgfas..",
"https://deno.land/std@0.160.0/http/file_server.ts": "asdwetsw44523asdfgfas.."
}
}
```
A follow up PR will add "npm" key that will be used to store information
related
to "npm" dependencies and their resolution.
The new format is used when `--lock-write` is present, if user tries to
load
a lock file using the old format it will still work.
A small cleanup that improves errors in the lockfile as well
as prepares for adding a new format of the lock file that will
allow to provide backward compatibility with existing format
(ie. "Lockfile::content" will be changed into an enum "LockfileContent"
that will have "V1" and "V2" variants).
`deno task` has been in use for a few months now. It was very
well received and there are not many complaints. I feel like
this warning might be discouraging for some users and we don't
really plan to make drastic changes to it (besides adding support
for globs in unspecified future).
Tests and implementation are found here:
https://github.com/denoland/deno_task_shell/pull/59
This is a breaking change, but `deno task` is unstable.
> This changes async commands so that on non-zero exit code they will
fail the entire task. For example:
>
> ```jsonc
> // task that asynchronously starts a server and starts a watcher for
the frontend
> "dev": "deno task server & deno task frontend:watch"
> ```
>
> Previously when running `deno task dev`, if `deno task server` failed,
the entire command would not fail, which kept in line with `sh`, but
it's not very practical. This change causes `deno task dev` to fail.
>
> To opt out, developers can add an `|| exit 0`:
>
> ```jsonc
> "dev": "deno task server || exit 0 & deno task frontend:watch"
> ```
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
This commit removes the calls to `expect()` on `std::rc::Rc`, which caused
Deno to panic under certain situations. We now return an error if `Rc`
is referenced by other variables.
Fixes #9360
Fixes #13345
Fixes #13926
Fixes #16241
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: bartlomieju <bartlomieju@users.noreply.github.com>
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
Co-authored-by: denobot <33910674+denobot@users.noreply.github.com>
Co-authored-by: bartlomieju <bartlomieju@users.noreply.github.com>
This change adds `windowsRawArguments` to `SpawnOptions`. The option enables
skipping the default quoting and escaping while creating the command on
windows.
The option works in a similar way as `windowsVerbatimArguments` in
child_process.spawn options in Node.js, and is necessary for simulating
it in `std/node`.
closes #8852
Fixes the error reported in #16304.
> = note:
/usr/bin/ld:/home/abotella/Projects/deno/cli/generated_symbol_exports_list_linux.def:1:
syntax error in dynamic list
collect2: error: ld returned 1 exit status
This was caused by the format of the symbols list on Linux being
malformed (as the error implies).
The format is documented in ld's
[VERSION](https://sourceware.org/binutils/docs/ld/VERSION.html) as well
as:
> --export-dynamic-symbol-list=file
Specify a --export-dynamic-symbol for each pattern in the
file. The format of the file is the same as the version node
without scope and node name. See VERSION for more
information.
Previously, the format for the Linux symbols list was simply a list of
symbols, now it follows the format:
```
{ symbol_name_a; ...; symbol_name_z };
```
Follow-up to #16208.
- Refactors build.rs behaviour to use `-exported_symbols_list` /
`--export-dynamic-symbol-list`
- Since all build systems now rely on a symbols list file, I have added
`generate_exported_symbols_list`, which derives the symbol list file
depending on the platform, which makes `tools/napi/generate_link_win.js`
redundant.
- Fixes a missed instance of `i8` being used instead of `c_char`
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Fixes #15136
Currently `UnsafeCallback` class' `ref()` and `unref()` methods rely on
the `event_loop_middleware` implementation in core. If even a single
`UnsafeCallback` is ref'ed, then the FFI event loop middleware will
always return `true` to signify that there may still be more work for
the event loop to do.
The middleware handling in core does not wait a moment to check again,
but will instead synchronously directly re-poll the event loop and
middlewares for more work. This becomes a live-loop.
This PR introduces a `Future` implementation for the `CallbackInfo`
struct that acts as the intermediary data storage between an
`UnsafeCallback` and the `libffi` C callback. Ref'ing a callback now
means calling an async op that binds to the `CallbackInfo` Future and
only resolves once the callback is unref'ed. The `libffi` C callback
will call the waker of this Future when it fires to make sure that the
main thread wakes up to receive the callback.
This commit prevents panics that `deno compile` command ran into under certain
conditions from occurring. Such conditions are as follows.
- the target file name begins with `@`, OR
- the stem part of the target file name is equal to one of
["main", "index", "mod", "index"] && the parent directory name starts with `@`
Fixes #16243
This PR fixes invalid header parsing which is flaky because `g` flag is
being used in the regex, which keeps track of `lastIndex`
```javascript
try {
new Headers([["x", "\u0000x"]]); // error
} catch(e) {}
new Headers([["x", "\u0000x"]]); // no error
```
This issue affects `Response` & `Request` constructors as well
Changes introduced in #13633 have broken the ability to compile for
linux/aarch64 - specifically the use of a `i8` as a char type, which is
an `u8` on linux/aarch64.
This PR:
- Replaces instances of `i8` with the architecture-aware wrapper type
`c_char`
- Skips the use of `--export-dynamic-symbol` on linux-aarch64, because
the target environments often rely on older libc/binutils versions
This commit introduces two new buffer wrapper types to `deno_core`. The
main benefit of these new wrappers is that they can wrap a number of
different underlying buffer types. This allows for a more flexible read
and write API on resources that will require less copying of data
between different buffer representations.
- `BufView` is a read-only view onto a buffer. It can be backed by
`ZeroCopyBuf`, `Vec<u8>`, and `bytes::Bytes`.
- `BufViewMut` is a read-write view onto a buffer. It can be cheaply
converted into a `BufView`. It can be backed by `ZeroCopyBuf` or
`Vec<u8>`.
Both new buffer views have a cursor. This means that the start point of
the view can be constrained to write / read from just a slice of the
view. Only the start point of the slice can be adjusted. The end point
is fixed. To adjust the end point, the underlying buffer needs to be
truncated.
Readable resources have been changed to better cater to resources that
do not support BYOB reads. The basic `read` method now returns a
`BufView` instead of taking a `ZeroCopyBuf` to fill. This allows the
operation to return buffers that the resource has already allocated,
instead of forcing the caller to allocate the buffer. BYOB reads are
still very useful for resources that support them, so a new `read_byob`
method has been added that takes a `BufViewMut` to fill. `op_read`
attempts to use `read_byob` if the resource supports it, which falls
back to `read` and performs an additional copy if it does not. For
Rust->JS reads this change should have no impact, but for Rust->Rust
reads, this allows the caller to avoid an additional copy in many
scenarios. This combined with the support for `BufView` to be backed by
`bytes::Bytes` allows us to avoid one data copy when piping from a
`fetch` response into an `ext/http` response.
Writable resources have been changed to take a `BufView` instead of a
`ZeroCopyBuf` as an argument. This allows for less copying of data in
certain scenarios, as described above. Additionally a new
`Resource::write_all` method has been added that takes a `BufView` and
continually attempts to write the resource until the entire buffer has
been written. Certain resources like files can override this method to
provide a more efficient `write_all` implementation.
The current location was causing failures during v1.26.1 publication.
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
This is the release commit being forwarded back to main for 1.26.1
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.26.1 && git checkout -b forward_v1.26.1 upstream/forward_v1.26.1
```
Don't need this PR? Close it.
cc @cjihrig
Co-authored-by: cjihrig <cjihrig@users.noreply.github.com>
Currently, we use `-rdynamic` for exporting Node API symbols to the
symbol table. `-rdynamic` will export *all* symbols, that means
previously unused functions will not be optimized away introducing a lot
of binary bloat.
This patch uses `-exported_symbol` and `--export-dynamic-symbol` link
flags (not as universal as `-rdynamic`) to only mark Node API symbols to
be put in the dynamic symbol table.
This PR implements the NAPI for loading native modules into Deno.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: DjDeveloper <43033058+DjDeveloperr@users.noreply.github.com>
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
Currently Content-Length is sent when the status code is 204. However,
according to the spec, this should not be sent.
Modify the if statement below to prevent the Content-Length from being
sent.
This commit adds a fast path to `Request` and `Response` that
make consuming request bodies much faster when using `Body#text`,
`Body#arrayBuffer`, and `Body#blob`, if the body is a FastStream.
Because the response bodies for `fetch` are FastStream, this speeds up
consuming `fetch` response bodies significantly.
This commit adds support for npm specifier in "deno cache" subcommand.
```
$ deno cache --unstable npm:vite npm:chalk https://deno.land/std/http/file_server.ts
```
Besides downloading requested npm package(s), it will also download
necessary code from "std/node/".
Make offering "virtual documents" via the lsp easier to parse. `deno:`
can be ambiguous to parse by editors (can conflict with linux paths)
Neovim recently landed a PR https://github.com/neovim/neovim/pull/19797
that allows it to parse `scheme:/` this PR should make deno lsp work
correctly in neovim
This commit adds a cache for CJS and ESM analysis that is backed by an
SQLite file.
The connection to the DB is lazily created on first use, so shouldn't
have impact on the startup time.
Benched with running Vite
Deno v1.26:
```
$ deno task dev
Warning deno task is unstable and may drastically change in the future
Task dev deno run -A --unstable --node-modules-dir npm:vite
VITE v3.1.4 ready in 961 ms
➜ Local: http://localhost:5173/
➜ Network: use --host to expose
```
This branch:
```
../deno/target/release/deno task dev
Warning deno task is unstable and may drastically change in the future
Task dev deno run -A --unstable --node-modules-dir npm:vite
VITE v3.1.4 ready in 330 ms
➜ Local: http://localhost:5173/
➜ Network: use --host to expose
```
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
