The `NpmPackageId` struct is being renamed to `NpmPackageNodeId`. In a
future PR it will be moved down into only npm dependency resolution and
a `NpmPackageId` struct will be introduced in `deno_graph` that only has
the name and version of the package (no peer dependency identifier
information). So a `NpmPackageReq` will map to an `NpmPackageId`, which
will map to an `NpmPackageNodeId` in the npm resolution.
Fixes denoland#16922.
The error messages in the `ffi` module are somewhat cryptic when passing
functions that have invalid `parameters` or `result` type strings. While
the generated serializer for the `ForeignFunction` struct correctly
outputs a correct and verbose message, the user sees a far less helpful
`data did not match any variant` message instead.
The underlying cause appears to be the fallback message in the
auto-derived deserializer for untagged enums [1] generated as a result
of `ForeignSymbol` being marked as `#[serde(untagged)]` [2]. Passing an
unexpected value for `NativeType` causes it to error out while
attempting to deserialize both enum variants -- once because it's not a
match for the `ForeignStatic` variant, and once because the
`ForeignFunction` deserializer rejects the invalid type for the
parameters/return type. This is currently open as [serde
#773](https://github.com/serde-rs/serde/issues/773), and not a trivial
exercise to fix generically.
[1]
https://github.com/serde-rs/serde/blob/v0.9.7/serde_derive/src/de.rs#L730
[2] https://github.com/denoland/deno/blob/main/ext/ffi/dlfcn.rs#L102
[3] https://github.com/serde-rs/serde/issues/773
Note that the auto-generated deserializer for untagged enums uses a
private API to buffer deserializer content that we don't have access to.
Instead, we can make use of the `serde_value` crate to buffer the
values. This can likely be removed once the official buffering API lands
(see [4] and [5]). In addition, this crate pulls in `serde_json` as a
cheap way to test that the deserializer works properly.
[4] https://github.com/serde-rs/serde/issues/741
[5] https://github.com/serde-rs/serde/pull/2348
This commit moves "deno_std/node" in "ext/node" crate. The code is
transpiled and snapshotted during the build process.
During the first pass a minimal amount of work was done to create the
snapshot, a lot of code in "ext/node" depends on presence of "Deno"
global. This code will be gradually fixed in the follow up PRs to migrate
it to import relevant APIs from "internal:" modules.
Currently the code from snapshot is not used in any way, and all
Node/npm compatibility still uses code from
"https://deno.land/std/node" (or from the location specified by
"DENO_NODE_COMPAT_URL"). This will also be handled in a follow
up PRs.
---------
Co-authored-by: crowlkats <crowlkats@toaxl.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This is a proof of concept for being able to snapshot TypeScript files.
Currently only a single runtime file is authored in TypeScript -
"runtime/js/01_version.ts".
Not needed infrastructure was removed from "core/snapshot_util.rs".
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit changes handling of config file to enable
specifying "imports" and "scopes" objects effectively making
the configuration file an import map.
"imports" and "scopes" take precedence over "importMap" configuration,
but have lower priority than "--importmap" CLI flag.
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
Co-authored-by: David Sherret <dsherret@gmail.com>
Bump the rsa crate to 0.7.0
The API for the `rsa` crate has changed significantly, but I have
verified that tests continue to pass throughout this update.
Adds support for passing and returning structs as buffers to FFI. This does not implement fastapi support for structs. Needed for certain system APIs such as AppKit on macOS.
Previously, errored streaming response bodies did not cause the HTTP
stream to be aborted. It instead caused the stream to be closed gracefully,
which had the result that the client could not detect the difference
between a successful response and an errored response.
This commit fixes the issue by aborting the stream on error.
This PR adds the concept of a global `DrawThread`, which can receive
multiple renderers to draw information on the screen (note: the
underlying thread is released back to tokio when it's not rendering). It
also separates the concept of progress bars from the existing "draw
thread". This makes it trivial for us to do stuff like show permission
prompts and progress bars at the same time in the future.
The reason this is global is because the process' tty stderr is also a
global concept.
Supports package names that aren't all lowercase.
This stores the package with a leading underscore (since that's not
allowed in npm's registry and no package exists with a leading
underscore) then base32 encoded (A-Z0-9) so it can be lowercased and
avoid collisions.
Global cache dir:
```
$DENO_DIR/npm/registry.npmjs.org/_{base32_encode(package_name).to_lowercase()}/{version}
```
node_modules dir `.deno` folder:
```
node_modules/.deno/_{base32_encode(package_name).to_lowercase()}@{version}/node_modules/<package-name>
```
Within node_modules folder:
```
node_modules/<package-name>
```
So, direct childs of the node_modules folder can have collisions between
packages like `JSON` vs `json`, but this is already something npm itself
doesn't handle well. Plus, Deno doesn't actually ever resolve to the
`node_modules/<package-name>` folder, but just has that for
compatibility. Additionally, packages in the `.deno` dir could have
collissions if they have multiple dependencies that only differ in
casing or a dependency that has different casing, but if someone is
doing that then they're already going to have trouble with npm and they
are asking for trouble in general.
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
* Use stack allocated array for 16 promises and spill rest to heap. the
exact number can change, maybe 128? (tokio's coop budget limit)
* Avoid v8::Global::clone for global context.
* Do not open global opresolve when its not needed.
This adds support for peer dependencies in npm packages.
1. If not found higher in the tree (ancestor and ancestor siblings),
peer dependencies are resolved like a dependency similar to npm 7.
2. Optional peer dependencies are only resolved if found higher in the
tree.
3. This creates "copy packages" or duplicates of a package when a
package has different resolution due to peer dependency resolution—see
https://pnpm.io/how-peers-are-resolved. Unlike pnpm though, duplicates
of packages will have `_1`, `_2`, etc. added to the end of the package
version in the directory in order to minimize the chance of hitting the
max file path limit on Windows. This is done for both the local
"node_modules" directory and also the global npm cache. The files are
hard linked in this case to reduce hard drive space.
This is a first pass and the code is definitely more inefficient than it
could be.
Closes #15823
The "proposed" feature that we depend upon in tower-lsp, turns on the
"proposed" feature in lsp-types which has breaking changes in patch
releases because it's explicitly unstable. We need to pin it to prevent
it breaking cargo publish.
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
Tests and implementation are found here:
https://github.com/denoland/deno_task_shell/pull/59
This is a breaking change, but `deno task` is unstable.
> This changes async commands so that on non-zero exit code they will
fail the entire task. For example:
>
> ```jsonc
> // task that asynchronously starts a server and starts a watcher for
the frontend
> "dev": "deno task server & deno task frontend:watch"
> ```
>
> Previously when running `deno task dev`, if `deno task server` failed,
the entire command would not fail, which kept in line with `sh`, but
it's not very practical. This change causes `deno task dev` to fail.
>
> To opt out, developers can add an `|| exit 0`:
>
> ```jsonc
> "dev": "deno task server || exit 0 & deno task frontend:watch"
> ```
Co-authored-by: bartlomieju <bartlomieju@users.noreply.github.com>
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
Co-authored-by: denobot <33910674+denobot@users.noreply.github.com>
Co-authored-by: bartlomieju <bartlomieju@users.noreply.github.com>
Follow-up to #16208.
- Refactors build.rs behaviour to use `-exported_symbols_list` /
`--export-dynamic-symbol-list`
- Since all build systems now rely on a symbols list file, I have added
`generate_exported_symbols_list`, which derives the symbol list file
depending on the platform, which makes `tools/napi/generate_link_win.js`
redundant.
- Fixes a missed instance of `i8` being used instead of `c_char`
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
This commit introduces two new buffer wrapper types to `deno_core`. The
main benefit of these new wrappers is that they can wrap a number of
different underlying buffer types. This allows for a more flexible read
and write API on resources that will require less copying of data
between different buffer representations.
- `BufView` is a read-only view onto a buffer. It can be backed by
`ZeroCopyBuf`, `Vec<u8>`, and `bytes::Bytes`.
- `BufViewMut` is a read-write view onto a buffer. It can be cheaply
converted into a `BufView`. It can be backed by `ZeroCopyBuf` or
`Vec<u8>`.
Both new buffer views have a cursor. This means that the start point of
the view can be constrained to write / read from just a slice of the
view. Only the start point of the slice can be adjusted. The end point
is fixed. To adjust the end point, the underlying buffer needs to be
truncated.
Readable resources have been changed to better cater to resources that
do not support BYOB reads. The basic `read` method now returns a
`BufView` instead of taking a `ZeroCopyBuf` to fill. This allows the
operation to return buffers that the resource has already allocated,
instead of forcing the caller to allocate the buffer. BYOB reads are
still very useful for resources that support them, so a new `read_byob`
method has been added that takes a `BufViewMut` to fill. `op_read`
attempts to use `read_byob` if the resource supports it, which falls
back to `read` and performs an additional copy if it does not. For
Rust->JS reads this change should have no impact, but for Rust->Rust
reads, this allows the caller to avoid an additional copy in many
scenarios. This combined with the support for `BufView` to be backed by
`bytes::Bytes` allows us to avoid one data copy when piping from a
`fetch` response into an `ext/http` response.
Writable resources have been changed to take a `BufView` instead of a
`ZeroCopyBuf` as an argument. This allows for less copying of data in
certain scenarios, as described above. Additionally a new
`Resource::write_all` method has been added that takes a `BufView` and
continually attempts to write the resource until the entire buffer has
been written. Certain resources like files can override this method to
provide a more efficient `write_all` implementation.
This is the release commit being forwarded back to main for 1.26.1
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.26.1 && git checkout -b forward_v1.26.1 upstream/forward_v1.26.1
```
Don't need this PR? Close it.
cc @cjihrig
Co-authored-by: cjihrig <cjihrig@users.noreply.github.com>
Currently, we use `-rdynamic` for exporting Node API symbols to the
symbol table. `-rdynamic` will export *all* symbols, that means
previously unused functions will not be optimized away introducing a lot
of binary bloat.
This patch uses `-exported_symbol` and `--export-dynamic-symbol` link
flags (not as universal as `-rdynamic`) to only mark Node API symbols to
be put in the dynamic symbol table.
This PR implements the NAPI for loading native modules into Deno.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: DjDeveloper <43033058+DjDeveloperr@users.noreply.github.com>
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
v0.1.3 contains code that will stop working with newer versions of
libstd because the layout of some std::net types changed.
Refs: https://github.com/bnoordhuis/netif/pull/10
- move errors related to Node compat from cli/node/errors.rs to "ext/node" crate
- remove dependency on "node_resolver" crate
- make some of structures private to the "cli/node" module
Previously `jsxImportSource` was resolved relative to the config file
during graph building, and relative to the emitted module during
runtime.
This is now fixed so that the JSX import source is resolved relative to
the module both during graph building and at runtime.
This commit adds "ext/node" extension that implementes CommonJS module system.
In the future this extension might be extended to actually contain implementation of
Node compatibility layer in favor of "deno_std/node".
Currently this functionality is not publicly exposed, it is available via "Deno[Deno.internal].require"
namespace and is meant to be used by other functionality to be landed soon.
This is a minimal first pass, things that still don't work:
support for dynamic imports in CJS
conditional exports
This commit fixes source maps for files that contain emojis.
This is done by updating "deno_ast" to "0.14.1" for the case
of "--no-check" flag (ie using SWC emit) and by overriding
TSC's default base64 encoder (which turned out to be buggy)
for the type checking case.
This commit changes "deno bench" subcommand, by updating
the "Deno.bench" API as follows:
- remove "Deno.BenchDefinition.n"
- remove "Deno.BenchDefintion.warmup"
- add "Deno.BenchDefinition.group"
- add "Deno.BenchDefintion.baseline"
This is done because bench cases are no longer run fixed amount
of iterations, but instead they are run until there is difference between
subsequent runs that is statistically insiginificant.
Additionally, console reporter was rewritten completely, to looks
similar to "hyperfine" reporter.
The following transformations gradually faced by "JsError" have all been
moved up front to "JsError::from_v8_exception()":
- finding the first non-"deno:" source line;
- moving "JsError::script_resource_name" etc. into the first error stack
in case of syntax errors;
- source mapping "JsError::script_resource_name" etc. when wrapping
the error even though the frame locations are source mapped earlier;
- removing "JsError::{script_resource_name,line_number,start_column,end_column}"
entirely in favour of "js_error.frames.get(0)".
We also no longer pass a js-side callback to "core/02_error.js" from cli.
I avoided doing this on previous occasions because the source map lookups
were in an awkward place.
This commit adds "deno bench" subcommand and "Deno.bench()"
API that allows to register bench cases.
The API is modelled after "Deno.test()" and "deno test" subcommand.
Currently the output is rudimentary and bench cases and not
subject to "ops" and "resource" sanitizers.
Co-authored-by: evan <github@evan.lol>
This commit adds CJS/ESM interoperability when running in --compat mode.
Before executing files, they are analyzed and all CommonJS modules are
transformed on the fly to a ES modules. This is done by utilizing analyze_cjs()
functionality from deno_ast. After discovering exports and reexports, an ES
module is rendered and saved in memory for later use.
There's a caveat that all files ending with ".js" extension are considered as
CommonJS modules (unless there's a related "package.json" with "type": "module").
Covered ranges were not merged and thus it appeared that some lines
might be uncovered. To fix this I used "v8-coverage" that takes care
of merging the ranges properly. With this change, coverage collected
from a file by multiple entrypoints is now correctly calculated.
I ended up forking https://github.com/demurgos/v8-coverage and adding
"cli/tools/coverage/merge.rs" and "cli/tools/coverage/range_tree.rs".
Add an op to list the network interfaces on the system.
Prep work for #8137 and `os.networkInterfaces()` Node compat in std.
Refs denoland/deno_std#1436.
This commit adds proper support for import assertions and JSON modules.
Implementation of "core/modules.rs" was changed to account for multiple possible
module types, instead of always assuming that the code is an "ES module". In
effect "ModuleMap" now has knowledge about each modules' type (stored via
"ModuleType" enum). Module loading pipeline now stores information about
expected module type for each request and validates that expected type matches
discovered module type based on file's "MediaType".
Relevant tests were added to "core/modules.rs" and integration tests,
additionally multiple WPT tests were enabled.
There are still some rough edges in the implementation and not all WPT were
enabled, due to:
a) unclear BOM handling in source code by "FileFetcher"
b) design limitation of Deno's "FileFetcher" that doesn't download the same
module multiple times in a single run
Co-authored-by: Kitson Kelly <me@kitsonkelly.com>
deno_fetch::init has a lot of parameters and generic on two types
that keeps expanding over time. This refactor adds deno_fetch::Options
struct for more clearly defining the various parameters.
This adds `.code` attributes to errors returned by the op-layer, facilitating classifying OS errors and helping node-compat.
Similar to Node, these `.code` attributes are stringified names of unix ERRNOs, the mapping tables are generated by [tools/codegen_error_codes.js](https://gist.github.com/AaronO/dfa1106cc6c7e2a6ebe4dba9d5248858) and derived from libuv and rust's std internals
This commit adds "--compat" flag. When the flag is passed a set of mappings for
built-in Node modules is injected into the import map. If user doesn't
explicitly provide an import map (using "--import-map" flag) then a map is
created on the fly. If there are already existing mappings in import map that
would clash with built-in Node modules a set of diagnostics is printed to the
terminal with suggestions how to proceed.
This adds support for the URLPattern API.
The API is added in --unstable only, as it has not yet shipped in any
browser. It is targeted for shipping in Chrome 95.
Spec: https://wicg.github.io/urlpattern/
Co-authored-by: crowlKats < crowlkats@toaxl.com >
This commit updated "deno_lint" crate to 0.15.1 and refactors
"cli/tools/lint.rs" to create only a single vector of lint rules,
instead of creating a vector for each linted file.
This splits the previous `op_url_parse` into:
- `op_url_parse`: parses a href with an optional base
- `op_url_reparse`: reparses a href with a modifier
This is a cleaner separation of concerns and it allows us to optimize & simplify args passed. Resulting in a 25% reduction in call overhead (~5000ns/call => ~3700ns/call in url_ops bench on my M1 Air)
This commit adds "DENO_TLS_CA_STORE" env variable to support
optionally loading certificates from the users local certificate store.
This will allow them to successfully connect via tls with corporate
and self signed certs provided they have them installed in their keystore.
It also allows them to deal with revoked certs by simply updating
their keystore without having to upgrade Deno.
Currently supported values are "mozilla", "system" or empty value.
This commit removes implementation of "native plugins" and replaces
it with FFI API.
Effectively "Deno.openPlugin" API was replaced with "Deno.dlopen" API.
This commits moves implementation of net related APIs available on "Deno"
namespace to "deno_net" extension.
Following APIs were moved:
- Deno.listen()
- Deno.connect()
- Deno.listenTls()
- Deno.serveHttp()
- Deno.shutdown()
- Deno.resolveDns()
- Deno.listenDatagram()
- Deno.startTls()
- Deno.Conn
- Deno.Listener
- Deno.DatagramConn
Waiting on next request in Deno.serveHttp() API hanged
when responses were using ReadableStream. This was caused
by op_http_request_next op that was never woken after
response was fully written. This commit adds waker field to
DenoService which is called after response is finished.
This commit removes all JS based text encoding / text decoding. Instead
encoding now happens in Rust via encoding_rs (already in tree). This
implementation retains stream support, but adds the last missing
encodings. We are incredibly close to 100% WPT on text encoding now.
This should reduce our baseline heap by quite a bit.
This speeds up incremental rebuild when only touching JS files by 30%
compared to #10786.
Rebuild time after touch 01_broadcast_channel.js:
main: run 1 49.18s, run 2 50.34s
#10786: run 1 43.12s, run 2 43.19s
this + #10786: run 1 30.30s, run 2 30.95s
Replaces the file-backed provider by an in-memory one because proper
file locking is a hard problem that detracts from the proof of concept.
Teach the WPT runner how to extract tests from .html files because all
the relevant tests in test_util/wpt/webmessaging/broadcastchannel are
inside basics.html and interface.html.
In #9118, TLS streams were split into a "read half" and a "write half"
using tokio::io::split() to allow concurrent Conn#read() and
Conn#write() calls without one blocking the other. However, this
introduced a bug: outgoing data gets discarded when the TLS stream is
gracefully closed, because the read half is closed too early, before all
TLS control data has been received.
Fixes: #9692
Fixes: #10049
Fixes: #10296
Fixes: denoland/deno_std#750
