Issue was main does canary builds, which broke this test because it
didn't handle searching for a canary release. Tested by building as
canary locally.
This PR updates the deprecation notices to point to the same replacement
APIs that the Standard Library points to. I've also tweaked the notices
to be a little more presentable/navigatable.
In particular, a follow-up PR in std will be made that documents the use
of `toArrayBuffer()`.
Closes #21193
Towards #20976
This commit adds unstable workspace support. This is extremely
bare-bones and
minimal first-pass at this.
With this change `deno.json` supports specifying `workspaces` key, that
accepts a list of subdirectories. Each workspace can have its own import
map. It's required to specify a `"name"` and `"version"` properties in the
configuration file for the workspace:
```jsonc
// deno.json
{
"workspaces": [
"a",
"b"
},
"imports": {
"express": "npm:express@5"
}
}
```
``` jsonc
// a/deno.json
{
"name": "a",
"version": "1.0.2",
"imports": {
"kleur": "npm:kleur"
}
}
```
```jsonc
// b/deno.json
{
"name": "b",
"version": "0.51.0",
"imports": {
"chalk": "npm:chalk"
}
}
```
`--unstable-workspaces` flag is required to use this feature:
```
$ deno run --unstable-workspaces mod.ts
```
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
This PR changes the `Deno.cron` API:
* Marks the existing function as deprecated
* Introduces 2 new overloads, where the handler arg is always last:
```ts
Deno.cron(
name: string,
schedule: string,
handler: () => Promise<void> | void,
)
Deno.cron(
name: string,
schedule: string,
options?: { backoffSchedule?: number[]; signal?: AbortSignal },
handler: () => Promise<void> | void,
)
```
This PR also fixes a bug, when other crons continue execution after one
of the crons was closed using `signal`.
Fixes #21121 and #19498
Migrates fully to rustls_tokio_stream. We no longer need to maintain our
own TlsStream implementation to properly support duplex.
This should fix a number of errors with TLS and websockets, HTTP and
"other" places where it's failing.
This PR uses the new `cancel` method of `TransformStream` to properly
clean up the internal `TextDecoder` used in `TextDecoderStream` if the
stream is cancelled.
Fixes #13142
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
We only want one zlib dependency.
Zlib dependencies are reorganized so they use a hidden
`__vendored_zlib_ng` flag in cli that enables zlib-ng for both libz-sys
(used by ext/node) and flate2 (used by deno_web).
This also updates deno_graph, which has the JSR change to use "exports".
It's not yet useful atm, so I've made this PR a fix about the deno doc
--lint error message improvements. I'll do a follow-up PR that adds
exports to the deno.json
This is the release commit being forwarded back to main for 1.38.1
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: littledivy <littledivy@users.noreply.github.com>
This PR reduces the time to run `queue persistence with inflight
messages` test from 60 seconds down to about 6 seconds.
The test simulates a crash to ensure that inflight messages are cleaned
up and re-queued when the new process starts. Since messages are
considered dead after 5 seconds of being queued, reopening the db within
5 seconds does not re-queue the messages on startup (they do get
re-queued after 60 seconds, which is the cleanup frequency). By waiting
for 5 seconds before reopening the db, the test ensures that the cleanup
happens quickly when the db is opened.
We can move all promise ID knowledge to deno_core, allowing us to better
experiment with promise implementation in deno_core.
`{un,}refOpPromise(promise)` is equivalent to
`{un,}refOp(promise[promiseIdSymbol])`
Remove tokio-rustls as a direct dependency of Deno and refactor
test_server to reduce code duplication.
All tcp and tls listener paths go through the same streams now, with the
exception of the simpler Hyper http-only handlers (those can be done in
a later follow-up).
Minor bugs fixed:
- gRPC server should only serve h2
- WebSocket over http/2 had a port overlap
- Restored missing eye-catchers for some servers (still missing on Hyper
ones)
This commit moves all Chrome Devtools Protocol messages to `cli/cdp.rs`
and refactors all places using these types to pull them from a common
place.
No functional changes.
As pointed out in https://github.com/denoland/deno/issues/21067 , it's
confusing that `deno doc --lint mod.ts` outputs the documentation to
stdout on success. Instead, it would be better if it outputted how many
files were checked similar to what `deno lint` does on success. It still
outputs the documentation if `--lint` or `--html` are provided so this
is non-breaking.
This commit adds granular `--unstable-*` flags:
- "--unstable-broadcast-channel"
- "--unstable-ffi"
- "--unstable-fs"
- "--unstable-http"
- "--unstable-kv"
- "--unstable-net"
- "--unstable-worker-options"
- "--unstable-cron"
These flags are meant to replace a "catch-all" flag - "--unstable", that
gives a binary control whether unstable features are enabled or not. The
downside of this flag that allowing eg. Deno KV API also enables the FFI
API (though the latter is still gated with a permission).
These flags can also be specified in `deno.json` file under `unstable`
key.
Currently, "--unstable" flag works the same way - I will open a follow
up PR that will print a warning when using "--unstable" and suggest to use
concrete "--unstable-*" flag instead. We plan to phase out "--unstable"
completely in Deno 2.
Implements `WebSocket` over http/2. This requires a conformant http/2
server supporting the extended connect protocol.
Passes approximately 100 new WPT tests (mostly `?wpt_flags=h2` versions
of existing websockets APIs).
This is implemented as a fallback when http/1.1 fails, so a server that
supports both h1 and h2 WebSockets will still end up on the http/1.1
upgrade path.
The patch also cleas up the websockets handshake to split it up into
http, https+http1 and https+http2, making it a little less intertwined.
This uncovered a likely bug in the WPT test server:
https://github.com/web-platform-tests/wpt/issues/42896
This PR adds unstable `Deno.cron` API to trigger execution of cron jobs.
* State: All cron state is in memory. Cron jobs are scheduled according
to the cron schedule expression and the current time. No state is
persisted to disk.
* Time zone: Cron expressions specify time in UTC.
* Overlapping executions: not permitted. If the next scheduled execution
time occurs while the same cron job is still executing, the scheduled
execution is skipped.
* Retries: failed jobs are automatically retried until they succeed or
until retry threshold is reached. Retry policy can be optionally
specified using `options.backoffSchedule`.
This change adds the `--env=[FILE]` flag to the `run`, `compile`,
`eval`, `install` and `repl` subcommands. Environment variables set in
the CLI overwrite those defined in the `.env` file.
Adds a new `--lint` flag to `deno doc` that surfaces three kinds of
diagnostics:
1. Diagnostic for non-exported type referenced in an exported type.
* Why? People often forget to export types from a module in TypeScript.
To supress this diagnostic, add an `@internal` jsdoc tag to the internal
type.
1. Diagnostic for missing return type or missing property type on a
**public** type.
* Why? Otherwise `deno doc` will not display good documentation. Adding
explicit types also helps with type checking performance.
1. Diagnostic for missing jsdoc on a **public** type.
* Why? Everything should be documented. This diagnostic can be supressed
by adding a jsdoc comment description.
If the lint passes, `deno doc` generates documentation as usual.
For example, checking for deno doc diagnostics on the CI:
```shellsession
$ deno doc --lint mod.ts second_entrypoint.ts > /dev/null
```
This feature is incredibly useful for library authors.
## Why not include this in `deno lint`?
1. The command needs the documenation output in order to figure out the
diagnostics.
1. `deno lint` doesn't understand where the entrypoints are. That's
critical for the diagnostics to be useful.
1. It's much more performant to do this while generating documentation.
1. There is precedence in rustdoc (ex. `#![warn(missing_docs)]`).
## Why not `--check`?
It is confusing with `deno run --check`, since that means to run type
checking (and confusing with `deno check --docs`).
## Output Future Improvement
The output is not ideal atm, but it's fine for a first pass. We will
improve it in the future.
Closes https://github.com/denoland/deno_lint/pull/972
Closes https://github.com/denoland/deno_lint/issues/970
Closes https://github.com/denoland/deno/issues/19356
Use new https://github.com/denoland/rustls-tokio-stream project instead
of tokio-rustls for direct websocket connections. This library was
written from the ground up to be more reliable and should help with
various bugs that may occur due to underlying bugs in the old library.
Believed to fix #20355, #18977, #20948
This commit updates the ext/kv module to use the denokv_* crates for
the protocol and the sqlite backend. This also fixes a couple of bugs in
the sqlite backend, and updates versionstamps to be updated less
linearly.
This commit adds `--unstable-hmr` flag, that enabled Hot Module Replacement.
This flag works like `--watch` and accepts the same arguments. If
HMR is not possible the process will be restarted instead.
Currently HMR is only supported in `deno run` subcommand.
Upon HMR a `CustomEvent("hmr")` will be dispatched that contains
information which file was changed in its `details` property.
---------
Co-authored-by: Valentin Anger <syrupthinker@gryphno.de>
Co-authored-by: David Sherret <dsherret@gmail.com>
This commit adds support for multiple entry points to `deno doc`.
Unfortunately to achieve that, I had to change the semantics of the
command to explicitly require `--filter` parameter for filtering
symbols, instead of treating second free argument as the filter argument.
`deno doc --builtin` is still supported, but cannot be mixed with
actual entrypoints.
Upgrades to deno_doc 0.70 which includes the feature for showing
non-exported types referenced in exported types as well as a much more
advanced deno doc that uses a symbol graph.
As title. This will help use the two independently from the other, which
will help in an upcoming deno doc PR where I need to parse the source
files with scope analysis.
This PR adds a new unstable "bring your own node_modules" (BYONM)
functionality currently behind a `--unstable-byonm` flag (`"unstable":
["byonm"]` in a deno.json).
This enables users to run a separate install command (ex. `npm install`,
`pnpm install`) then run `deno run main.ts` and Deno will respect the
layout of the node_modules directory as setup by the separate install
command. It also works with npm/yarn/pnpm workspaces.
For this PR, the behaviour is opted into by specifying
`--unstable-byonm`/`"unstable": ["byonm"]`, but in the future we may
make this the default behaviour as outlined in
https://github.com/denoland/deno/issues/18967#issuecomment-1761248941
This is an extremely rough initial implementation. Errors are
terrible in this and the LSP requires frequent restarts. Improvements
will be done in follow up PRs.
This commit introduces "WatcherCommunicator" struct that
is used facilitate bi-directional communication between CLI
file watcher and the watched function.
Prerequisite for https://github.com/denoland/deno/pull/20876
This brings in [`display`](https://github.com/rgbkrk/display.js) as part
of the `Deno.jupyter` namespace.
Additionally these APIs were added:
- "Deno.jupyter.md"
- "Deno.jupyter.html"
- "Deno.jupyter.svg"
- "Deno.jupyter.format"
These APIs greatly extend capabilities of rendering output in Jupyter
notebooks.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit improves "node:http2" module implementation, by enabling
to use "options.createConnection" callback when starting an HTTP2
session.
This change enables to pass basic client-side test with "grpc-js/grpc"
package.
Smaller fixes like "Http2Session.unref()" and "Http2Session.setTimeout()"
were handled as well.
Fixes #16647
Otherwise you can not return `Deno.Server` from async functions.
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This fixes #20767.
We were losing `this` and then when an exception was happening, it
didn't show up in the output because we weren't bubbling up exceptions
from within a user defined function for displaying. I thought about
doing a `.call(object)` but didn't want to get in the way of a bound
`this` that a user or library was already putting on the function.
Adds `buffers` to the `Deno.jupyter.broadcast` API to send binary data
via comms. This affords the ability to send binary data via websockets
to the jupyter widget frontend.
fixes #20454
Current KV queues implementation assumes that `enqueue` and
`listenQueue` are called on the same instance of `Deno.Kv`. It's
possible that the same Deno process opens multiple KV instances pointing
to the same fs path, and in that case `listenQueue` should still get
notified of messages enqueued through a different KV instance.
This makes `CliNpmResolver` a trait. The terminology used is:
- **managed** - Deno manages the node_modules folder and does an
auto-install (ex. `ManagedCliNpmResolver`)
- **byonm** - "Bring your own node_modules" (ex. `ByonmCliNpmResolver`,
which is in this PR, but unimplemented at the moment)
Part of #18967
This API is providing hoops to jump through with undergoing migration to
`#[op2]` macro.
The overhead of supporting this API is non-trivial and besides internal
use of it in test sanitizers is very rarely used in the wild.
This helps reduce flakes where a test starts an HTTP server and makes a
request using fetch, then shuts down the server, then starting a new
test with a new server, but the connection pool still has a "not quite
closed yet" connection to the old server, and a new request to the new
server gets sent on the closed connection, which obviously errors out.
Previously could flake on the op sanitizer because the
`await makeTempFile()` promise could leak out of the test. Now we ensure
the request is fully handled before returning.
We can go one level down in abstraction and avoid using the public
`ReadableStream` APIs.
This patch ~5% perf boost on small ReadableStream:
```
Running 10s test @ http://localhost:8080/
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 148.32us 108.95us 3.88ms 95.71%
Req/Sec 33.24k 2.68k 37.94k 73.76%
668188 requests in 10.10s, 77.74MB read
Requests/sec: 66162.91
Transfer/sec: 7.70MB
```
main:
```
Running 10s test @ http://localhost:8080/
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 150.23us 67.61us 4.39ms 94.80%
Req/Sec 31.81k 1.55k 35.56k 83.17%
639078 requests in 10.10s, 74.36MB read
Requests/sec: 63273.72
Transfer/sec: 7.36MB
```
When sending configuration requests to the client, reads `javascript`
and `typescript` sections in addition to `deno`.
The LSP's initialization options now accepts `javascript` and
`typescript` namespaces.
Give auto-import completion entries a sort-text suffix depending on if
the specifier parses as a URL. This will favour relative and bare
(likely import-mapped) specifiers.
Adds an experimental unstable built-in package manager to Deno, but it is
currently not usable because the registry infrastructure hasn't been
setup and it points to a non-existent url by default. The default
registry url can be configured via the `DENO_REGISTRY_URL` environment
variable.
This commit changes ordering of quickfix actions, by sorting them in
following order:
- TSC fixes
- Deno fixes
- deno_lint fixes
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
Closes #20535.
# Screenshots
## JSON
<img width="779" alt="image"
src="https://github.com/denoland/deno/assets/836375/668bb1a6-3f76-4b36-974e-cdc6c93f94c3">
## Vegalite
<img width="558" alt="image"
src="https://github.com/denoland/deno/assets/836375/a5e70908-6b87-42d8-85c3-1323ad52a00f">
# Implementation
Instead of going the route of recursively getting all the objects under
`application/.*json` keys, I went with `JSON.stringify`ing in denospace
then parsing it from rust. One of the key benefits of serializing and
deserializing is that non-JSON-able entries will get stripped
automatically. This also keeps the code pretty simple.
In the future we should _only_ do this for `application/.*json` keys.
cc @mmastrac
"Fixes" the exception display issue of #20524 on older versions of
Jupyter that required `evalue` to be truthy. For now, until we can do
proper processing of the `ExceptionDetails` this will make Jupyter
Notebook 6.5.1 and below happy.
This is the alternative "just work now" PR to #20530
This commit improves async op sanitizer speed by only delaying metrics
collection if there are pending ops. This
results in a speedup of around 30% for small CPU bound unit tests.
It performs this check and possible delay on every collection now,
fixing an issue with parent test leaks into steps.
This commit adds "deno jupyter" subcommand which
provides a Deno kernel for Jupyter notebooks.
The implementation is mostly based on Deno's REPL and
reuses large parts of it (though there's some clean up that
needs to happen in follow up PRs). Not all functionality of
Jupyter kernel is implemented and some message type
are still not implemented (eg. "inspect_request") but
the kernel is fully working and provides all the capatibilities
that the Deno REPL has; including TypeScript transpilation
and npm packages support.
Closes https://github.com/denoland/deno/issues/13016
---------
Co-authored-by: Adam Powers <apowers@ato.ms>
Co-authored-by: Kyle Kelley <rgbkrk@gmail.com>
This commit improves compatibility of "node:http2" module by polyfilling
"connect" method and "ClientHttp2Session" class. Basic operations like
streaming, header and trailer handling are working correctly.
Refing/unrefing is still a TODO and "npm:grpc-js/grpc" is not yet working
correctly.
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
This adds the ability to pattern match unordered lines. For example, the
downloading messages may appear in any order
```
[UNORDERED_START]
Download https://localhost:4546/a.ts
Download https://localhost:4546/b.ts
[UNORDERED_END]
Hello!
```
Additionally, I've made the pattern matching slightly more strict and the output better.
TaskQueue is being removed from `deno_core` and replaced with an unsync
version in deno_unsyc.
https://github.com/denoland/deno_core/pull/193
This is a change in preparation for that. The remaining
`deno_core::TaskQueue` usage in this repo should be replaced with
`deno_core::unsync::TaskQueue` once upgraded.
### What
Skip writing files from the template if the files already exist in the
project directory.
### Why
When I run deno init in a directory that already has a main.ts, or one
of the other template files, I usually want to initialize a workspace
around a file I've started working in. A hard error in this case seems
counter productive. An informational message about what's being skipped
seems sufficient.
Close #20433
Removes usage of `serde_json::Value` in several ops used in TSC, in
favor of using strongly typed structs. This will unblock more
changes in https://github.com/denoland/deno/pull/20462.
This PR implements a graceful shutdown API for Deno.serve, allowing all
current connections to drain from the server before shutting down, while
preventing new connections from being started or new transactions on
existing connections from being created.
We split the cancellation handle into two parts: a listener handle, and
a connection handle. A graceful shutdown cancels the listener only,
while allowing the connections to drain. The connection handle aborts
all futures. If the listener handle is cancelled, we put the connections
into graceful shutdown mode, which disables keep-alive on http/1.1 and
uses http/2 mechanisms for http/2 connections.
In addition, we now guarantee that all connections are complete or
cancelled, and all resources are cleaned up when the server `finished`
promise resolves -- we use a Rust-side server refcount for this.
Performance impact: does not appear to affect basic serving performance
by more than 1% (~126k -> ~125k)
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
LSP testing APIs now obey the various file inclusion settings:
- Modules shown in the text explorer now respect the `exclude`,
`test.exclude` and `test.include` fields in `deno.json`, as well as
`deno.enablePaths` in VSCode settings.
- Modules with testing code lens now respect the `"exclude"`,
`test.exclude` and `test.include` fields in `deno.json`. Code lens
already respects `deno.enablePaths`.
This allows us to opt in to extremely detailed tracing from dependency
libraries, like so:
```
cargo run --features tracing/log,tracing/max_level_trace -- test --log-level=trace -A --unstable ./cli/tests/unit/serve_test.ts
```
It will not impact normal operation as it requires the
`tracing/max_level_trace` and `tracing/log` to be active.
Note that tracing is already a dependency -- this just makes it a direct
dep of cli so we can access its features more easily.
Fixes https://github.com/denoland/deno/issues/19816
In that issue, I suggest switching over the other brotli functionality
to the Rust API provided by the `brotli` crate. Here, I only do that
with the `brotli_decompress` function to fix the bug with buffers longer
than 4096 bytes.
Previously we pre-computed enabled paths into `Config::enabled_paths`,
and had to keep updating it. Now we determine enabled paths directly
from `Config::settings` on demand as a single source of truth.
Removes `Config::root_uri`. If `InitializeParams::rootUri` is given, and
it doesn't correspond to a folder in
`InitializeParams::workspaceFolders`, prepend it to
`Config::workspace_folders` as a mocked folder.
Includes groundwork for
https://github.com/denoland/vscode_deno/issues/908. In a minor version
cycle or two we can fix that in vscode_deno, and it won't break for Deno
versions post this patch due to the corrected deserialization logic for
`enablePaths`.
Keys are expensive metadata. We track it for various purposes, e.g.
transaction conflict check, and key expiration.
This patch limits the total key size in an atomic operation to 80 KiB
(81920 bytes). This helps ensure efficiency in implementations.
Fixes #19802.
Properly respect when clients do not have the `workspace/configuration`
capability, a.k.a. when an editor cannot provide scoped settings on
request from the LSP.
- Fix one spot where we weren't checking for the capability before
sending this request.
- For `enablePaths`, fall back to the settings passed in the
initialization options in more cases.
- Respect the `workspace/configuration` capability in the test harness
client.
See:
https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#workspace_configuration.
Closes #14122.
Adds two extensions to `--allow-run` behaviour:
- When `--allow-run=foo` is specified and `foo` is found in the `PATH`
at startup, `RunDescriptor::Path(which("foo"))` is added to the
allowlist alongside `RunDescriptor::Name("foo")`. Currently only the
latter is.
- When run permission for `foo` is queried and `foo` is found in the
`PATH` at runtime, either `RunDescriptor::Path(which("foo"))` or
`RunDescriptor::Name("foo")` would qualify in the allowlist. Currently
only the latter does.
Previously:
```rust
pub struct TestDefinition {
pub id: String,
pub name: String,
pub range: SourceRange,
pub steps: Vec<TestDefinition>,
}
pub struct TestDefinitions {
pub discovered: Vec<TestDefinition>,
pub injected: Vec<lsp_custom::TestData>,
pub script_version: String,
}
```
Now:
```rust
pub struct TestDefinition {
pub id: String,
pub name: String,
pub range: Option<Range>,
pub is_dynamic: bool, // True for 'injected' module, not statically detected but added at runtime.
pub parent_id: Option<String>,
pub step_ids: HashSet<String>,
}
pub struct TestModule {
pub specifier: ModuleSpecifier,
pub script_version: String,
pub defs: HashMap<String, TestDefinition>,
}
```
Storing the test tree as a literal tree diminishes the value of IDs,
even though vscode stores them that way. This makes all data easily
accessible from `TestModule`. It unifies the interface between
'discovered' and 'injected' tests. This unblocks some enhancements wrt
syncing tests between the LSP and extension, such as this TODO:
61f08d5a71/client/src/testing.ts (L251-L259)
and https://github.com/denoland/vscode_deno/issues/900. We should also
get more flexibility overall.
`TestCollector` is cleaned up, now stores a `&mut TestModule` directly
and registers tests as it comes across them with
`TestModule::register()`. This method ensures sanity in the redundant
data from having both of `TestDefinition::{parent_id,step_ids}`.
All of the messy conversions between `TestDescription`,
`LspTestDescription`, `TestDefinition`, `TestData` and `TestIdentifier`
are cleaned up. They shouldn't have been using `impl From` and now the
full list of tests is available to their implementations.
We never want tests to hit the real npm registry because this causes
test flakes. In addition, we set a sentinal "unset" value for
`NPM_CONFIG_REGISTRY` to ensure that all tests requiring npm go through
the test server.
Adds `runtime/shared.rs` which is imported by both `build.rs` and the
rest of the crate, containing utilities used by both.
Renames the `snapshot_from_snapshot` feature to
`exclude_runtime_main_js` since that's what it does and it's relevant
outside of snapshotting when `__runtime_js_sources` is specified.
The fix for #20188 was not entirely correct -- we were unlocking the
global buffer incorrectly. This PR introduces a lock state that ensures
we only unlock a lock we have taken out.
When a TCP connection is force-closed (ie: browser refresh), the
underlying future we pass to Hyper is dropped which may cause us to try
to drop the body resource while the OpState lock is still held.
Preconditions for this bug to trigger:
- The body resource must have been taken
- The response must return a resource (which requires us to take the
OpState lock)
- The TCP connection must have been dropped before this
Fixes #20315 and #20298
The motivation is If I'm using deno lint --rules, I want to see all the
rules especially the one that have no tags, since the recommend ones are
already active
This change also prints the tags associated with the rule inline.
<!--
Before submitting a PR, please read https://deno.com/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
As the title.
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
Disables `BenchContext::start()` and `BenchContext::end()` for low
precision benchmarks (less than 0.01s per iteration). Prints a warning
when they are used in such benchmarks, suggesting to remove them.
```ts
Deno.bench("noop", { group: "noops" }, () => {});
Deno.bench("noop with start/end", { group: "noops" }, (b) => {
b.start();
b.end();
});
```
Before:
```
cpu: 12th Gen Intel(R) Core(TM) i9-12900K
runtime: deno 1.36.2 (x86_64-unknown-linux-gnu)
file:///home/nayeem/projects/deno/temp3.ts
benchmark time (avg) iter/s (min … max) p75 p99 p995
----------------------------------------------------------------------------- -----------------------------
noop 2.63 ns/iter 380,674,131.4 (2.45 ns … 27.78 ns) 2.55 ns 4.03 ns 5.33 ns
noop with start and end 302.47 ns/iter 3,306,146.0 (200 ns … 151.2 µs) 300 ns 400 ns 400 ns
summary
noop
115.14x faster than noop with start and end
```
After:
```
cpu: 12th Gen Intel(R) Core(TM) i9-12900K
runtime: deno 1.36.1 (x86_64-unknown-linux-gnu)
file:///home/nayeem/projects/deno/temp3.ts
benchmark time (avg) iter/s (min … max) p75 p99 p995
----------------------------------------------------------------------------- -----------------------------
noop 3.01 ns/iter 332,565,561.7 (2.73 ns … 29.54 ns) 2.93 ns 5.29 ns 7.45 ns
noop with start and end 7.73 ns/iter 129,291,091.5 (6.61 ns … 46.76 ns) 7.87 ns 13.12 ns 15.32 ns
Warning start() and end() calls in "noop with start and end" are ignored because it averages less than 0.01s per iteration. Remove them for better results.
summary
noop
2.57x faster than noop with start and end
```
Fixes https://github.com/denoland/vscode_deno/issues/743.
```ts
const items: string[] = ['foo', 'bar', 'baz'];
items.map
// ->
items.map(callbackfn) // auto-completes with argument placeholders.
```
---
We have our own setting for `suggest.completeFunctionCalls`, which must
be enabled:
```js
{
"deno.suggest.completeFunctionCalls": true,
// Re-implementation of:
// "javascript.suggest.completeFunctionCalls": true,
// "typescript.suggest.completeFunctionCalls": true,
}
```
But before this commit the actual implementation had been left as a TODO.
This PR adds a test reporter for the [Test Anything
Protocol](https://testanything.org).
It makes the following implementation decisions:
- No TODO pragma, as there is no such marker in `Deno.test`
- SKIP pragma for `ignore`d tests
- Test steps are treated as TAP14 subtests
- Support for this in consumers seems spotty
- Some consumers will incorrectly interpret these markers, resulting in
unexpected output
- Considering the lack of support, and to avoid implementation
complexity,
subtests are at most one level deep (all test steps are in the same
subtest)
- To accommodate consumers that use comments to indicate test-suites
(unspecced)
- The test module path is output as a comment
- This is disabled for `--parallel` testing
- Failure diagnostics are output as JSON, which is also valid YAML
- The structure is not specified, so the format roughly follows the spec
example:
```
---
message: "Failed with error 'hostname peebles.example.com not found'"
severity: fail
found:
hostname: 'peebles.example.com'
address: ~
wanted:
hostname: 'peebles.example.com'
address: '85.193.201.85'
at:
file: test/dns-resolve.c
line: 142
...
```
Fixes https://github.com/denoland/vscode_deno/issues/843.
Prevents step results from being reported twice. Refactors
`LspTestReporter` to use a complete `(test_id, descriptor)` map instead
of a brittle `LspTestReporter::stack`.
Bumps [trust-dns-server](https://github.com/bluejekyll/trust-dns) from
0.22.0 to 0.22.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/bluejekyll/trust-dns/releases">trust-dns-server's
releases</a>.</em></p>
<blockquote>
<h2>v0.22.1 - Deny response requests on the server</h2>
<h2>0.22.1</h2>
<h3>Fixed</h3>
<ul>
<li>(server) drop response messages <a
href="https://redirect.github.com/bluejekyll/trust-dns/issues/1952">#1952</a>
by <a href="https://github.com/djc"><code>@djc</code></a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/bluejekyll/trust-dns/blob/main/CHANGELOG.md">trust-dns-server's
changelog</a>.</em></p>
<blockquote>
<h2>0.22.1</h2>
<h3>Fixed</h3>
<ul>
<li>(server) drop response messages <a
href="https://redirect.github.com/bluejekyll/trust-dns/issues/1952">#1952</a>
by <a href="https://github.com/djc"><code>@djc</code></a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="9f344b54cd"><code>9f344b5</code></a>
bump server and bin crates to 0.22.1</li>
<li><a
href="9e6e77293b"><code>9e6e772</code></a>
update CHANGELOG for 0.22.1</li>
<li><a
href="5f6278154d"><code>5f62781</code></a>
Regenerate the test SSL certificates</li>
<li><a
href="2fd2603171"><code>2fd2603</code></a>
server: drop response messages</li>
<li>See full diff in <a
href="https://github.com/bluejekyll/trust-dns/compare/v0.22.0...v0.22.1">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts page](https://github.com/denoland/deno/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This patch adds a `remote` backend for `ext/kv`. This supports
connection to Deno Deploy and potentially other services compatible with
the KV Connect protocol.
Few improvements to FFI types:
1. Export `PointerObject` for convenience. It's fairly commonly used in
library code and thus should be exported.
2. Fix various comments around `PointerValue` and `UnsafePointer` and
expand upon them to better reflect reality.
3. Instead of using a `Record<"value", type>[T]` for determining the
type of an FFI symbol parameter use direct `T extends "value" ? type :
never` comparison.
The last part enables smuggling extra information into the parameter and
return value string declarations at the type level. eg. Instead of just
`"u8"` the parameter can be `"u8" & { [brand]: T }` for some `T extends
number`. That `T` can then be extracted from the parameter to form the
TypeScript function's parameter or return value type. Essentially, this
enables type-safe FFI!
The foremost use-cases for this are enums and pointer safety. These are
implemented in the second commit which should enable, in a backwards
compatible way, for pointer parameters to declare what sort of pointer
they mean, functions to declare what the API definition of the native
function is, and for numbers to declare what Enum they stand for (if
any).
Deno.serve's fast streaming implementation was not keeping the request
body resource ID alive. We were taking the `Rc<Resource>` from the
resource table during the response, so a hairpin duplex response that
fed back the request body would work.
However, if any JS code attempted to read from the request body (which
requires the resource ID to be valid), the response would fail with a
difficult-to-diagnose "EOF" error.
This was affecting more complex duplex uses of `Deno.fetch` (though as
far as I can tell was unreported).
Simple test:
```ts
const reader = request.body.getReader();
return new Response(
new ReadableStream({
async pull(controller) {
const { done, value } = await reader.read();
if (done) {
controller.close();
} else {
controller.enqueue(value);
}
},
}),
```
And then attempt to use the stream in duplex mode:
```ts
async function testDuplex(
reader: ReadableStreamDefaultReader<Uint8Array>,
writable: WritableStreamDefaultWriter<Uint8Array>,
) {
await writable.write(new Uint8Array([1]));
const chunk1 = await reader.read();
assert(!chunk1.done);
assertEquals(chunk1.value, new Uint8Array([1]));
await writable.write(new Uint8Array([2]));
const chunk2 = await reader.read();
assert(!chunk2.done);
assertEquals(chunk2.value, new Uint8Array([2]));
await writable.close();
const chunk3 = await reader.read();
assert(chunk3.done);
}
```
In older versions of Deno, this would just lock up. I believe after
23ff0e722e, it started throwing a more
explicit error:
```
httpServerStreamDuplexJavascript => ./cli/tests/unit/serve_test.ts:1339:6
error: TypeError: request or response body error: error reading a body from connection: Connection reset by peer (os error 54)
at async Object.pull (ext:deno_web/06_streams.js:810:27)
```
Some people might get think they need to import from this directory,
which could cause confusion and duplicate dependencies. Additionally,
the `vendor` directory has special behaviour in the language server, so
importing from the folder will definitely cause confusion and issues
there.
Fixes this error message:
```
error: missing field `now` at line 32 column 1
```
This would occur if someone used an old version of the deno_cache
library to cache information in the cache then tried to load it with the
latest CLI. Regression in the last patch when migrating to the
deno_cache_dir crate.
Properly handle the `SQLITE_BUSY` error code by retrying the
transaction.
Also wraps database initialization logic in a transaction to protect
against incomplete/concurrent initializations.
Fixes https://github.com/denoland/deno/issues/20116.
The goal of this PR is to address issue #20106 where a `TypeError`
occurs when the variables `uid` and `gid` from `userInfo()` in `node:os`
are reassigned if the user is on Windows. Both `uid` and `gid` are
marked as `const` therefore producing a `TypeError` when the two are
reassigned.
This PR achieves that goal by marking `uid` and `gid` as `let`
To fix bugs around detection of when node emulation is required, we will
just eagerly initialize it. The improvements we make to reduce the
impact of the startup time:
- [x] Process stdin/stdout/stderr are lazily created
- [x] node.js global proxy no longer allocates on each access check
- [x] Process checks for `beforeExit` listeners before doing expensive
shutdown work
- [x] Process should avoid adding global event handlers until listeners
are added
Benchmarking this PR (`89de7e1ff`) vs main (`41cad2179`)
```
12:36 $ third_party/prebuilt/mac/hyperfine --warmup 100 -S none './deno-41cad2179 run ./empty.js' './deno-89de7e1ff run ./empty.js'
Benchmark 1: ./deno-41cad2179 run ./empty.js
Time (mean ± σ): 24.3 ms ± 1.6 ms [User: 16.2 ms, System: 6.0 ms]
Range (min … max): 21.1 ms … 29.1 ms 115 runs
Benchmark 2: ./deno-89de7e1ff run ./empty.js
Time (mean ± σ): 24.0 ms ± 1.4 ms [User: 16.3 ms, System: 5.6 ms]
Range (min … max): 21.3 ms … 28.6 ms 126 runs
```
Fixes https://github.com/denoland/deno/issues/20142
Fixes https://github.com/denoland/deno/issues/15826
Fixes https://github.com/denoland/deno/issues/20028
The goal of this PR is to address issue #19520 where Deno panics when
encountering an invalid SSL certificate.
This PR achieves that goal by removing an `.expect()` statement and
implementing a match statement on `tsl_config` (found in
[/ext/net/ops_tsl.rs](e071382768/ext/net/ops_tls.rs (L1058)))
to check whether the desired configuration is valid
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
Fix #20022, fix #19627 (duplicate)
#17333 upgraded clap from version 3.1 to version 4. clap version 3.2.0
(intentionally) broke a behavior that deno was relying on to make `deno
run --v8-flags=--help` work without specifying a file, see
clap-rs/clap#3793. The workaround was to make the script argument
required _unless_ `--v8-flags` is present. This broke the expectation
that all successfully parsed `run` commands have the script argument
set, leading to the panic on
`matches.remove_many::<String>("script_arg").unwrap()`.
Clap, as far as I was able to find out, does not currently offer a neat
solution to this problem. This PR adds logic to create and return a
custom clap error when a parsed run command does not have the script
argument.
I added an appropriate test.
Handles ASCCI espace chars in test and bench name making
test and bench reporting more reliable. This one is also tested
in the fixture of "node:test" module.
Rename some of the helper methods on the Fs trait to be suffixed with
`_sync` / `_async`, in preparation of the introduction of more async
methods for some helpers.
Also adds a `read_text_file_async` helper to complement the renamed
`read_text_file_sync` helper.
This commit moves `snapshot_from_lockfile` function to [deno_npm
crate](https://github.com/denoland/deno_npm). This allows this function
to be called outside Deno CLI (in particular, Deno Deploy).
Not sure why `40_testing.js` is there. The other two `00_typescript.js`
and `99_main_compiler.js` should be covered by
`files_loaded_during_snapshot` at the end.
This helps with `__runtime_js_sources` wrt changing `40_testing.js`.
Renames the unstable `deno_modules` directory and corresponding settings
to `vendor` after feedback. Also causes the vendoring of the
`node_modules` directory which can be disabled via
`--node-modules-dir=false` or `"nodeModulesDir": false`.
This changes the design of the manifest.json file to have a separate
"folders" map for mapping hashed directories. This allows, for example,
to add files in a folder like `http_localhost_8000/#testing_5de71/` and
have them be resolved automatically as long as their remaining
components are identity-mappable to the file system (not hashed). It
also saves space in the manifest.json file by only including the hashed
directory instead of each descendant file.
```
// manifest.json
{
"folders": {
"https://localhost/NOT_MAPPABLE/": "localhost/#not_mappable_5cefgh"
},
"modules": {
"https://localhost/folder/file": {
"headers": {
"content-type": "application/javascript"
}
},
}
}
// folder structure
localhost
- folder
- #file_2defn (note: I've made up the hashes in these examples)
- #not_mappable_5cefgh
- mod.ts
- etc.ts
- more_files.ts
```
Closes #19399 (running without snapshots at all was suggested as an
alternative solution).
Adds a `__runtime_js_sources` pseudo-private feature to load extension
JS sources at runtime for faster development, instead of building and
loading snapshots or embedding sources in the binary. Will only work in
a development environment obviously.
Try running `cargo test --features __runtime_js_sources
integration::node_unit_tests::os_test`. Then break some behaviour in
`ext/node/polyfills/os.ts` e.g. make `function cpus() {}` return an
empty array, and run it again. Fix and then run again. No more build
time in between.
Chipping away at making tests faster. Appears we don't need double
timeout before sanitizing ops. This should cut baseline cost of running a test
by half.
This commit adds new "--deny-*" permission flags. These are complimentary to
"--allow-*" flags.
These flags can be used to restrict access to certain resources, even if they
were granted using "--allow-*" flags or the "--allow-all" ("-A") flag.
Eg. specifying "--allow-read --deny-read" will result in a permission error,
while "--allow-read --deny-read=/etc" will allow read access to all FS but the
"/etc" directory.
Runtime permissions APIs ("Deno.permissions") were adjusted as well, mainly
by adding, a new "PermissionStatus.partial" field. This field denotes that
while permission might be granted to requested resource, it's only partial (ie.
a "--deny-*" flag was specified that excludes some of the requested resources).
Eg. specifying "--allow-read=foo/ --deny-read=foo/bar" and then querying for
permissions like "Deno.permissions.query({ name: "read", path: "foo/" })"
will return "PermissionStatus { state: "granted", onchange: null, partial: true }",
denoting that some of the subpaths don't have read access.
Closes #18804.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
This commit adds a "dot" reporter to "deno test" subcommand,
that can be activated using "--dot" flag.
It provides a concise output using:
- "." for passing test
- "," for ignored test
- "!" for failing test
User output is silenced and not printed to the console.
In non-TTY environments each result is printed on a separate line.
This commit provides basic polyfill for "node:test" module. Currently
only top-level "test" function is polyfilled, all remaining functions from
that module throw not implemented errors.
Closes https://github.com/denoland/deno/issues/17251
Closes #19970
This commits adds logic to retry failed module downloads once.
Both request and server errors are handled and the retry is done after
50 ms wait time.
I'm not sure why, but sending SIGABRT to Deno on my machine as part of
this test causes it to lock up very badly, leaving it in an unkillable
`UE+` state.
This showed up after #19333, but was not caused by it.
Closes #17589.
```ts
Deno.bench("foo", async (t) => {
const resource = setup(); // not included in measurement
t.start();
measuredOperation(resource);
t.end();
resource.close(); // not included in measurement
});
```
This PR fixes #19818. The problem was that the new InnerRequest class does not initialize the fields urlList and urlListProcessed that are used during a request clone. The solution aims to be straightforward by simply initializing the missing properties during the clone process. I also implemented a "cache" to the url getter of the new InnerRequest, avoiding the cost of calling op_http_get_request_method_and_url.
Source code is now not cloned anymore between eszip and deno_core, as
eszip now
returns `Arc<[u8]>`, that can be trivially casted into `Arc<str>`, which
`deno_core` can consume without copying.
This commit makes the following changes
- Created a `CompoundTestReporter` to allow us to use multiple reporters
- Implements `JUnitTestReporter` which writes JUnit XML to a path
- Added a CLI flag/option `--junit` that enables JUnit reporting. By
default this writes the report to `stdout` (and disables pretty
reporting). If a path is provided, it will write the JUnit report to
that file while the pretty reporter writes to stdout like normal
Output of `deno -- test --allow-all --unstable
--location=http://js-unit-tests/foo/bar --junit
cli/tests/unit/testing_test.ts `
```xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="deno test" tests="7" failures="0" errors="0" time="0.176">
<testsuite name="file:///Users/cooper/deno/deno/cli/tests/unit/testing_test.ts" tests="7" disabled="0" errors="0" failures="0">
<testcase name="testWrongOverloads" time="0.012">
</testcase>
<testcase name="nameOfTestCaseCantBeEmpty" time="0.009">
</testcase>
<testcase name="invalidStepArguments" time="0.008">
</testcase>
<testcase name="nameOnTextContext" time="0.029">
<properties>
<property name="step[passed]" value="step ... nested step"/>
<property name="step[passed]" value="step"/>
</properties>
</testcase>
<testcase name="originOnTextContext" time="0.030">
<properties>
<property name="step[passed]" value="step ... nested step"/>
<property name="step[passed]" value="step"/>
</properties>
</testcase>
<testcase name="parentOnTextContext" time="0.030">
<properties>
<property name="step[passed]" value="step ... nested step"/>
<property name="step[passed]" value="step"/>
</properties>
</testcase>
<testcase name="explicit undefined for boolean options" time="0.009">
</testcase>
</testsuite>
</testsuites>
```
We weren't auto-discovering the deno.json in two cases:
1. A project that didn't have a deno.json and just added one.
2. After a syntax error in the deno.json.
This now rediscovers it in both these cases.
Closes https://github.com/denoland/vscode_deno/issues/867
Code run within Deno-mode and Node-mode should have access to a
slightly different set of globals. Previously this was done through a
compile time code-transform for Node-mode, but this is not ideal and has
many edge cases, for example Node's globalThis having a different
identity than Deno's globalThis.
This commit makes the `globalThis` of the entire runtime a semi-proxy.
This proxy returns a different set of globals depending on the caller's
mode. This is not a full proxy, because it is shadowed by "real"
properties on globalThis. This is done to avoid the overhead of a full
proxy for all globalThis operations.
The globals between Deno-mode and Node-mode are now properly segregated.
This means that code running in Deno-mode will not have access to Node's
globals, and vice versa. Deleting a managed global in Deno-mode will
NOT delete the corresponding global in Node-mode, and vice versa.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: Aapo Alasuutari <aapo.alasuutari@gmail.com>
Closes https://github.com/denoland/deno/issues/15277
This commit adds a single "warmup" run of empty function when running
`deno bench`.
This change will break so-called "JIT bias" which makes V8 optimize the
first function
and then bail out of optimization on second function. In essence the
"warmup" function
is getting optimized and then all user benches are bailed out of
optimization.
This PR breaks the addition of the `TestReporter` trait and refactoring
of `PrettyTestReporter` out of #19747. The goal is to enable the
addition of test reporters, including machine readable output.
Part of #19774. This makes it twice as fast on my machine.
Stores a file at `node_modules/.deno/setup-cache.bin`, which contains
information about how the node_modules folder is currently setup.
Obviously there is a risk that this information will get out of date
with the current folder structure.
This commit adds some regression tests for using `jsxImportSource` in
the config file in combination with an import map.
These underlying issues were fixed by #15561.
Closes #13389
Closes #14723
---------
Co-authored-by: Aapo Alasuutari <aapo.alasuutari@gmail.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit stabilizes "Deno.serve()", which becomes the
preferred way to create HTTP servers in Deno.
Documentation was adjusted for each overload of "Deno.serve()"
API and the API always binds to "127.0.0.1:8000" by default.
