This commit adds some regression tests for using `jsxImportSource` in
the config file in combination with an import map.
These underlying issues were fixed by #15561.
Closes #13389
Closes #14723
---------
Co-authored-by: Aapo Alasuutari <aapo.alasuutari@gmail.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit stabilizes "Deno.serve()", which becomes the
preferred way to create HTTP servers in Deno.
Documentation was adjusted for each overload of "Deno.serve()"
API and the API always binds to "127.0.0.1:8000" by default.
This PR changes Web IDL interfaces to be declared with `var` instead of
`class`, so that accessing them via `globalThis` does not raise type
errors.
Closes #13390.
This is a fix for issue #19644, concerning the `parseCssColor` function
in the file `ext/console/01_console.js`. Changes made on lines
2756-2758. To sum it up:
> The internal `parseCssColor` function currently parses 3/4-digit hex
colors incorrectly. For example, it parses the string `#FFFFFF` as
`[255, 255, 255]` (as expected), but returns `[240, 240, 240]` for
`#FFF`, when it should return the same triplet as the former.
While it's not going to cause a fatal runtime error, it did bug me
enough to fix it real quick.
…nclusion" (#19519)"
This reverts commit 28a4f3d0f5.
This change causes failures when used outside Deno repo:
```
============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.
Platform: linux x86_64
Version: 1.34.3+b37b286
Args: ["/opt/hostedtoolcache/deno/0.0.0-b37b286f7fa68d5656f7c180f6127bdc38cf2cf5/x64/deno", "test", "--doc", "--unstable", "--allow-all", "--coverage=./cov"]
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Failed to read "/home/runner/work/deno/deno/core/00_primordials.js"
Caused by:
No such file or directory (os error 2)', core/runtime/jsruntime.rs:699:8
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
Relands #19463. This time the `ExtensionFileSourceCode` enum is
preserved, so this effectively just splits feature
`include_js_for_snapshotting` into `exclude_js_sources` and
`runtime_js_sources`, adds a `force_include_js_sources` option on
`extension!()`, and unifies `ext::Init_ops_and_esm()` and
`ext::init_ops()` into `ext::init()`.
Fixes #19568
Values are not coerced to the desired type during deserialisation. This
makes serde_v8 stricter.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
… (#19463)"
This reverts commit ceb03cfb03.
This is being reverted because it causes 3.5Mb increase in the binary
size,
due to runtime JS code being included in the binary, even though it's
already snapshotted.
CC @nayeemrmn
This prevents documents specified in a deno.json's "exclude" from being
pre-loaded by the lsp.
For example, someone may have something like:
```jsonc
// deno.json
{
"exclude": [
"dist" // build directory
]
}
```
Remove `ExtensionFileSourceCode::LoadedFromFsDuringSnapshot` and feature
`include_js_for_snapshotting` since they leak paths that are only
applicable in this repo to embedders. Replace with feature
`exclude_js_sources`. Additionally the feature
`force_include_js_sources` allows negating it, if both features are set.
We need both of these because features are additive and there must be a
way of force including sources for snapshot creation while still having
the `exclude_js_sources` feature. `force_include_js_sources` is only set
for build deps, so sources are still excluded from the final binary.
You can also specify `force_include_js_sources` on any extension to
override the above features for that extension. Towards #19398.
But there was still the snapshot-from-snapshot situation where code
could be executed twice, I addressed that by making `mod_evaluate()` and
scripts like `core/01_core.js` behave idempotently. This allowed
unifying `ext::init_ops()` and `ext::init_ops_and_esm()` into
`ext::init()`.
I'm unsure why we canonicalize the config file path when loading and the
canonicalization is causing issues in #19431 because everything in the
lsp is not canonicalized except the config file (actually, the config
file is only canonicalized when auto-discovered and not whens pecified).
We also don't canonicalize module paths when loading them.
Canonicalization was added in https://github.com/denoland/deno/pull/7621
This commit adds ability to print metrics of the Tokio
runtime to the console by passing "DENO_TOKIO_METRICS=1"
env var.
Metrics will be printed every second, but this can be changed
by "DENO_TOKIO_METRICS_INTERVAL=500" env var.
Reduce the GC pressure from the websocket event method by splitting it
into an event getter and a buffer getter.
Before:
165.9k msg/sec
After:
169.9k msg/sec
This commit migrates "deno_core" from using "FuturesUnordered" to
"tokio::task::JoinSet". This makes every op to be a separate Tokio task
and should unlock better utilization of kqueue/epoll.
There were two quirks added to this PR:
- because of the fact that "JoinSet" immediately polls spawn tasks,
op sanitizers can give false positives in some cases, this was
alleviated by polling event loop once before running a test with
"deno test", which gives canceled ops an opportunity to settle
- "JsRuntimeState::waker" was moved to "OpState::waker" so that FFI
API can still use threadsafe functions - without this change the
registered wakers were wrong as they would not wake up the
whole "JsRuntime" but the task associated with an op
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
This commit adds basic support for "node:http2" module. Not
all APIs have been yet implemented, but this change already
allows to use this module for some basic functions.
The "grpc" package is still not working, but it's a good stepping
stone.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
## WHY
ref: https://github.com/denoland/deno/issues/19165
The FileHandle class has many missing methods compared to node.
Add these.
## WHAT
- Add close method
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
Related issue: https://github.com/denoland/deno/issues/19358.
This is a regression that seems to have been introduced in
https://github.com/denoland/deno/pull/18905. It looks to have been a
performance optimization.
The issue is probably easiest described with some code:
```ts
const target = new EventTarget();
const event = new Event("foo");
target.addEventListener("foo", () => {
console.log('base');
target.addEventListener("foo", () => {
console.log('nested');
});
});
target.dispatchEvent(event);
```
Essentially, the second event listener is being attached while the `foo`
event is still being dispatched. It should then not fire that second
event listener, but Deno currently does.
<!--
Before submitting a PR, please read https://deno.com/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
Internally, `node-tap` spawns a child process with `stdio: [0, 1, 2]`.
Whilst we don't support passing fd numbers as an argument so far, it
turns out that `[0, 1, 2]` is equivalent to `"inherit"` which we already
support. See: https://nodejs.org/api/child_process.html#optionsstdio
Mapping it to `"inherit"` is fine for us and gets us one step closer in
getting `node-tap` working. I'm now at the stage where already the
coverage table is shown 🎉
<!--
Before submitting a PR, please read https://deno.com/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
## WHY
ref: https://github.com/denoland/deno/issues/19165
Node's fs/promises includes a FileHandle class, but deno does not. The
open function in Node's fs/promises returns a FileHandle, which provides
an IO interface to the file. However, deno's open function returns a
resource id.
### deno
```js
> const fs = await import("node:fs/promises");
undefined
> const file3 = await fs.open("./README.md");
undefined
> file3
3
> file3.read
undefined
Node:
```
### Node
```js
> const fs = await import("fs/promises");
undefined
> const file3 = await fs.open("./tests/e2e_unit/testdata/file.txt");
undefined
> file3
FileHandle {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
close: [Function: close],
[Symbol(kCapture)]: false,
[Symbol(kHandle)]: FileHandle {},
[Symbol(kFd)]: 24,
[Symbol(kRefs)]: 1,
[Symbol(kClosePromise)]: null
}
> file3.read
[Function: read]
```
To be compatible with Node, deno's open function should also return a
FileHandle.
## WHAT
I have implemented the first step in adding a FileHandle.
- Changed the return value of the open function to a FileHandle object
- Implemented the readFile method in FileHandle
- Add test code
## What to do next
This PR is the first step in adding a FileHandle, and there are things
that should be done next.
- Add functionality equivalent to Node's FileHandle to FileHandle
(currently there is only readFile)
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
It's not used anymore. Subsequently allows removing
`ModuleMap::op_state`, allowing `ModuleMap` to have a sane default so
`JsRuntime::module_map` no longer needs to be optional.
For the first implementation of node:http2, we'll use the internal
version of `Deno.serve` which allows us to listen on a raw TCP
connection rather than a listener.
This is mostly a refactoring, and hooking up of `op_http_serve_on` that
was never previously exposed (but designed for this purpose).
This cleans up `JsRuntime` a bit more:
* We no longer print cargo's rerun-if-changed messages in `JsRuntime` --
those are printed elsewhere
* We no longer special case the OwnedIsolate for snapshots. Instead we
make use of an inner object that has the `Drop` impl and allows us to
`std::mem::forget` it if we need to extract the isolate for a snapshot
* The `snapshot` method is only available on `JsRuntimeForSnapshot`, not
`JsRuntime`.
* `OpState` construction is slightly cleaner, though I'd still like to
extract more
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This PR adds the missing `process.reallyExit()` method to node's
`process` object.
Was [pinged on
twitter](https://twitter.com/biwanczuk/status/1663326659787862017)
regarding running the `fastify` test suite in node. They use `node-tap`
which has been around arguably the longest of the test frameworks and
relies on a couple of old APIs. They have `signal-exit` as a dependency
which in turn [makes use of
`process.reallyExit()`](8fa7fc9a9c/src/index.ts (L19)).
That function cannot be found anywhere in their documentation, but
exists at runtime. See
6a6b3c5402/lib/internal/bootstrap/node.js (L172)
This doesn't yet make `node-tap` work, but gets us one step closer.
Part of #17840
I haven't made any changes or additions to the tests themselves, just
moved the tests over and updated to match.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Rather than disallowing `ext:` resolution, clear the module map after
initializing extensions so extension modules are anonymized. This
operation is explicitly called in `deno_runtime`. Re-inject `node:`
specifiers into the module map after doing this.
Fixes #17717.
If a symlink within the `node_modules` directory lies outside that
directory, it will now warn and inline the file. For directories, it
will just warn for now.
Probably fixes #19251 (I'm still unable to reproduce).
This commit fixes problem with loading N-API modules that use
the "old" way of registration (using "napi_module_register" API).
The slot was not cleared after loading modules, causing subsequent
calls that use the new way of registration (using
"napi_register_module_v1" API) to try and load the previous module.
Ref https://github.com/denoland/deno/issues/16460
---------
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
`isFile`, `isDirectory`, `isSymlink` are defined in `Deno.FileInfo`, but
`isBlockDevice`, `isCharacterDevice`, `isFIFO`, `isSocket` are not
defined.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This is what pnpm does and we were missing it. It makes modules work
which have a dependency on something, but don't say they have that
dependency, but that dep is still in the tree somewhere.
This commit fixes emitting "unhandledrejection" event when there are
"node:" or "npm:" imports.
Before this commit the Node "unhandledRejection" event was emitted
using a regular listener for Web "unhandledrejection" event. This
listener was installed before any user listener had a chance to be
installed which effectively prevent emitting "unhandledrejection"
events to user code.
Closes https://github.com/denoland/deno/issues/16928
When someone explicitly opts into using the node_modules dir via
`--node-modules-dir` or setting `"nodeModulesDir": true` in the
deno.json file, we should eagerly ensure a top level package.json
install is done on startup. This was initially always done when we added
package.json support and a package.json was auto-discovered, but scaled
it back to be lazily done when a bare specifier matched an entry in the
package.json because of how disruptive it was for people using Deno
scripts in Node projects. That said, it does not make sense for someone
to opt-into having deno control and use their node_modules directory and
not want a package.json install to occur. If such a rare scenario
exists, the `DENO_NO_PACKAGE_JSON=1` environment variable can be set.
Ideally, we would only ever use a node_modules directory with this
explicit opt-in so everything is very clear, but we still have this
automatic scenario when there's a package.json in order to make more
node projects work out of the box.
Before:
```
$ cargo run -- test "foo/*******/bar.ts"
error: Pattern syntax error near position 6: wildcards are either regular `*` or recursive `**`
```
After:
```
$ cargo run -- test "foo/*******/bar.ts"
error: Failed to expand glob: "foo/*******/bar.ts"
Caused by:
Pattern syntax error near position 6: wildcards are either regular `*` or recursive `**`
```
---------
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
Noticed that we're checking more module paths than necessary. In
particular the module path array contains a couple of entries with a
duplicated `node_modules/node_modules` suffix.
```js
[
// ... more entries before here, where some also contain duplicate suffixes
"/Users/marvinhagemeister/dev/preact-render-to-string/node_modules/.deno/node_modules",
"/Users/marvinhagemeister/dev/preact-render-to-string/node_modules/node_modules", // <-- duplicate suffix
"/Users/marvinhagemeister/dev/preact-render-to-string/node_modules",
"/Users/marvinhagemeister/dev/node_modules",
"/Users/marvinhagemeister/node_modules",
"/Users/node_modules",
"/node_modules",
"/node_modules" // <-- duplicate entry
]
```
This was caused by a misunderstanding in how Rust's
[`Path::ends_with()`](https://doc.rust-lang.org/std/path/struct.Path.html#method.ends_with)
works. It's designed to match on whole path segments and the suffix
`/node_modules` is not that, except for the root entry. This meant that
our check for if the path already ended with `node_module` always
returned `false`. Removing the leading slash fixes that.
While we're at it, we can remove the last condition where we explicitly
added the root `/node_modules` entry since the while loop prior to that
takes care of it already.
Follow up to https://github.com/denoland/deno/pull/19084.
This commit adds support for globs in the configuration file as well
as CLI arguments for files.
With this change users can now use glob syntax for "include" and
"exclude" fields, like so:
```json
{
"lint": {
"include": [
"directory/test*.ts",
"other_dir/"
],
"exclude": [
"other_dir/foo*.ts",
"nested/nested2/*"
]
},
"test": {
"include": [
"data/test*.ts",
"nested/",
"tests/test[1-9].ts"
],
"exclude": [
"nested/foo?.ts",
"nested/nested2/*"
]
}
}
```
Or in CLI args like so:
```
// notice quotes here; these values will be passed to Deno verbatim
// and deno will perform glob expansion
$ deno fmt --ignore="data/*.ts"
$ deno lint "data/**/*.ts"
```
Closes https://github.com/denoland/deno/issues/17971
Closes https://github.com/denoland/deno/issues/6365
This adds support for the lockfile and node_modules directory to the
lsp.
In the case of the node_modules directory, it is only enabled when
explicitly opted into via `"nodeModulesDir": true` in the configuration
file. This is to reduce the language server automatically modifying the
node_modules directory when the user doesn't want it to.
Closes #16510
Closes #16373
This runs our `js_unit_tests` and `node_unit_tests` in parallel, one
rust test per JS unit test file. Some of our JS tests don't like running
in parallel due to port requirements, so this also makes those use a
specific port-per-file. This does not attempt to make the node-compat
tests work.
We don't need to use the `deno` command here to test kill permissions
and it's awkward to get right without passing `-A`. `cat` works, but for
platforms other than windows. This test should have plenty of coverage
on other platforms.
This commit changes the implementation of `ext/web` timers, by using
"op_void_async_deferred" for timeouts of 0ms.
0ms timeout is meant to be run at the end of the event loop tick and
currently Tokio timers that we use to back timeouts have at least 1ms
resolution. That means that 0ms timeout actually take >1ms. This
commit changes that and runs 0ms timeout at the end of the event
loop tick.
One consequence is that "unrefing" a 0ms timer will actually keep
the event loop alive (which I believe actually makes sense, the test
we had only worked because the timeout took more than 1ms).
Ref https://github.com/denoland/deno/issues/19034
We never properly added support for this. This fixes vendoring when it
has npm or node specifiers. Vendoring occurs by adding a
`"nodeModulesDir": true` property to deno.json then it uses a local
node_modules directory. This can be opted out by setting
`"nodeModulesDir": false` or running with `--node-modules-dir=false`.
Closes #18090
Closes #17210
Closes #17619
Closes #16778
This reverts commit 798c1ad0f1.
Reverting because this change caused a spike in memory usage, but we
can't fully realise gains from lower GC pressure from more optimal
malloc/ free provided by "jemalloc".
We might revisit the topic in future.
This commit changes the return type of an unstable `Deno.serve()` API
to instead return a `Deno.Server` object that has a `finished` field.
This change is done in preparation to be able to ref/unref the HTTP
server.
**THIS PR HAS GIT CONFLICTS THAT MUST BE RESOLVED**
This is the release commit being forwarded back to main for 1.33.4
Please ensure:
- [x] Everything looks ok in the PR
- [ ] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.33.4 && git checkout -b forward_v1.33.4 upstream/forward_v1.33.4
```
Don't need this PR? Close it.
cc @levex
Co-authored-by: levex <levex@users.noreply.github.com>
Co-authored-by: Levente Kurusa <lkurusa@kernelstuff.org>
This is a bit bare bones but gets `npm-run-all` working. For full stdio
compatibility with node more work is needed which is probably better
done in follow up PRs.
Fixes #19159
Doesn't make the API bullet-proof and there are some TODOs left,
but greatly improves the situation. Tests were ported from Node.js.
Closes https://github.com/denoland/deno/issues/18276.
Note: If the package information has already been cached, then this
requires running with `--reload` or for the registry information to be
fetched some other way (ex. the cache busting).
Closes #15544
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit reimplements most of "node:http" client APIs using
"ext/fetch".
There is some duplicated code and two removed Node compat tests that
will be fixed in follow up PRs.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Fixes for various `Attemped to access invalid request` bugs (#19058,
#15427, #17213).
We did not wait for both a drop event and a completion event before
removing items from the slab table. This ensures that we do so.
In addition, the slab methods are refactored out into `slab.rs` for
maintainability.
Currently the `multipart/form-data` parser in
`Request.prototype.formData` and `Response.prototype.formData` decodes
non-ASCII filenames incorrectly, as if they were encoded in Latin-1
rather than UTF-8. This happens because the header section of each
`multipart/form-data` entry is decoded as Latin-1 in order to be parsed
with `Headers`, which only allows `ByteString`s, but the names and
filenames are never decoded correctly. This PR fixes this as a
post-processing step.
Note that the `multipart/form-data` parsing for this APIs in the Fetch
spec is very much underspecified, and it does not specify that names and
filenames must be decoded as UTF-8. However, it does require that the
bodies of non-`File` entries are decoded as UTF-8, and in browsers,
names and filenames always use the same encoding as the body.
Closes #19142.
Partially supersedes #19016.
This migrates `spawn` and `spawn_blocking` to `deno_core`, and removes
the requirement for `spawn` tasks to be `Send` given our single-threaded
executor.
While we don't need to technically do anything w/`spawn_blocking`, this
allows us to have a single `JoinHandle` type that works for both cases,
and allows us to more easily experiment with alternative
`spawn_blocking` implementations that do not require tokio (ie: rayon).
Async ops (+~35%):
Before:
```
time 1310 ms rate 763358
time 1267 ms rate 789265
time 1259 ms rate 794281
time 1266 ms rate 789889
```
After:
```
time 956 ms rate 1046025
time 954 ms rate 1048218
time 924 ms rate 1082251
time 920 ms rate 1086956
```
HTTP serve (+~4.4%):
Before:
```
Running 10s test @ http://localhost:4500
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 68.78us 19.77us 1.43ms 86.84%
Req/Sec 68.78k 5.00k 73.84k 91.58%
1381833 requests in 10.10s, 167.36MB read
Requests/sec: 136823.29
Transfer/sec: 16.57MB
```
After:
```
Running 10s test @ http://localhost:4500
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 63.12us 17.43us 1.11ms 85.13%
Req/Sec 71.82k 3.71k 77.02k 79.21%
1443195 requests in 10.10s, 174.79MB read
Requests/sec: 142921.99
Transfer/sec: 17.31MB
```
Suggested-By: alice@ryhl.io
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
**THIS PR HAS GIT CONFLICTS THAT MUST BE RESOLVED**
This is the release commit being forwarded back to main for 1.33.3
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.33.3 && git checkout -b forward_v1.33.3 upstream/forward_v1.33.3
```
Don't need this PR? Close it.
cc @levex
Co-authored-by: Levente Kurusa <lkurusa@kernelstuff.org>
Adds a `deno.preloadLimit` option (ex. `"deno.preloadLimit": 2000`)
which specifies how many file entries to traverse on the file system
when the lsp loads or its configuration changes.
Closes #18955
This PR ensures that node's `worker_threads` module exports
`MessageChannel`, `MessagePort` and the `BroadcastChannel` API. Fixing
these won't make `esbuild` work, but brings us one step closer 🎉
Fixes #19028 .
This is the initial support for npm and node specifiers in `deno
compile`. The npm packages are included in the binary and read from it via
a virtual file system. This also supports the `--node-modules-dir` flag,
dependencies specified in a package.json, and npm binary commands (ex.
`deno compile --unstable npm:cowsay`)
Closes #16632
`Content-Encoding: gzip` support for `Deno.serve`. This doesn't support
Brotli (`br`) yet, however it should not be difficult to add. Heuristics
for compression are modelled after those in `Deno.serveHttp`.
Tests are provided to ensure that the gzip compression is correct. We
chunk a number of different streams (zeros, hard-to-compress data,
already-gzipped data) in a number of different ways (regular, random,
large/small, small/large).
This now allows circular imports across extensions.
Instead of load + eval of all ESM files in declaration order, all files
are only loaded. Eval is done recursively by V8, only evaluating
files that are listed in `Extension::esm_entry_point` fields.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Fixes #16699 and #18960 by ensuring that we release our HTTP
`spawn_local` tasks when the HTTP resource is dropped.
Because our cancel handle was being projected from the resource via
`RcMap`, the resource was never `Drop`ped. By splitting the handle out
into its own `Rc`, we can avoid keeping the resource alive and let it
drop to cancel everything.
1. Boxed `File` and `FileSystem` to allow more easily passing this
through the CLI code (as shown within this pr).
2. `StdFileResource` is now `FileResource`. `FileResource` now contains
an `Rc<dyn File>`.
**THIS PR HAS GIT CONFLICTS THAT MUST BE RESOLVED**
This is the release commit being forwarded back to main for 1.33.2
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.33.2 && git checkout -b forward_v1.33.2 upstream/forward_v1.33.2
```
Don't need this PR? Close it.
cc @levex
Co-authored-by: levex <levex@users.noreply.github.com>
Co-authored-by: Levente Kurusa <lkurusa@kernelstuff.org>
Fixes #18979.
This changes the predicate for allowing `ext:` specifier resolution from
`snapshot_loaded_and_not_snapshotting` to `ext_resolution_allowed` which
is only set to true during the extension module loading phase. Module
loaders as used in core
are now declared as `ExtModuleLoader` rather than `dyn ModuleLoader`.
`new Deno.KvU64(1n) + 2n == 3n` is now true.
`new Deno.KvU64(1n)` is now inspected as `[Deno.KvU64: 1n]`
(`Object(1n)` is inspected as `[BigInt: 1n]`).
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit changes how paths for npm packages are handled,
by canonicalizing them when resolving. This is done so that instead
of returning
"node_modules/<package_name>@<version>/node_modules/<dep>/index.js"
(which is a symlink) we "node_modules/<dep>@<dep_version>/index.js.
Fixes https://github.com/denoland/deno/issues/18924
Fixes https://github.com/bluwy/create-vite-extra/issues/31
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
This removes `ProcState` and replaces it with a new `CliFactory` which
initializes our "service structs" on demand. This isn't a performance
improvement at the moment for `deno run`, but might unlock performance
improvements in the future.
- Do not use `ReflectHas` in `isNode`.
- Avoid copying handler array when handlers.length == 1
- Avoid searching for path target when path.length == 1
```
Linux divy-2 5.19.0-1022-gcp #24~22.04.1-Ubuntu SMP Sun Apr 23 09:51:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
32GiB System memory
Intel(R) Xeon(R) CPU @ 3.10GHz
# main + https://github.com/denoland/deno/pull/18904
Msg/sec: 89326.750000
Msg/sec: 90320.000000
Msg/sec: 89576.250000
# this patch
Msg/sec: 97250.000000
Msg/sec: 97125.500000
Msg/sec: 97964.500000
```
Fixes regression from #18878 where `Promise.reject()`,
`Promise.reject(undefined)` and `reportError(undefined)` panic in the
REPL.
Fixes `throw undefined` printing `Uncaught Unknown exception` instead of
`Uncaught undefined`.
Fixes #8858.
Fixes #8869.
```
$ target/debug/deno
Deno 1.32.5
exit using ctrl+d, ctrl+c, or close()
REPL is running with all permissions allowed.
To specify permissions, run `deno repl` with allow flags.
> Promise.reject(new Error("bar"));
Promise { <rejected> Error: bar
at <anonymous>:2:16 }
Uncaught (in promise) Error: bar
at <anonymous>:2:16
> reportError(new Error("baz"));
undefined
Uncaught Error: baz
at <anonymous>:2:13
>
Towards https://github.com/denoland/deno/issues/18455
`safe`, `add` and `rem` options are not implemented because there is no
rust crate that provides this functionality (except rust-openssl maybe)
and its just not clear if this API is used widely.
Hey there! I took a crack at improving these embedded docs [as requested
here](https://github.com/denoland/deno/issues/18685). These should
accurately reflect the functionality of the permission-related flags for
`deno run`.
### Highlights
* Adds human-readable argument string in the format [prescribed in the
docs](https://docs.rs/clap/latest/clap/struct.Arg.html#method.value_name)
* Keeps text description terse, but includes a relevant copy/pasteable
docs link
* Includes example argument usage/formatting
This commit adds `@deprecated` comments to `Deno.run` API declarations.
Since stabilization of `Deno.Command` API in [Deno
v1.31](https://deno.com/blog/v1.31#api-stabilizations), `Deno.Command`
is the preferred (more reliable) API to interact with subprocesses.
This is the preparation for the removal of `Deno.run` API in Deno 2.0.
We can make `NodePermissions` rely on interior mutability (which the
`PermissionsContainer` is already doing) in order to not have to clone
everything all the time. This also reduces the chance of an accidental
`borrow` while `borrrow_mut`.
This implements HTTP/2 prior-knowledge connections, allowing clients to
request HTTP/2 over plaintext or TLS-without-ALPN connections. If a
client requests a specific protocol via ALPN (`h2` or `http/1.1`),
however, the protocol is forced and must be used.
We were indeterministically including packages in the top level
`node_modules/` folder when using a local node_modules directory. This
change aligns with pnpm and only includes top level packages in this
folder. This should be faster for initializing the folder, but may
expose issues in packages that reference other packages not defined in
their dependencies. That said, the behaviour previously was previously
broken.
This has exposed a bug in the require implementation where it doesn't
find a package (which is the main underlying issue here). There is a
failing test already for this in the test suite after this change.
Closes #18822
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This is a rewrite of the `Deno.serve` API to live on top of hyper
1.0-rc3. The code should be more maintainable long-term, and avoids some
of the slower mpsc patterns that made the older code less efficient than
it could have been.
Missing features:
- `upgradeHttp` and `upgradeHttpRaw` (`upgradeWebSocket` is available,
however).
- Automatic compression is unavailable on responses.
This is just a straight refactor and I didn't do any cleanup in
ext/node. After this PR we can start to clean it up and make things
private that don't need to be public anymore.
1. Adds cli/standalone folder
2. Writes the bytes directly to the output file. When adding npm
packages this might get quite large, so let's not keep the final output
in memory just in case.
Towards #18455
This commit implements the keypair generation for asymmetric keys for
the `generateKeyPair` API.
See how key material is managed in this implementation:
https://www.notion.so/denolandinc/node-crypto-design-99fc33f568d24e47a5e4b36002c5325d?pvs=4
Private and public key encoding depend on `KeyObject#export` which is
not implemented. I've also skipped ED448 and X448 since we need a crate
for that in WebCrypto too.
```
./target/release/deno run cli/bench/async_ops_deferred.js
time 794 ms rate 1259445
time 786 ms rate 1272264
time 770 ms rate 1298701
time 784 ms rate 1275510
time 775 ms rate 1290322
time 786 ms rate 1272264
time 773 ms rate 1293661
time 771 ms rate 1297016
time 774 ms rate 1291989
time 767 ms rate 1303780
time 764 ms rate 1308900
time 768 ms rate 1302083
time 763 ms rate 1310615
time 761 ms rate 1314060
time 761 ms rate 1314060
time 762 ms rate 1312335
time 763 ms rate 1310615
time 759 ms rate 1317523
time 760 ms rate 1315789
time 761 ms rate 1314060
time 769 ms rate 1300390
time 763 ms rate 1310615
time 760 ms rate 1315789
time 763 ms rate 1310615
time 761 ms rate 1314060
time 759 ms rate 1317523
time 765 ms rate 1307189
time 760 ms rate 1315789
time 764 ms rate 1308900
time 763 ms rate 1310615
time 760 ms rate 1315789
time 757 ms rate 1321003
time 763 ms rate 1310615
time 759 ms rate 1317523
time 771 ms rate 1297016
time 759 ms rate 1317523
time 759 ms rate 1317523
time 763 ms rate 1310615
time 754 ms rate 1326259
time 755 ms rate 1324503
time 762 ms rate 1312335
time 752 ms rate 1329787
time 755 ms rate 1324503
time 754 ms rate 1326259
time 759 ms rate 1317523
time 754 ms rate 1326259
time 749 ms rate 1335113
time 753 ms rate 1328021
time 756 ms rate 1322751
time 753 ms rate 1328021
```
```
samply record -r 20000 target/release/deno run cli/bench/async_ops_deferred.js
```
https://share.firefox.dev/43Efvm6
1. Breaks up functionality within `ProcState` into several other structs
to break out the responsibilities (`ProcState` is only a data struct
now).
2. Moves towards being able to inject dependencies more easily and have
functionality only require what it needs.
3. Exposes `Arc<T>` around the "service structs" instead of it being
embedded within them. The idea behind embedding them was to reduce the
verbosity of needing to pass around `Arc<...>`, but I don't think it was
exactly working and as we move more of these structs to be more
injectable I don't think the extra verbosity will be a big deal.
Fixes https://github.com/denoland/deno/issues/18700
Timeline of the events that lead to the bug.
1. WebSocket handshake complete
2. Server on `read_frame` holding an AsyncRefCell borrow of the
WebSocket stream.
3. Client sends a TXT frame after a some time
4. Server recieves the frame and goes back to `read_frame`.
5. After some time, Server starts a `write_frame` but `read_frame` is
still holding a borrow!
^--- Locked. read_frame needs to complete so we can resume the write.
This commit changes all writes to directly borrow the
`fastwebsocket::WebSocket` resource under the assumption that it won't
affect ongoing reads.
This commit refactors "deno_core" to do fewer boundary crossings
from Rust to V8. In other words we are now calling V8 from Rust fewer
times.
This is done by merging 3 distinct callbacks into a single one. Instead
of having "op resolve" callback, "next tick" callback and "macrotask
queue" callback, we now have only "Deno.core.eventLoopTick" callback,
which is responsible for doing the same actions previous 3 callbacks.
On each of the event loop we were doing at least 2 boundary crosses
(timers macrotask queue callback and unhandled promise rejection
callback) and up to 4 crosses if there were op response and next tick
callbacks coming from Node.js compatibility layer. Now this is all done
in a single callback.
Closes https://github.com/denoland/deno/issues/18620
Removes the functions in the `emit` module and replaces them with an
`Emitter` struct that can have "ctor dependencies" injected rather than
using functions to pass along the dependencies.
This is part of a long term refactor to move more functionality out of
proc state.
Stores the test/bench functions in rust op state during registration.
The functions are wrapped in JS first so that they return a directly
convertible `TestResult`/`BenchResult`. Test steps are still mostly
handled in JS since they are pretty much invoked by the user. Allows
removing a bunch of infrastructure for communicating between JS and
rust. Allows using rust utilities for things like shuffling tests
(`Vec::shuffle`). We can progressively move op and resource sanitization
to rust as well.
Fixes #17122.
Fixes #17312.
This commit changes "eager ops" to directly return a response value
instead of calling "opresponse" callback in JavaScript. This saves
one boundary crossing and has a fantastic impact on the "async_ops.js"
benchmark:
```
v1.32.4
$ deno run cli/bench/async_ops.js
time 329 ms rate 3039513
time 322 ms rate 3105590
time 307 ms rate 3257328
time 301 ms rate 3322259
time 303 ms rate 3300330
time 306 ms rate 3267973
time 300 ms rate 3333333
time 301 ms rate 3322259
time 301 ms rate 3322259
time 301 ms rate 3322259
time 302 ms rate 3311258
time 301 ms rate 3322259
time 302 ms rate 3311258
time 302 ms rate 3311258
time 303 ms rate 3300330
```
```
this branch
$ ./target/release/deno run -A cli/bench/async_ops.js
time 257 ms rate 3891050
time 248 ms rate 4032258
time 251 ms rate 3984063
time 246 ms rate 4065040
time 238 ms rate 4201680
time 227 ms rate 4405286
time 228 ms rate 4385964
time 229 ms rate 4366812
time 228 ms rate 4385964
time 226 ms rate 4424778
time 226 ms rate 4424778
time 227 ms rate 4405286
time 228 ms rate 4385964
time 227 ms rate 4405286
time 228 ms rate 4385964
time 227 ms rate 4405286
time 229 ms rate 4366812
time 228 ms rate 4385964
```
Prerequisite for https://github.com/denoland/deno/pull/18652
The CLI docs suggested that all deno subcommands no longer type-check by
default. This is only the case for some subcommands, and this PR
clarifies the CLI docs in this regard.
- bump deps: the newest `lazy-regex` need newer `oncecell` and
`regex`
- reduce `unwrap`
- remove dep `lazy_static`
- make more regex cached
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
`current_dir().unwrap()` joined with a Path is equivalent to the
implementation in `resolve_from_cwd()`. Manually tested on Ubuntu 22.04
and Windows 11.
Signed-off-by: Elijah Conners <business@elijahpepe.com>
This commit abstracts out the specifics of the underlying system calls
FS operations behind a new `FileSystem` and `File` trait in the
`ext/fs` extension.
This allows other embedders to re-use ext/fs, but substituting in a
different FS backend.
This is likely not the final form of these traits. Eventually they will
be entirely `deno_core::Resource` agnostic, and will live in a seperate
crate.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Fixes #6259.
Adds the location for v8 syntax errors to the message (`message += " at
{location}"`) when rethrowing them for dynamic imports.
Discussing with @bartlomieju on discord I proposed just preserving v8's
error and not reconstructing it, allowing the standard stack trace to
just point to the syntax error instead of the dynamic import. But on
further thought this way has parity with SWC's syntax errors + has the
advantage of showing both the syntax error and dynamic import location.
```ts
// temp.js
await import("./temp2.js");
// temp2.js
function foo() {
await Promise.resolve();
}
// Before:
// error: Uncaught (in promise) SyntaxError: Unexpected reserved word
// await import("./temp2.js");
// ^
// at async file:///.../temp.js:1:1
// After:
// error: Uncaught (in promise) SyntaxError: Unexpected reserved word at file:///.../temp2.js:2:3
// await import("./temp2.js");
// ^
// at async file:///.../temp.js:1:1
```
This reloads an npm package's dependency's information when a
version/version req/tag is not found.
This PR applies only to dependencies of npm packages. It does NOT yet
cause npm specifiers to have their dependency information cache busted.
That requires a different solution, but this should help cache bust in
more scenarios.
Part of #16901, but doesn't close it yet
Turns out `autoprefixer` is a better reproduction case then
`microbundle`.
Fixes #18535
Fixes #18600
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This introduces a new built-in `unset` command to `deno task` by bumping
deno_task_shell to the latest version 0.11.0. Also this includes a fix
on how `deno task` handles empty environment variables (see
https://github.com/denoland/deno_task_shell/pull/80 for details).
This is a follow-on to the earlier work in reducing string copies,
mainly focused on ensuring that ASCII strings are easy to provide to the
JS runtime.
While we are replacing a 16-byte reference in a number of places with a
24-byte structure (measured via `std::mem::size_of`), the reduction in
copies wins out over the additional size of the arguments passed into
functions.
Benchmarking shows approximately the same if not slightly less wallclock
time/instructions retired, but I believe this continues to open up
further refactoring opportunities.
This commit updates the `Deno.Kv` API to return the new commited
versionstamp for the mutated data from `db.set` and `ao.commit`. This is
returned in the form of a `Deno.KvCommitResult` object that has a
`versionstamp` property.
This reverts commit a3529d0232.
This change made debugging Node tests very hard - `AssertionError` is
now printed as `[Circular *1]` giving no visibility what failed.
We need to align two implementations together and remove this one then.
1. Fixes a cosmetic issue in the repl where it would display lsp warning
messages.
2. Lazily loads dependencies from the package.json on use.
3. Supports using bare specifiers from package.json in the REPL.
Closes #17929
Closes #18494
We currently run the all test cases in `parallel` category at the same
time, which invokes hundreds process at the same time, and that seems
causing some flakiness in CI. (maybe related to #18487)
This PR limits the concurrency to the number of cpu cores. This is more
aligned to how Node.js run their `parallel` test in their repository.
42c4a35952/Makefile (L356)
This commit adds associated type to "NodeEnv" trait, called "Fs".
The "Fs" type has a trait bound on "NodeFs", which specifies APIs
required for all ops and resolution APIs to function.
A "RealFs" implementation of "NodeFs" is exported from the "deno_node"
crate, that provides a default implementation for the trait.
All code in "deno_node" extension was changed to use the "NodeFs" trait
to handle file system operations, instead of relying on APIs from the
standard library.
This commit changes the type parameter for "deno_node" extension, from
`P: NodePermission` to `Env: NodeEnv`.
`NodeEnv` is a new trait that has associated type `P: NodePermission`.
This is a stepping stone to support swappable file system for the
extension, that will be added as a second associated type to the
`NodeEnv` trait.
1. Rewrites the tests to be more back and forth rather than getting the
output all at once (which I believe was causing the hangs on linux and
maybe mac)
2. Runs the pty tests on the linux ci.
3. Fixes a bunch of tests that were just wrong.
4. Adds timeouts on the pty tests.
Fast-follow on #18401 -- the reason that some tests were panicking in
the `CacheDB` `impl Drop` was that the cache itself was being dropped
during panic and the runtime may or may not still exist at that point.
We can reduce the actual tokio runtime testing to where it's needed.
In addition, we return the journal mode to `TRUNCATE` to avoid the risk
of data corruption.
Towards #18455
This commit implements `checkPrimeSync` and `checkPrime` in node:crypto
using the Miller-Rabin primality test (fun fact: it actually is a test
for composite numbers)
It first compares the candidate against many known small primes and if
not, proceeds to run the Miller-Rabin primality test.
http://nickle.org/examples/miller-rabin.5c used as reference
implementation.
This gets SQLite off the flamegraph and reduces initialization time by
somewhere between 0.2ms and 0.5ms. In addition, I took the opportunity
to move all the cache management code to a single place and reduce
duplication. While the PR has a net gain of lines, much of that is just
being a bit more deliberate with how we're recovering from errors.
The existing caches had various policies for dealing with cache
corruption, so I've unified them and tried to isolate the decisions we
make for recovery in a single place (see `open_connection` in
`CacheDB`). The policy I chose was:
1. Retry twice to open on-disk caches
2. If that fails, try to delete the file and recreate it on-disk
3. If we fail to delete the file or re-create a new cache, use a
fallback strategy that can be chosen per-cache: InMemory (temporary
cache for the process run), BlackHole (ignore writes, return empty
reads), or Error (fail on every operation).
The caches all use the same general code now, and share the cache
failure recovery policy.
In addition, it cleans up a TODO in the `NodeAnalysisCache`.
This internal node hook is used by libraries such as `ts-node` when used
as a require hook `node -r ts-node/register`. That combination is often
used with test frameworks like `mocha` or `jasmine`.
We had a reference to `Module._preloadModules` in our code, but the
implementation was missing. While fixing this I also noticed that the
`fakeParent` module that we create internally always threw because of
the `pathDirname` check on the module id in the constructor of `Mdoule`.
So this code path was probably broken for a while.
```txt
✖ ERROR: Error: Empty filepath.
at pathDirname (ext:deno_node/01_require.js:245:11)
at new Module (ext:deno_node/01_require.js:446:15)
at Function.Module._resolveFilename (ext:deno_node/01_require.js:754:28)
at Function.resolve (ext:deno_node/01_require.js:1015:19)
```
This commit adds the `crypto.createSecretKey` API.
Key management: This follows the same approach as our WebCrypto
CryptoKey impl where we use WeakMap for storing key material and a
handle is passed around, such that (only internal) JS can access the key
material and we don't have to explicitly close a Rust resource.
As a result, `createHmac` now accepts a secret KeyObject.
Closes https://github.com/denoland/deno/issues/17844
This will improve diagnostics and catch any non-ASCII extension code
early.
This will use `debug_assert!` rather than `assert!` to avoid runtime
costs, and ensures (in debug_assert mode only) that all extension source
files are ASCII as we load them.
This will make it a bit harder to accidentally use a client url in the
wrong place. I don't fully understand why we do this mapping, but this
will help prevent bugs like #18373
Closes #18374
No need for two almost identical implementations of the same thing
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
Co-authored-by: Aapo Alasuutari <aapo.alasuutari@gmail.com>
Previously the mapping between `AnyValue::Bool` and `KeyPart::Bool` was
inverted.
This patch also allows using the empty key `[]` as range start/end to
`snapshot_read`.
Currently `Deno.openKv(":memory:")` requests read+write permissions for
`./:memory:` even though no file is read or written. Also added some
guards for special sqlite paths that were unintentionally opted into.
This commit adds unstable "Deno.openKv()" API that allows to open
a key-value database at a specified path.
---------
Co-authored-by: Luca Casonato <hello@lcas.dev>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
…'t automatically re-created (#18330)"
This reverts commit 2ef8269fdb.
Printing these messages by default (instead of requiring `-L debug`
flag) caused
various tests to start printing it and mismatch in output assertions.
Reduce the number of copies and allocations of script code by carrying
around ownership/reference information from creation time.
As an advantage, this allows us to maintain the identity of `&'static
str`-based scripts and use v8's external 1-byte strings (to avoid
incorrectly passing non-ASCII strings, debug `assert!`s gate all string
reference paths).
Benchmark results:
Perf improvements -- ~0.1 - 0.2ms faster, but should reduce garbage
w/external strings and reduces data copies overall. May also unlock some
more interesting optimizations in the future.
This requires adding some generics to functions, but manual
monomorphization has been applied (outer/inner function) to avoid code
bloat.
This commit disables compression of the TSC snapshot.
The compression only decreased the size of snapshot by 0.5Mb
and it took about 40s during release build to compress.
With recent gains in TS 5.0 upgrade in terms of size and performance
it makes sense to remove this compression.
This PR changes the inspect result of anonymous functions from
`[Function]` to `[Function (anonymous)]`. This behavior is aligned
to `util.inspect` of Node.js.
Moving some code around in `ext/node` is it's a bit better well defined
and makes it possible for others to embed it.
I expect to see no difference in startup perf with this change.
These functions don't need to be async, as they are only calling
synchronous JavaScript code. As a follow up, all 3 functions
should be merge together - this will reduce roundtrips for
calling V8 from Rust, which is somewhat expensive
This commit adds support for spawning Web Workers in self-contained
binaries created with "deno compile" subcommand.
As long as module requested in "new Worker" constructor is part of the
eszip (by means of statically importing it beforehand, or using "--include"
flag), then the worker can be spawned.
This change will enable dynamic imports and web workers to use modules
not reachable from the main module, by passing a list of extra side
module roots as options to `deno compile`.
This can be done by specifying "--include" flag that accepts a file path or a
URL. This flag can be specified multiple times, to include several modules.
The modules specified with "--include" flag, will be added to the produced
"eszip".
This commit changes the build process in a way that preserves already
registered ops in the snapshot. This allows us to skip creating hundreds of
"v8::String" on each startup, but sadly there is still some op registration
going on startup (however we're registering 49 ops instead of >200 ops).
This situation could be further improved, by moving some of the ops
from "runtime/" to a separate extension crates.
---------
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Follow-up to #18210:
* we are passing the generated `cfg` object into the state function
rather than passing individual config fields
* reduce cloning dramatically by making the state_fn `FnOnce`
* `take` for `ExtensionBuilder` to avoid more unnecessary copies
* renamed `config` to `options`
This implements two macros to simplify extension registration and centralize a lot of the boilerplate as a base for future improvements:
* `deno_core::ops!` registers a block of `#[op]`s, optionally with type
parameters, useful for places where we share lists of ops
* `deno_core::extension!` is used to register an extension, and creates
two methods that can be used at runtime/snapshot generation time:
`init_ops` and `init_ops_and_esm`.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit removes compression for the runtime JS code.
It means that we will have a bigger binary, but faster startup. After
several discussion in the CLI team we decided it's worth to trade
about 3Mb of binary size for 2ms faster startup time. With WebGPU
removed in 35196eab27
it shouldn't have such a big impact on the binary size.
This PR _**temporarily**_ removes WebGPU (which has behind the
`--unstable` flag in Deno), due to performance complications due to its
presence.
It will be brought back in the future; as a point of reference, Chrome
will ship WebGPU to stable on 26/04/2023.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Closes #17831. This change hides the indices of any indexed collection
when triggering tab completion for object properties in the REPL.
An example is shown in the issue, but for verbosity here is another.
Before the change:
```
> const arr = new Uint8ClampedArray([1, 2, 3])
undefined
> arr.
0 map
1 reverse
2 reduce
...
```
After the change:
```
> const arr = new Uint8ClampedArray([1, 2, 3])
undefined
> arr.
constructor reduce
BYTES_PER_ELEMENT reduceRight
buffer set
...
```
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
To be able to preserve "Deno.core.ops" we need to ensure that
ops are registered in the same order in various places, otherwise
we will get mismatch in external references ordering.
Prerequisite for https://github.com/denoland/deno/pull/18080
This commit adds support for retrieving `dev` information
when stating files on Windows.
Additionally `Deno.FileInfo` interfaces was changed to always
return 0 for fields that we don't retrieve information for on Windows.
Closes https://github.com/denoland/deno/issues/18053
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
These methods are confusing because the arguments are backwards. I feel
like they should have never been added to `Option<T>` and that clippy
should suggest rewriting to
`map(...).unwrap_or(...)`/`map(...).unwrap_or_else(|| ...)`
https://github.com/rust-lang/rfcs/issues/1025
Chrono's `clock` feature pulls in `iana-time-zone` which links to macOS
core_foundation. This PR itself is not enough to get rid of
CoreFoundation. Removal depends on getting rid of security framework,
see #18071
This commit changes "deno_core" to not rely on implicitly calling
"std::env::current_dir()" when resolving module specifiers using
APIs from "deno_core::modules_specifier".
Supersedes https://github.com/denoland/deno/pull/15454
Remove remaining usages of "resolve_url_or_path_deprecated" in favor
of "resolve_url_or_path" with explicit calls to
"std::env::current_dir()".
Towards landing https://github.com/denoland/deno/pull/15454
This commit changes various CLI subcommands that have support for
the "--watch" flag to use initial current working directory when
resolving "main module".
This is part of migration towards explicitly passing current working
directory to "deno_core::resolve_url_or_path" API.
As a side effect this makes "deno <subcommand> --watch" more aligned to
user expectations, where calling "Deno.chdir()" during program doesn't
break watcher.
Towards landing https://github.com/denoland/deno/pull/15454
This commit changes current "deno_core::resolve_url_or_path" API to
"resolve_url_or_path_deprecated" and adds new "resolve_url_or_path"
API that requires to explicitly pass the directory from which paths
should be resolved to.
Some of the call sites were updated to use the new API, the reminder
of them will be updated in a follow up PR.
Towards landing https://github.com/denoland/deno/pull/15454
Creating the node_modules folder when the packages are already
downloaded can take a bit of time and not knowing what is going on can
be confusing. It's better to show a progress bar.
This has been bothering me for a while and it became more painful while
working on #18136 because injecting the shared progress bar became very
verbose. Basically we should move the creation of all these npm structs
up to a higher level.
This is a stepping stone for a future refactor where we can improve how
we create all our structs.
This commit removes "deno_core::RuntimeOptions::extensions_with_js".
Now it's embedders' responsibility to properly register extensions
that will not contains JavaScript sources when running from an existing
snapshot.
Prerequisite for https://github.com/denoland/deno/pull/18080
This commit partially reverts changes from
https://github.com/denoland/deno/pull/18095.
Turns out I made a mistake that became apparent when working
on removing "RuntimeOptions::extensions_with_js" in a follow up.
This commit splits "<ext_name>::init" functions into "init_ops" and
"init_ops_and_esm". That way we don't have to construct list of
ESM sources on each startup if we're running with a snapshot.
In a follow up commit "deno_core" will be changed to not have a split
between "extensions" and "extensions_with_js" - it will be embedders'
responsibility to pass appropriately configured extensions.
Prerequisite for https://github.com/denoland/deno/pull/18080
This improves peer dependency resolution yet again. We did not handle
scenarios like the following:
```
// a -> b -> c -> d -> c -> b (peer)
```
...which would maybe work ok the first time its run in some cases, but
then lead to a lockfile that would error on load.
This now keeps track of circular dependencies and updates nodes
accordingly. That said, there is still a lurking bug in this code
somewhere that I've added a comment for (there is a mitigation on the
tail end that seems to work well). The current state is much better than
before and I can look into it later. I think it's something small that's
incorrect.
```
Benchmark 1: deno run -A ../empty.js
Time (mean ± σ): 20.5 ms ± 0.5 ms [User: 13.4 ms, System: 5.1 ms]
Range (min … max): 19.8 ms … 24.0 ms 119 runs
Benchmark 2: target/release/deno run -A ../empty.js
Time (mean ± σ): 18.8 ms ± 0.3 ms [User: 13.0 ms, System: 4.9 ms]
Range (min … max): 18.3 ms … 19.9 ms 129 runs
```
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This is implemented in such a way that it should still allow processes
to go through when a file lock wasn't properly cleaned up and the OS
hasn't released it yet (but with a 200ms-ish delay).
Closes #18039
This commit renames "deno_core::InternalModuleLoader" to
"ExtModuleLoader" and changes the specifiers used by the
modules loaded from this loader to "ext:".
"internal:" scheme was really ambiguous and it's more characters than
"ext:", which should result in slightly smaller snapshot size.
Closes https://github.com/denoland/deno/issues/18020
Since we are snapshotting extension source at build time, there's no
need to define list of sources for the extension at runtime.
This commit changes "deno_node" extension by removing "init_polyfill"
function in favor of "init_polyfill_ops_and_esm()" and "init_polyfill_ops()".
The former is used during snapshot and when "deno_runtime" is compiled
with "dont_create_runtime_snapshot" cargo feature flag. The latter is used
when running a worker from an existing snapshot.
This is a start of a bigger refactor to all extensions - thanks to this
change, we don't have to iterate over all defined source files for extension at
runtime, and because of that we don't have to create a filepath for each of the
source files. It's not a big deal, but we are iterating over 300 files on each start,
and concatenating 3 strings before creating a "PathBuf" for ~200 of them.
This is already visible on the startup flamegraphs and should be avoided.
There's no point for this API to expect result. If something fails it should
result in a panic during build time to signal to embedder that setup is
wrong.
We use information about build in several extension crates like
"ext/node" or "runtime/". In an effort to move "fs" APIs to a separate
crate it is a prerequisite to have this information available outside
of the "runtime/" crate.
This commit moves definition of "build" object to "Deno.core" that is
later forwarded to "Deno.build".
JavaScript APIs from "runtime/js/40_files.js" combined abstractions
for stdio streams ("Stdout", "Stderr", "Stdin") and file system file
("File", "FsFile"). APIs from "runtime/js/40_read_file.js" and
"runtime/js/40_write_file.js" were implemented using ops from
"runtime/ops/fs.rs".
This file was removed and relevant APIs were moved to "deno_io/12_io.js"
and "runtime/js/30_fs.js".
This work is meant to enable factoring out "deno_fs" crate.
This commit changes "include_js_files!" macro from "deno_core"
in a way that "dir" option doesn't cause specifiers to be rewritten
to include it.
Example:
```
include_js_files! {
dir "js",
"hello.js",
}
```
The above definition required embedders to use:
`import ... from "internal:<ext_name>/js/hello.js"`.
But with this change, the "js" directory in which the files are stored
is an implementation detail, which for embedders results in:
`import ... from "internal:<ext_name>/hello.js"`.
The directory the files are stored in, is an implementation detail and
in some cases might result in a significant size difference for the
snapshot. As an example, in "deno_node" extension, we store the
source code in "polyfills" directory; which resulted in each specifier
to look like "internal:deno_node/polyfills/<module_name>", but with
this change it's "internal:deno_node/<module_name>".
Given that "deno_node" has over 100 files, many of them having
several import specifiers to the same extension, this change removes
10 characters from each import specifier.
denoland/eszip#115 added support for statically-analyzed dynamic imports
in eszip, which made `deno compile` support dynamic imports starting
from #17858. This PR adds a test for it.
----
This test is adapted from PR #17663.
Closes #17908
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
This commit updates deno_lint crate to 0.41.0. The new version contains
a braking change that requries a minor code fix here, which is also
addressed in this commit.
This lazily does an "npm install" when any package name matches what's
found in the package.json or when running a script from package.json
with deno task.
Part of #17916
Closes #17928
This is a super basic initial implementation. We don't create a
`node_modules/.bin` folder at the moment and add it to the PATH like we
should which is necessary to make command name resolution in the
subprocess work properly (ex. you run a script that launches another
script that then tries to launch an "npx command"... this won't work
atm).
Closes #17492
This commit enables resolution of "bare specifiers" (eg. "import express
from 'express';") if a "package.json" file is discovered.
It's a step towards being able to run projects authored for Node.js
without any changes.
With this commit we are able to successfully run Vite projects without
any changes to the user code.
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
This commit adds new "A" option to the interactive permission prompt, that will
allow all subsequent permissions for given group (domain). Ie. when querying for
permissions to access eg. env variables responding with "A" will allow access
to all environmental variables.
This works for all permission domains and should make permission prompts
more ergonomic for users.
This changes npm specifiers to be handled by deno_graph and resolved to
an npm package name and version when the specifier is encountered. It
also slightly changes how npm specifier resolution occurs—previously it
would collect all the npm specifiers and resolve them all at once, but
now it resolves them on the fly as they are encountered in the module
graph.
https://github.com/denoland/deno_graph/pull/232
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Deno.Command needs to be stabilized first and allow people to upgrade to
it before we can deprecate Deno.run. Otherwise lint will suddenly fail
with deprecated errors without giving people a chance to update.
This PR fixes peer dependency resolution to only resolve peers based on
the current graph traversal path. Previously, it would resolve a peers
by looking at a graph node's ancestors, which is not correct because
graph nodes are shared by different resolutions.
It also stores more information about peer dependency resolution in the
lockfile.
This allows to not include source code into the binary (because
it will already be included in the V8 snapshot).
Nothing changes for the embedders - everything should still build the
same.
This commit brings the binary size from 87Mb to 82Mb on M1.
Alternative to https://github.com/denoland/deno/pull/17820 and
https://github.com/denoland/deno/pull/17653
---------
Co-authored-by: Leo Kettmeir <crowlkats@toaxl.com>
This commits adds auto-discovery of "package.json" file when running
"deno run" and "deno task" subcommands. In case of "deno run" the
"package.json" is being looked up starting from the directory of the
script that is being run, stopping early if "deno.json(c)" file is found
(ie. FS tree won't be traversed "up" from "deno.json").
When "package.json" is discovered the "--node-modules-dir" flag is
implied, leading to creation of local "node_modules/" directory - we
did that, because most tools relying on "package.json" will expect
"node_modules/" directory to be present (eg. Vite). Additionally
"dependencies" and "devDependencies" specified in the "package.json"
are downloaded on startup.
This is a stepping stone to supporting bare specifier imports, but
the actual integration will be done in a follow up commit.
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
This PR adds the remaining ~650 Node.js compat test cases from std/node.
Among these 650 cases, about 130 cases are now failing. These failing
cases are prefixed with `TODO:` in `tests/node_compat/config.json`.
These will be addressed in later PRs.
This commit changes definition of "ExtensionFileSource", by changing
"code" field to being "ExtensionFileSourceCode" enum. Currently the enum
has only a single variant "IncludedInBinary". It is done in preparation
to allow embedders to decide if they want to include the source code in the
binary when snapshotting (in most cases they shouldn't do that).
In the follow up commit we'll add more variants to
"ExtensionFileSourceCode".
"include_js_files_dir!" macro was removed in favor "include_js_files!"
macro which can now accept "dir" option.
The `NpmPackageId` struct is being renamed to `NpmPackageNodeId`. In a
future PR it will be moved down into only npm dependency resolution and
a `NpmPackageId` struct will be introduced in `deno_graph` that only has
the name and version of the package (no peer dependency identifier
information). So a `NpmPackageReq` will map to an `NpmPackageId`, which
will map to an `NpmPackageNodeId` in the npm resolution.
Adds two test files: "cli/tests/unit_node/process_test.ts" and
"cli/tests/unit_node/child_process_test.ts"
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This PR changes Node.js/npm compatibility layer to use polyfills for
built-in Node.js
embedded in the snapshot (that are coming from "ext/node" extension).
As a result loading `std/node`, either from
"https://deno.land/std@<latest>/" or
from "DENO_NODE_COMPAT_URL" env variable were removed. All code that is
imported via "npm:" specifiers now uses code embedded in the snapshot.
Several fixes were applied to various modules in "ext/node" to make
tests pass.
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
This commit moves "deno_std/node" in "ext/node" crate. The code is
transpiled and snapshotted during the build process.
During the first pass a minimal amount of work was done to create the
snapshot, a lot of code in "ext/node" depends on presence of "Deno"
global. This code will be gradually fixed in the follow up PRs to migrate
it to import relevant APIs from "internal:" modules.
Currently the code from snapshot is not used in any way, and all
Node/npm compatibility still uses code from
"https://deno.land/std/node" (or from the location specified by
"DENO_NODE_COMPAT_URL"). This will also be handled in a follow
up PRs.
---------
Co-authored-by: crowlkats <crowlkats@toaxl.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This commit stabilizes Node-API, the "--unstable" flag is no longer
required to load native extensions. "--allow-ffi" permission is still
required to load them.
Compressing the TSC snapshot in debug build took
~45s on M1 MacBook Pro; without compression it took ~1s.
Thus we're not not using compressed snapshot, trading off
a lot of build time for some startup time in debug build.
This was not caught in the previous test case, as the response body was
smaller than the size of `HEAD` response.
This made `nwritten < responseLen` check in `writeFixedResponse` to
fail, and not trigger `op_flash_respond_async` as a result.
When the response body is larger than the `HEAD` though, as in the
updated test case (`HEAD` i 120 bytes, where our response is 300 bytes),
it would think that we still have something to send, and effectively
panic, as `op_flash_respond` already removed the request from the pool.
This change, makes the `handleResponse` function always calculate the
number of bytes to transmit when `HEAD` request is encountered.
Effectively ignoring `Content-Length` of the body, but still setting it
correctly in the request header itself.
Fixes https://github.com/denoland/deno/issues/17737
This commit moves some code around from "cli/node/mod.rs" to
"ext/node". Additionally "ext/node" was changed to factor out
"ops.rs" and "polyfill.rs" modules.
This change makes absolute urls, that contain no path like `deno install
https://my-cli.io` to follow redirects and extract the name from it.
It allows modifies `test_util` server listener on port `4550`
(`REDIRECT_ABSOLUTE_PORT`) to allow for specifying `redirect_to` query
param, that fill use that value for it's next redirect.
Fixes https://github.com/denoland/deno/issues/17409
This commit does preparatory work to allow snapshotting Node.js
compatibility layer, that currently lives in `std/node`. The logic was
changed to allow loading some modules from the snapshot and
some from the remote URL.
Additionally "module_es_shim.js" that provides exports for "node:module"
is now snapshotted.
This is a proof of concept for being able to snapshot TypeScript files.
Currently only a single runtime file is authored in TypeScript -
"runtime/js/01_version.ts".
Not needed infrastructure was removed from "core/snapshot_util.rs".
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This PR refactors all internal js files (except core) to be written as
ES modules.
`__bootstrap`has been mostly replaced with static imports in form in
`internal:[path to file from repo root]`.
To specify if files are ESM, an `esm` method has been added to
`Extension`, similar to the `js` method.
A new ModuleLoader called `InternalModuleLoader` has been added to
enable the loading of internal specifiers, which is used in all
situations except when a snapshot is only loaded, and not a new one is
created from it.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
- Generalizes the npm version code (ex. `NpmVersion` -> `Version`,
`NpmVersionReq` -> `VersionReq`). This is a slow refactor towards
extracting out this code for deno specifiers and better usage in
deno_graph.
- Removes `SpecifierVersionReq`. Consolidates `NpmVersionReq` and
`SpecifierVersionReq` to just `VersionReq`
- Removes `NpmVersionMatcher`. This now just looks at `VersionReq`.
- Paves the way to allow us to create `NpmPackageReference`'s from a
package.json's dependencies/dev dependencies
(`VersionReq::parse_from_npm`).
This commit changes handling of config file to enable
specifying "imports" and "scopes" objects effectively making
the configuration file an import map.
"imports" and "scopes" take precedence over "importMap" configuration,
but have lower priority than "--importmap" CLI flag.
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
Co-authored-by: David Sherret <dsherret@gmail.com>
This commit adds sync versions of async APIs to "Deno.permissions"
namespace.
Following APIs were added:
- "Deno.permissions.querySync"
- "Deno.permissions.requestSync"
- "Deno.permissions.revokeSync"
Allows to change behavior of `deno fmt` to use "ASI" setting for
semicolons instead of always prefering them, this is done
by "--options-semi=asi" flag or `"semi": "asi"` setting
in the config file.
This commit removes "Deno.core" namespace. It is strictly private API
that has no stability guarantees, we were supposed to remove it long time ago.
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This change prints a link to the release notes to `deno upgrade` output
and its variations.
Release notes aren't printed for commands relating to canary versions.
Closes #16350.
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
The way the standalone mode handles the `--cert` flag is different to
all other modes. This is because `--cert` takes a path to the
certificate file, which is directly added to the root cert store; except
for compile mode, where its byte contents are stored in the standalone
metadata, and they are added to the root cert store after the
`ProcState` is created.
This change instead changes `Flags::ca_file` (an `Option<String>`) into
`Flags::ca_data`, which can represent a `String` file path or a
`Vec<u8>` with the certificate contents. That way, standalone mode can
create a `ProcState` whose root cert store alreay contains the
certificate.
This change also adds a tests for certificates in standalone mode, since
there weren't any before.
This refactor will help with implementing web workers in standalone mode
in the future.
Updated third_party dlint to v0.37.0 for GitHub Actions. This PR
includes following changes:
* fix(prefer-primordials): Stop using array pattern assignments
* fix(prefer-primordials): Stop using global intrinsics except for
`SharedArrayBuffer`
* feat(guard-for-in): Apply new guard-for-in rule
I'm not sure this properly handles scenarios where an npm package uses
an alias that resolves to itself, we can fix that if we find a package
that actually depends on that behavior.
Closes #17420
The leading cause of the problem was that `handleResponse` has
`tryRespondChunked` passed as an argument, which in turn is implemented
as a call to `core.ops.op_try_flash_respond_chuncked`, that throws in
the repro code.
`handleResponse` was not handled correctly, as it not returned any
value, and had no `catch` attached to it.
It also effectively was never correctly handled inside two other blocks
with `resp.then` and `PromisePrototypeCatch(PromisePrototypeThen(resp,
"..."))` as well, as it just short-circuited the promise with an empty
resolve, instead of relying on the last `(async () => {})` block.
This change makes `handleResponse` return a correct value and attach
`onError` handler to the "non-thenable" variant of response handling
code.
This commit removes the libuv pollyfills introduced with Node-API
support.
It is too much Node-specific. Most Node-API modules that depend on libuv
are already giving up the benefits of Node-API. We should rather
encourage modules to use `ThreadSafeFunction` or `AsyncWork` to
interface with the event loop.
Relevant discussion a few months ago:
https://github.com/denoland/deno/pull/13633#discussion_r904916178
cc @bartlomieju
This commit changes "ProcState" to store "file_fetcher" field in an "Arc",
allowing it to be preserved between restarts and thus keeping the state
alive between the restarts. File watchers for "deno test" and "deno bench"
now reset "ProcState" between restarts.
This commit changes signature of "deno_core::ModuleLoader::resolve" to pass
an enum indicating whether or not we're resolving a specifier for dynamic import.
Additionally "CliModuleLoader" was changes to store both "parent permissions" (or
"root permissions") as well as "dynamic permissions" that allow to check for permissions
in top-level module load an dynamic imports.
Then all code paths that have anything to do with Node/npm compat are now checking
for permissions which are passed from module loader instance associated with given
worker.
This PR updates the name used in `clap::Arg::value_name` for the
`--inspect*` flags from `HOST:PORT` to `HOST_AND_PORT` because the
former causes an arguments error when using shell completions in the
`zsh` shell.
Adds support for passing and returning structs as buffers to FFI. This does not implement fastapi support for structs. Needed for certain system APIs such as AppKit on macOS.
Turns out we were cloning permissions which after prompting were discarded,
so the state of permissions was never preserved. To handle that we need to store
all permissions behind "Arc<Mutex<>>" (because there are situations where we
need to send them to other thread).
Testing and benching code still uses "Permissions" in most places - it's undesirable
to share the same permission set between various test/bench files - otherwise
granting or revoking permissions in one file would influence behavior of other test
files.
If an optional peer dependency entry previously wasn't resolved and it's
now being resolved, then it will add it as if it were a dependency of
the previously resolved package instead of creating a new "copy package"
(seems to be what npm and pnpm does).
Closes #17240
Previously, `Deno.permissions.[revoke|request]()` wouldn't correctly
process the `path: URL` when `name` was `ffi`. This change fixes that
behaviour and adds a new function, `formDescriptor()`, to ensure `URL`
arguments are consistently handled across
`Deno.permissions.[query|revoke|request]()`.
This commit fixes "Add all missing imports" quick fix; before
it was replacing all occurrences with the same specifier. Now
every line returned from TSC is processed individually.
This commit adds "Deno.Conn.ref()" and "Deno.Conn.unref()" methods.
These methods can be used to make connection block or not block the
event loop from finishing. Refing/unrefing only influences "read"
operations - ie. scheduling writes to a connection _do_ keep event
loop alive.
Required for https://github.com/denoland/deno/issues/16710
This commit fixes handling of rejected promises in dynamic imports
evaluation.
Previously we were running callbacks for next ticks and macrotasks
_before_ polling
dynamic imports and checked for unhandled rejections immediately after.
This is wrong,
as `unhandledrejection` event is dispatched and its callbacks are run as
macrotasks.
This commit changes order of actions performed by the event loop to
following:
- poll async ops
- poll dynamic imports
- run next tick callbacks
- run macrotask callbacks
- check for unhandled promise rejections
This commit fixes formatting of JSError with "errors" property. Before this
commit all instances of "Error" were treated as if they were "AggregateError"
if they had "errors" property. After this commit only actual instances of
"AggregateError" are formatted in such a way, while instances of "Error"
that have "errors" property are formatted without showing details of "errors".
Previously, errored streaming response bodies did not cause the HTTP
stream to be aborted. It instead caused the stream to be closed gracefully,
which had the result that the client could not detect the difference
between a successful response and an errored response.
This commit fixes the issue by aborting the stream on error.
This PR adds the concept of a global `DrawThread`, which can receive
multiple renderers to draw information on the screen (note: the
underlying thread is released back to tokio when it's not rendering). It
also separates the concept of progress bars from the existing "draw
thread". This makes it trivial for us to do stuff like show permission
prompts and progress bars at the same time in the future.
The reason this is global is because the process' tty stderr is also a
global concept.
Right now an error in a request body stream causes an uncatchable
global promise rejection. This PR fixes this to instead propagate the
error correctly into the promise returned from `fetch`.
It additionally fixes errored readable stream bodies being treated as
successfully completed bodies by Rust.
The output of `init` are commands, so this should be treated as a "Shell
script". In Shell script, comments must start with `#`, not `//`. (This
also makes the commands example easier to be copied to somewhere.)
This commit changes implementation of "Deno.memoryUsage()" to return
correct value for "rss" field. To do that we implement a specialized function
per os to retrieve this information.
In our `require()` implementation we use a special logic to resolve
"base path" when looking for matching packages, however this logic
is in contradiction to what needs to happen if there's a local
"node_modules"
directory used. This commit changes require implementation to be aware
if we're running off of global node modules cache or a local one.
This commit adds new "--inspect-wait" flag which works similarly
to "--inspect-brk" in that it waits for inspector session to be
established before running code. However it doesn't break on the first
statement of user code, but instead runs it as soon as a session
is established.
This commit removes three unstable Deno APIs:
- "Deno.spawn()"
- "Deno.spawnSync()"
- "Deno.spawnChild()"
These APIs were replaced by a unified "Deno.Command" API.
This allows the user to completely opt out from the lock file or rename
it without having to use `--no-lock` and/or `--lock` in all commands.
## Don’t Use Lock File
```json
{
"lock": false
}
```
## Use Lock File With a Different Name
```json
{
"lock": "deno2.lock"
}
```
The CLI args `--no-lock` and `--lock` will always override what is in
the config file.
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
This commit changes "deno repl" command to run with no permissions by
default and accept "--allow-*" flags.
This change is dictated by the fact that currently there is no way to
run REPL with limited permissions. Technically it's a breaking
change in the CLI command, but there's agreement in the team
that it has merit and it's a good solution.
Running just "deno" command still starts the REPL with full permissions
allowed, but now a banner is printed to inform users about that:
We currently only do this for fmt. This makes it so they're excluded by
default, but you can still opt into these directories by explicitly
specifying them.
This commit adds "InspectorTester" struct which is used in
inspector tests - it encapsulated various functionalities that
we need (like reading/writing to WebSocket), but also adds
better error handling which should help with debugging flaky
tests.
Previously the inner request object of the original and the new request
were the same, causing the requests to be entangled and mutable changes
to one to be visible to the other. This fixes that.
Currently runtime exception are only displayed at the program end in
terminal, which makes it only a partial fix, as a full fix requires
https://github.com/denoland/rusty_v8/pull/1149 which adds new bindings
to the inspector that allows to notify it about thrown exceptions.
This will be handled in a follow up commit.
This commit completely rewrites inspector session polling.
Until now, there was a single function responsible for polling inspector
sessions which could have been called when polling the "JsRuntime"
as well as from internal inspector functions. There are some cases
where it's required to have reentrant polling of sessions (eg. when
"debugger" statement is run) which should be blocking until inspector
sends appropriate message to continue execution. This was not possible
before, because polling of sessions didn't have reentry ability.
As a consequence, session polling was split into two separate functions:
a) one to be used when polling from async context (on each tick of event
loop in "JsRuntime")
b) one to be used when polling synchronously and potentially blocking
(used by various inspector methods).
There are further cleanups and simplifications to be made in inspector
code, but this rewrite solves the problem at hand (being able to
evaluate
"debugger" JS statement and continue inspector functionality).
Co-authored-by: Bert Belder <bertbelder@gmail.com>
Since "Deno.spawn()", "Deno.spawnSync()" and "Deno.spawnChild"
are getting deprecated, this commits rewrites all tests and utilities to
use "Deno.Command" API instead.
Uses SeqOneByteString optimization to do zero-copy `&str` arguments in
fast calls.
- [x] Depends on https://github.com/denoland/rusty_v8/pull/1129
- [x] Depends on
https://chromium-review.googlesource.com/c/v8/v8/+/4036884
- [x] Disable in async ops
- [x] Make it work with owned `String` with an extra alloc in fast path.
- [x] Support `Cow<'_, str>`. Owned for slow case, Borrowed for fast
case
```rust
#[op]
fn op_string_len(s: &str) -> u32 {
str.len() as u32
}
```
This commit updates unhelpful messages that are raised when event loop
stalls on unresolved top-level promises.
Instead of "Module evaluation is still pending but there are no pending
ops or dynamic imports. This situation is often caused by unresolved
promises." and "Dynamically imported module evaluation is still pending
but there are no pending ops. This situation is often caused by
unresolved promises." we are now printing a message like:
error: Top-level await promise never resolved
[SOURCE LINE]
^
at [FUNCTION NAME] ([FILENAME])
eg:
error: Top-level await promise never resolved
await new Promise((_resolve, _reject) => {});
^
at <anonymous>
(file:///Users/ib/dev/deno/cli/tests/testdata/test/unresolved_promise.ts:1:1)
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
Refactors the `Deno.Command` class to not handle any state, but only being an intermediary to calling its methods, and as such any methods and properties besides `output`, `outputSync` & `spawn` have been removed. Interracting with a `spawn`ed subprocess now works by using the methods and properties on the returned class of the `spawn` method.
…ed promises in mind (#16616)"
This reverts commit fd023cf793.
There are reports saying that Vite is often hanging in 1.28.2 and this
is
the only PR that changed something with HTTP server. I think we should
hold off on trying to fix this and instead focus on #16787
CC @magurotuna
This commit allows to execute more JS code from extensions when
creating a snapshot from an existing snapshot.
"deno_core::RuntimeOptions::extensions_with_js" field was added
that is used to pass a list of extensions whose both "ops" and
associated JS source should be executed upon start.
Co-authored-by: crowlkats <crowlkats@toaxl.com>
This commit changes "JsRuntime" to send "executionContextDestroyed"
notification when the program finishes and shows a prompt informing
that runtime is waiting for inspector to disconnect.
With trial and error I found that most debuggers expect "isDefault" to be sent
in "auxData" field of "executionContextCreated" notification. This stems from
the fact that Node.js sends this data and eg. VSCode requires it to close
connection to the debugger when the program finishes execution.
This commit changes history handling of the REPL.
There were some situations were history wasn't properly saved and flushed to a
file, making history very spotty. This commit changes it to save every line into
the history file and flush it to disk before being evaluated.
Thanks to this all lines, including "close()" will be stored in the history
file.
If for any reason we're not able to save history file, a single warning will be
printed to the REPL and it will continue to work, even if subsequent tries will
fail to save to disk.
This PR resets the revert commit made by #16610, bringing back #16383
which attempts to fix the issue happening when we use the flash server
with `--watch` option enabled.
Also, some code changes are made to pass the regression test added in
#16610.
This code checks if permission flags are incorrectly defined after the
module name (e.g. `deno run mod.ts --allow-read` instead of the correct
`deno run --allow-read mod.ts`). If so, a simple warning is displayed.
For CommonJS packages we were not trying different extensions for files
specified as subpath of the package ([package_name]/[subpath]).
This commit fixes that.
If "--lock-write" flag was present we never passed instance of the lockfile to
the npm resolver, which made it skip adding discovered npm packages to
the lockfile. This commit fixes that, by always passing lockfile to the npm
resolver and only regenerating resolver snapshot is "--lock-write" is not
present.
Closes https://github.com/denoland/deno/issues/16666
Supports package names that aren't all lowercase.
This stores the package with a leading underscore (since that's not
allowed in npm's registry and no package exists with a leading
underscore) then base32 encoded (A-Z0-9) so it can be lowercased and
avoid collisions.
Global cache dir:
```
$DENO_DIR/npm/registry.npmjs.org/_{base32_encode(package_name).to_lowercase()}/{version}
```
node_modules dir `.deno` folder:
```
node_modules/.deno/_{base32_encode(package_name).to_lowercase()}@{version}/node_modules/<package-name>
```
Within node_modules folder:
```
node_modules/<package-name>
```
So, direct childs of the node_modules folder can have collisions between
packages like `JSON` vs `json`, but this is already something npm itself
doesn't handle well. Plus, Deno doesn't actually ever resolve to the
`node_modules/<package-name>` folder, but just has that for
compatibility. Additionally, packages in the `.deno` dir could have
collissions if they have multiple dependencies that only differ in
casing or a dependency that has different casing, but if someone is
doing that then they're already going to have trouble with npm and they
are asking for trouble in general.
If resolving types for an npm package, we didn't find "types" entry in
the conditional exports declaration we were falling-through to regular
resolution, instead of short-circuiting and giving up on resolving
types, which might lead to unwarranted errors.
Closes https://github.com/denoland/deno/issues/16649
**This patch**
```
benchmark time (avg) (min … max) p75 p99 p995
------------------------------------------------- -----------------------------
echo deno 23.99 ms/iter (22.51 ms … 33.61 ms) 23.97 ms 33.61 ms 33.61 ms
cat 16kb 24.27 ms/iter (22.5 ms … 35.21 ms) 24.2 ms 35.21 ms 35.21 ms
cat 1mb 25.88 ms/iter (25.04 ms … 30.28 ms) 26.12 ms 30.28 ms 30.28 ms
cat 15mb 38.41 ms/iter (35.7 ms … 50 ms) 38.31 ms 50 ms 50 ms
```
**main**
```
benchmark time (avg) (min … max) p75 p99 p995
------------------------------------------------- -----------------------------
echo deno 35.66 ms/iter (34.53 ms … 41.84 ms) 35.79 ms 41.84 ms 41.84 ms
cat 16kb 35.99 ms/iter (34.52 ms … 44.94 ms) 36.05 ms 44.94 ms 44.94 ms
cat 1mb 38.68 ms/iter (36.67 ms … 50.44 ms) 37.95 ms 50.44 ms 50.44 ms
cat 15mb 48.4 ms/iter (46.19 ms … 58.41 ms) 49.16 ms 58.41 ms 58.41 ms
```
Supports npm specifiers for `deno install`. This will by default always
use a lockfile (which is generated on first run) unless `--no-lock` is
specified.
Peer dependency resolution wasn't handling a peer dependency being
resolved without a dep higher in the tree and then with one being found
higher in the tree.
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
1. There was a lot of cloning going on with `NpmPackageInfo`. This is
now stored in an `Arc<NpmPackageInfo>` and cloning only happens on the
individual version.
2. The package cache is now cleared from memory after resolution.
3. This surfaced a bug in `deno cache` and I noticed it can be more
efficient if we have multiple root specifiers if we provide all the
specifiers as roots.
This PR adds copies of several unstable APIs that are available
in "Deno[Deno.internal].nodeUnstable" namespace.
These copies do not perform unstable check (ie. don't require
"--unstable" flag to be present). Otherwise they work exactly
the same, including permission checks.
These APIs are not meant to be used by users directly and
can change at any time.
Copies of following APIs are available in that namespace:
- Deno.spawnChild
- Deno.spawn
- Deno.spawnSync
- Deno.serve
- Deno.upgradeHttpRaw
- Deno.listenDatagram
This commit makes "npm:" specifiers not require "--unstable" flag.
At the moment some APIs used by Node polyfills still require
"--unstable" which will be addressed in follow up PRs.
This PR makes it possible for applications to create workers from custom
snapshots to improve runtime performance (without having to fork/copy
`runtime/workers.rs`).
This adds support for peer dependencies in npm packages.
1. If not found higher in the tree (ancestor and ancestor siblings),
peer dependencies are resolved like a dependency similar to npm 7.
2. Optional peer dependencies are only resolved if found higher in the
tree.
3. This creates "copy packages" or duplicates of a package when a
package has different resolution due to peer dependency resolution—see
https://pnpm.io/how-peers-are-resolved. Unlike pnpm though, duplicates
of packages will have `_1`, `_2`, etc. added to the end of the package
version in the directory in order to minimize the chance of hitting the
max file path limit on Windows. This is done for both the local
"node_modules" directory and also the global npm cache. The files are
hard linked in this case to reduce hard drive space.
This is a first pass and the code is definitely more inefficient than it
could be.
Closes #15823
This commit fixes CJS resolution when there's a local "node_modules/"
directory.
Before this commit relative imports from CJS files where resolved
relative to
root directory of the package instead of relative to referrer file.
When streaming a resource in ext/http, with compression enabled, we
didn't flush individual chunks. This became very problematic when we
enabled `req.body` from `fetch` for FastStream recently.
This commit now correctly flushes each resource chunk after compression.
This commit adds autodiscovery of lockfile.
This only happens if Deno discovers the configuration file (either
"deno.json" or "deno.jsonc"). In such case Deno tries to load
"deno.lock"
file that sits next to the configuration file, or creates one for user
if
the lockfile doesn't exist yet.
As a consequence, "--lock" and "--lock-write" flags had been updated.
"--lock" no longer requires a value, if one is not provided, it defaults
to "./deno.lock" resolved from the current working directory.
"--lock-write"
description was updated to say that it forces to overwrite a lockfile.
Autodiscovery is currently not handled by the LSP.
This commit changes lockfile to be "additive" - ie. integrity check only fails if
file/package is already specified in the lockfile, but its integrity doesn't match.
If file/package is not present in the lockfile, it will be added to the lockfile and
the lockfile will be written to disk.
In order for test cases to pass regardless of each individual's environment,
this commit adds calls to `slice` method when printing the filenames so
we can avoid getting `console.log` to truncate them.
Fixes #16305
This API needs `--allow-sys` permissions nowadays, but the docs still
mention `--allow-env` permissions.
```
deno run .\file.ts
⚠️ ┌ Deno requests sys access to "osRelease".
├ Requested by `Deno.osRelease()` API
├ Run again with --allow-sys to bypass this prompt.
└ Allow? [y/n] (y = yes, allow; n = no, deny) >
```
This test has hung a lot recently on macOS. I am not sure if this is
because of a bug in the test or because of the macOS runner that is extremely
slow and flaky in general.
The "proposed" feature that we depend upon in tower-lsp, turns on the
"proposed" feature in lsp-types which has breaking changes in patch
releases because it's explicitly unstable. We need to pin it to prevent
it breaking cargo publish.
V8's JIT can do a better job knowing the argument count and also enable
fast call path (in future).
This also lets us call async ops without `opAsync`:
```js
const { ops } = Deno.core;
await ops.op_void_async();
```
this patch: 4405286 ops/sec
main: 3508771 ops/sec
This commit stabilizes "Deno.consoleSize()" API.
There is one change compared to previous unstable API,
in that the API doesn't accept any arguments. Console size
is established by querying syscalls for stdio streams at fd
0, 1 and 2.
This commit adds a `reuseAddress` option for UDP sockets. When this
option is enabled, one can listen on an address even though it is
already being listened on from a different process or thread. The new
socket will steal the address from the existing socket.
On Windows and Linux this uses the `SO_REUSEADDR` option, while on other
Unixes this is done with `SO_REUSEPORT`.
This behavior aligns with what libuv does.
TCP sockets still unconditionally set the `SO_REUSEADDR` flag - this
behavior matches Node.js and Go. This PR does not change this behaviour.
Co-authored-by: Luca Casonato <hello@lcas.dev>
When listening on a UNIX socket path, Deno currently tries to unlink
this path prior to actually listening. The implementation of this
behaviour is VERY racy, involves 2 additional syscalls, and does not
match the behaviour of any other runtime (Node.js, Go, Rust, etc).
This commit removes this behaviour. If a user wants to listen on an
existing socket, they must now unlink the file themselves prior to
listening.
This change in behaviour only impacts --unstable APIs, so it is not
a breaking change.
This PR fixes a regression that caused deno binaries produced by the CI
release workflows to be larger than expected.
**The problem:** The build script will determine whether the linker
supports the `--export-dynamic-symbol-list` flag by looking at the glibc
version installed on the system. Ubuntu 20.04 ships with glibc 2.31,
which does not support this flag. Upon investigation, I discovered that
the CI pipeline does not use the gcc compiler provided by the
`build-essential` package, and instead uses *clang-14*, which does
support the new flag.
**The solution:** Whenever a custom C Compiler is configured, the build
script now assumes the compiler supports the
`--export-dynamic-symbol-list` flag. This is not always going to be the
case (you could use clang-8, for example), but it puts the onus on the
user making the override to ensure the compiler has support.
This will return deno builds for Linux to their previous size of ~100MB,
and also allow builds under older glibc/gcc versions to succeed. If a
user is compiling deno with a custom compiler that does not support this
new flag, however, their build will fail. I expect this is a rare
scenario, however, and suggest we cross that bridge if and when we come
to it.
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
There were still remaining bigint usages for pointers. This now finally
fixes all of them, there is only the one `type PointerValue = number |
bigint;` line that references `bigint` in the unstable type definition
file.
Potential fix for type-code mismatch in FFI buffer types. The code
supports ArrayBuffers, but types only reflect TypedArray support.
There's also an existing type for this sort of stuff: `BufferSource`.
(Although, it uses `ArrayBufferView` which doesn't actually connect with
the TypedArray interfaces specifically, but it's just a type inheritance
difference and nothing more.)
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
Makes `op_ffi_ptr_of` fast. One of the tests changed from printing
`false` to `true` as the fast `&[u8]` slice path creates the slice with
a null pointer. Thus the `op_ffi_ptr_of` will now return a null pointer
value whereas previously it returned a dangling pointer value.
This PR makes pointer read methods of `Deno.UnsafePointerView` Fast API
compliant, with the exception of `getCString` which cannot be made fast
with current V8 Fast API.
Introduces a new lockfile format that will be used to support locking
"npm" dependencies.
Currently the format looks as follows:
```
// This file is automatically generated by Deno, do not edit its contents
// manually. This file should be commited to your repository.
{
"version": "2",
"remote": {
"https://deno.land/std@0.160.0/http/server.ts": "asdwetsw44523asdfgfas..",
"https://deno.land/std@0.160.0/http/file_server.ts": "asdwetsw44523asdfgfas.."
}
}
```
A follow up PR will add "npm" key that will be used to store information
related
to "npm" dependencies and their resolution.
The new format is used when `--lock-write` is present, if user tries to
load
a lock file using the old format it will still work.
A small cleanup that improves errors in the lockfile as well
as prepares for adding a new format of the lock file that will
allow to provide backward compatibility with existing format
(ie. "Lockfile::content" will be changed into an enum "LockfileContent"
that will have "V1" and "V2" variants).
`deno task` has been in use for a few months now. It was very
well received and there are not many complaints. I feel like
this warning might be discouraging for some users and we don't
really plan to make drastic changes to it (besides adding support
for globs in unspecified future).
Tests and implementation are found here:
https://github.com/denoland/deno_task_shell/pull/59
This is a breaking change, but `deno task` is unstable.
> This changes async commands so that on non-zero exit code they will
fail the entire task. For example:
>
> ```jsonc
> // task that asynchronously starts a server and starts a watcher for
the frontend
> "dev": "deno task server & deno task frontend:watch"
> ```
>
> Previously when running `deno task dev`, if `deno task server` failed,
the entire command would not fail, which kept in line with `sh`, but
it's not very practical. This change causes `deno task dev` to fail.
>
> To opt out, developers can add an `|| exit 0`:
>
> ```jsonc
> "dev": "deno task server || exit 0 & deno task frontend:watch"
> ```
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
This commit removes the calls to `expect()` on `std::rc::Rc`, which caused
Deno to panic under certain situations. We now return an error if `Rc`
is referenced by other variables.
Fixes #9360
Fixes #13345
Fixes #13926
Fixes #16241
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: bartlomieju <bartlomieju@users.noreply.github.com>
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
Co-authored-by: denobot <33910674+denobot@users.noreply.github.com>
Co-authored-by: bartlomieju <bartlomieju@users.noreply.github.com>
This change adds `windowsRawArguments` to `SpawnOptions`. The option enables
skipping the default quoting and escaping while creating the command on
windows.
The option works in a similar way as `windowsVerbatimArguments` in
child_process.spawn options in Node.js, and is necessary for simulating
it in `std/node`.
closes #8852
Fixes the error reported in #16304.
> = note:
/usr/bin/ld:/home/abotella/Projects/deno/cli/generated_symbol_exports_list_linux.def:1:
syntax error in dynamic list
collect2: error: ld returned 1 exit status
This was caused by the format of the symbols list on Linux being
malformed (as the error implies).
The format is documented in ld's
[VERSION](https://sourceware.org/binutils/docs/ld/VERSION.html) as well
as:
> --export-dynamic-symbol-list=file
Specify a --export-dynamic-symbol for each pattern in the
file. The format of the file is the same as the version node
without scope and node name. See VERSION for more
information.
Previously, the format for the Linux symbols list was simply a list of
symbols, now it follows the format:
```
{ symbol_name_a; ...; symbol_name_z };
```
Follow-up to #16208.
- Refactors build.rs behaviour to use `-exported_symbols_list` /
`--export-dynamic-symbol-list`
- Since all build systems now rely on a symbols list file, I have added
`generate_exported_symbols_list`, which derives the symbol list file
depending on the platform, which makes `tools/napi/generate_link_win.js`
redundant.
- Fixes a missed instance of `i8` being used instead of `c_char`
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Fixes #15136
Currently `UnsafeCallback` class' `ref()` and `unref()` methods rely on
the `event_loop_middleware` implementation in core. If even a single
`UnsafeCallback` is ref'ed, then the FFI event loop middleware will
always return `true` to signify that there may still be more work for
the event loop to do.
The middleware handling in core does not wait a moment to check again,
but will instead synchronously directly re-poll the event loop and
middlewares for more work. This becomes a live-loop.
This PR introduces a `Future` implementation for the `CallbackInfo`
struct that acts as the intermediary data storage between an
`UnsafeCallback` and the `libffi` C callback. Ref'ing a callback now
means calling an async op that binds to the `CallbackInfo` Future and
only resolves once the callback is unref'ed. The `libffi` C callback
will call the waker of this Future when it fires to make sure that the
main thread wakes up to receive the callback.
This commit prevents panics that `deno compile` command ran into under certain
conditions from occurring. Such conditions are as follows.
- the target file name begins with `@`, OR
- the stem part of the target file name is equal to one of
["main", "index", "mod", "index"] && the parent directory name starts with `@`
Fixes #16243
This PR fixes invalid header parsing which is flaky because `g` flag is
being used in the regex, which keeps track of `lastIndex`
```javascript
try {
new Headers([["x", "\u0000x"]]); // error
} catch(e) {}
new Headers([["x", "\u0000x"]]); // no error
```
This issue affects `Response` & `Request` constructors as well
Changes introduced in #13633 have broken the ability to compile for
linux/aarch64 - specifically the use of a `i8` as a char type, which is
an `u8` on linux/aarch64.
This PR:
- Replaces instances of `i8` with the architecture-aware wrapper type
`c_char`
- Skips the use of `--export-dynamic-symbol` on linux-aarch64, because
the target environments often rely on older libc/binutils versions
This commit introduces two new buffer wrapper types to `deno_core`. The
main benefit of these new wrappers is that they can wrap a number of
different underlying buffer types. This allows for a more flexible read
and write API on resources that will require less copying of data
between different buffer representations.
- `BufView` is a read-only view onto a buffer. It can be backed by
`ZeroCopyBuf`, `Vec<u8>`, and `bytes::Bytes`.
- `BufViewMut` is a read-write view onto a buffer. It can be cheaply
converted into a `BufView`. It can be backed by `ZeroCopyBuf` or
`Vec<u8>`.
Both new buffer views have a cursor. This means that the start point of
the view can be constrained to write / read from just a slice of the
view. Only the start point of the slice can be adjusted. The end point
is fixed. To adjust the end point, the underlying buffer needs to be
truncated.
Readable resources have been changed to better cater to resources that
do not support BYOB reads. The basic `read` method now returns a
`BufView` instead of taking a `ZeroCopyBuf` to fill. This allows the
operation to return buffers that the resource has already allocated,
instead of forcing the caller to allocate the buffer. BYOB reads are
still very useful for resources that support them, so a new `read_byob`
method has been added that takes a `BufViewMut` to fill. `op_read`
attempts to use `read_byob` if the resource supports it, which falls
back to `read` and performs an additional copy if it does not. For
Rust->JS reads this change should have no impact, but for Rust->Rust
reads, this allows the caller to avoid an additional copy in many
scenarios. This combined with the support for `BufView` to be backed by
`bytes::Bytes` allows us to avoid one data copy when piping from a
`fetch` response into an `ext/http` response.
Writable resources have been changed to take a `BufView` instead of a
`ZeroCopyBuf` as an argument. This allows for less copying of data in
certain scenarios, as described above. Additionally a new
`Resource::write_all` method has been added that takes a `BufView` and
continually attempts to write the resource until the entire buffer has
been written. Certain resources like files can override this method to
provide a more efficient `write_all` implementation.
The current location was causing failures during v1.26.1 publication.
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
-->
This is the release commit being forwarded back to main for 1.26.1
Please ensure:
- [x] Everything looks ok in the PR
- [x] The release has been published
To make edits to this PR:
```shell
git fetch upstream forward_v1.26.1 && git checkout -b forward_v1.26.1 upstream/forward_v1.26.1
```
Don't need this PR? Close it.
cc @cjihrig
Co-authored-by: cjihrig <cjihrig@users.noreply.github.com>
Currently, we use `-rdynamic` for exporting Node API symbols to the
symbol table. `-rdynamic` will export *all* symbols, that means
previously unused functions will not be optimized away introducing a lot
of binary bloat.
This patch uses `-exported_symbol` and `--export-dynamic-symbol` link
flags (not as universal as `-rdynamic`) to only mark Node API symbols to
be put in the dynamic symbol table.
This PR implements the NAPI for loading native modules into Deno.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: DjDeveloper <43033058+DjDeveloperr@users.noreply.github.com>
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
Currently Content-Length is sent when the status code is 204. However,
according to the spec, this should not be sent.
Modify the if statement below to prevent the Content-Length from being
sent.
This commit adds a fast path to `Request` and `Response` that
make consuming request bodies much faster when using `Body#text`,
`Body#arrayBuffer`, and `Body#blob`, if the body is a FastStream.
Because the response bodies for `fetch` are FastStream, this speeds up
consuming `fetch` response bodies significantly.
This commit adds support for npm specifier in "deno cache" subcommand.
```
$ deno cache --unstable npm:vite npm:chalk https://deno.land/std/http/file_server.ts
```
Besides downloading requested npm package(s), it will also download
necessary code from "std/node/".
Make offering "virtual documents" via the lsp easier to parse. `deno:`
can be ambiguous to parse by editors (can conflict with linux paths)
Neovim recently landed a PR https://github.com/neovim/neovim/pull/19797
that allows it to parse `scheme:/` this PR should make deno lsp work
correctly in neovim
This commit adds a cache for CJS and ESM analysis that is backed by an
SQLite file.
The connection to the DB is lazily created on first use, so shouldn't
have impact on the startup time.
Benched with running Vite
Deno v1.26:
```
$ deno task dev
Warning deno task is unstable and may drastically change in the future
Task dev deno run -A --unstable --node-modules-dir npm:vite
VITE v3.1.4 ready in 961 ms
➜ Local: http://localhost:5173/
➜ Network: use --host to expose
```
This branch:
```
../deno/target/release/deno task dev
Warning deno task is unstable and may drastically change in the future
Task dev deno run -A --unstable --node-modules-dir npm:vite
VITE v3.1.4 ready in 330 ms
➜ Local: http://localhost:5173/
➜ Network: use --host to expose
```
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Stop allowing clippy::derive-partial-eq-without-eq and fix warnings
about deriving PartialEq without also deriving Eq.
In one case I removed the PartialEq because it a) wasn't necessary,
and b) sketchy because it was comparing floating point numbers.
IMO, that's a good argument for enforcing the lint rule, because it
would most likely have been caught during review if it had been enabled.
This commit allows the Node compatibility layer to skip
environment variable permission checks when --unstable
is passed and the variable name is one that Node uses.
Fixes: https://github.com/denoland/deno/issues/15890
This commit makes error objects more resistant to
prototype tampering.
This bug was found when updating the deno_std Node compatibility
layer to Node 18. The Node test 'parallel/test-assert-fail.js'
was breaking std's assertion library.
Refs: https://github.com/denoland/deno_std/pull/2585
Changes how built-in Node modules are mapped to polyfills
from "deno_std". Instead of intertwining this logic into Node
resolution logic, we map them to "NodeResolution::BuiltIn"
which are remapped to "deno_std" URLs in ProcState.
- move errors related to Node compat from cli/node/errors.rs to "ext/node" crate
- remove dependency on "node_resolver" crate
- make some of structures private to the "cli/node" module
This commit removes "compat" mode. We shipped support for "npm:" specifier
support in v1.25 and that is preferred way to interact with Node code that we
will iterate and improve upon.
Previously `jsxImportSource` was resolved relative to the config file
during graph building, and relative to the emitted module during
runtime.
This is now fixed so that the JSX import source is resolved relative to
the module both during graph building and at runtime.
This commit splits `Deno.upgradeHttp` into two different APIs, because
the same API is currently overloaded with two different functions. Flash
requests upgrade immediately, with no need to return a `Response`
object. Instead you have to manually write the response to the socket.
Hyper requests only upgrade once a `Response` object has been sent.
These two behaviours are now split into `Deno.upgradeHttp` and
`Deno.upgradeHttpRaw`. The latter is flash only. The former only
supports hyper requests at the moment, but can be updated to support
flash in the future.
Additionally this removes `void | Promise<void>` as valid return types
for the handler function. If one wants to use `Deno.upgradeHttpRaw`,
they will have to type cast the handler signature - the signature is
meant for the 99.99%, and should not be complicated for the 0.01% that
use `Deno.upgradeHttpRaw()`.
This commit changes the `Deno.serve` function signature to be more
versatile and easier to use. It is now a drop in replacement for
std/http's `serve`.
The input validation has also been reworked.
Previously if a user specified a content-length header for an POST
request without a body, the request would contain two `content-length`
headers. One added by us, and one added by the user.
This commit ignores all content-length headers coming from the user,
because we need to have the sole authority on the content-length because
we transmit the body.
This commit changes "npm:" specifier handling to respect "--cached-only" flags and adds "Download" messages for npm registry api calls.
Co-authored-by: David Sherret <dsherret@gmail.com>
Adds error event dispatching for queueMicrotask(). Consequently unhandled errors are now reported with Deno.core.terminate(), which is immune to the existing quirk with plainly thrown errors (#14158).
This adds an init subcommand to that creates a project starter similar to cargo init.
```
$ deno init my_project
Project initialized
Run these commands to get started:
cd my_project
deno run main.ts
deno run main_test.ts
$ deno run main.ts
Add 2 + 3 5
$ cat main.ts
export function add(a: number, b: number): number {
return a + b;
}
if (import.meta.main) {
console.log("Add 2 + 3", add(2, 3));
}
$ cat main_test.ts
import { assertEquals } from "https://deno.land/std@0.151.0/testing/asserts.ts";
import { add } from "./main.ts";
Deno.test(function addTest() {
assertEquals(add(2, 3), 5);
});
```
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
- Merge "Deno.serve()" and "Deno.serveTls()" API
- Remove first argument and use "fetch" field options instead
- Update type declarations
- Add more documentation
Welcome to better optimised op calls! Currently opSync is called with parameters of every type and count. This most definitely makes the call megamorphic. Additionally, it seems that spread params leads to V8 not being able to optimise the calls quite as well (apparently Fast Calls cannot be used with spread params).
Monomorphising op calls should lead to some improved performance. Now that unwrapping of sync ops results is done on Rust side, this is pretty simple:
```
opSync("op_foo", param1, param2);
// -> turns to
ops.op_foo(param1, param2);
```
This means sync op calls are now just directly calling the native binding function. When V8 Fast API Calls are enabled, this will enable those to be called on the optimised path.
Monomorphising async ops likely requires using callbacks and is left as an exercise to the reader.
This commit adds "ext/node" extension that implementes CommonJS module system.
In the future this extension might be extended to actually contain implementation of
Node compatibility layer in favor of "deno_std/node".
Currently this functionality is not publicly exposed, it is available via "Deno[Deno.internal].require"
namespace and is meant to be used by other functionality to be landed soon.
This is a minimal first pass, things that still don't work:
support for dynamic imports in CJS
conditional exports
This commit adds new "import.meta.resolve()" API which
allows to resolve specifiers relative to the module the API
is called in. This API supports resolving using import maps.
- "SpawnOutput" extends "ChildStatus" instead of composing it
- "SpawnOutput::stdout", "SpawnOutput::stderr", "Child::stdin",
"Child::stdout" and "Child::stderr" are no longer optional, instead
made them getters that throw at runtime if that stream wasn't set
to "piped".
- Remove the complicated "<T extends SpawnOptions = SpawnOptions>"
which we currently need to give proper type hints for the availability of
these fields. Their typings for these would get increasingly complex
if it became dependent on more options (e.g. "SpawnOptions::pty"
which if set should make the stdio streams unavailable)
Relanding #12994
This commit adds support for "unhandledrejection" event.
This event will trigger event listeners registered using:
"globalThis.addEventListener("unhandledrejection")
"globalThis.onunhandledrejection"
This is done by registering a default handler using
"Deno.core.setPromiseRejectCallback" that allows to
handle rejected promises in JavaScript instead of Rust.
This commit will make it possible to polyfill
"process.on("unhandledRejection")" in the Node compat
layer.
Co-authored-by: Colin Ihrig <cjihrig@gmail.com>
This commit adds support for "unhandledrejection" event.
This event will trigger event listeners registered using:
"globalThis.addEventListener("unhandledrejection")
"globalThis.onunhandledrejection"
This is done by registering a default handler using
"Deno.core.setPromiseRejectCallback" that allows to
handle rejected promises in JavaScript instead of Rust.
This commit will make it possible to polyfill
"process.on("unhandledRejection")" in the Node compat
layer.
Co-authored-by: Colin Ihrig <cjihrig@gmail.com>
When a dynamically imported module gets resolved, any code that comes after an
await import() to that module will continue running. However, if that is the
last code in the evaluation of another dynamically imported module, that second
module will not resolve until the next iteration of the event loop, even though
it does not depend on the event loop at all.
When the event loop is being blocked by a long-running operation, such as a
long-running timer, or by an async op that might never end, such as with workers
or BroadcastChannels, that will result in the second dynamically imported module
not being resolved for a while, or ever.
This change fixes this by running the dynamic module loading steps in a loop
until no more dynamic modules can be resolved.
This commit simplifies the TypeScript types used for interacting with Deno FFI. The basis is that types are now first grouped into logical wholes, NativeNumberType, NativeBigIntType etc. These wholes are combined into the NativeType and NativeResultType general types.
Additionally, this PR removes the { function: { parameters: [], result: "void" } } type declaration from parameters (and result types. Now functions are merely passed and returned as "function".
Keep a cache for source maps and source lines.
We sort of already had a cache argument for source map lookup
functions but we just passed an empty map instead of storing it.
Extended it to cache source line lookups as well and plugged it
into runtime state.
This commit adds support for unstable FFI
callbacks. A callback is registered using
the `Deno.UnsafeCallback` API.
The backing memory for the callback can
be disposed of using `Deno.UnsafeCallback#close`.
It is not safe to pass the callback after calling
close.
Callbacks from other than the isolate thread
are not supported.
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Bert Belder <bertbelder@gmail.com>
This commit adds brand checking to EventTarget. It also fixes a
bug where deno would crash if an abort signal was aborted on the
global addEventListener().
This commit updates the custom inspect function for URL objects
to pass the inspect options through so that the context is
propagated and the resulting indentation is correct.
Fixes: https://github.com/denoland/deno/issues/14171
This commit changes default default behavior of type checking
for several subcommands.
Instead of type checking and reporting type errors only for local
files, the type checking is skipped entirely. Type checking can
still be enabled using the "--check" flag.
Following subcomands are affected:
- deno cache
- deno install
- deno eval
- deno run
This commit adds "Deno.core.setFormatExceptionCallback" which
can be used to provide custom formatting for errors. It is useful
in cases when user throws something that is non-Error (eg.
a string, plain object, etc).
This commit improves the permission descriptor validation by
explicitly checking for object types and using optional
chaining when creating error messages in case the descriptor
is not an object.
Fixes: https://github.com/denoland/deno/issues/14675
This commit changes default mode of type-checking to "local"
and adds "--check" flag to following subcommands:
- deno bench
- deno bundle
- deno cache
- deno compile
- deno eval
- deno install
- deno test
This commit removes "WorkerOptions.deno" option as a boolean,
as well as "WorkerOptions.deno.namespace" settings. Starting
with this commit all workers have access to "Deno" namespace
by default.
This commit fixes source maps for files that contain emojis.
This is done by updating "deno_ast" to "0.14.1" for the case
of "--no-check" flag (ie using SWC emit) and by overriding
TSC's default base64 encoder (which turned out to be buggy)
for the type checking case.
This flag disables loading of configuration file, ie. it will not be
automatically discovered and loaded. Of course this flag conflicts
with "--config" flag and they cannot be used together.
This commit adds better reporting of uncaught errors
in top level scope of testing files. This change affects
both console runner as well as LSP runner.
This commit moves "op_format_location" to "core/ops_builtin.rs"
and removes "Deno.core.createPrepareStackTrace" in favor of
"Deno.core.prepareStackTrace".
Co-authored-by: Aaron O'Mullan <aaron.omullan@gmail.com>
Calling `worker.terminate()` used to kill the worker's isolate and
then block until the worker's thread finished. This blocks the calling
thread if the worker's event loop was blocked in a sync op (as with
`Deno.sleepSync`), which wasn't realized at the time, but since the
worker's isolate was killed at that moment, it would not block the
calling thread if the worker was in a JS endless loop.
However, in #12831, in order to work around a V8 bug, worker
termination was changed to first set a signal to let the worker event
loop know that termination has been requested, and only kill the
isolate if the event loop has not finished after 2 seconds. However,
this change kept the blocking, which meant that JS endless loops in
the worker now blocked the parent for 2 seconds.
As it turns out, after #12831 it is fine to signal termination and
even kill the worker's isolate without waiting for the thread to
finish, so this change does that. However, that might leave the async
ops that receive messages and control data from the worker pending
after `worker.terminate()`, which leads to odd results from the op
sanitizer. Therefore, we set up a `CancelHandler` to cancel those ops
when the worker is terminated.
This commit:
- removes "fmt_errors::PrettyJsError" in favor of "format_js_error" fn
- removes "deno_core::JsError::create" and
"deno_core::RuntimeOptions::js_error_create_fn"
- adds new option to "deno_runtime::ops::worker_host::init"
stream shutdown wasn't happening correctly (moved it to call op_http_shutdown) & extra zeroed bytes were being sent for when body length not a multiple of 64*1024
This commit changes "deno bench" subcommand, by updating
the "Deno.bench" API as follows:
- remove "Deno.BenchDefinition.n"
- remove "Deno.BenchDefintion.warmup"
- add "Deno.BenchDefinition.group"
- add "Deno.BenchDefintion.baseline"
This is done because bench cases are no longer run fixed amount
of iterations, but instead they are run until there is difference between
subsequent runs that is statistically insiginificant.
Additionally, console reporter was rewritten completely, to looks
similar to "hyperfine" reporter.
This commit adds support for "--eval-file" in "deno repl" subcommand.
This flag can be used to pass paths or URLs to files, that will be executed
on REPL startup. All files will be executed in the same context as the REPL
(ie. as "plain old scripts", not ES modules), sharing the global scope.
This feature allows to implement custom REPLs on top of Deno's REPL.
This commit changes "deno test" to filter out stack frames if it is beneficial to the user.
This is the case when there are stack frames coming from "internal" code
below frames coming from user code.
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
This commit fixes and edge case, where testing/benching code could pledge new
permission set before restoring the previous pledge.
Appropriate panics were added and tests that assert that process is killed
in case of "recursive pledge".
This commit rewrites test runner to send structured error data from JavaScript
to Rust instead of passing strings. This will allow to customize display of errors
in test report (which will be addressed in follow up commits).
This commit adds "aggregated" field to "deno_core::JsError" that stores
instances of "JsError" recursively to properly handle "AggregateError"
formatting. Appropriate logics was added to "PrettyJsError" and
"console" API to format AggregateErrors.
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
The following transformations gradually faced by "JsError" have all been
moved up front to "JsError::from_v8_exception()":
- finding the first non-"deno:" source line;
- moving "JsError::script_resource_name" etc. into the first error stack
in case of syntax errors;
- source mapping "JsError::script_resource_name" etc. when wrapping
the error even though the frame locations are source mapped earlier;
- removing "JsError::{script_resource_name,line_number,start_column,end_column}"
entirely in favour of "js_error.frames.get(0)".
We also no longer pass a js-side callback to "core/02_error.js" from cli.
I avoided doing this on previous occasions because the source map lookups
were in an awkward place.
This commit changes "deno test" to better denote user output coming
from test cases.
This is done by printing "---- output ----" and "---- output end ----"
markers if an output is produced. The output from "console" and
"Deno.core.print" is captured, as well as direct writes to "Deno.stdout"
and "Deno.stderr".
To achieve that new APIs were added to "deno_core" crate, that allow
to replace an existing resource with a different one (while keeping resource
ids intact). Resources for stdout and stderr are replaced by pipes.
Co-authored-by: David Sherret <dsherret@gmail.com>
Following changes were done in this commit:
- remove "test" prefix before each test
- use gray color for "running N tests from ..." prompt
- use relative path or remote URL instead of full URL in "running N tests from ..." prompt
- in "failures" section, add file path/remote URL before the test name
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This commit adds new "deno check" subcommand.
Currently it is an alias for "deno cache" with the difference that remote
modules don't emit TS diagnostics by default.
Prints warning for "deno run" subcommand if "--check" flag is not present
and there's no "--no-check" flag. Adds "DENO_FUTURE_CHECK" env
variable that allows to opt into new behavior now.
This commit adds following fields to "Deno.TestContext" interface:
- name
- origin
- parent
These are prerequisites for supporting snapshot functionality in
"std/testing".
`handleWasmStreaming` is the function that provides the binding with
the `fetch` API needed for `WebAssembly.instantiateStreaming()` and
`WebAssembly.compileStreaming()`. When I implemented it in #11200, I
thought V8 was calling these functions with the argument of the
`WebAssembly` streaming functions, without doing any resolving, and so
`handleWasmStreaming` awaits for the parameter to resolve. However,
as discovered in
https://github.com/denoland/deno/issues/13917#issuecomment-1065805565,
V8 does in fact resolve the parameter if it's a promise (and handles
rejections arising from that).
This change removes the `async` IIFE inside `handleWasmStreaming`,
letting initial errors be handled synchronously (which will however
not throw synchronously from the `WebAssembly` namespace functions).
Awaiting is still necessary for reading the bytes of the response,
though, and so there is an `async` IIFE for that.
When an exception is thrown during the processing of streaming WebAssembly,
`op_wasm_streaming_abort` is called. This op calls into V8, which synchronously
rejects the promise and calls into the promise rejection handler, if applicable.
But calling an op borrows the isolate's `JsRuntimeState` for the duration of the
op, which means it is borrowed when V8 calls into `promise_reject_callback`,
which tries to borrow it again, panicking.
This change changes `op_wasm_streaming_abort` from an op to a binding
(`Deno.core.abortWasmStreaming`). Although that binding must borrow the
`JsRuntimeState` in order to access the `WasmStreamingResource` stored in the
`OpTable`, it also takes ownership of that `WasmStreamingResource` instance,
which means it can drop any borrows of the `JsRuntimeState` before calling into
V8.
This commit adds "Deno.upgradeHttp" API, which
allows to "hijack" connection and switch protocols, to eg.
implement WebSocket required for Node compat.
Co-authored-by: crowlkats <crowlkats@toaxl.com>
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Previously specifying permissions: {} was the same as specifying
permissions: "inherit". Now it will be the same as permissions: "none".
Not specifying any permissions (permissions: undefined) still means
permissions: "inherit".
In the implementation of structured serialization in
`Deno.core.serialize`, whenever there is a serialization error, an
exception will be thrown with the message "Failed to serialize
response", even though V8 provides a message to use in such cases.
This change instead throws an exception with the V8-provided message,
if there is one.
This commit adds "deno bench" subcommand and "Deno.bench()"
API that allows to register bench cases.
The API is modelled after "Deno.test()" and "deno test" subcommand.
Currently the output is rudimentary and bench cases and not
subject to "ops" and "resource" sanitizers.
Co-authored-by: evan <github@evan.lol>
This commit fixes CJS/ESM interop in compat mode for dynamically
imported modules.
"ProcState::prepare_module_load" was changed to accept a list
of "graph roots" without associated "module kind". That module kind
was always hardcoded to "ESM" which is not true for CJS/ESM interop -
a CommonJs module might be imported using "import()" function. In
such case the root of the graph should have "CommonJs" module kind
instead of "ESM".
This commit adds CJS/ESM interoperability when running in --compat mode.
Before executing files, they are analyzed and all CommonJS modules are
transformed on the fly to a ES modules. This is done by utilizing analyze_cjs()
functionality from deno_ast. After discovering exports and reexports, an ES
module is rendered and saved in memory for later use.
There's a caveat that all files ending with ".js" extension are considered as
CommonJS modules (unless there's a related "package.json" with "type": "module").
This commit adds "--trace-ops" flag to "deno test" subcommand.
This flag enables saving of stack traces for async ops, that before were always
saved. While the feature proved to be very useful it comes with a significant performance
hit, it's caused by excessive source mapping of stack frames.
This commit improves the error messages for the `deno test` async op
sanitizer. It does this in two ways:
- it uses handwritten error messages for each op that could be leaking
- it includes traces showing where each op was started
This "async op tracing" functionality is a new feature in deno_core.
It likely has a significant performance impact, which is why it is only
enabled in tests.
This commit adds "--output" to "deno coverage" subcommand.
It can be used instead of piping output to a file.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds `readable` and `writable` properties to `Deno.File` and
`Deno.Conn`. This makes it very simple to use files and network sockets
with fetch or the native HTTP server.
Adds another callback to WebWorkerOptions that allows to execute
some modules before actual worker code executes. This allows to set up Node
global using std/node.
This commit makes the errors produced from the resource sanitizer much
more human readable. It does this by using real words rather than our
"resource names" when referring to resources, and by giving helpful
hints on how to clean up each of the resources.
This commit fixes an error when user deletes "window" global JS
variable. Instead of relying on "window" or "globalThis" to dispatch
"load" and "unload" events, we are default to global scope of the
worker.
V8 has supported `Intl.ListFormat` since version 7.2, but TypeScript doesn't
have typings for it yet. This PR manually adds those typings, copying them from
microsoft/TypeScript#47254.
Deno's module loader currently strips a shebang if a module file
starts with one. However, this is no longer necessary, since there is
a stage-3 TC39 that adds support for shebangs (or "hashbangs") to the
language (https://github.com/tc39/proposal-hashbang), and V8, `tsc`
and `swc` all support it.
Furthermore, stripping shebangs causes a correctness bug with JSON
modules, since a JSON file with a shebang should not parse as a JSON
module, yet it does with this stripping. This change fixes this.
This commit adds lint and fmt ignore directives to bundled
code as well as a comment stating that the code was bundled
and shouldn't be edited manually.
Covered ranges were not merged and thus it appeared that some lines
might be uncovered. To fix this I used "v8-coverage" that takes care
of merging the ranges properly. With this change, coverage collected
from a file by multiple entrypoints is now correctly calculated.
I ended up forking https://github.com/demurgos/v8-coverage and adding
"cli/tools/coverage/merge.rs" and "cli/tools/coverage/range_tree.rs".
This commit changes "deno coverage" command not to type check.
Instead of relying on infrastructure for module loading in "deno run";
the code now directly reaches into cache for original and transpiled
sources. In case sources are not available the error is returned to the
user, suggesting to first run "deno test --coverage" command.
This commit changes flow in inspector code to no longer require
"Runtime.runIfWaitingForDebugger" message to complete a handshake.
Even though clients like Chrome DevTools always send this message on startup,
it is against the protocol to require this message to start an inspector
session.
Instead "Runtime.runIfWaitingForDebugger" is required only when running with
"--inspect-brk" flag, which matches behavior of Node.js.
This commit fixes prompts printed to the terminal when
running with "--inspect" or "--inspect-brk" flags.
When debugger disconnects error is no longer printed as
users don't care about the reason debugger did disconnect.
A message suggesting to go to "chrome://inspect" is printed
if debugger is active.
Additionally and information that process is waiting for
debugger to connect is printed if running with "--inspect-brk"
flag.
This commit fixes inspector integration with "deno test" subcommand
by waiting for inspector sessions to connect if "--inspect-brk" flag
is passed.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit adds proper support for import assertions and JSON modules.
Implementation of "core/modules.rs" was changed to account for multiple possible
module types, instead of always assuming that the code is an "ES module". In
effect "ModuleMap" now has knowledge about each modules' type (stored via
"ModuleType" enum). Module loading pipeline now stores information about
expected module type for each request and validates that expected type matches
discovered module type based on file's "MediaType".
Relevant tests were added to "core/modules.rs" and integration tests,
additionally multiple WPT tests were enabled.
There are still some rough edges in the implementation and not all WPT were
enabled, due to:
a) unclear BOM handling in source code by "FileFetcher"
b) design limitation of Deno's "FileFetcher" that doesn't download the same
module multiple times in a single run
Co-authored-by: Kitson Kelly <me@kitsonkelly.com>
std/http/server knows how to handle "Listener has been closed"
exceptions but not "operation canceled" errors.
Make "unix" listen sockets throw the same exception as "tcp" listen
sockets when the socket is closed and has a pending accept operation.
There is still a discrepancy when multiple accept requests are posted
but that's probably a less visible issue and something for another day.
Fixes #13033
This commit adds support for exporting RSA JWKs in the Web Crypto API.
It also does some minor fixes for RSA JWK imports.
Co-authored-by: Sean Michael Wykes <sean.wykes@nascent.com.br>
Fixes "op_set_exit_code" by sharing a single "Arc" between
all workers (via "op state") instead of having a "global" value stored in
"deno_runtime" crate. As a consequence setting an exit code is always
scoped to a tree of workers, instead of being overridable if there are
multiple worker tree (like in "deno test --jobs" subcommand).
Refactored "cli/main.rs" functions to return "Result<i32, AnyError>" instead
of "Result<(), AnyError>" so they can return exit code.
This change also makes the timers implementation closer to the spec, and
sets up the stage to implement AbortSignal.timeout() (whatwg/dom#1032).
Fixes #8965
Fixes #10974
Fixes #11398
Although not easy to replicate in the wild, the `deno test` op sanitizer
can fail when there are intervals that started before a test runs, since
the op sanitizer can end up running in the time between the timer op for
an interval's run resolves and the op for the next run starts.
This change fixes that by adding a new macrotask callback that will run
after the timer macrotask queue has drained. This ensures that there is
a timer op if there are any timers which are unresolved by the time the
op sanitizer runs.
Due to a bug in V8, terminating an isolate while a module with top-level
await is being evaluated would crash the process. This change makes it
so calling `worker.terminate()` will signal the worker to terminate at
the next iteration of the event loop, and it schedules a proper
termination of the worker's isolate after 2 seconds.
Although not easy to replicate in the wild, the `deno test` op sanitizer
can fail when there are intervals that started before a test runs, since
the op sanitizer can end up running in the time between the timer op for
an interval's run resolves and the op for the next run starts.
This change fixes that by adding a new macrotask callback that will run
after the timer macrotask queue has drained. This ensures that there is
a timer op if there are any timers which are unresolved by the time the
op sanitizer runs.
Set the exit code to use if none is provided to Deno.exit(), or when
Deno exits naturally.
Needed for process.exitCode Node compat. Paves the way for #12888.
Fetching of local files, added in #12545, returns a response with no
headers, including the `Content-Type` header. This currently makes it
not work with the WebAssembly streaming APIs, which require the response
to have a content type of `application/wasm`.
Since the only way to obtain a `Response` object with a non-empty `url`
field is via `fetch()`, this change changes the content type requirement
to only apply to responses whose url has the `file:` scheme.
Reduce the number of iterations from 1,024 to 128.
On my big bruiser of a desktop machine it already takes up close to a
minute to complete when nothing else is running so no way it's going to
finish in the allotted time on the CI.
The fact that the test used to pass may be indicative of a performance
regression somewhere but it's not clear to me when or where that would
have been introduced.
Fixes #12887.
This commit introduces "ProcState::maybe_resolver" field, which
stores a single instance of resolver for the whole lifetime of the
process, instead of creating these resolvers for each creation
of module graph. As a result, this resolver can be used in fallback
case where graph is not constructed (REPL, loading modules using
"require") unifying resolution logic.
This commit speeds up compat tests by using local copy of "deno_std"
instead of downloading it from https://deno.land for each test.
Additionally type checking is skipped.
The tests expect 3 publish notifications. It was possible for less than 3 to occur if two or more tasks set the diagnostics in the collection, exited the lock at the same time, then called `publish_diagnostics`
This allows resources to be "streams" by implementing read/write/shutdown. These streams are implicit since their nature (read/write/duplex) isn't known until called, but we could easily add another method to explicitly tag resources as streams.
`op_read/op_write/op_shutdown` are now builtin ops provided by `deno_core`
Note: this current implementation is simple & straightforward but it results in an additional alloc per read/write call
Closes #12556
Previously just a generic "error: No such file or directory (os error
2)" was printed. Now "`unzip` was not found on your PATH, please install
`unzip`" will be printed.
This adds `.code` attributes to errors returned by the op-layer, facilitating classifying OS errors and helping node-compat.
Similar to Node, these `.code` attributes are stringified names of unix ERRNOs, the mapping tables are generated by [tools/codegen_error_codes.js](https://gist.github.com/AaronO/dfa1106cc6c7e2a6ebe4dba9d5248858) and derived from libuv and rust's std internals
This commit changes formatting of elapsed time in test
runner output.
Instead of "XXXms", reporter outputs one of:
- "XXXms" for <1000ms
- "XXs" for <60s
- "XXXmYYs" for >=60s
This commit integrates import map and "classic" resolutions in
the "--compat" mode when using ES modules; in effect
"http:", "https:" and "blob:" imports now work in compat mode.
The algorithm works as follows:
1. If there's an import map, try to resolve using it and if succeeded
return the specifier
2. Try to resolve using "Node ESM resolution", and if succeeded return
the specifier
3. Fall back to regular ESM resolution
Closes #11826
**BREAKING CHANGE** this behaviour was disable when introduced in Deno 1.14/TypeScript 4.4. It will highlight code that unsafely handles variables that are caught, and will cause type errors in unsafe code.
Closes #11882
BREAKING CHANGE: Previously when `--location` was set, the unique storage key was derived from the the URL of the location instead of just the origin. This change correctly uses just the origin. This may cause previously persisted storage to change its key and data to not be available with the same location as before.
A `handshake()` method was added that returns when the TLS handshake is
complete. The `TlsListener` and `TlsConn` interfaces were added to
accomodate this new method.
Closes: #11759.
`fetch()` and client-side websocket used to support HTTP/2, but this
regressed in #11491. This patch reenables it by explicitly adding `h2`
and `http/1.1` to the list of ALPN protocols on the HTTP and websocket
clients.
A bug was fixed that could cause a hang when a method was
called on a TlsConn object that had thrown an exception earlier.
Additionally, a bug was fixed that caused TlsConn.write() to not
completely flush large buffers (>64kB) to the socket.
The public `TlsConn.handshake()` API is scheduled for inclusion in the
next minor release. See https://github.com/denoland/deno/pull/12467.
This commit adds CJS and ESM Node resolvers to the "--compat" mode.
The functionality is spread across "cli/compat" module and Node compatibility
layer in "deno_std/node"; this stems from the fact that ES module resolution
can only be implemented in Rust as it needs to directly integrated with
"deno_core"; however "deno_std/node" already provided CJS module resolution.
Currently this resolution is only active when running a files using
"deno run --compat --unstable <filename>", and is not available in other
subcommands, which will be changed in follow up commits.
Returns empty values in case of errors, source lines are non-essential anyway. These errors can happen e.g. when source files change at runtime. A warning is also printed to help us track when it happens in unexpected cases besides this.
Currently all async ops are polled lazily, which means that op
initialization code is postponed until control is yielded to the event
loop. This has some weird consequences, e.g.
```js
let listener = Deno.listen(...);
let conn_promise = listener.accept();
listener.close();
// `BadResource` is thrown. A reasonable error would be `Interrupted`.
let conn = await conn_promise;
```
JavaScript promises are expected to be eagerly evaluated. This patch
makes ops actually do that.
On OS X, the watcher sometimes witnesses the creation of it's own root
directory. Creating that directory using a sync op instead of an async
op sidesteps the issue.
This commit fixes problem with LSP where diagnostics coming
from "deno lint" don't respect configuration file.
LSP was changed to store "Option<ConfigFile>", "Option<LintConfig>"
and "Option<FmtConfig>" on "Inner"; as well as storing "Option<LintConfig>"
and "Option<FmtConfig>" on "StateSnapshot".
Co-authored-by: Kitson Kelly <me@kitsonkelly.com>
`Window`'s `self` property and `DedicatedWorkerGlobalScope`'s `name`
property are defined as Web IDL read-only attributes with the
`[Replaceable]` extended attribute, meaning that their setter will
redefine the property as a data property with the set value, rather than
changing some internal state. Deno currently defines them as read-only
data properties instead.
Given that Web IDL requires all attributes to be accessor properties
rather than data properties, but Deno exposes almost all of those
properties as either read-only or writable data properties, it makes
sense to expose `[Replaceable]` properties as writable as well – as is
already the case with `WindowOrWorkerGlobalScope`'s `performance`
property.
GET/HEAD requests can't have bodies according to `fetch` spec. This
commit changes the HTTP server to hide request bodies for requests with
GET or HEAD methods.
This commit annotates errors returned from FS Deno APIs to include
paths that were passed to the API calls.
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
WebAssembly modules compiled through `WebAssembly.compile()` and similar
non-streaming APIs don't have a URL associated to them, because they
have been compiled from a buffer source. In stack traces, V8 will use
a URL such as `wasm://wasm/d1c677ea`, with a hash of the module.
However, wasm modules compiled through streaming APIs, like
`WebAssembly.compileStreaming()`, do have a known URL, which can be
obtained from the `Response` object passed into the streaming APIs. And
as per the developer-facing display conventions in the WebAssembly
Web API spec, this URL should be used in stack traces. This change
implements that.
