This commits adds auto-discovery of "package.json" file when running
"deno run" and "deno task" subcommands. In case of "deno run" the
"package.json" is being looked up starting from the directory of the
script that is being run, stopping early if "deno.json(c)" file is found
(ie. FS tree won't be traversed "up" from "deno.json").
When "package.json" is discovered the "--node-modules-dir" flag is
implied, leading to creation of local "node_modules/" directory - we
did that, because most tools relying on "package.json" will expect
"node_modules/" directory to be present (eg. Vite). Additionally
"dependencies" and "devDependencies" specified in the "package.json"
are downloaded on startup.
This is a stepping stone to supporting bare specifier imports, but
the actual integration will be done in a follow up commit.
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
This PR changes Node.js/npm compatibility layer to use polyfills for
built-in Node.js
embedded in the snapshot (that are coming from "ext/node" extension).
As a result loading `std/node`, either from
"https://deno.land/std@<latest>/" or
from "DENO_NODE_COMPAT_URL" env variable were removed. All code that is
imported via "npm:" specifiers now uses code embedded in the snapshot.
Several fixes were applied to various modules in "ext/node" to make
tests pass.
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
This PR refactors all internal js files (except core) to be written as
ES modules.
`__bootstrap`has been mostly replaced with static imports in form in
`internal:[path to file from repo root]`.
To specify if files are ESM, an `esm` method has been added to
`Extension`, similar to the `js` method.
A new ModuleLoader called `InternalModuleLoader` has been added to
enable the loading of internal specifiers, which is used in all
situations except when a snapshot is only loaded, and not a new one is
created from it.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit changes handling of config file to enable
specifying "imports" and "scopes" objects effectively making
the configuration file an import map.
"imports" and "scopes" take precedence over "importMap" configuration,
but have lower priority than "--importmap" CLI flag.
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
Co-authored-by: David Sherret <dsherret@gmail.com>
This commit adds sync versions of async APIs to "Deno.permissions"
namespace.
Following APIs were added:
- "Deno.permissions.querySync"
- "Deno.permissions.requestSync"
- "Deno.permissions.revokeSync"
Allows to change behavior of `deno fmt` to use "ASI" setting for
semicolons instead of always prefering them, this is done
by "--options-semi=asi" flag or `"semi": "asi"` setting
in the config file.
This commit removes "Deno.core" namespace. It is strictly private API
that has no stability guarantees, we were supposed to remove it long time ago.
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This commit changes signature of "deno_core::ModuleLoader::resolve" to pass
an enum indicating whether or not we're resolving a specifier for dynamic import.
Additionally "CliModuleLoader" was changes to store both "parent permissions" (or
"root permissions") as well as "dynamic permissions" that allow to check for permissions
in top-level module load an dynamic imports.
Then all code paths that have anything to do with Node/npm compat are now checking
for permissions which are passed from module loader instance associated with given
worker.
This commit fixes "Add all missing imports" quick fix; before
it was replacing all occurrences with the same specifier. Now
every line returned from TSC is processed individually.
This commit fixes handling of rejected promises in dynamic imports
evaluation.
Previously we were running callbacks for next ticks and macrotasks
_before_ polling
dynamic imports and checked for unhandled rejections immediately after.
This is wrong,
as `unhandledrejection` event is dispatched and its callbacks are run as
macrotasks.
This commit changes order of actions performed by the event loop to
following:
- poll async ops
- poll dynamic imports
- run next tick callbacks
- run macrotask callbacks
- check for unhandled promise rejections
This commit fixes formatting of JSError with "errors" property. Before this
commit all instances of "Error" were treated as if they were "AggregateError"
if they had "errors" property. After this commit only actual instances of
"AggregateError" are formatted in such a way, while instances of "Error"
that have "errors" property are formatted without showing details of "errors".
This commit adds new "--inspect-wait" flag which works similarly
to "--inspect-brk" in that it waits for inspector session to be
established before running code. However it doesn't break on the first
statement of user code, but instead runs it as soon as a session
is established.
This allows the user to completely opt out from the lock file or rename
it without having to use `--no-lock` and/or `--lock` in all commands.
## Don’t Use Lock File
```json
{
"lock": false
}
```
## Use Lock File With a Different Name
```json
{
"lock": "deno2.lock"
}
```
The CLI args `--no-lock` and `--lock` will always override what is in
the config file.
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
Currently runtime exception are only displayed at the program end in
terminal, which makes it only a partial fix, as a full fix requires
https://github.com/denoland/rusty_v8/pull/1149 which adds new bindings
to the inspector that allows to notify it about thrown exceptions.
This will be handled in a follow up commit.
Since "Deno.spawn()", "Deno.spawnSync()" and "Deno.spawnChild"
are getting deprecated, this commits rewrites all tests and utilities to
use "Deno.Command" API instead.
This commit updates unhelpful messages that are raised when event loop
stalls on unresolved top-level promises.
Instead of "Module evaluation is still pending but there are no pending
ops or dynamic imports. This situation is often caused by unresolved
promises." and "Dynamically imported module evaluation is still pending
but there are no pending ops. This situation is often caused by
unresolved promises." we are now printing a message like:
error: Top-level await promise never resolved
[SOURCE LINE]
^
at [FUNCTION NAME] ([FILENAME])
eg:
error: Top-level await promise never resolved
await new Promise((_resolve, _reject) => {});
^
at <anonymous>
(file:///Users/ib/dev/deno/cli/tests/testdata/test/unresolved_promise.ts:1:1)
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
This code checks if permission flags are incorrectly defined after the
module name (e.g. `deno run mod.ts --allow-read` instead of the correct
`deno run --allow-read mod.ts`). If so, a simple warning is displayed.
For CommonJS packages we were not trying different extensions for files
specified as subpath of the package ([package_name]/[subpath]).
This commit fixes that.
Supports package names that aren't all lowercase.
This stores the package with a leading underscore (since that's not
allowed in npm's registry and no package exists with a leading
underscore) then base32 encoded (A-Z0-9) so it can be lowercased and
avoid collisions.
Global cache dir:
```
$DENO_DIR/npm/registry.npmjs.org/_{base32_encode(package_name).to_lowercase()}/{version}
```
node_modules dir `.deno` folder:
```
node_modules/.deno/_{base32_encode(package_name).to_lowercase()}@{version}/node_modules/<package-name>
```
Within node_modules folder:
```
node_modules/<package-name>
```
So, direct childs of the node_modules folder can have collisions between
packages like `JSON` vs `json`, but this is already something npm itself
doesn't handle well. Plus, Deno doesn't actually ever resolve to the
`node_modules/<package-name>` folder, but just has that for
compatibility. Additionally, packages in the `.deno` dir could have
collissions if they have multiple dependencies that only differ in
casing or a dependency that has different casing, but if someone is
doing that then they're already going to have trouble with npm and they
are asking for trouble in general.
If resolving types for an npm package, we didn't find "types" entry in
the conditional exports declaration we were falling-through to regular
resolution, instead of short-circuiting and giving up on resolving
types, which might lead to unwarranted errors.
Closes https://github.com/denoland/deno/issues/16649
1. There was a lot of cloning going on with `NpmPackageInfo`. This is
now stored in an `Arc<NpmPackageInfo>` and cloning only happens on the
individual version.
2. The package cache is now cleared from memory after resolution.
3. This surfaced a bug in `deno cache` and I noticed it can be more
efficient if we have multiple root specifiers if we provide all the
specifiers as roots.
This commit makes "npm:" specifiers not require "--unstable" flag.
At the moment some APIs used by Node polyfills still require
"--unstable" which will be addressed in follow up PRs.
This adds support for peer dependencies in npm packages.
1. If not found higher in the tree (ancestor and ancestor siblings),
peer dependencies are resolved like a dependency similar to npm 7.
2. Optional peer dependencies are only resolved if found higher in the
tree.
3. This creates "copy packages" or duplicates of a package when a
package has different resolution due to peer dependency resolution—see
https://pnpm.io/how-peers-are-resolved. Unlike pnpm though, duplicates
of packages will have `_1`, `_2`, etc. added to the end of the package
version in the directory in order to minimize the chance of hitting the
max file path limit on Windows. This is done for both the local
"node_modules" directory and also the global npm cache. The files are
hard linked in this case to reduce hard drive space.
This is a first pass and the code is definitely more inefficient than it
could be.
Closes #15823
This commit fixes CJS resolution when there's a local "node_modules/"
directory.
Before this commit relative imports from CJS files where resolved
relative to
root directory of the package instead of relative to referrer file.
This commit adds autodiscovery of lockfile.
This only happens if Deno discovers the configuration file (either
"deno.json" or "deno.jsonc"). In such case Deno tries to load
"deno.lock"
file that sits next to the configuration file, or creates one for user
if
the lockfile doesn't exist yet.
As a consequence, "--lock" and "--lock-write" flags had been updated.
"--lock" no longer requires a value, if one is not provided, it defaults
to "./deno.lock" resolved from the current working directory.
"--lock-write"
description was updated to say that it forces to overwrite a lockfile.
Autodiscovery is currently not handled by the LSP.
In order for test cases to pass regardless of each individual's environment,
this commit adds calls to `slice` method when printing the filenames so
we can avoid getting `console.log` to truncate them.
Fixes #16305
Makes `op_ffi_ptr_of` fast. One of the tests changed from printing
`false` to `true` as the fast `&[u8]` slice path creates the slice with
a null pointer. Thus the `op_ffi_ptr_of` will now return a null pointer
value whereas previously it returned a dangling pointer value.
This PR makes pointer read methods of `Deno.UnsafePointerView` Fast API
compliant, with the exception of `getCString` which cannot be made fast
with current V8 Fast API.
Introduces a new lockfile format that will be used to support locking
"npm" dependencies.
Currently the format looks as follows:
```
// This file is automatically generated by Deno, do not edit its contents
// manually. This file should be commited to your repository.
{
"version": "2",
"remote": {
"https://deno.land/std@0.160.0/http/server.ts": "asdwetsw44523asdfgfas..",
"https://deno.land/std@0.160.0/http/file_server.ts": "asdwetsw44523asdfgfas.."
}
}
```
A follow up PR will add "npm" key that will be used to store information
related
to "npm" dependencies and their resolution.
The new format is used when `--lock-write` is present, if user tries to
load
a lock file using the old format it will still work.
`deno task` has been in use for a few months now. It was very
well received and there are not many complaints. I feel like
this warning might be discouraging for some users and we don't
really plan to make drastic changes to it (besides adding support
for globs in unspecified future).
This commit adds support for npm specifier in "deno cache" subcommand.
```
$ deno cache --unstable npm:vite npm:chalk https://deno.land/std/http/file_server.ts
```
Besides downloading requested npm package(s), it will also download
necessary code from "std/node/".
This commit allows the Node compatibility layer to skip
environment variable permission checks when --unstable
is passed and the variable name is one that Node uses.
Fixes: https://github.com/denoland/deno/issues/15890
This commit removes "compat" mode. We shipped support for "npm:" specifier
support in v1.25 and that is preferred way to interact with Node code that we
will iterate and improve upon.
Previously `jsxImportSource` was resolved relative to the config file
during graph building, and relative to the emitted module during
runtime.
This is now fixed so that the JSX import source is resolved relative to
the module both during graph building and at runtime.
This commit changes "npm:" specifier handling to respect "--cached-only" flags and adds "Download" messages for npm registry api calls.
Co-authored-by: David Sherret <dsherret@gmail.com>
Adds error event dispatching for queueMicrotask(). Consequently unhandled errors are now reported with Deno.core.terminate(), which is immune to the existing quirk with plainly thrown errors (#14158).
Welcome to better optimised op calls! Currently opSync is called with parameters of every type and count. This most definitely makes the call megamorphic. Additionally, it seems that spread params leads to V8 not being able to optimise the calls quite as well (apparently Fast Calls cannot be used with spread params).
Monomorphising op calls should lead to some improved performance. Now that unwrapping of sync ops results is done on Rust side, this is pretty simple:
```
opSync("op_foo", param1, param2);
// -> turns to
ops.op_foo(param1, param2);
```
This means sync op calls are now just directly calling the native binding function. When V8 Fast API Calls are enabled, this will enable those to be called on the optimised path.
Monomorphising async ops likely requires using callbacks and is left as an exercise to the reader.
This commit adds "ext/node" extension that implementes CommonJS module system.
In the future this extension might be extended to actually contain implementation of
Node compatibility layer in favor of "deno_std/node".
Currently this functionality is not publicly exposed, it is available via "Deno[Deno.internal].require"
namespace and is meant to be used by other functionality to be landed soon.
This is a minimal first pass, things that still don't work:
support for dynamic imports in CJS
conditional exports
This commit adds new "import.meta.resolve()" API which
allows to resolve specifiers relative to the module the API
is called in. This API supports resolving using import maps.
- "SpawnOutput" extends "ChildStatus" instead of composing it
- "SpawnOutput::stdout", "SpawnOutput::stderr", "Child::stdin",
"Child::stdout" and "Child::stderr" are no longer optional, instead
made them getters that throw at runtime if that stream wasn't set
to "piped".
- Remove the complicated "<T extends SpawnOptions = SpawnOptions>"
which we currently need to give proper type hints for the availability of
these fields. Their typings for these would get increasingly complex
if it became dependent on more options (e.g. "SpawnOptions::pty"
which if set should make the stdio streams unavailable)
Relanding #12994
This commit adds support for "unhandledrejection" event.
This event will trigger event listeners registered using:
"globalThis.addEventListener("unhandledrejection")
"globalThis.onunhandledrejection"
This is done by registering a default handler using
"Deno.core.setPromiseRejectCallback" that allows to
handle rejected promises in JavaScript instead of Rust.
This commit will make it possible to polyfill
"process.on("unhandledRejection")" in the Node compat
layer.
Co-authored-by: Colin Ihrig <cjihrig@gmail.com>
This commit adds support for "unhandledrejection" event.
This event will trigger event listeners registered using:
"globalThis.addEventListener("unhandledrejection")
"globalThis.onunhandledrejection"
This is done by registering a default handler using
"Deno.core.setPromiseRejectCallback" that allows to
handle rejected promises in JavaScript instead of Rust.
This commit will make it possible to polyfill
"process.on("unhandledRejection")" in the Node compat
layer.
Co-authored-by: Colin Ihrig <cjihrig@gmail.com>
When a dynamically imported module gets resolved, any code that comes after an
await import() to that module will continue running. However, if that is the
last code in the evaluation of another dynamically imported module, that second
module will not resolve until the next iteration of the event loop, even though
it does not depend on the event loop at all.
When the event loop is being blocked by a long-running operation, such as a
long-running timer, or by an async op that might never end, such as with workers
or BroadcastChannels, that will result in the second dynamically imported module
not being resolved for a while, or ever.
This change fixes this by running the dynamic module loading steps in a loop
until no more dynamic modules can be resolved.
This commit adds support for unstable FFI
callbacks. A callback is registered using
the `Deno.UnsafeCallback` API.
The backing memory for the callback can
be disposed of using `Deno.UnsafeCallback#close`.
It is not safe to pass the callback after calling
close.
Callbacks from other than the isolate thread
are not supported.
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
Co-authored-by: Bert Belder <bertbelder@gmail.com>
This commit updates the custom inspect function for URL objects
to pass the inspect options through so that the context is
propagated and the resulting indentation is correct.
Fixes: https://github.com/denoland/deno/issues/14171
This commit changes default default behavior of type checking
for several subcommands.
Instead of type checking and reporting type errors only for local
files, the type checking is skipped entirely. Type checking can
still be enabled using the "--check" flag.
Following subcomands are affected:
- deno cache
- deno install
- deno eval
- deno run
This commit adds "Deno.core.setFormatExceptionCallback" which
can be used to provide custom formatting for errors. It is useful
in cases when user throws something that is non-Error (eg.
a string, plain object, etc).
This commit changes default mode of type-checking to "local"
and adds "--check" flag to following subcommands:
- deno bench
- deno bundle
- deno cache
- deno compile
- deno eval
- deno install
- deno test
This commit removes "WorkerOptions.deno" option as a boolean,
as well as "WorkerOptions.deno.namespace" settings. Starting
with this commit all workers have access to "Deno" namespace
by default.
This commit fixes source maps for files that contain emojis.
This is done by updating "deno_ast" to "0.14.1" for the case
of "--no-check" flag (ie using SWC emit) and by overriding
TSC's default base64 encoder (which turned out to be buggy)
for the type checking case.
This flag disables loading of configuration file, ie. it will not be
automatically discovered and loaded. Of course this flag conflicts
with "--config" flag and they cannot be used together.
This commit adds better reporting of uncaught errors
in top level scope of testing files. This change affects
both console runner as well as LSP runner.
Calling `worker.terminate()` used to kill the worker's isolate and
then block until the worker's thread finished. This blocks the calling
thread if the worker's event loop was blocked in a sync op (as with
`Deno.sleepSync`), which wasn't realized at the time, but since the
worker's isolate was killed at that moment, it would not block the
calling thread if the worker was in a JS endless loop.
However, in #12831, in order to work around a V8 bug, worker
termination was changed to first set a signal to let the worker event
loop know that termination has been requested, and only kill the
isolate if the event loop has not finished after 2 seconds. However,
this change kept the blocking, which meant that JS endless loops in
the worker now blocked the parent for 2 seconds.
As it turns out, after #12831 it is fine to signal termination and
even kill the worker's isolate without waiting for the thread to
finish, so this change does that. However, that might leave the async
ops that receive messages and control data from the worker pending
after `worker.terminate()`, which leads to odd results from the op
sanitizer. Therefore, we set up a `CancelHandler` to cancel those ops
when the worker is terminated.
This commit:
- removes "fmt_errors::PrettyJsError" in favor of "format_js_error" fn
- removes "deno_core::JsError::create" and
"deno_core::RuntimeOptions::js_error_create_fn"
- adds new option to "deno_runtime::ops::worker_host::init"
This commit changes "deno bench" subcommand, by updating
the "Deno.bench" API as follows:
- remove "Deno.BenchDefinition.n"
- remove "Deno.BenchDefintion.warmup"
- add "Deno.BenchDefinition.group"
- add "Deno.BenchDefintion.baseline"
This is done because bench cases are no longer run fixed amount
of iterations, but instead they are run until there is difference between
subsequent runs that is statistically insiginificant.
Additionally, console reporter was rewritten completely, to looks
similar to "hyperfine" reporter.
This commit changes "deno test" to filter out stack frames if it is beneficial to the user.
This is the case when there are stack frames coming from "internal" code
below frames coming from user code.
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
This commit fixes and edge case, where testing/benching code could pledge new
permission set before restoring the previous pledge.
Appropriate panics were added and tests that assert that process is killed
in case of "recursive pledge".
This commit rewrites test runner to send structured error data from JavaScript
to Rust instead of passing strings. This will allow to customize display of errors
in test report (which will be addressed in follow up commits).
This commit adds "aggregated" field to "deno_core::JsError" that stores
instances of "JsError" recursively to properly handle "AggregateError"
formatting. Appropriate logics was added to "PrettyJsError" and
"console" API to format AggregateErrors.
Co-authored-by: Nayeem Rahman <nayeemrmn99@gmail.com>
The following transformations gradually faced by "JsError" have all been
moved up front to "JsError::from_v8_exception()":
- finding the first non-"deno:" source line;
- moving "JsError::script_resource_name" etc. into the first error stack
in case of syntax errors;
- source mapping "JsError::script_resource_name" etc. when wrapping
the error even though the frame locations are source mapped earlier;
- removing "JsError::{script_resource_name,line_number,start_column,end_column}"
entirely in favour of "js_error.frames.get(0)".
We also no longer pass a js-side callback to "core/02_error.js" from cli.
I avoided doing this on previous occasions because the source map lookups
were in an awkward place.
This commit changes "deno test" to better denote user output coming
from test cases.
This is done by printing "---- output ----" and "---- output end ----"
markers if an output is produced. The output from "console" and
"Deno.core.print" is captured, as well as direct writes to "Deno.stdout"
and "Deno.stderr".
To achieve that new APIs were added to "deno_core" crate, that allow
to replace an existing resource with a different one (while keeping resource
ids intact). Resources for stdout and stderr are replaced by pipes.
Co-authored-by: David Sherret <dsherret@gmail.com>
Following changes were done in this commit:
- remove "test" prefix before each test
- use gray color for "running N tests from ..." prompt
- use relative path or remote URL instead of full URL in "running N tests from ..." prompt
- in "failures" section, add file path/remote URL before the test name
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
This commit adds new "deno check" subcommand.
Currently it is an alias for "deno cache" with the difference that remote
modules don't emit TS diagnostics by default.
Prints warning for "deno run" subcommand if "--check" flag is not present
and there's no "--no-check" flag. Adds "DENO_FUTURE_CHECK" env
variable that allows to opt into new behavior now.
`handleWasmStreaming` is the function that provides the binding with
the `fetch` API needed for `WebAssembly.instantiateStreaming()` and
`WebAssembly.compileStreaming()`. When I implemented it in #11200, I
thought V8 was calling these functions with the argument of the
`WebAssembly` streaming functions, without doing any resolving, and so
`handleWasmStreaming` awaits for the parameter to resolve. However,
as discovered in
https://github.com/denoland/deno/issues/13917#issuecomment-1065805565,
V8 does in fact resolve the parameter if it's a promise (and handles
rejections arising from that).
This change removes the `async` IIFE inside `handleWasmStreaming`,
letting initial errors be handled synchronously (which will however
not throw synchronously from the `WebAssembly` namespace functions).
Awaiting is still necessary for reading the bytes of the response,
though, and so there is an `async` IIFE for that.
