Step 1 of the Rustification of sanitizers, which unblocks the faster
timers.
This replaces the resource sanitizer with a Rust one, using the new APIs
in deno_core.
This commit adds support for [TC39 Decorator
Proposal](https://github.com/tc39/proposal-decorators).
These decorators are only available in transpiled sources - ie.
non-JavaScript files (because of lack of support in V8).
This entails that "experimental TypeScript decorators" are not available
by default
and require to be configured, with a configuration like this:
```
{
"compilerOptions": {
"experimentalDecorators": true
}
}
```
Closes https://github.com/denoland/deno/issues/19160
---------
Signed-off-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: crowlkats <crowlkats@toaxl.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
This change sets the removal version for the `deno bundle` sub-command
for Deno v2. The warnings appear when `deno bundle` is run and in the
`--help` menu.
This initially uses the new diagnostic printer in `deno lint`,
`deno doc` and `deno publish`. In the limit we should also update
`deno check` to use this printer.
This commit removes the requirement for `--unstable` flag in `deno
jupyter` subcommand. The process will no longer exit if this flag is not
provided, however the subcommand itself is still considered unstable
and might change in the future.
Required for https://github.com/denoland/deno/pull/21452
This commit removes conditional type-checking of unstable APIs.
Before this commit `deno check` (or any other type-checking command and
the LSP) would error out if there was an unstable API in the code, but not
`--unstable` flag provided.
This situation hinders DX and makes it harder to configure Deno. Failing
during runtime unless `--unstable` flag is provided is enough in this case.
We were calling `expand_glob` on our excludes, which is very expensive
and unnecessary because we can pattern match while traversing instead.
1. Doesn't expand "exclude" globs. Instead pattern matches while walking
the directory.
2. Splits up the "include" into base paths and applicable file patterns.
This causes less pattern matching to occur because we're only pattern
matching on patterns that might match and not ones in completely
unrelated directories.
Main change is that:
- "hyper" has been renamed to "hyper_v014" to signal that it's legacy
- "hyper1" has been renamed to "hyper" and should be the default
Adds an `--unstable-sloppy-imports` flag which supports the
following for `file:` specifiers:
* Allows writing `./mod` in a specifier to do extension probing.
- ex. `import { Example } from "./example"` instead of `import { Example
} from "./example.ts"`
* Allows writing `./routes` to do directory extension probing for files
like `./routes/index.ts`
* Allows writing `./mod.js` for *mod.ts* files.
This functionality is **NOT RECOMMENDED** for general use with Deno:
1. It's not as optimal for perf:
https://marvinh.dev/blog/speeding-up-javascript-ecosystem-part-2/
1. It makes tooling in the ecosystem more complex in order to have to
understand this.
1. The "Deno way" is to be explicit about what you're doing. It's better
in the long run.
1. It doesn't work if published to the Deno registry because doing stuff
like extension probing with remote specifiers would be incredibly slow.
This is instead only recommended to help with migrating existing
projects to Deno. For example, it's very useful for getting CJS projects
written with import/export declaration working in Deno without modifying
module specifiers and for supporting TS ESM projects written with
`./mod.js` specifiers.
This feature will output warnings to guide the user towards correcting
their specifiers. Additionally, quick fixes are provided in the LSP to
update these specifiers:
A bit hacky, but it works. Essentially, this will check for all the
scripts in the node_modules/.bin directory then force them to run with
Deno via deno_task_shell.
This PR causes Deno to include more files in the graph based on how a
template literal looks that's provided to a dynamic import:
```ts
const file = await import(`./dir/${expr}`);
```
In this case, it will search the `dir` directory and descendant
directories for any .js/jsx/etc modules and include them in the graph.
To opt out of this behaviour, move the template literal to a separate
line:
```ts
const specifier = `./dir/${expr}`
const file = await import(specifier);
```
The self-upgrade feature is undesirable when deno is installed from
(Linux) distribution repository - using a system package manager. This
change will allow package maintainers to build deno with the "upgrade"
subcommand and background check disabled.
When the user runs `deno upgrade <args>` and the upgrade feature is
disabled, it will exit with error message explaining that this deno
binary was built without the upgrade feature.
Note: This patch is already used in the Alpine Linux’s
[deno](https://pkgs.alpinelinux.org/packages?name=deno) package.
This also updates deno_graph, which has the JSR change to use "exports".
It's not yet useful atm, so I've made this PR a fix about the deno doc
--lint error message improvements. I'll do a follow-up PR that adds
exports to the deno.json
This commit moves all Chrome Devtools Protocol messages to `cli/cdp.rs`
and refactors all places using these types to pull them from a common
place.
No functional changes.
As pointed out in https://github.com/denoland/deno/issues/21067 , it's
confusing that `deno doc --lint mod.ts` outputs the documentation to
stdout on success. Instead, it would be better if it outputted how many
files were checked similar to what `deno lint` does on success. It still
outputs the documentation if `--lint` or `--html` are provided so this
is non-breaking.
Adds a new `--lint` flag to `deno doc` that surfaces three kinds of
diagnostics:
1. Diagnostic for non-exported type referenced in an exported type.
* Why? People often forget to export types from a module in TypeScript.
To supress this diagnostic, add an `@internal` jsdoc tag to the internal
type.
1. Diagnostic for missing return type or missing property type on a
**public** type.
* Why? Otherwise `deno doc` will not display good documentation. Adding
explicit types also helps with type checking performance.
1. Diagnostic for missing jsdoc on a **public** type.
* Why? Everything should be documented. This diagnostic can be supressed
by adding a jsdoc comment description.
If the lint passes, `deno doc` generates documentation as usual.
For example, checking for deno doc diagnostics on the CI:
```shellsession
$ deno doc --lint mod.ts second_entrypoint.ts > /dev/null
```
This feature is incredibly useful for library authors.
## Why not include this in `deno lint`?
1. The command needs the documenation output in order to figure out the
diagnostics.
1. `deno lint` doesn't understand where the entrypoints are. That's
critical for the diagnostics to be useful.
1. It's much more performant to do this while generating documentation.
1. There is precedence in rustdoc (ex. `#![warn(missing_docs)]`).
## Why not `--check`?
It is confusing with `deno run --check`, since that means to run type
checking (and confusing with `deno check --docs`).
## Output Future Improvement
The output is not ideal atm, but it's fine for a first pass. We will
improve it in the future.
Closes https://github.com/denoland/deno_lint/pull/972
Closes https://github.com/denoland/deno_lint/issues/970
Closes https://github.com/denoland/deno/issues/19356
This commit adds `--unstable-hmr` flag, that enabled Hot Module Replacement.
This flag works like `--watch` and accepts the same arguments. If
HMR is not possible the process will be restarted instead.
Currently HMR is only supported in `deno run` subcommand.
Upon HMR a `CustomEvent("hmr")` will be dispatched that contains
information which file was changed in its `details` property.
---------
Co-authored-by: Valentin Anger <syrupthinker@gryphno.de>
Co-authored-by: David Sherret <dsherret@gmail.com>
This commit adds support for multiple entry points to `deno doc`.
Unfortunately to achieve that, I had to change the semantics of the
command to explicitly require `--filter` parameter for filtering
symbols, instead of treating second free argument as the filter argument.
`deno doc --builtin` is still supported, but cannot be mixed with
actual entrypoints.
Upgrades to deno_doc 0.70 which includes the feature for showing
non-exported types referenced in exported types as well as a much more
advanced deno doc that uses a symbol graph.
As title. This will help use the two independently from the other, which
will help in an upcoming deno doc PR where I need to parse the source
files with scope analysis.
This PR adds a new unstable "bring your own node_modules" (BYONM)
functionality currently behind a `--unstable-byonm` flag (`"unstable":
["byonm"]` in a deno.json).
This enables users to run a separate install command (ex. `npm install`,
`pnpm install`) then run `deno run main.ts` and Deno will respect the
layout of the node_modules directory as setup by the separate install
command. It also works with npm/yarn/pnpm workspaces.
For this PR, the behaviour is opted into by specifying
`--unstable-byonm`/`"unstable": ["byonm"]`, but in the future we may
make this the default behaviour as outlined in
https://github.com/denoland/deno/issues/18967#issuecomment-1761248941
This is an extremely rough initial implementation. Errors are
terrible in this and the LSP requires frequent restarts. Improvements
will be done in follow up PRs.
This commit introduces "WatcherCommunicator" struct that
is used facilitate bi-directional communication between CLI
file watcher and the watched function.
Prerequisite for https://github.com/denoland/deno/pull/20876
This brings in [`display`](https://github.com/rgbkrk/display.js) as part
of the `Deno.jupyter` namespace.
Additionally these APIs were added:
- "Deno.jupyter.md"
- "Deno.jupyter.html"
- "Deno.jupyter.svg"
- "Deno.jupyter.format"
These APIs greatly extend capabilities of rendering output in Jupyter
notebooks.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This fixes #20767.
We were losing `this` and then when an exception was happening, it
didn't show up in the output because we weren't bubbling up exceptions
from within a user defined function for displaying. I thought about
doing a `.call(object)` but didn't want to get in the way of a bound
`this` that a user or library was already putting on the function.
Adds `buffers` to the `Deno.jupyter.broadcast` API to send binary data
via comms. This affords the ability to send binary data via websockets
to the jupyter widget frontend.
This makes `CliNpmResolver` a trait. The terminology used is:
- **managed** - Deno manages the node_modules folder and does an
auto-install (ex. `ManagedCliNpmResolver`)
- **byonm** - "Bring your own node_modules" (ex. `ByonmCliNpmResolver`,
which is in this PR, but unimplemented at the moment)
Part of #18967
Closes #20535.
# Screenshots
## JSON
<img width="779" alt="image"
src="https://github.com/denoland/deno/assets/836375/668bb1a6-3f76-4b36-974e-cdc6c93f94c3">
## Vegalite
<img width="558" alt="image"
src="https://github.com/denoland/deno/assets/836375/a5e70908-6b87-42d8-85c3-1323ad52a00f">
# Implementation
Instead of going the route of recursively getting all the objects under
`application/.*json` keys, I went with `JSON.stringify`ing in denospace
then parsing it from rust. One of the key benefits of serializing and
deserializing is that non-JSON-able entries will get stripped
automatically. This also keeps the code pretty simple.
In the future we should _only_ do this for `application/.*json` keys.
cc @mmastrac
"Fixes" the exception display issue of #20524 on older versions of
Jupyter that required `evalue` to be truthy. For now, until we can do
proper processing of the `ExceptionDetails` this will make Jupyter
Notebook 6.5.1 and below happy.
This is the alternative "just work now" PR to #20530
This commit adds "deno jupyter" subcommand which
provides a Deno kernel for Jupyter notebooks.
The implementation is mostly based on Deno's REPL and
reuses large parts of it (though there's some clean up that
needs to happen in follow up PRs). Not all functionality of
Jupyter kernel is implemented and some message type
are still not implemented (eg. "inspect_request") but
the kernel is fully working and provides all the capatibilities
that the Deno REPL has; including TypeScript transpilation
and npm packages support.
Closes https://github.com/denoland/deno/issues/13016
---------
Co-authored-by: Adam Powers <apowers@ato.ms>
Co-authored-by: Kyle Kelley <rgbkrk@gmail.com>
### What
Skip writing files from the template if the files already exist in the
project directory.
### Why
When I run deno init in a directory that already has a main.ts, or one
of the other template files, I usually want to initialize a workspace
around a file I've started working in. A hard error in this case seems
counter productive. An informational message about what's being skipped
seems sufficient.
Close #20433
Previously:
```rust
pub struct TestDefinition {
pub id: String,
pub name: String,
pub range: SourceRange,
pub steps: Vec<TestDefinition>,
}
pub struct TestDefinitions {
pub discovered: Vec<TestDefinition>,
pub injected: Vec<lsp_custom::TestData>,
pub script_version: String,
}
```
Now:
```rust
pub struct TestDefinition {
pub id: String,
pub name: String,
pub range: Option<Range>,
pub is_dynamic: bool, // True for 'injected' module, not statically detected but added at runtime.
pub parent_id: Option<String>,
pub step_ids: HashSet<String>,
}
pub struct TestModule {
pub specifier: ModuleSpecifier,
pub script_version: String,
pub defs: HashMap<String, TestDefinition>,
}
```
Storing the test tree as a literal tree diminishes the value of IDs,
even though vscode stores them that way. This makes all data easily
accessible from `TestModule`. It unifies the interface between
'discovered' and 'injected' tests. This unblocks some enhancements wrt
syncing tests between the LSP and extension, such as this TODO:
61f08d5a71/client/src/testing.ts (L251-L259)
and https://github.com/denoland/vscode_deno/issues/900. We should also
get more flexibility overall.
`TestCollector` is cleaned up, now stores a `&mut TestModule` directly
and registers tests as it comes across them with
`TestModule::register()`. This method ensures sanity in the redundant
data from having both of `TestDefinition::{parent_id,step_ids}`.
All of the messy conversions between `TestDescription`,
`LspTestDescription`, `TestDefinition`, `TestData` and `TestIdentifier`
are cleaned up. They shouldn't have been using `impl From` and now the
full list of tests is available to their implementations.
The motivation is If I'm using deno lint --rules, I want to see all the
rules especially the one that have no tags, since the recommend ones are
already active
This change also prints the tags associated with the rule inline.
<!--
Before submitting a PR, please read https://deno.com/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
As the title.
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
Disables `BenchContext::start()` and `BenchContext::end()` for low
precision benchmarks (less than 0.01s per iteration). Prints a warning
when they are used in such benchmarks, suggesting to remove them.
```ts
Deno.bench("noop", { group: "noops" }, () => {});
Deno.bench("noop with start/end", { group: "noops" }, (b) => {
b.start();
b.end();
});
```
Before:
```
cpu: 12th Gen Intel(R) Core(TM) i9-12900K
runtime: deno 1.36.2 (x86_64-unknown-linux-gnu)
file:///home/nayeem/projects/deno/temp3.ts
benchmark time (avg) iter/s (min … max) p75 p99 p995
----------------------------------------------------------------------------- -----------------------------
noop 2.63 ns/iter 380,674,131.4 (2.45 ns … 27.78 ns) 2.55 ns 4.03 ns 5.33 ns
noop with start and end 302.47 ns/iter 3,306,146.0 (200 ns … 151.2 µs) 300 ns 400 ns 400 ns
summary
noop
115.14x faster than noop with start and end
```
After:
```
cpu: 12th Gen Intel(R) Core(TM) i9-12900K
runtime: deno 1.36.1 (x86_64-unknown-linux-gnu)
file:///home/nayeem/projects/deno/temp3.ts
benchmark time (avg) iter/s (min … max) p75 p99 p995
----------------------------------------------------------------------------- -----------------------------
noop 3.01 ns/iter 332,565,561.7 (2.73 ns … 29.54 ns) 2.93 ns 5.29 ns 7.45 ns
noop with start and end 7.73 ns/iter 129,291,091.5 (6.61 ns … 46.76 ns) 7.87 ns 13.12 ns 15.32 ns
Warning start() and end() calls in "noop with start and end" are ignored because it averages less than 0.01s per iteration. Remove them for better results.
summary
noop
2.57x faster than noop with start and end
```
This PR adds a test reporter for the [Test Anything
Protocol](https://testanything.org).
It makes the following implementation decisions:
- No TODO pragma, as there is no such marker in `Deno.test`
- SKIP pragma for `ignore`d tests
- Test steps are treated as TAP14 subtests
- Support for this in consumers seems spotty
- Some consumers will incorrectly interpret these markers, resulting in
unexpected output
- Considering the lack of support, and to avoid implementation
complexity,
subtests are at most one level deep (all test steps are in the same
subtest)
- To accommodate consumers that use comments to indicate test-suites
(unspecced)
- The test module path is output as a comment
- This is disabled for `--parallel` testing
- Failure diagnostics are output as JSON, which is also valid YAML
- The structure is not specified, so the format roughly follows the spec
example:
```
---
message: "Failed with error 'hostname peebles.example.com not found'"
severity: fail
found:
hostname: 'peebles.example.com'
address: ~
wanted:
hostname: 'peebles.example.com'
address: '85.193.201.85'
at:
file: test/dns-resolve.c
line: 142
...
```
Some people might get think they need to import from this directory,
which could cause confusion and duplicate dependencies. Additionally,
the `vendor` directory has special behaviour in the language server, so
importing from the folder will definitely cause confusion and issues
there.
To fix bugs around detection of when node emulation is required, we will
just eagerly initialize it. The improvements we make to reduce the
impact of the startup time:
- [x] Process stdin/stdout/stderr are lazily created
- [x] node.js global proxy no longer allocates on each access check
- [x] Process checks for `beforeExit` listeners before doing expensive
shutdown work
- [x] Process should avoid adding global event handlers until listeners
are added
Benchmarking this PR (`89de7e1ff`) vs main (`41cad2179`)
```
12:36 $ third_party/prebuilt/mac/hyperfine --warmup 100 -S none './deno-41cad2179 run ./empty.js' './deno-89de7e1ff run ./empty.js'
Benchmark 1: ./deno-41cad2179 run ./empty.js
Time (mean ± σ): 24.3 ms ± 1.6 ms [User: 16.2 ms, System: 6.0 ms]
Range (min … max): 21.1 ms … 29.1 ms 115 runs
Benchmark 2: ./deno-89de7e1ff run ./empty.js
Time (mean ± σ): 24.0 ms ± 1.4 ms [User: 16.3 ms, System: 5.6 ms]
Range (min … max): 21.3 ms … 28.6 ms 126 runs
```
Fixes https://github.com/denoland/deno/issues/20142
Fixes https://github.com/denoland/deno/issues/15826
Fixes https://github.com/denoland/deno/issues/20028
Renames the unstable `deno_modules` directory and corresponding settings
to `vendor` after feedback. Also causes the vendoring of the
`node_modules` directory which can be disabled via
`--node-modules-dir=false` or `"nodeModulesDir": false`.
This commit adds a "dot" reporter to "deno test" subcommand,
that can be activated using "--dot" flag.
It provides a concise output using:
- "." for passing test
- "," for ignored test
- "!" for failing test
User output is silenced and not printed to the console.
In non-TTY environments each result is printed on a separate line.
This commit makes the following changes
- Created a `CompoundTestReporter` to allow us to use multiple reporters
- Implements `JUnitTestReporter` which writes JUnit XML to a path
- Added a CLI flag/option `--junit` that enables JUnit reporting. By
default this writes the report to `stdout` (and disables pretty
reporting). If a path is provided, it will write the JUnit report to
that file while the pretty reporter writes to stdout like normal
Output of `deno -- test --allow-all --unstable
--location=http://js-unit-tests/foo/bar --junit
cli/tests/unit/testing_test.ts `
```xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="deno test" tests="7" failures="0" errors="0" time="0.176">
<testsuite name="file:///Users/cooper/deno/deno/cli/tests/unit/testing_test.ts" tests="7" disabled="0" errors="0" failures="0">
<testcase name="testWrongOverloads" time="0.012">
</testcase>
<testcase name="nameOfTestCaseCantBeEmpty" time="0.009">
</testcase>
<testcase name="invalidStepArguments" time="0.008">
</testcase>
<testcase name="nameOnTextContext" time="0.029">
<properties>
<property name="step[passed]" value="step ... nested step"/>
<property name="step[passed]" value="step"/>
</properties>
</testcase>
<testcase name="originOnTextContext" time="0.030">
<properties>
<property name="step[passed]" value="step ... nested step"/>
<property name="step[passed]" value="step"/>
</properties>
</testcase>
<testcase name="parentOnTextContext" time="0.030">
<properties>
<property name="step[passed]" value="step ... nested step"/>
<property name="step[passed]" value="step"/>
</properties>
</testcase>
<testcase name="explicit undefined for boolean options" time="0.009">
</testcase>
</testsuite>
</testsuites>
```
Closes https://github.com/denoland/deno/issues/15277
This commit adds a single "warmup" run of empty function when running
`deno bench`.
This change will break so-called "JIT bias" which makes V8 optimize the
first function
and then bail out of optimization on second function. In essence the
"warmup" function
is getting optimized and then all user benches are bailed out of
optimization.
This PR breaks the addition of the `TestReporter` trait and refactoring
of `PrettyTestReporter` out of #19747. The goal is to enable the
addition of test reporters, including machine readable output.
…nclusion" (#19519)"
This reverts commit 28a4f3d0f5.
This change causes failures when used outside Deno repo:
```
============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.
Platform: linux x86_64
Version: 1.34.3+b37b286
Args: ["/opt/hostedtoolcache/deno/0.0.0-b37b286f7fa68d5656f7c180f6127bdc38cf2cf5/x64/deno", "test", "--doc", "--unstable", "--allow-all", "--coverage=./cov"]
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Failed to read "/home/runner/work/deno/deno/core/00_primordials.js"
Caused by:
No such file or directory (os error 2)', core/runtime/jsruntime.rs:699:8
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
Relands #19463. This time the `ExtensionFileSourceCode` enum is
preserved, so this effectively just splits feature
`include_js_for_snapshotting` into `exclude_js_sources` and
`runtime_js_sources`, adds a `force_include_js_sources` option on
`extension!()`, and unifies `ext::Init_ops_and_esm()` and
`ext::init_ops()` into `ext::init()`.
… (#19463)"
This reverts commit ceb03cfb03.
This is being reverted because it causes 3.5Mb increase in the binary
size,
due to runtime JS code being included in the binary, even though it's
already snapshotted.
CC @nayeemrmn
Remove `ExtensionFileSourceCode::LoadedFromFsDuringSnapshot` and feature
`include_js_for_snapshotting` since they leak paths that are only
applicable in this repo to embedders. Replace with feature
`exclude_js_sources`. Additionally the feature
`force_include_js_sources` allows negating it, if both features are set.
We need both of these because features are additive and there must be a
way of force including sources for snapshot creation while still having
the `exclude_js_sources` feature. `force_include_js_sources` is only set
for build deps, so sources are still excluded from the final binary.
You can also specify `force_include_js_sources` on any extension to
override the above features for that extension. Towards #19398.
But there was still the snapshot-from-snapshot situation where code
could be executed twice, I addressed that by making `mod_evaluate()` and
scripts like `core/01_core.js` behave idempotently. This allowed
unifying `ext::init_ops()` and `ext::init_ops_and_esm()` into
`ext::init()`.
This commit migrates "deno_core" from using "FuturesUnordered" to
"tokio::task::JoinSet". This makes every op to be a separate Tokio task
and should unlock better utilization of kqueue/epoll.
There were two quirks added to this PR:
- because of the fact that "JoinSet" immediately polls spawn tasks,
op sanitizers can give false positives in some cases, this was
alleviated by polling event loop once before running a test with
"deno test", which gives canceled ops an opportunity to settle
- "JsRuntimeState::waker" was moved to "OpState::waker" so that FFI
API can still use threadsafe functions - without this change the
registered wakers were wrong as they would not wake up the
whole "JsRuntime" but the task associated with an op
---------
Co-authored-by: Matt Mastracci <matthew@mastracci.com>
When someone explicitly opts into using the node_modules dir via
`--node-modules-dir` or setting `"nodeModulesDir": true` in the
deno.json file, we should eagerly ensure a top level package.json
install is done on startup. This was initially always done when we added
package.json support and a package.json was auto-discovered, but scaled
it back to be lazily done when a bare specifier matched an entry in the
package.json because of how disruptive it was for people using Deno
scripts in Node projects. That said, it does not make sense for someone
to opt-into having deno control and use their node_modules directory and
not want a package.json install to occur. If such a rare scenario
exists, the `DENO_NO_PACKAGE_JSON=1` environment variable can be set.
Ideally, we would only ever use a node_modules directory with this
explicit opt-in so everything is very clear, but we still have this
automatic scenario when there's a package.json in order to make more
node projects work out of the box.
We never properly added support for this. This fixes vendoring when it
has npm or node specifiers. Vendoring occurs by adding a
`"nodeModulesDir": true` property to deno.json then it uses a local
node_modules directory. This can be opted out by setting
`"nodeModulesDir": false` or running with `--node-modules-dir=false`.
Closes #18090
Closes #17210
Closes #17619
Closes #16778
Note: If the package information has already been cached, then this
requires running with `--reload` or for the registry information to be
fetched some other way (ex. the cache busting).
Closes #15544
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Partially supersedes #19016.
This migrates `spawn` and `spawn_blocking` to `deno_core`, and removes
the requirement for `spawn` tasks to be `Send` given our single-threaded
executor.
While we don't need to technically do anything w/`spawn_blocking`, this
allows us to have a single `JoinHandle` type that works for both cases,
and allows us to more easily experiment with alternative
`spawn_blocking` implementations that do not require tokio (ie: rayon).
Async ops (+~35%):
Before:
```
time 1310 ms rate 763358
time 1267 ms rate 789265
time 1259 ms rate 794281
time 1266 ms rate 789889
```
After:
```
time 956 ms rate 1046025
time 954 ms rate 1048218
time 924 ms rate 1082251
time 920 ms rate 1086956
```
HTTP serve (+~4.4%):
Before:
```
Running 10s test @ http://localhost:4500
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 68.78us 19.77us 1.43ms 86.84%
Req/Sec 68.78k 5.00k 73.84k 91.58%
1381833 requests in 10.10s, 167.36MB read
Requests/sec: 136823.29
Transfer/sec: 16.57MB
```
After:
```
Running 10s test @ http://localhost:4500
2 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 63.12us 17.43us 1.11ms 85.13%
Req/Sec 71.82k 3.71k 77.02k 79.21%
1443195 requests in 10.10s, 174.79MB read
Requests/sec: 142921.99
Transfer/sec: 17.31MB
```
Suggested-By: alice@ryhl.io
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This is the initial support for npm and node specifiers in `deno
compile`. The npm packages are included in the binary and read from it via
a virtual file system. This also supports the `--node-modules-dir` flag,
dependencies specified in a package.json, and npm binary commands (ex.
`deno compile --unstable npm:cowsay`)
Closes #16632
This removes `ProcState` and replaces it with a new `CliFactory` which
initializes our "service structs" on demand. This isn't a performance
improvement at the moment for `deno run`, but might unlock performance
improvements in the future.
Fixes regression from #18878 where `Promise.reject()`,
`Promise.reject(undefined)` and `reportError(undefined)` panic in the
REPL.
Fixes `throw undefined` printing `Uncaught Unknown exception` instead of
`Uncaught undefined`.
Fixes #8858.
Fixes #8869.
```
$ target/debug/deno
Deno 1.32.5
exit using ctrl+d, ctrl+c, or close()
REPL is running with all permissions allowed.
To specify permissions, run `deno repl` with allow flags.
> Promise.reject(new Error("bar"));
Promise { <rejected> Error: bar
at <anonymous>:2:16 }
Uncaught (in promise) Error: bar
at <anonymous>:2:16
> reportError(new Error("baz"));
undefined
Uncaught Error: baz
at <anonymous>:2:13
>
This is just a straight refactor and I didn't do any cleanup in
ext/node. After this PR we can start to clean it up and make things
private that don't need to be public anymore.
1. Adds cli/standalone folder
2. Writes the bytes directly to the output file. When adding npm
packages this might get quite large, so let's not keep the final output
in memory just in case.
1. Breaks up functionality within `ProcState` into several other structs
to break out the responsibilities (`ProcState` is only a data struct
now).
2. Moves towards being able to inject dependencies more easily and have
functionality only require what it needs.
3. Exposes `Arc<T>` around the "service structs" instead of it being
embedded within them. The idea behind embedding them was to reduce the
verbosity of needing to pass around `Arc<...>`, but I don't think it was
exactly working and as we move more of these structs to be more
injectable I don't think the extra verbosity will be a big deal.
Removes the functions in the `emit` module and replaces them with an
`Emitter` struct that can have "ctor dependencies" injected rather than
using functions to pass along the dependencies.
This is part of a long term refactor to move more functionality out of
proc state.
Stores the test/bench functions in rust op state during registration.
The functions are wrapped in JS first so that they return a directly
convertible `TestResult`/`BenchResult`. Test steps are still mostly
handled in JS since they are pretty much invoked by the user. Allows
removing a bunch of infrastructure for communicating between JS and
rust. Allows using rust utilities for things like shuffling tests
(`Vec::shuffle`). We can progressively move op and resource sanitization
to rust as well.
Fixes #17122.
Fixes #17312.
- bump deps: the newest `lazy-regex` need newer `oncecell` and
`regex`
- reduce `unwrap`
- remove dep `lazy_static`
- make more regex cached
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This is a follow-on to the earlier work in reducing string copies,
mainly focused on ensuring that ASCII strings are easy to provide to the
JS runtime.
While we are replacing a 16-byte reference in a number of places with a
24-byte structure (measured via `std::mem::size_of`), the reduction in
copies wins out over the additional size of the arguments passed into
functions.
Benchmarking shows approximately the same if not slightly less wallclock
time/instructions retired, but I believe this continues to open up
further refactoring opportunities.
1. Fixes a cosmetic issue in the repl where it would display lsp warning
messages.
2. Lazily loads dependencies from the package.json on use.
3. Supports using bare specifiers from package.json in the REPL.
Closes #17929
Closes #18494
1. Rewrites the tests to be more back and forth rather than getting the
output all at once (which I believe was causing the hangs on linux and
maybe mac)
2. Runs the pty tests on the linux ci.
3. Fixes a bunch of tests that were just wrong.
4. Adds timeouts on the pty tests.
This gets SQLite off the flamegraph and reduces initialization time by
somewhere between 0.2ms and 0.5ms. In addition, I took the opportunity
to move all the cache management code to a single place and reduce
duplication. While the PR has a net gain of lines, much of that is just
being a bit more deliberate with how we're recovering from errors.
The existing caches had various policies for dealing with cache
corruption, so I've unified them and tried to isolate the decisions we
make for recovery in a single place (see `open_connection` in
`CacheDB`). The policy I chose was:
1. Retry twice to open on-disk caches
2. If that fails, try to delete the file and recreate it on-disk
3. If we fail to delete the file or re-create a new cache, use a
fallback strategy that can be chosen per-cache: InMemory (temporary
cache for the process run), BlackHole (ignore writes, return empty
reads), or Error (fail on every operation).
The caches all use the same general code now, and share the cache
failure recovery policy.
In addition, it cleans up a TODO in the `NodeAnalysisCache`.
Reduce the number of copies and allocations of script code by carrying
around ownership/reference information from creation time.
As an advantage, this allows us to maintain the identity of `&'static
str`-based scripts and use v8's external 1-byte strings (to avoid
incorrectly passing non-ASCII strings, debug `assert!`s gate all string
reference paths).
Benchmark results:
Perf improvements -- ~0.1 - 0.2ms faster, but should reduce garbage
w/external strings and reduces data copies overall. May also unlock some
more interesting optimizations in the future.
This requires adding some generics to functions, but manual
monomorphization has been applied (outer/inner function) to avoid code
bloat.
Moving some code around in `ext/node` is it's a bit better well defined
and makes it possible for others to embed it.
I expect to see no difference in startup perf with this change.
These functions don't need to be async, as they are only calling
synchronous JavaScript code. As a follow up, all 3 functions
should be merge together - this will reduce roundtrips for
calling V8 from Rust, which is somewhat expensive
This change will enable dynamic imports and web workers to use modules
not reachable from the main module, by passing a list of extra side
module roots as options to `deno compile`.
This can be done by specifying "--include" flag that accepts a file path or a
URL. This flag can be specified multiple times, to include several modules.
The modules specified with "--include" flag, will be added to the produced
"eszip".
This implements two macros to simplify extension registration and centralize a lot of the boilerplate as a base for future improvements:
* `deno_core::ops!` registers a block of `#[op]`s, optionally with type
parameters, useful for places where we share lists of ops
* `deno_core::extension!` is used to register an extension, and creates
two methods that can be used at runtime/snapshot generation time:
`init_ops` and `init_ops_and_esm`.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Closes #17831. This change hides the indices of any indexed collection
when triggering tab completion for object properties in the REPL.
An example is shown in the issue, but for verbosity here is another.
Before the change:
```
> const arr = new Uint8ClampedArray([1, 2, 3])
undefined
> arr.
0 map
1 reverse
2 reduce
...
```
After the change:
```
> const arr = new Uint8ClampedArray([1, 2, 3])
undefined
> arr.
constructor reduce
BYTES_PER_ELEMENT reduceRight
buffer set
...
```
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
These methods are confusing because the arguments are backwards. I feel
like they should have never been added to `Option<T>` and that clippy
should suggest rewriting to
`map(...).unwrap_or(...)`/`map(...).unwrap_or_else(|| ...)`
https://github.com/rust-lang/rfcs/issues/1025
Chrono's `clock` feature pulls in `iana-time-zone` which links to macOS
core_foundation. This PR itself is not enough to get rid of
CoreFoundation. Removal depends on getting rid of security framework,
see #18071
This commit changes various CLI subcommands that have support for
the "--watch" flag to use initial current working directory when
resolving "main module".
This is part of migration towards explicitly passing current working
directory to "deno_core::resolve_url_or_path" API.
As a side effect this makes "deno <subcommand> --watch" more aligned to
user expectations, where calling "Deno.chdir()" during program doesn't
break watcher.
Towards landing https://github.com/denoland/deno/pull/15454
This commit changes current "deno_core::resolve_url_or_path" API to
"resolve_url_or_path_deprecated" and adds new "resolve_url_or_path"
API that requires to explicitly pass the directory from which paths
should be resolved to.
Some of the call sites were updated to use the new API, the reminder
of them will be updated in a follow up PR.
Towards landing https://github.com/denoland/deno/pull/15454
This commit renames "deno_core::InternalModuleLoader" to
"ExtModuleLoader" and changes the specifiers used by the
modules loaded from this loader to "ext:".
"internal:" scheme was really ambiguous and it's more characters than
"ext:", which should result in slightly smaller snapshot size.
Closes https://github.com/denoland/deno/issues/18020
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
This commit updates deno_lint crate to 0.41.0. The new version contains
a braking change that requries a minor code fix here, which is also
addressed in this commit.
This lazily does an "npm install" when any package name matches what's
found in the package.json or when running a script from package.json
with deno task.
Part of #17916
Closes #17928
This is a super basic initial implementation. We don't create a
`node_modules/.bin` folder at the moment and add it to the PATH like we
should which is necessary to make command name resolution in the
subprocess work properly (ex. you run a script that launches another
script that then tries to launch an "npx command"... this won't work
atm).
Closes #17492
This commit enables resolution of "bare specifiers" (eg. "import express
from 'express';") if a "package.json" file is discovered.
It's a step towards being able to run projects authored for Node.js
without any changes.
With this commit we are able to successfully run Vite projects without
any changes to the user code.
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
This changes npm specifiers to be handled by deno_graph and resolved to
an npm package name and version when the specifier is encountered. It
also slightly changes how npm specifier resolution occurs—previously it
would collect all the npm specifiers and resolve them all at once, but
now it resolves them on the fly as they are encountered in the module
graph.
https://github.com/denoland/deno_graph/pull/232
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This PR fixes peer dependency resolution to only resolve peers based on
the current graph traversal path. Previously, it would resolve a peers
by looking at a graph node's ancestors, which is not correct because
graph nodes are shared by different resolutions.
It also stores more information about peer dependency resolution in the
lockfile.
This commits adds auto-discovery of "package.json" file when running
"deno run" and "deno task" subcommands. In case of "deno run" the
"package.json" is being looked up starting from the directory of the
script that is being run, stopping early if "deno.json(c)" file is found
(ie. FS tree won't be traversed "up" from "deno.json").
When "package.json" is discovered the "--node-modules-dir" flag is
implied, leading to creation of local "node_modules/" directory - we
did that, because most tools relying on "package.json" will expect
"node_modules/" directory to be present (eg. Vite). Additionally
"dependencies" and "devDependencies" specified in the "package.json"
are downloaded on startup.
This is a stepping stone to supporting bare specifier imports, but
the actual integration will be done in a follow up commit.
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
The `NpmPackageId` struct is being renamed to `NpmPackageNodeId`. In a
future PR it will be moved down into only npm dependency resolution and
a `NpmPackageId` struct will be introduced in `deno_graph` that only has
the name and version of the package (no peer dependency identifier
information). So a `NpmPackageReq` will map to an `NpmPackageId`, which
will map to an `NpmPackageNodeId` in the npm resolution.
This PR changes Node.js/npm compatibility layer to use polyfills for
built-in Node.js
embedded in the snapshot (that are coming from "ext/node" extension).
As a result loading `std/node`, either from
"https://deno.land/std@<latest>/" or
from "DENO_NODE_COMPAT_URL" env variable were removed. All code that is
imported via "npm:" specifiers now uses code embedded in the snapshot.
Several fixes were applied to various modules in "ext/node" to make
tests pass.
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
This commit moves some code around from "cli/node/mod.rs" to
"ext/node". Additionally "ext/node" was changed to factor out
"ops.rs" and "polyfill.rs" modules.
This change makes absolute urls, that contain no path like `deno install
https://my-cli.io` to follow redirects and extract the name from it.
It allows modifies `test_util` server listener on port `4550`
(`REDIRECT_ABSOLUTE_PORT`) to allow for specifying `redirect_to` query
param, that fill use that value for it's next redirect.
Fixes https://github.com/denoland/deno/issues/17409
This commit changes handling of config file to enable
specifying "imports" and "scopes" objects effectively making
the configuration file an import map.
"imports" and "scopes" take precedence over "importMap" configuration,
but have lower priority than "--importmap" CLI flag.
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
Co-authored-by: David Sherret <dsherret@gmail.com>
Allows to change behavior of `deno fmt` to use "ASI" setting for
semicolons instead of always prefering them, this is done
by "--options-semi=asi" flag or `"semi": "asi"` setting
in the config file.
This change prints a link to the release notes to `deno upgrade` output
and its variations.
Release notes aren't printed for commands relating to canary versions.
Closes #16350.
The way the standalone mode handles the `--cert` flag is different to
all other modes. This is because `--cert` takes a path to the
certificate file, which is directly added to the root cert store; except
for compile mode, where its byte contents are stored in the standalone
metadata, and they are added to the root cert store after the
`ProcState` is created.
This change instead changes `Flags::ca_file` (an `Option<String>`) into
`Flags::ca_data`, which can represent a `String` file path or a
`Vec<u8>` with the certificate contents. That way, standalone mode can
create a `ProcState` whose root cert store alreay contains the
certificate.
This change also adds a tests for certificates in standalone mode, since
there weren't any before.
This refactor will help with implementing web workers in standalone mode
in the future.
This commit changes "ProcState" to store "file_fetcher" field in an "Arc",
allowing it to be preserved between restarts and thus keeping the state
alive between the restarts. File watchers for "deno test" and "deno bench"
now reset "ProcState" between restarts.
Turns out we were cloning permissions which after prompting were discarded,
so the state of permissions was never preserved. To handle that we need to store
all permissions behind "Arc<Mutex<>>" (because there are situations where we
need to send them to other thread).
Testing and benching code still uses "Permissions" in most places - it's undesirable
to share the same permission set between various test/bench files - otherwise
granting or revoking permissions in one file would influence behavior of other test
files.
The output of `init` are commands, so this should be treated as a "Shell
script". In Shell script, comments must start with `#`, not `//`. (This
also makes the commands example easier to be copied to somewhere.)
In our `require()` implementation we use a special logic to resolve
"base path" when looking for matching packages, however this logic
is in contradiction to what needs to happen if there's a local
"node_modules"
directory used. This commit changes require implementation to be aware
if we're running off of global node modules cache or a local one.
This commit changes "deno repl" command to run with no permissions by
default and accept "--allow-*" flags.
This change is dictated by the fact that currently there is no way to
run REPL with limited permissions. Technically it's a breaking
change in the CLI command, but there's agreement in the team
that it has merit and it's a good solution.
Running just "deno" command still starts the REPL with full permissions
allowed, but now a banner is printed to inform users about that:
We currently only do this for fmt. This makes it so they're excluded by
default, but you can still opt into these directories by explicitly
specifying them.
With trial and error I found that most debuggers expect "isDefault" to be sent
in "auxData" field of "executionContextCreated" notification. This stems from
the fact that Node.js sends this data and eg. VSCode requires it to close
connection to the debugger when the program finishes execution.
This commit changes history handling of the REPL.
There were some situations were history wasn't properly saved and flushed to a
file, making history very spotty. This commit changes it to save every line into
the history file and flush it to disk before being evaluated.
Thanks to this all lines, including "close()" will be stored in the history
file.
If for any reason we're not able to save history file, a single warning will be
printed to the REPL and it will continue to work, even if subsequent tries will
fail to save to disk.
Supports npm specifiers for `deno install`. This will by default always
use a lockfile (which is generated on first run) unless `--no-lock` is
specified.
This adds support for peer dependencies in npm packages.
1. If not found higher in the tree (ancestor and ancestor siblings),
peer dependencies are resolved like a dependency similar to npm 7.
2. Optional peer dependencies are only resolved if found higher in the
tree.
3. This creates "copy packages" or duplicates of a package when a
package has different resolution due to peer dependency resolution—see
https://pnpm.io/how-peers-are-resolved. Unlike pnpm though, duplicates
of packages will have `_1`, `_2`, etc. added to the end of the package
version in the directory in order to minimize the chance of hitting the
max file path limit on Windows. This is done for both the local
"node_modules" directory and also the global npm cache. The files are
hard linked in this case to reduce hard drive space.
This is a first pass and the code is definitely more inefficient than it
could be.
Closes #15823
