This is just a straight refactor and I didn't do any cleanup in
ext/node. After this PR we can start to clean it up and make things
private that don't need to be public anymore.
1. Adds cli/standalone folder
2. Writes the bytes directly to the output file. When adding npm
packages this might get quite large, so let's not keep the final output
in memory just in case.
1. Breaks up functionality within `ProcState` into several other structs
to break out the responsibilities (`ProcState` is only a data struct
now).
2. Moves towards being able to inject dependencies more easily and have
functionality only require what it needs.
3. Exposes `Arc<T>` around the "service structs" instead of it being
embedded within them. The idea behind embedding them was to reduce the
verbosity of needing to pass around `Arc<...>`, but I don't think it was
exactly working and as we move more of these structs to be more
injectable I don't think the extra verbosity will be a big deal.
Removes the functions in the `emit` module and replaces them with an
`Emitter` struct that can have "ctor dependencies" injected rather than
using functions to pass along the dependencies.
This is part of a long term refactor to move more functionality out of
proc state.
Stores the test/bench functions in rust op state during registration.
The functions are wrapped in JS first so that they return a directly
convertible `TestResult`/`BenchResult`. Test steps are still mostly
handled in JS since they are pretty much invoked by the user. Allows
removing a bunch of infrastructure for communicating between JS and
rust. Allows using rust utilities for things like shuffling tests
(`Vec::shuffle`). We can progressively move op and resource sanitization
to rust as well.
Fixes #17122.
Fixes #17312.
- bump deps: the newest `lazy-regex` need newer `oncecell` and
`regex`
- reduce `unwrap`
- remove dep `lazy_static`
- make more regex cached
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This is a follow-on to the earlier work in reducing string copies,
mainly focused on ensuring that ASCII strings are easy to provide to the
JS runtime.
While we are replacing a 16-byte reference in a number of places with a
24-byte structure (measured via `std::mem::size_of`), the reduction in
copies wins out over the additional size of the arguments passed into
functions.
Benchmarking shows approximately the same if not slightly less wallclock
time/instructions retired, but I believe this continues to open up
further refactoring opportunities.
1. Fixes a cosmetic issue in the repl where it would display lsp warning
messages.
2. Lazily loads dependencies from the package.json on use.
3. Supports using bare specifiers from package.json in the REPL.
Closes #17929
Closes #18494
1. Rewrites the tests to be more back and forth rather than getting the
output all at once (which I believe was causing the hangs on linux and
maybe mac)
2. Runs the pty tests on the linux ci.
3. Fixes a bunch of tests that were just wrong.
4. Adds timeouts on the pty tests.
This gets SQLite off the flamegraph and reduces initialization time by
somewhere between 0.2ms and 0.5ms. In addition, I took the opportunity
to move all the cache management code to a single place and reduce
duplication. While the PR has a net gain of lines, much of that is just
being a bit more deliberate with how we're recovering from errors.
The existing caches had various policies for dealing with cache
corruption, so I've unified them and tried to isolate the decisions we
make for recovery in a single place (see `open_connection` in
`CacheDB`). The policy I chose was:
1. Retry twice to open on-disk caches
2. If that fails, try to delete the file and recreate it on-disk
3. If we fail to delete the file or re-create a new cache, use a
fallback strategy that can be chosen per-cache: InMemory (temporary
cache for the process run), BlackHole (ignore writes, return empty
reads), or Error (fail on every operation).
The caches all use the same general code now, and share the cache
failure recovery policy.
In addition, it cleans up a TODO in the `NodeAnalysisCache`.
Reduce the number of copies and allocations of script code by carrying
around ownership/reference information from creation time.
As an advantage, this allows us to maintain the identity of `&'static
str`-based scripts and use v8's external 1-byte strings (to avoid
incorrectly passing non-ASCII strings, debug `assert!`s gate all string
reference paths).
Benchmark results:
Perf improvements -- ~0.1 - 0.2ms faster, but should reduce garbage
w/external strings and reduces data copies overall. May also unlock some
more interesting optimizations in the future.
This requires adding some generics to functions, but manual
monomorphization has been applied (outer/inner function) to avoid code
bloat.
Moving some code around in `ext/node` is it's a bit better well defined
and makes it possible for others to embed it.
I expect to see no difference in startup perf with this change.
These functions don't need to be async, as they are only calling
synchronous JavaScript code. As a follow up, all 3 functions
should be merge together - this will reduce roundtrips for
calling V8 from Rust, which is somewhat expensive
This change will enable dynamic imports and web workers to use modules
not reachable from the main module, by passing a list of extra side
module roots as options to `deno compile`.
This can be done by specifying "--include" flag that accepts a file path or a
URL. This flag can be specified multiple times, to include several modules.
The modules specified with "--include" flag, will be added to the produced
"eszip".
This implements two macros to simplify extension registration and centralize a lot of the boilerplate as a base for future improvements:
* `deno_core::ops!` registers a block of `#[op]`s, optionally with type
parameters, useful for places where we share lists of ops
* `deno_core::extension!` is used to register an extension, and creates
two methods that can be used at runtime/snapshot generation time:
`init_ops` and `init_ops_and_esm`.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Closes #17831. This change hides the indices of any indexed collection
when triggering tab completion for object properties in the REPL.
An example is shown in the issue, but for verbosity here is another.
Before the change:
```
> const arr = new Uint8ClampedArray([1, 2, 3])
undefined
> arr.
0 map
1 reverse
2 reduce
...
```
After the change:
```
> const arr = new Uint8ClampedArray([1, 2, 3])
undefined
> arr.
constructor reduce
BYTES_PER_ELEMENT reduceRight
buffer set
...
```
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
These methods are confusing because the arguments are backwards. I feel
like they should have never been added to `Option<T>` and that clippy
should suggest rewriting to
`map(...).unwrap_or(...)`/`map(...).unwrap_or_else(|| ...)`
https://github.com/rust-lang/rfcs/issues/1025
Chrono's `clock` feature pulls in `iana-time-zone` which links to macOS
core_foundation. This PR itself is not enough to get rid of
CoreFoundation. Removal depends on getting rid of security framework,
see #18071
This commit changes various CLI subcommands that have support for
the "--watch" flag to use initial current working directory when
resolving "main module".
This is part of migration towards explicitly passing current working
directory to "deno_core::resolve_url_or_path" API.
As a side effect this makes "deno <subcommand> --watch" more aligned to
user expectations, where calling "Deno.chdir()" during program doesn't
break watcher.
Towards landing https://github.com/denoland/deno/pull/15454
This commit changes current "deno_core::resolve_url_or_path" API to
"resolve_url_or_path_deprecated" and adds new "resolve_url_or_path"
API that requires to explicitly pass the directory from which paths
should be resolved to.
Some of the call sites were updated to use the new API, the reminder
of them will be updated in a follow up PR.
Towards landing https://github.com/denoland/deno/pull/15454
This commit renames "deno_core::InternalModuleLoader" to
"ExtModuleLoader" and changes the specifiers used by the
modules loaded from this loader to "ext:".
"internal:" scheme was really ambiguous and it's more characters than
"ext:", which should result in slightly smaller snapshot size.
Closes https://github.com/denoland/deno/issues/18020
<!--
Before submitting a PR, please read http://deno.land/manual/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
This commit updates deno_lint crate to 0.41.0. The new version contains
a braking change that requries a minor code fix here, which is also
addressed in this commit.
This lazily does an "npm install" when any package name matches what's
found in the package.json or when running a script from package.json
with deno task.
Part of #17916
Closes #17928
This is a super basic initial implementation. We don't create a
`node_modules/.bin` folder at the moment and add it to the PATH like we
should which is necessary to make command name resolution in the
subprocess work properly (ex. you run a script that launches another
script that then tries to launch an "npx command"... this won't work
atm).
Closes #17492
This commit enables resolution of "bare specifiers" (eg. "import express
from 'express';") if a "package.json" file is discovered.
It's a step towards being able to run projects authored for Node.js
without any changes.
With this commit we are able to successfully run Vite projects without
any changes to the user code.
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
This changes npm specifiers to be handled by deno_graph and resolved to
an npm package name and version when the specifier is encountered. It
also slightly changes how npm specifier resolution occurs—previously it
would collect all the npm specifiers and resolve them all at once, but
now it resolves them on the fly as they are encountered in the module
graph.
https://github.com/denoland/deno_graph/pull/232
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This PR fixes peer dependency resolution to only resolve peers based on
the current graph traversal path. Previously, it would resolve a peers
by looking at a graph node's ancestors, which is not correct because
graph nodes are shared by different resolutions.
It also stores more information about peer dependency resolution in the
lockfile.
This commits adds auto-discovery of "package.json" file when running
"deno run" and "deno task" subcommands. In case of "deno run" the
"package.json" is being looked up starting from the directory of the
script that is being run, stopping early if "deno.json(c)" file is found
(ie. FS tree won't be traversed "up" from "deno.json").
When "package.json" is discovered the "--node-modules-dir" flag is
implied, leading to creation of local "node_modules/" directory - we
did that, because most tools relying on "package.json" will expect
"node_modules/" directory to be present (eg. Vite). Additionally
"dependencies" and "devDependencies" specified in the "package.json"
are downloaded on startup.
This is a stepping stone to supporting bare specifier imports, but
the actual integration will be done in a follow up commit.
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
The `NpmPackageId` struct is being renamed to `NpmPackageNodeId`. In a
future PR it will be moved down into only npm dependency resolution and
a `NpmPackageId` struct will be introduced in `deno_graph` that only has
the name and version of the package (no peer dependency identifier
information). So a `NpmPackageReq` will map to an `NpmPackageId`, which
will map to an `NpmPackageNodeId` in the npm resolution.
This PR changes Node.js/npm compatibility layer to use polyfills for
built-in Node.js
embedded in the snapshot (that are coming from "ext/node" extension).
As a result loading `std/node`, either from
"https://deno.land/std@<latest>/" or
from "DENO_NODE_COMPAT_URL" env variable were removed. All code that is
imported via "npm:" specifiers now uses code embedded in the snapshot.
Several fixes were applied to various modules in "ext/node" to make
tests pass.
---------
Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
This commit moves some code around from "cli/node/mod.rs" to
"ext/node". Additionally "ext/node" was changed to factor out
"ops.rs" and "polyfill.rs" modules.
This change makes absolute urls, that contain no path like `deno install
https://my-cli.io` to follow redirects and extract the name from it.
It allows modifies `test_util` server listener on port `4550`
(`REDIRECT_ABSOLUTE_PORT`) to allow for specifying `redirect_to` query
param, that fill use that value for it's next redirect.
Fixes https://github.com/denoland/deno/issues/17409
This commit changes handling of config file to enable
specifying "imports" and "scopes" objects effectively making
the configuration file an import map.
"imports" and "scopes" take precedence over "importMap" configuration,
but have lower priority than "--importmap" CLI flag.
Co-authored-by: David Sherret <dsherret@users.noreply.github.com>
Co-authored-by: David Sherret <dsherret@gmail.com>
Allows to change behavior of `deno fmt` to use "ASI" setting for
semicolons instead of always prefering them, this is done
by "--options-semi=asi" flag or `"semi": "asi"` setting
in the config file.
This change prints a link to the release notes to `deno upgrade` output
and its variations.
Release notes aren't printed for commands relating to canary versions.
Closes #16350.
The way the standalone mode handles the `--cert` flag is different to
all other modes. This is because `--cert` takes a path to the
certificate file, which is directly added to the root cert store; except
for compile mode, where its byte contents are stored in the standalone
metadata, and they are added to the root cert store after the
`ProcState` is created.
This change instead changes `Flags::ca_file` (an `Option<String>`) into
`Flags::ca_data`, which can represent a `String` file path or a
`Vec<u8>` with the certificate contents. That way, standalone mode can
create a `ProcState` whose root cert store alreay contains the
certificate.
This change also adds a tests for certificates in standalone mode, since
there weren't any before.
This refactor will help with implementing web workers in standalone mode
in the future.
This commit changes "ProcState" to store "file_fetcher" field in an "Arc",
allowing it to be preserved between restarts and thus keeping the state
alive between the restarts. File watchers for "deno test" and "deno bench"
now reset "ProcState" between restarts.
Turns out we were cloning permissions which after prompting were discarded,
so the state of permissions was never preserved. To handle that we need to store
all permissions behind "Arc<Mutex<>>" (because there are situations where we
need to send them to other thread).
Testing and benching code still uses "Permissions" in most places - it's undesirable
to share the same permission set between various test/bench files - otherwise
granting or revoking permissions in one file would influence behavior of other test
files.
The output of `init` are commands, so this should be treated as a "Shell
script". In Shell script, comments must start with `#`, not `//`. (This
also makes the commands example easier to be copied to somewhere.)
In our `require()` implementation we use a special logic to resolve
"base path" when looking for matching packages, however this logic
is in contradiction to what needs to happen if there's a local
"node_modules"
directory used. This commit changes require implementation to be aware
if we're running off of global node modules cache or a local one.
This commit changes "deno repl" command to run with no permissions by
default and accept "--allow-*" flags.
This change is dictated by the fact that currently there is no way to
run REPL with limited permissions. Technically it's a breaking
change in the CLI command, but there's agreement in the team
that it has merit and it's a good solution.
Running just "deno" command still starts the REPL with full permissions
allowed, but now a banner is printed to inform users about that:
We currently only do this for fmt. This makes it so they're excluded by
default, but you can still opt into these directories by explicitly
specifying them.
With trial and error I found that most debuggers expect "isDefault" to be sent
in "auxData" field of "executionContextCreated" notification. This stems from
the fact that Node.js sends this data and eg. VSCode requires it to close
connection to the debugger when the program finishes execution.
This commit changes history handling of the REPL.
There were some situations were history wasn't properly saved and flushed to a
file, making history very spotty. This commit changes it to save every line into
the history file and flush it to disk before being evaluated.
Thanks to this all lines, including "close()" will be stored in the history
file.
If for any reason we're not able to save history file, a single warning will be
printed to the REPL and it will continue to work, even if subsequent tries will
fail to save to disk.
