This changes the lockfile to not store JSR specifiers in the "remote"
section. Instead a single JSR integrity is stored per package in the
lockfile, which is a hash of the version's `x.x.x_meta.json` file, which
contains hashes for every file in the package. The hashes in this file
are then compared against when loading.
Additionally, when using `{ "vendor": true }` in a deno.json, the files
can be modified without causing lockfile errors—the checksum is only
checked when copying into the vendor folder and not afterwards
(eventually we should add this behaviour for non-jsr specifiers as
well). As part of this change, the `vendor` folder creation is not
always automatic in the LSP and running an explicit cache command is
necessary. The code required to track checksums in the LSP would have
been too complex for this PR, so that all goes through deno_graph now.
The vendoring is still automatic when running from the CLI.
We were calling `expand_glob` on our excludes, which is very expensive
and unnecessary because we can pattern match while traversing instead.
1. Doesn't expand "exclude" globs. Instead pattern matches while walking
the directory.
2. Splits up the "include" into base paths and applicable file patterns.
This causes less pattern matching to occur because we're only pattern
matching on patterns that might match and not ones in completely
unrelated directories.
Adds an `--unstable-sloppy-imports` flag which supports the
following for `file:` specifiers:
* Allows writing `./mod` in a specifier to do extension probing.
- ex. `import { Example } from "./example"` instead of `import { Example
} from "./example.ts"`
* Allows writing `./routes` to do directory extension probing for files
like `./routes/index.ts`
* Allows writing `./mod.js` for *mod.ts* files.
This functionality is **NOT RECOMMENDED** for general use with Deno:
1. It's not as optimal for perf:
https://marvinh.dev/blog/speeding-up-javascript-ecosystem-part-2/
1. It makes tooling in the ecosystem more complex in order to have to
understand this.
1. The "Deno way" is to be explicit about what you're doing. It's better
in the long run.
1. It doesn't work if published to the Deno registry because doing stuff
like extension probing with remote specifiers would be incredibly slow.
This is instead only recommended to help with migrating existing
projects to Deno. For example, it's very useful for getting CJS projects
written with import/export declaration working in Deno without modifying
module specifiers and for supporting TS ESM projects written with
`./mod.js` specifiers.
This feature will output warnings to guide the user towards correcting
their specifiers. Additionally, quick fixes are provided in the LSP to
update these specifiers:
This PR causes Deno to include more files in the graph based on how a
template literal looks that's provided to a dynamic import:
```ts
const file = await import(`./dir/${expr}`);
```
In this case, it will search the `dir` directory and descendant
directories for any .js/jsx/etc modules and include them in the graph.
To opt out of this behaviour, move the template literal to a separate
line:
```ts
const specifier = `./dir/${expr}`
const file = await import(specifier);
```
This commit adds unstable workspace support. This is extremely
bare-bones and
minimal first-pass at this.
With this change `deno.json` supports specifying `workspaces` key, that
accepts a list of subdirectories. Each workspace can have its own import
map. It's required to specify a `"name"` and `"version"` properties in the
configuration file for the workspace:
```jsonc
// deno.json
{
"workspaces": [
"a",
"b"
},
"imports": {
"express": "npm:express@5"
}
}
```
``` jsonc
// a/deno.json
{
"name": "a",
"version": "1.0.2",
"imports": {
"kleur": "npm:kleur"
}
}
```
```jsonc
// b/deno.json
{
"name": "b",
"version": "0.51.0",
"imports": {
"chalk": "npm:chalk"
}
}
```
`--unstable-workspaces` flag is required to use this feature:
```
$ deno run --unstable-workspaces mod.ts
```
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
This commit adds `--unstable-hmr` flag, that enabled Hot Module Replacement.
This flag works like `--watch` and accepts the same arguments. If
HMR is not possible the process will be restarted instead.
Currently HMR is only supported in `deno run` subcommand.
Upon HMR a `CustomEvent("hmr")` will be dispatched that contains
information which file was changed in its `details` property.
---------
Co-authored-by: Valentin Anger <syrupthinker@gryphno.de>
Co-authored-by: David Sherret <dsherret@gmail.com>
As title. This will help use the two independently from the other, which
will help in an upcoming deno doc PR where I need to parse the source
files with scope analysis.
This PR adds a new unstable "bring your own node_modules" (BYONM)
functionality currently behind a `--unstable-byonm` flag (`"unstable":
["byonm"]` in a deno.json).
This enables users to run a separate install command (ex. `npm install`,
`pnpm install`) then run `deno run main.ts` and Deno will respect the
layout of the node_modules directory as setup by the separate install
command. It also works with npm/yarn/pnpm workspaces.
For this PR, the behaviour is opted into by specifying
`--unstable-byonm`/`"unstable": ["byonm"]`, but in the future we may
make this the default behaviour as outlined in
https://github.com/denoland/deno/issues/18967#issuecomment-1761248941
This is an extremely rough initial implementation. Errors are
terrible in this and the LSP requires frequent restarts. Improvements
will be done in follow up PRs.
This commit introduces "WatcherCommunicator" struct that
is used facilitate bi-directional communication between CLI
file watcher and the watched function.
Prerequisite for https://github.com/denoland/deno/pull/20876
This makes `CliNpmResolver` a trait. The terminology used is:
- **managed** - Deno manages the node_modules folder and does an
auto-install (ex. `ManagedCliNpmResolver`)
- **byonm** - "Bring your own node_modules" (ex. `ByonmCliNpmResolver`,
which is in this PR, but unimplemented at the moment)
Part of #18967
Adds an experimental unstable built-in package manager to Deno, but it is
currently not usable because the registry infrastructure hasn't been
setup and it points to a non-existent url by default. The default
registry url can be configured via the `DENO_REGISTRY_URL` environment
variable.
TaskQueue is being removed from `deno_core` and replaced with an unsync
version in deno_unsyc.
https://github.com/denoland/deno_core/pull/193
This is a change in preparation for that. The remaining
`deno_core::TaskQueue` usage in this repo should be replaced with
`deno_core::unsync::TaskQueue` once upgraded.
Rather than disallowing `ext:` resolution, clear the module map after
initializing extensions so extension modules are anonymized. This
operation is explicitly called in `deno_runtime`. Re-inject `node:`
specifiers into the module map after doing this.
Fixes #17717.
When someone explicitly opts into using the node_modules dir via
`--node-modules-dir` or setting `"nodeModulesDir": true` in the
deno.json file, we should eagerly ensure a top level package.json
install is done on startup. This was initially always done when we added
package.json support and a package.json was auto-discovered, but scaled
it back to be lazily done when a bare specifier matched an entry in the
package.json because of how disruptive it was for people using Deno
scripts in Node projects. That said, it does not make sense for someone
to opt-into having deno control and use their node_modules directory and
not want a package.json install to occur. If such a rare scenario
exists, the `DENO_NO_PACKAGE_JSON=1` environment variable can be set.
Ideally, we would only ever use a node_modules directory with this
explicit opt-in so everything is very clear, but we still have this
automatic scenario when there's a package.json in order to make more
node projects work out of the box.
This is the initial support for npm and node specifiers in `deno
compile`. The npm packages are included in the binary and read from it via
a virtual file system. This also supports the `--node-modules-dir` flag,
dependencies specified in a package.json, and npm binary commands (ex.
`deno compile --unstable npm:cowsay`)
Closes #16632
This is just a straight refactor and I didn't do any cleanup in
ext/node. After this PR we can start to clean it up and make things
private that don't need to be public anymore.
1. Breaks up functionality within `ProcState` into several other structs
to break out the responsibilities (`ProcState` is only a data struct
now).
2. Moves towards being able to inject dependencies more easily and have
functionality only require what it needs.
3. Exposes `Arc<T>` around the "service structs" instead of it being
embedded within them. The idea behind embedding them was to reduce the
verbosity of needing to pass around `Arc<...>`, but I don't think it was
exactly working and as we move more of these structs to be more
injectable I don't think the extra verbosity will be a big deal.
This gets SQLite off the flamegraph and reduces initialization time by
somewhere between 0.2ms and 0.5ms. In addition, I took the opportunity
to move all the cache management code to a single place and reduce
duplication. While the PR has a net gain of lines, much of that is just
being a bit more deliberate with how we're recovering from errors.
The existing caches had various policies for dealing with cache
corruption, so I've unified them and tried to isolate the decisions we
make for recovery in a single place (see `open_connection` in
`CacheDB`). The policy I chose was:
1. Retry twice to open on-disk caches
2. If that fails, try to delete the file and recreate it on-disk
3. If we fail to delete the file or re-create a new cache, use a
fallback strategy that can be chosen per-cache: InMemory (temporary
cache for the process run), BlackHole (ignore writes, return empty
reads), or Error (fail on every operation).
The caches all use the same general code now, and share the cache
failure recovery policy.
In addition, it cleans up a TODO in the `NodeAnalysisCache`.
This change will enable dynamic imports and web workers to use modules
not reachable from the main module, by passing a list of extra side
module roots as options to `deno compile`.
This can be done by specifying "--include" flag that accepts a file path or a
URL. This flag can be specified multiple times, to include several modules.
The modules specified with "--include" flag, will be added to the produced
"eszip".
This has been bothering me for a while and it became more painful while
working on #18136 because injecting the shared progress bar became very
verbose. Basically we should move the creation of all these npm structs
up to a higher level.
This is a stepping stone for a future refactor where we can improve how
we create all our structs.
This lazily does an "npm install" when any package name matches what's
found in the package.json or when running a script from package.json
with deno task.
Part of #17916
Closes #17928
This commit enables resolution of "bare specifiers" (eg. "import express
from 'express';") if a "package.json" file is discovered.
It's a step towards being able to run projects authored for Node.js
without any changes.
With this commit we are able to successfully run Vite projects without
any changes to the user code.
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
This changes npm specifiers to be handled by deno_graph and resolved to
an npm package name and version when the specifier is encountered. It
also slightly changes how npm specifier resolution occurs—previously it
would collect all the npm specifiers and resolve them all at once, but
now it resolves them on the fly as they are encountered in the module
graph.
https://github.com/denoland/deno_graph/pull/232
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commits adds auto-discovery of "package.json" file when running
"deno run" and "deno task" subcommands. In case of "deno run" the
"package.json" is being looked up starting from the directory of the
script that is being run, stopping early if "deno.json(c)" file is found
(ie. FS tree won't be traversed "up" from "deno.json").
When "package.json" is discovered the "--node-modules-dir" flag is
implied, leading to creation of local "node_modules/" directory - we
did that, because most tools relying on "package.json" will expect
"node_modules/" directory to be present (eg. Vite). Additionally
"dependencies" and "devDependencies" specified in the "package.json"
are downloaded on startup.
This is a stepping stone to supporting bare specifier imports, but
the actual integration will be done in a follow up commit.
---------
Co-authored-by: David Sherret <dsherret@gmail.com>
This commit changes "ProcState" to store "file_fetcher" field in an "Arc",
allowing it to be preserved between restarts and thus keeping the state
alive between the restarts. File watchers for "deno test" and "deno bench"
now reset "ProcState" between restarts.
Turns out we were cloning permissions which after prompting were discarded,
so the state of permissions was never preserved. To handle that we need to store
all permissions behind "Arc<Mutex<>>" (because there are situations where we
need to send them to other thread).
Testing and benching code still uses "Permissions" in most places - it's undesirable
to share the same permission set between various test/bench files - otherwise
granting or revoking permissions in one file would influence behavior of other test
files.
This commit removes "compat" mode. We shipped support for "npm:" specifier
support in v1.25 and that is preferred way to interact with Node code that we
will iterate and improve upon.
This commit adds CJS/ESM interoperability when running in --compat mode.
Before executing files, they are analyzed and all CommonJS modules are
transformed on the fly to a ES modules. This is done by utilizing analyze_cjs()
functionality from deno_ast. After discovering exports and reexports, an ES
module is rendered and saved in memory for later use.
There's a caveat that all files ending with ".js" extension are considered as
CommonJS modules (unless there's a related "package.json" with "type": "module").
