Fixes #23053.
Two small bugs here:
- the existing condition for printing out the group header was broken.
it worked in the reproducer (in the issue above) without filtering only
by accident, due to setting `self.has_ungrouped = true` once we see the
warmup bench. Knowing that we sort benchmarks to put ungrouped benches
first, there are only two cases: 1) we are starting the first group 2)
we are ending the previous group and starting a new group
- when you passed `--filter` we were applying that filter to the warmup
bench (which is not visible to users), so we suffered from jit bias if
you were filtering (unless your filter was `<warmup>`)
TLDR;
Running
```bash
deno bench main.js --filter="G"
```
```js
// main.js
Deno.bench({
group: "G1",
name: "G1-A",
fn() {},
});
Deno.bench({
group: "G1",
name: "G1-B",
fn() {},
});
```
Before this PR:
```
benchmark time (avg) iter/s (min … max) p75 p99 p995
--------------------------------------------------------------- -----------------------------
G1-A 303.52 ps/iter3,294,726,102.1 (254.2 ps … 7.8 ns) 287.5 ps 391.7 ps 437.5 ps
G1-B 3.8 ns/iter 263,360,635.9 (2.24 ns … 8.36 ns) 3.84 ns 4.73 ns 4.94 ns
summary
G1-A
12.51x faster than G1-B
```
After this PR:
```
benchmark time (avg) iter/s (min … max) p75 p99 p995
--------------------------------------------------------------- -----------------------------
group G1
G1-A 3.85 ns/iter 259,822,096.0 (2.42 ns … 9.03 ns) 3.83 ns 4.62 ns 4.83 ns
G1-B 3.84 ns/iter 260,458,274.5 (3.55 ns … 7.05 ns) 3.83 ns 4.45 ns 4.7 ns
summary
G1-B
1x faster than G1-A
```
Before this PR, we didn't have any integration tests set up for the
`jupyter` subcommand.
This PR adds a basic jupyter client and helpers for writing integration
tests for the jupyter kernel. A lot of the code here is boilerplate,
mainly around the message format for jupyter.
This also adds a few basic integration tests, most notably for
requesting execution of a snippet of code and getting the correct
results.
This patch gets JUnit reporter to output more detailed information for
test steps (subtests).
## Issue with previous implementation
In the previous implementation, the test hierarchy was represented using
several XML tags like the following:
- `<testsuites>` corresponds to the entire test (one execution of `deno
test` has exactly one `<testsuites>` tag)
- `<testsuite>` corresponds to one file, such as `main_test.ts`
- `<testcase>` corresponds to one `Deno.test(...)`
- `<property>` corresponds to one `t.step(...)`
This structure describes the test layers but one problem is that
`<property>` tag is used for any use cases so some tools that can ingest
a JUnit XML file might not be able to interpret `<property>` as
subtests.
## How other tools address it
Some of the testing frameworks in the ecosystem address this issue by
fitting subtests into the `<testcase>` layer. For instance, take a look
at the following Go test file:
```go
package main_test
import "testing"
func TestMain(t *testing.T) {
t.Run("child 1", func(t *testing.T) {
// OK
})
t.Run("child 2", func(t *testing.T) {
// Error
t.Fatal("error")
})
}
```
Running [gotestsum], we can get the output like this:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="3" failures="2" errors="0" time="1.013694">
<testsuite tests="3" failures="2" time="0.510000" name="example/gosumtest" timestamp="2024-03-11T12:26:39+09:00">
<properties>
<property name="go.version" value="go1.22.1 darwin/arm64"></property>
</properties>
<testcase classname="example/gosumtest" name="TestMain/child_2" time="0.000000">
<failure message="Failed" type="">=== RUN TestMain/child_2
 main_test.go:12: error
--- FAIL: TestMain/child_2 (0.00s)
</failure>
</testcase>
<testcase classname="example/gosumtest" name="TestMain" time="0.000000">
<failure message="Failed" type="">=== RUN TestMain
--- FAIL: TestMain (0.00s)
</failure>
</testcase>
<testcase classname="example/gosumtest" name="TestMain/child_1" time="0.000000"></testcase>
</testsuite>
</testsuites>
```
This output shows that nested test cases are squashed into the
`<testcase>` layer by treating them as the same layer as their parent,
`TestMain`. We can still distinguish nested ones by their `name`
attributes that look like `TestMain/<subtest_name>`.
As described in #22795, [vitest] solves the issue in the same way as
[gotestsum].
One downside of this would be that one test failure that happens in a
nested test case will end up being counted multiple times, because not
only the subtest but also its wrapping container(s) are considered to be
failures. In fact, in the [gotestsum] output above, `TestMain/child_2`
failed (which is totally expected) while its parent, `TestMain`, was
also counted as failure. As
https://github.com/denoland/deno/pull/20273#discussion_r1307558757
pointed out, there is a test runner that offers flexibility to prevent
this, but I personally don't think the "duplicate failure count" issue
is a big deal.
## How to fix the issue in this patch
This patch fixes the issue with the same approach as [gotestsum] and
[vitest].
More specifically, nested test cases are put into the `<testcase>` level
and their names are now represented as squashed test names concatenated
by `>` (e.g. `parent 2 > child 1 > grandchild 1`). This change also
allows us to put a detailed error message as `<failure>` tag within the
`<testcase>` tag, which should be handled nicely by third-party tools
supporting JUnit XML.
## Extra fix
Also, file paths embedded into XML outputs are changed from absolute
path to relative path, which is helpful when running the test suites in
several different environments like CI.
Resolves #22795
[gotestsum]: https://github.com/gotestyourself/gotestsum
[vitest]: https://vitest.dev/
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Fixes #22941.
In that case, the only file with coverage was the `test.ts` file. The
coverage reporter filters out test files before compiling its report, so
after filtering we were left with an empty set of files. Later on it's
assumed that there is at least 1 file to be reported on, and we panic.
Instead of panicking, just issue an error after filtering.
1. Stops `deno publish` using some custom include/exclude behaviour from
other sub commands
2. Takes ancestor directories into account when resolving gitignore
3. Backards compatible change that adds ability to unexclude an exclude
by using a negated glob at a more specific level for all sub commands
(see https://github.com/denoland/deno_config/pull/44).
We emitted `import "./` rather than `import "./$NAME"`. This is now
fixed.
Also makes a cosmetic change so that `../` imports are now just imported
as `../`, not `./../`.
Improves #19100
Fixes #20356
Replaces #20428
Changes made in deno_core to support this:
- [x] Errors must be handled in setTimeout callbacks
- [x] Microtask ordering is not-quite-right
- [x] Timer cancellation must be checked right before dispatch
- [x] Timer sanitizer
- [x] Move high-res timer to deno_core
- [x] Timers need opcall tracing
This commit adds "deno add" subcommand that has a basic support for
adding "jsr:" packages to "deno.json" file.
This currently doesn't support "npm:" specifiers and specifying version
constraints.
Some `deno_std` tests were failing to print output that was resolved
after the last test finished. In addition, output printed before tests
began would sometimes appear above the "running X tests ..." line, and
sometimes below it depending on timing.
We now guarantee that all output is flushed before and after tests run,
making the output consistent.
Pre-test and post-test output are captured in `------ pre-test output
------` and `------ post-test output ------` blocks to differentiate
them from the regular output blocks.
Here's an example of a test (that is much noisier than normal, but an
example of what the output will look like):
```
Check ./load_unload.ts
------- pre-test output -------
load
----- output end -----
running 1 test from ./load_unload.ts
test ...
------- output -------
test
----- output end -----
test ... ok ([WILDCARD])
------- post-test output -------
unload
----- output end -----
```
As we add tracing to more types of runtime activity, `--trace-ops` is
less useful of a name. `--trace-leaks` better reflects that this feature
traces both ops and timers, and will eventually trace resource opening
as well.
This keeps `--trace-ops` as an alias for `--trace-leaks`, but prints a
warning to the console suggesting migration to `--trace-leaks`.
One test continues to use `--trace-ops` to test the deprecation warning.
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
- Removes the origin call, since all origins are the same for an isolate
(ie: the main module)
- Collects the `TestDescription`s and sends them all at the same time
inside of an Arc, allowing us to (later on) re-use these instead of
cloning.
Needs a follow-up pass to remove all the cloning, but that's a thread
that is pretty long to pull
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Supply chain security for JSR.
```
$ deno publish --provenance
Successfully published @divy/test_provenance@0.0.3
Provenance transparency log available at https://search.sigstore.dev/?logIndex=73657418
```
0. Package has been published.
1. Fetches the version manifest and verifies it's matching with uploaded
files and exports.
2. Builds the attestation SLSA payload using Github actions env.
3. Creates an ephemeral key pair for signing the github token
(aud=sigstore) and DSSE pre authentication tag.
4. Requests a X.509 signing certificate from Fulcio using the challenge
and ephemeral public key PEM.
5. Prepares a DSSE envelop for Rekor to witness. Posts an intoto entry
to Rekor and gets back the transparency log index.
6. Builds the provenance bundle and posts it to JSR.
Gets us closer to solving #20707.
Rewrites the `TestEventSender`:
- Allow for explicit creation of multiple streams. This will allow for
one-std{out,err}-per-worker
- All test events are received along with a worker ID, allowing for
eventual, proper parallel threading of test events.
In theory this should open up proper interleaving of test output,
however that is left for a future PR.
I had some plans for a better performing synchronization primitive, but
the inter-thread communication is tricky. This does, however, speed up
the processing of large numbers of tests 15-25% (possibly even more on
100,000+).
Before
```
ok | 1000 passed | 0 failed (32ms)
ok | 10000 passed | 0 failed (276ms)
```
After
```
ok | 1000 passed | 0 failed (25ms)
ok | 10000 passed | 0 failed (230ms)
```
<!--
Before submitting a PR, please read
https://docs.deno.com/runtime/manual/references/contributing
1. Give the PR a descriptive title.
Examples of good title:
- fix(std/http): Fix race condition in server
- docs(console): Update docstrings
- feat(doc): Handle nested reexports
Examples of bad title:
- fix #7123
- update docs
- fix bugs
2. Ensure there is a related issue and it is referenced in the PR text.
3. Ensure there are tests that cover the changes.
4. Ensure `cargo test` passes.
5. Ensure `./tools/format.js` passes without changing files.
6. Ensure `./tools/lint.js` passes.
7. Open as a draft PR if your work is still in progress. The CI won't
run
all steps, but you can add '[ci]' to a commit message to force it to.
8. If you would like to run the benchmarks on the CI, add the 'ci-bench'
label.
-->
This PR enhances the `deno publish` command to infer dependencies from
`package.json` if present.
1. Renames zap/fast-check to instead be a `no-slow-types` lint rule.
1. This lint rule is automatically run when doing `deno lint` for
packages (deno.json files with a name, version, and exports field)
1. This lint rules still occurs on publish. It can be skipped by running
with `--no-slow-types`
The format of the sanitizers will change a little bit:
- If multiple async ops leak and traces are on, we repeat the async op
header once per stack trace.
- All leaks are aggregated under a "Leaks detected:" banner as the new
timers are eventually going to be added, and these are neither ops nor
resources.
- `1 async op` is now `An async op`
- If ops and resources leak, we show both (rather than op leaks masking
resources)
Follow-on to https://github.com/denoland/deno/pull/22226
Splitting the sleep and interval ops allows us to detect an interval
timer. We also remove the use of the `op_async_void_deferred` call.
A future PR will be able to split the op sanitizer messages for timers
and intervals.
This changes the lockfile to not store JSR specifiers in the "remote"
section. Instead a single JSR integrity is stored per package in the
lockfile, which is a hash of the version's `x.x.x_meta.json` file, which
contains hashes for every file in the package. The hashes in this file
are then compared against when loading.
Additionally, when using `{ "vendor": true }` in a deno.json, the files
can be modified without causing lockfile errors—the checksum is only
checked when copying into the vendor folder and not afterwards
(eventually we should add this behaviour for non-jsr specifiers as
well). As part of this change, the `vendor` folder creation is not
always automatic in the LSP and running an explicit cache command is
necessary. The code required to track checksums in the LSP would have
been too complex for this PR, so that all goes through deno_graph now.
The vendoring is still automatic when running from the CLI.
This introduces the `denort` binary - a slim version of deno without
tooling. The binary is used as the default for `deno compile`.
Improves `deno compile` final size by ~2.5x (141 MB -> 61 MB) on Linux
x86_64.
Removes the `FileFetcher`'s internal cache because I don't believe it's
necessary (we already cache this kind of stuff in places like deno_graph
or config files in different places). Removing it fixes this bug because
this functionality was already implemented in deno_graph and lowers
memory usage of the CLI a little bit.
This moves the op sanitizer descriptions into Rust code and prepares for
eventual op import from `ext:core/ops`. We cannot import these ops from
`ext:core/ops` as the testing infrastructure ops are not always present.
Changes:
- Op descriptions live in `cli` code and are currently accessible via an
op for the older sanitizer code
- `phf` dep moved to workspace root so we can use it here
- `ops.op_XXX` changed to to `op_XXX` to prepare for op imports later
on.
Step 1 of the Rustification of sanitizers, which unblocks the faster
timers.
This replaces the resource sanitizer with a Rust one, using the new APIs
in deno_core.
This commit adds support for [TC39 Decorator
Proposal](https://github.com/tc39/proposal-decorators).
These decorators are only available in transpiled sources - ie.
non-JavaScript files (because of lack of support in V8).
This entails that "experimental TypeScript decorators" are not available
by default
and require to be configured, with a configuration like this:
```
{
"compilerOptions": {
"experimentalDecorators": true
}
}
```
Closes https://github.com/denoland/deno/issues/19160
---------
Signed-off-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Co-authored-by: crowlkats <crowlkats@toaxl.com>
Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
This change sets the removal version for the `deno bundle` sub-command
for Deno v2. The warnings appear when `deno bundle` is run and in the
`--help` menu.
This initially uses the new diagnostic printer in `deno lint`,
`deno doc` and `deno publish`. In the limit we should also update
`deno check` to use this printer.
This commit removes the requirement for `--unstable` flag in `deno
jupyter` subcommand. The process will no longer exit if this flag is not
provided, however the subcommand itself is still considered unstable
and might change in the future.
Required for https://github.com/denoland/deno/pull/21452
This commit removes conditional type-checking of unstable APIs.
Before this commit `deno check` (or any other type-checking command and
the LSP) would error out if there was an unstable API in the code, but not
`--unstable` flag provided.
This situation hinders DX and makes it harder to configure Deno. Failing
during runtime unless `--unstable` flag is provided is enough in this case.
We were calling `expand_glob` on our excludes, which is very expensive
and unnecessary because we can pattern match while traversing instead.
1. Doesn't expand "exclude" globs. Instead pattern matches while walking
the directory.
2. Splits up the "include" into base paths and applicable file patterns.
This causes less pattern matching to occur because we're only pattern
matching on patterns that might match and not ones in completely
unrelated directories.
Main change is that:
- "hyper" has been renamed to "hyper_v014" to signal that it's legacy
- "hyper1" has been renamed to "hyper" and should be the default
