foster/denoland-deno
1
0
Fork 0
mirror of https://github.com/denoland/deno.git synced 2025-01-11 08:33:43 -05:00
Code Issues Packages 1 Wiki Activity

fix(cli): output more detailed information for steps when using JUnit reporter (#22797)

Browse source 
This patch gets JUnit reporter to output more detailed information for
test steps (subtests).

## Issue with previous implementation

In the previous implementation, the test hierarchy was represented using
several XML tags like the following:

- `<testsuites>` corresponds to the entire test (one execution of `deno
test` has exactly one `<testsuites>` tag)
- `<testsuite>` corresponds to one file, such as `main_test.ts`
- `<testcase>` corresponds to one `Deno.test(...)`
- `<property>` corresponds to one `t.step(...)`

This structure describes the test layers but one problem is that
`<property>` tag is used for any use cases so some tools that can ingest
a JUnit XML file might not be able to interpret `<property>` as
subtests.

## How other tools address it

Some of the testing frameworks in the ecosystem address this issue by
fitting subtests into the `<testcase>` layer. For instance, take a look
at the following Go test file:

```go
package main_test

import "testing"

func TestMain(t *testing.T) {
  t.Run("child 1", func(t *testing.T) {
    // OK
  })

  t.Run("child 2", func(t *testing.T) {
    // Error
    t.Fatal("error")
  })
}
```

Running [gotestsum], we can get the output like this:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="3" failures="2" errors="0" time="1.013694">
	<testsuite tests="3" failures="2" time="0.510000" name="example/gosumtest" timestamp="2024-03-11T12:26:39+09:00">
		<properties>
			<property name="go.version" value="go1.22.1 darwin/arm64"></property>
		</properties>
		<testcase classname="example/gosumtest" name="TestMain/child_2" time="0.000000">
			<failure message="Failed" type="">=== RUN   TestMain/child_2&#xA;    main_test.go:12: error&#xA;--- FAIL: TestMain/child_2 (0.00s)&#xA;</failure>
		</testcase>
		<testcase classname="example/gosumtest" name="TestMain" time="0.000000">
			<failure message="Failed" type="">=== RUN   TestMain&#xA;--- FAIL: TestMain (0.00s)&#xA;</failure>
		</testcase>
		<testcase classname="example/gosumtest" name="TestMain/child_1" time="0.000000"></testcase>
	</testsuite>
</testsuites>
``` 

This output shows that nested test cases are squashed into the
`<testcase>` layer by treating them as the same layer as their parent,
`TestMain`. We can still distinguish nested ones by their `name`
attributes that look like `TestMain/<subtest_name>`.

As described in #22795, [vitest] solves the issue in the same way as
[gotestsum].

One downside of this would be that one test failure that happens in a
nested test case will end up being counted multiple times, because not
only the subtest but also its wrapping container(s) are considered to be
failures. In fact, in the [gotestsum] output above, `TestMain/child_2`
failed (which is totally expected) while its parent, `TestMain`, was
also counted as failure. As
https://github.com/denoland/deno/pull/20273#discussion_r1307558757
pointed out, there is a test runner that offers flexibility to prevent
this, but I personally don't think the "duplicate failure count" issue
is a big deal.

## How to fix the issue in this patch

This patch fixes the issue with the same approach as [gotestsum] and
[vitest].
More specifically, nested test cases are put into the `<testcase>` level
and their names are now represented as squashed test names concatenated
by `>` (e.g. `parent 2 > child 1 > grandchild 1`). This change also
allows us to put a detailed error message as `<failure>` tag within the
`<testcase>` tag, which should be handled nicely by third-party tools
supporting JUnit XML.

## Extra fix

Also, file paths embedded into XML outputs are changed from absolute
path to relative path, which is helpful when running the test suites in
several different environments like CI.

Resolves #22795

[gotestsum]: https://github.com/gotestyourself/gotestsum
[vitest]: https://vitest.dev/

---------

Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit is contained in:
Yusuke Tanaka 2024-03-26 00:08:46 +09:00 committed by GitHub
parent 5c1fa0cf9c
commit 64e8c36805
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
13 changed files with 599 additions and 122 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

11
cli/tools/jupyter/mod.rs
View file

@ -11,11 +11,13 @@ use crate::tools::test::TestEventWorkerSender;
use crate::util::logger;
use crate::CliFactory;
use deno_core::anyhow::Context;
use deno_core::error::generic_error;
use deno_core::error::AnyError;
use deno_core::located_script_name;
use deno_core::resolve_url_or_path;
use deno_core::serde::Deserialize;
use deno_core::serde_json;
use deno_core::url::Url;
use deno_runtime::deno_io::Stdio;
use deno_runtime::deno_io::StdioPipe;
use deno_runtime::permissions::Permissions;
@ -129,9 +131,16 @@ pub async fn kernel(
Ok(())
}
}
let cwd_url =
Url::from_directory_path(cli_options.initial_cwd()).map_err(|_| {
generic_error(format!(
"Unable to construct URL from the path of cwd: {}",
cli_options.initial_cwd().to_string_lossy(),
))
})?;
repl_session.set_test_reporter_factory(Box::new(move || {
Box::new(
PrettyTestReporter::new(false, true, false, true)
PrettyTestReporter::new(false, true, false, true, cwd_url.clone())
.with_writer(Box::new(TestWriter(stdio_tx.clone()))),
)
}));

19
cli/tools/repl/session.rs
View file

@ -30,6 +30,7 @@ use deno_ast::ParsedSource;
use deno_ast::SourcePos;
use deno_ast::SourceRangedForSpanned;
use deno_ast::SourceTextInfo;
use deno_core::error::generic_error;
use deno_core::error::AnyError;
use deno_core::futures::channel::mpsc::UnboundedReceiver;
use deno_core::futures::FutureExt;
@ -37,6 +38,7 @@ use deno_core::futures::StreamExt;
use deno_core::serde_json;
use deno_core::serde_json::Value;
use deno_core::unsync::spawn;
use deno_core::url::Url;
use deno_core::LocalInspectorSession;
use deno_core::PollEventLoopOptions;
use deno_graph::source::ResolutionMode;
@ -243,6 +245,13 @@ impl ReplSession {
deno_core::resolve_path("./$deno$repl.ts", cli_options.initial_cwd())
.unwrap();
let cwd_url =
Url::from_directory_path(cli_options.initial_cwd()).map_err(|_| {
generic_error(format!(
"Unable to construct URL from the path of cwd: {}",
cli_options.initial_cwd().to_string_lossy(),
))
})?;
let ts_config_for_emit = cli_options
.resolve_ts_config_for_emit(deno_config::TsConfigType::Emit)?;
let emit_options =
@ -257,8 +266,14 @@ impl ReplSession {
language_server,
referrer,
notifications: Arc::new(Mutex::new(notification_rx)),
test_reporter_factory: Box::new(|| {
Box::new(PrettyTestReporter::new(false, true, false, true))
test_reporter_factory: Box::new(move || {
Box::new(PrettyTestReporter::new(
false,
true,
false,
true,
cwd_url.clone(),
))
}),
main_module,
test_event_sender,

98
cli/tools/test/mod.rs
View file

@ -298,43 +298,77 @@ pub enum TestFailure {
HasSanitizersAndOverlaps(IndexSet<String>), // Long names of overlapped tests
}
impl ToString for TestFailure {
fn to_string(&self) -> String {
impl std::fmt::Display for TestFailure {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
TestFailure::JsError(js_error) => format_test_error(js_error),
TestFailure::FailedSteps(1) => "1 test step failed.".to_string(),
TestFailure::FailedSteps(n) => format!("{} test steps failed.", n),
TestFailure::IncompleteSteps => "Completed while steps were still running. Ensure all steps are awaited with `await t.step(...)`.".to_string(),
TestFailure::Incomplete => "Didn't complete before parent. Await step with `await t.step(...)`.".to_string(),
TestFailure::JsError(js_error) => {
write!(f, "{}", format_test_error(js_error))
}
TestFailure::FailedSteps(1) => write!(f, "1 test step failed."),
TestFailure::FailedSteps(n) => write!(f, "{n} test steps failed."),
TestFailure::IncompleteSteps => {
write!(f, "Completed while steps were still running. Ensure all steps are awaited with `await t.step(...)`.")
}
TestFailure::Incomplete => {
write!(
f,
"Didn't complete before parent. Await step with `await t.step(...)`."
)
}
TestFailure::Leaked(details, trailer_notes) => {
let mut string = "Leaks detected:".to_string();
write!(f, "Leaks detected:")?;
for detail in details {
string.push_str(&format!("\n - {detail}"));
write!(f, "\n - {}", detail)?;
}
for trailer in trailer_notes {
string.push_str(&format!("\n{trailer}"));
write!(f, "\n{}", trailer)?;
}
string
Ok(())
}
TestFailure::OverlapsWithSanitizers(long_names) => {
let mut string = "Started test step while another test step with sanitizers was running:".to_string();
write!(f, "Started test step while another test step with sanitizers was running:")?;
for long_name in long_names {
string.push_str(&format!("\n * {}", long_name));
write!(f, "\n * {}", long_name)?;
}
string
Ok(())
}
TestFailure::HasSanitizersAndOverlaps(long_names) => {
let mut string = "Started test step with sanitizers while another test step was running:".to_string();
write!(f, "Started test step with sanitizers while another test step was running:")?;
for long_name in long_names {
string.push_str(&format!("\n * {}", long_name));
write!(f, "\n * {}", long_name)?;
}
string
Ok(())
}
}
}
}
impl TestFailure {
pub fn overview(&self) -> String {
match self {
TestFailure::JsError(js_error) => js_error.exception_message.clone(),
TestFailure::FailedSteps(1) => "1 test step failed".to_string(),
TestFailure::FailedSteps(n) => format!("{n} test steps failed"),
TestFailure::IncompleteSteps => {
"Completed while steps were still running".to_string()
}
TestFailure::Incomplete => "Didn't complete before parent".to_string(),
TestFailure::Leaked(_, _) => "Leaks detected".to_string(),
TestFailure::OverlapsWithSanitizers(_) => {
"Started test step while another test step with sanitizers was running"
.to_string()
}
TestFailure::HasSanitizersAndOverlaps(_) => {
"Started test step with sanitizers while another test step was running"
.to_string()
}
}
}
pub fn detail(&self) -> String {
self.to_string()
}
fn format_label(&self) -> String {
match self {
TestFailure::Incomplete => colors::gray("INCOMPLETE").to_string(),
@ -465,6 +499,7 @@ pub struct TestSummary {
#[derive(Debug, Clone)]
struct TestSpecifiersOptions {
cwd: Url,
concurrent_jobs: NonZeroUsize,
fail_fast: Option<NonZeroUsize>,
log_level: Option<log::Level>,
@ -506,23 +541,30 @@ impl TestSummary {
fn get_test_reporter(options: &TestSpecifiersOptions) -> Box<dyn TestReporter> {
let parallel = options.concurrent_jobs.get() > 1;
let reporter: Box<dyn TestReporter> = match &options.reporter {
TestReporterConfig::Dot => Box::new(DotTestReporter::new()),
TestReporterConfig::Dot => {
Box::new(DotTestReporter::new(options.cwd.clone()))
}
TestReporterConfig::Pretty => Box::new(PrettyTestReporter::new(
parallel,
options.log_level != Some(Level::Error),
options.filter,
false,
options.cwd.clone(),
)),
TestReporterConfig::Junit => {
Box::new(JunitTestReporter::new("-".to_string()))
Box::new(JunitTestReporter::new(options.cwd.clone(), "-".to_string()))
}
TestReporterConfig::Tap => Box::new(TapTestReporter::new(
options.cwd.clone(),
options.concurrent_jobs > NonZeroUsize::new(1).unwrap(),
)),
};
if let Some(junit_path) = &options.junit_path {
let junit = Box::new(JunitTestReporter::new(junit_path.to_string()));
let junit = Box::new(JunitTestReporter::new(
options.cwd.clone(),
junit_path.to_string(),
));
return Box::new(CompoundTestReporter::new(vec![reporter, junit]));
}
@ -1641,6 +1683,14 @@ pub async fn run_tests(
})
.collect(),
TestSpecifiersOptions {
cwd: Url::from_directory_path(cli_options.initial_cwd()).map_err(
|_| {
generic_error(format!(
"Unable to construct URL from the path of cwd: {}",
cli_options.initial_cwd().to_string_lossy(),
))
},
)?,
concurrent_jobs: test_options.concurrent_jobs,
fail_fast: test_options.fail_fast,
log_level,
@ -1780,6 +1830,14 @@ pub async fn run_tests_with_watch(
})
.collect(),
TestSpecifiersOptions {
cwd: Url::from_directory_path(cli_options.initial_cwd()).map_err(
|_| {
generic_error(format!(
"Unable to construct URL from the path of cwd: {}",
cli_options.initial_cwd().to_string_lossy(),
))
},
)?,
concurrent_jobs: test_options.concurrent_jobs,
fail_fast: test_options.fail_fast,
log_level,

9
cli/tools/test/reporters/common.rs
View file

@ -136,13 +136,8 @@ pub(super) fn report_summary(
if !failure.hide_in_summary() {
let failure_title = format_test_for_summary(cwd, description);
writeln!(writer, "{}", &failure_title).unwrap();
writeln!(
writer,
"{}: {}",
colors::red_bold("error"),
failure.to_string()
)
.unwrap();
writeln!(writer, "{}: {}", colors::red_bold("error"), failure)
.unwrap();
writeln!(writer).unwrap();
failure_titles.push(failure_title);
}

4
cli/tools/test/reporters/dot.rs
View file

@ -12,7 +12,7 @@ pub struct DotTestReporter {
}
impl DotTestReporter {
pub fn new() -> DotTestReporter {
pub fn new(cwd: Url) -> DotTestReporter {
let console_width = if let Some(size) = crate::util::console::console_size()
{
size.cols as usize
@ -23,7 +23,7 @@ impl DotTestReporter {
DotTestReporter {
n: 0,
width: console_width,
cwd: Url::from_directory_path(std::env::current_dir().unwrap()).unwrap(),
cwd,
summary: TestSummary::new(),
}
}

353
cli/tools/test/reporters/junit.rs
View file

@ -1,20 +1,29 @@
// Copyright 2018-2024 the Deno authors. All rights reserved. MIT license.
use std::collections::VecDeque;
use std::path::PathBuf;
use super::fmt::to_relative_path_or_remote_url;
use super::*;
pub struct JunitTestReporter {
path: String,
cwd: Url,
output_path: String,
// Stores TestCases (i.e. Tests) by the Test ID
cases: IndexMap<usize, quick_junit::TestCase>,
// Stores nodes representing test cases in such a way that can be traversed
// from child to parent to build the full test name that reflects the test
// hierarchy.
test_name_tree: TestNameTree,
}
impl JunitTestReporter {
pub fn new(path: String) -> Self {
pub fn new(cwd: Url, output_path: String) -> Self {
Self {
path,
cwd,
output_path,
cases: IndexMap::new(),
test_name_tree: TestNameTree::new(),
}
}
@ -24,9 +33,9 @@ impl JunitTestReporter {
TestResult::Ignored => quick_junit::TestCaseStatus::skipped(),
TestResult::Failed(failure) => quick_junit::TestCaseStatus::NonSuccess {
kind: quick_junit::NonSuccessKind::Failure,
message: Some(failure.to_string()),
message: Some(failure.overview()),
ty: None,
description: None,
description: Some(failure.detail()),
reruns: vec![],
},
TestResult::Cancelled => quick_junit::TestCaseStatus::NonSuccess {
@ -38,6 +47,24 @@ impl JunitTestReporter {
},
}
}
fn convert_step_status(
status: &TestStepResult,
) -> quick_junit::TestCaseStatus {
match status {
TestStepResult::Ok => quick_junit::TestCaseStatus::success(),
TestStepResult::Ignored => quick_junit::TestCaseStatus::skipped(),
TestStepResult::Failed(failure) => {
quick_junit::TestCaseStatus::NonSuccess {
kind: quick_junit::NonSuccessKind::Failure,
message: Some(failure.overview()),
ty: None,
description: Some(failure.detail()),
reruns: vec![],
}
}
}
}
}
impl TestReporter for JunitTestReporter {
@ -46,11 +73,10 @@ impl TestReporter for JunitTestReporter {
description.name.clone(),
quick_junit::TestCaseStatus::skipped(),
);
let file_name = description.location.file_name.clone();
let file_name = file_name.strip_prefix("file://").unwrap_or(&file_name);
case
.extra
.insert(String::from("filename"), String::from(file_name));
case.classname = Some(to_relative_path_or_remote_url(
&self.cwd,
&description.location.file_name,
));
case.extra.insert(
String::from("line"),
description.location.line_number.to_string(),
@ -60,6 +86,8 @@ impl TestReporter for JunitTestReporter {
description.location.column_number.to_string(),
);
self.cases.insert(description.id, case);
self.test_name_tree.add_node(description.clone().into());
}
fn report_plan(&mut self, _plan: &TestPlan) {}
@ -89,7 +117,29 @@ impl TestReporter for JunitTestReporter {
fn report_uncaught_error(&mut self, _origin: &str, _error: Box<JsError>) {}
fn report_step_register(&mut self, _description: &TestStepDescription) {}
fn report_step_register(&mut self, description: &TestStepDescription) {
self.test_name_tree.add_node(description.clone().into());
let test_case_name =
self.test_name_tree.construct_full_test_name(description.id);
let mut case = quick_junit::TestCase::new(
test_case_name,
quick_junit::TestCaseStatus::skipped(),
);
case.classname = Some(to_relative_path_or_remote_url(
&self.cwd,
&description.location.file_name,
));
case.extra.insert(
String::from("line"),
description.location.line_number.to_string(),
);
case.extra.insert(
String::from("col"),
description.location.column_number.to_string(),
);
self.cases.insert(description.id, case);
}
fn report_step_wait(&mut self, _description: &TestStepDescription) {}
@ -97,43 +147,13 @@ impl TestReporter for JunitTestReporter {
&mut self,
description: &TestStepDescription,
result: &TestStepResult,
_elapsed: u64,
elapsed: u64,
_tests: &IndexMap<usize, TestDescription>,
test_steps: &IndexMap<usize, TestStepDescription>,
_test_steps: &IndexMap<usize, TestStepDescription>,
) {
let status = match result {
TestStepResult::Ok => "passed",
TestStepResult::Ignored => "skipped",
TestStepResult::Failed(_) => "failure",
};
let root_id: usize;
let mut name = String::new();
{
let mut ancestors = vec![];
let mut current_desc = description;
loop {
if let Some(d) = test_steps.get(&current_desc.parent_id) {
ancestors.push(&d.name);
current_desc = d;
} else {
root_id = current_desc.parent_id;
break;
}
}
ancestors.reverse();
for n in ancestors {
name.push_str(n);
name.push_str(" ... ");
}
name.push_str(&description.name);
}
if let Some(case) = self.cases.get_mut(&root_id) {
case.add_property(quick_junit::Property::new(
format!("step[{}]", status),
name,
));
if let Some(case) = self.cases.get_mut(&description.id) {
case.status = Self::convert_step_status(result);
case.set_time(Duration::from_millis(elapsed));
}
}
@ -167,44 +187,239 @@ impl TestReporter for JunitTestReporter {
&mut self,
elapsed: &Duration,
tests: &IndexMap<usize, TestDescription>,
_test_steps: &IndexMap<usize, TestStepDescription>,
test_steps: &IndexMap<usize, TestStepDescription>,
) -> anyhow::Result<()> {
let mut suites: IndexMap<String, quick_junit::TestSuite> = IndexMap::new();
for (id, case) in &self.cases {
if let Some(test) = tests.get(id) {
suites
.entry(test.location.file_name.clone())
.and_modify(|s| {
s.add_test_case(case.clone());
})
.or_insert_with(|| {
quick_junit::TestSuite::new(test.location.file_name.clone())
.add_test_case(case.clone())
.to_owned()
});
}
let abs_filename = match (tests.get(id), test_steps.get(id)) {
(Some(test), _) => &test.location.file_name,
(_, Some(step)) => &step.location.file_name,
(None, None) => {
unreachable!("Unknown test ID '{id}' provided");
}
};
let filename = to_relative_path_or_remote_url(&self.cwd, abs_filename);
suites
.entry(filename.clone())
.and_modify(|s| {
s.add_test_case(case.clone());
})
.or_insert_with(|| {
let mut suite = quick_junit::TestSuite::new(filename);
suite.add_test_case(case.clone());
suite
});
}
let mut report = quick_junit::Report::new("deno test");
report.set_time(*elapsed).add_test_suites(
suites
.values()
.cloned()
.collect::<Vec<quick_junit::TestSuite>>(),
);
report
.set_time(*elapsed)
.add_test_suites(suites.into_values());
if self.path == "-" {
if self.output_path == "-" {
report
.serialize(std::io::stdout())
.with_context(|| "Failed to write JUnit report to stdout")?;
} else {
let file = crate::util::fs::create_file(&PathBuf::from(&self.path))
.context("Failed to open JUnit report file.")?;
let file =
crate::util::fs::create_file(&PathBuf::from(&self.output_path))
.context("Failed to open JUnit report file.")?;
report.serialize(file).with_context(|| {
format!("Failed to write JUnit report to {}", self.path)
format!("Failed to write JUnit report to {}", self.output_path)
})?;
}
Ok(())
}
}
#[derive(Debug, Default)]
struct TestNameTree(IndexMap<usize, TestNameTreeNode>);
impl TestNameTree {
fn new() -> Self {
// Pre-allocate some space to avoid excessive reallocations.
Self(IndexMap::with_capacity(256))
}
fn add_node(&mut self, node: TestNameTreeNode) {
self.0.insert(node.id, node);
}
/// Constructs the full test name by traversing the tree from the specified
/// node as a child to its parent nodes.
/// If the provided ID is not found in the tree, or the tree is broken (e.g.
/// a child node refers to a parent node that doesn't exist), this method
/// just panics.
fn construct_full_test_name(&self, id: usize) -> String {
let mut current_id = Some(id);
let mut name_pieces = VecDeque::new();
loop {
let Some(id) = current_id else {
break;
};
let Some(node) = self.0.get(&id) else {
// The ID specified as a parent node by the child node should exist in
// the tree, but it doesn't. In this case we give up constructing the
// full test name.
unreachable!("Unregistered test ID '{id}' provided");
};
name_pieces.push_front(node.test_name.as_str());
current_id = node.parent_id;
}
if name_pieces.is_empty() {
unreachable!("Unregistered test ID '{id}' provided");
}
let v: Vec<_> = name_pieces.into();
v.join(" > ")
}
}
#[derive(Debug)]
struct TestNameTreeNode {
id: usize,
parent_id: Option<usize>,
test_name: String,
}
impl From<TestDescription> for TestNameTreeNode {
fn from(description: TestDescription) -> Self {
Self {
id: description.id,
parent_id: None,
test_name: description.name,
}
}
}
impl From<TestStepDescription> for TestNameTreeNode {
fn from(description: TestStepDescription) -> Self {
Self {
id: description.id,
parent_id: Some(description.parent_id),
test_name: description.name,
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn construct_full_test_name_one_node() {
let mut tree = TestNameTree::new();
tree.add_node(TestNameTreeNode {
id: 0,
parent_id: None,
test_name: "root".to_string(),
});
assert_eq!(tree.construct_full_test_name(0), "root".to_string());
}
#[test]
fn construct_full_test_name_two_level_hierarchy() {
let mut tree = TestNameTree::new();
tree.add_node(TestNameTreeNode {
id: 0,
parent_id: None,
test_name: "root".to_string(),
});
tree.add_node(TestNameTreeNode {
id: 1,
parent_id: Some(0),
test_name: "child".to_string(),
});
assert_eq!(tree.construct_full_test_name(0), "root".to_string());
assert_eq!(tree.construct_full_test_name(1), "root > child".to_string());
}
#[test]
fn construct_full_test_name_three_level_hierarchy() {
let mut tree = TestNameTree::new();
tree.add_node(TestNameTreeNode {
id: 0,
parent_id: None,
test_name: "root".to_string(),
});
tree.add_node(TestNameTreeNode {
id: 1,
parent_id: Some(0),
test_name: "child".to_string(),
});
tree.add_node(TestNameTreeNode {
id: 2,
parent_id: Some(1),
test_name: "grandchild".to_string(),
});
assert_eq!(tree.construct_full_test_name(0), "root".to_string());
assert_eq!(tree.construct_full_test_name(1), "root > child".to_string());
assert_eq!(
tree.construct_full_test_name(2),
"root > child > grandchild".to_string()
);
}
#[test]
fn construct_full_test_name_one_root_two_chains() {
// 0
// / \
// 1 2
// / \
// 3 4
let mut tree = TestNameTree::new();
tree.add_node(TestNameTreeNode {
id: 0,
parent_id: None,
test_name: "root".to_string(),
});
tree.add_node(TestNameTreeNode {
id: 1,
parent_id: Some(0),
test_name: "child 1".to_string(),
});
tree.add_node(TestNameTreeNode {
id: 2,
parent_id: Some(0),
test_name: "child 2".to_string(),
});
tree.add_node(TestNameTreeNode {
id: 3,
parent_id: Some(1),
test_name: "grandchild 1".to_string(),
});
tree.add_node(TestNameTreeNode {
id: 4,
parent_id: Some(1),
test_name: "grandchild 2".to_string(),
});
assert_eq!(tree.construct_full_test_name(0), "root".to_string());
assert_eq!(
tree.construct_full_test_name(1),
"root > child 1".to_string(),
);
assert_eq!(
tree.construct_full_test_name(2),
"root > child 2".to_string(),
);
assert_eq!(
tree.construct_full_test_name(3),
"root > child 1 > grandchild 1".to_string(),
);
assert_eq!(
tree.construct_full_test_name(4),
"root > child 1 > grandchild 2".to_string(),
);
}
}

3
cli/tools/test/reporters/pretty.rs
View file

@ -28,6 +28,7 @@ impl PrettyTestReporter {
echo_output: bool,
filter: bool,
repl: bool,
cwd: Url,
) -> PrettyTestReporter {
PrettyTestReporter {
parallel,
@ -37,7 +38,7 @@ impl PrettyTestReporter {
filter,
repl,
scope_test_id: None,
cwd: Url::from_directory_path(std::env::current_dir().unwrap()).unwrap(),
cwd,
did_have_user_output: false,
started_tests: false,
ended_tests: false,

4
cli/tools/test/reporters/tap.rs
View file

@ -23,9 +23,9 @@ pub struct TapTestReporter {
}
impl TapTestReporter {
pub fn new(is_concurrent: bool) -> TapTestReporter {
pub fn new(cwd: Url, is_concurrent: bool) -> TapTestReporter {
TapTestReporter {
cwd: Url::from_directory_path(std::env::current_dir().unwrap()).unwrap(),
cwd,
is_concurrent,
header: false,
planned: 0,

12
tests/integration/test_tests.rs
View file

@ -283,6 +283,18 @@ itest!(junit {
output: "test/pass.junit.out",
});
itest!(junit_nested {
args: "test --reporter junit test/nested_failures.ts",
output: "test/nested_failures.junit.out",
exit_code: 1,
});
itest!(junit_multiple_test_files {
args: "test --reporter junit test/pass.ts test/fail.ts",
output: "test/junit_multiple_test_files.junit.out",
exit_code: 1,
});
#[test]
fn junit_path() {
let context = TestContextBuilder::new().use_temp_cwd().build();

102
tests/testdata/test/junit_multiple_test_files.junit.out vendored Normal file
View file

@ -0,0 +1,102 @@
Check file:///[WILDCARD]/test/pass.ts
Check file:///[WILDCARD]/test/fail.ts
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="deno test" tests="26" failures="10" errors="0" time="[WILDCARD]">
<testsuite name="./test/pass.ts" tests="16" disabled="0" errors="0" failures="0">
<testcase name="test 0" classname="./test/pass.ts" time="[WILDCARD]" line="1" col="6">
</testcase>
<testcase name="test 1" classname="./test/pass.ts" time="[WILDCARD]" line="2" col="6">
</testcase>
<testcase name="test 2" classname="./test/pass.ts" time="[WILDCARD]" line="3" col="6">
</testcase>
<testcase name="test 3" classname="./test/pass.ts" time="[WILDCARD]" line="4" col="6">
</testcase>
<testcase name="test 4" classname="./test/pass.ts" time="[WILDCARD]" line="5" col="6">
</testcase>
<testcase name="test 5" classname="./test/pass.ts" time="[WILDCARD]" line="6" col="6">
</testcase>
<testcase name="test 6" classname="./test/pass.ts" time="[WILDCARD]" line="7" col="6">
</testcase>
<testcase name="test 7" classname="./test/pass.ts" time="[WILDCARD]" line="8" col="6">
</testcase>
<testcase name="test 8" classname="./test/pass.ts" time="[WILDCARD]" line="9" col="6">
</testcase>
<testcase name="test 9" classname="./test/pass.ts" time="[WILDCARD]" line="12" col="6">
</testcase>
<testcase name="test\b" classname="./test/pass.ts" time="[WILDCARD]" line="16" col="6">
</testcase>
<testcase name="test\f" classname="./test/pass.ts" time="[WILDCARD]" line="19" col="6">
</testcase>
<testcase name="test\t" classname="./test/pass.ts" time="[WILDCARD]" line="23" col="6">
</testcase>
<testcase name="test\n" classname="./test/pass.ts" time="[WILDCARD]" line="27" col="6">
</testcase>
<testcase name="test\r" classname="./test/pass.ts" time="[WILDCARD]" line="31" col="6">
</testcase>
<testcase name="test\v" classname="./test/pass.ts" time="[WILDCARD]" line="35" col="6">
</testcase>
</testsuite>
<testsuite name="./test/fail.ts" tests="10" disabled="0" errors="0" failures="10">
<testcase name="test 0" classname="./test/fail.ts" time="[WILDCARD]" line="1" col="6">
<failure message="Uncaught Error">Error
throw new Error();
^
at file:///[WILDCARD]/test/fail.ts:2:9</failure>
</testcase>
<testcase name="test 1" classname="./test/fail.ts" time="[WILDCARD]" line="4" col="6">
<failure message="Uncaught Error">Error
throw new Error();
^
at file:///[WILDCARD]/test/fail.ts:5:9</failure>
</testcase>
<testcase name="test 2" classname="./test/fail.ts" time="[WILDCARD]" line="7" col="6">
<failure message="Uncaught Error">Error
throw new Error();
^
at file:///[WILDCARD]/test/fail.ts:8:9</failure>
</testcase>
<testcase name="test 3" classname="./test/fail.ts" time="[WILDCARD]" line="10" col="6">
<failure message="Uncaught Error">Error
throw new Error();
^
at file:///[WILDCARD]/test/fail.ts:11:9</failure>
</testcase>
<testcase name="test 4" classname="./test/fail.ts" time="[WILDCARD]" line="13" col="6">
<failure message="Uncaught Error">Error
throw new Error();
^
at file:///[WILDCARD]/test/fail.ts:14:9</failure>
</testcase>
<testcase name="test 5" classname="./test/fail.ts" time="[WILDCARD]" line="16" col="6">
<failure message="Uncaught Error">Error
throw new Error();
^
at file:///[WILDCARD]/test/fail.ts:17:9</failure>
</testcase>
<testcase name="test 6" classname="./test/fail.ts" time="[WILDCARD]" line="19" col="6">
<failure message="Uncaught Error">Error
throw new Error();
^
at file:///[WILDCARD]/test/fail.ts:20:9</failure>
</testcase>
<testcase name="test 7" classname="./test/fail.ts" time="[WILDCARD]" line="22" col="6">
<failure message="Uncaught Error">Error
throw new Error();
^
at file:///[WILDCARD]/test/fail.ts:23:9</failure>
</testcase>
<testcase name="test 8" classname="./test/fail.ts" time="[WILDCARD]" line="25" col="6">
<failure message="Uncaught Error">Error
throw new Error();
^
at file:///[WILDCARD]/test/fail.ts:26:9</failure>
</testcase>
<testcase name="test 9" classname="./test/fail.ts" time="[WILDCARD]" line="28" col="6">
<failure message="Uncaught Error">Error
throw new Error();
^
at file:///[WILDCARD]/test/fail.ts:29:9</failure>
</testcase>
</testsuite>
</testsuites>
error: Test failed

47
tests/testdata/test/nested_failures.junit.out vendored Normal file
View file

@ -0,0 +1,47 @@
Check file:///[WILDCARD]/test/nested_failures.ts
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="deno test" tests="11" failures="6" errors="0" time="[WILDCARD]">
<testsuite name="./test/nested_failures.ts" tests="11" disabled="0" errors="0" failures="6">
<testcase name="parent 1" classname="./test/nested_failures.ts" time="[WILDCARD]" line="1" col="6">
<failure message="1 test step failed">1 test step failed.</failure>
</testcase>
<testcase name="parent 2" classname="./test/nested_failures.ts" time="[WILDCARD]" line="8" col="6">
<failure message="2 test steps failed">2 test steps failed.</failure>
</testcase>
<testcase name="parent 3" classname="./test/nested_failures.ts" time="[WILDCARD]" line="20" col="6">
</testcase>
<testcase name="parent 1 &gt; child 1" classname="./test/nested_failures.ts" time="[WILDCARD]" line="2" col="11">
</testcase>
<testcase name="parent 1 &gt; child 2" classname="./test/nested_failures.ts" time="[WILDCARD]" line="3" col="11">
<failure message="Uncaught Error: Fail.">Error: Fail.
throw new Error(&quot;Fail.&quot;);
^
at file:///[WILDCARD]/test/nested_failures.ts:4:11
[WILDCARD]</failure>
</testcase>
<testcase name="parent 2 &gt; child 1" classname="./test/nested_failures.ts" time="[WILDCARD]" line="9" col="11">
<failure message="1 test step failed">1 test step failed.</failure>
</testcase>
<testcase name="parent 2 &gt; child 1 &gt; grandchild 1" classname="[WILDCARD]/test/nested_failures.ts" time="[WILDCARD]" line="10" col="13">
</testcase>
<testcase name="parent 2 &gt; child 1 &gt; grandchild 2" classname="[WILDCARD]/test/nested_failures.ts" time="[WILDCARD]" line="11" col="13">
<failure message="Uncaught Error: Fail.">Error: Fail.
throw new Error(&quot;Fail.&quot;);
^
at file:///[WILDCARD]/test/nested_failures.ts:12:13
[WILDCARD]</failure>
</testcase>
<testcase name="parent 2 &gt; child 2" classname="./test/nested_failures.ts" time="[WILDCARD]" line="15" col="11">
<failure message="Uncaught Error: Fail.">Error: Fail.
throw new Error(&quot;Fail.&quot;);
^
at file:///[WILDCARD]/test/nested_failures.ts:16:11
[WILDCARD]</failure>
</testcase>
<testcase name="parent 3 &gt; child 1" classname="./test/nested_failures.ts" time="[WILDCARD]" line="21" col="11">
</testcase>
<testcase name="parent 3 &gt; child 2" classname="./test/nested_failures.ts" time="[WILDCARD]" line="22" col="11">
</testcase>
</testsuite>
</testsuites>
error: Test failed

23
tests/testdata/test/nested_failures.ts vendored Normal file
View file

@ -0,0 +1,23 @@
Deno.test("parent 1", async (t) => {
await t.step("child 1", () => {});
await t.step("child 2", () => {
throw new Error("Fail.");
});
});
Deno.test("parent 2", async (t) => {
await t.step("child 1", async (t) => {
await t.step("grandchild 1", () => {});
await t.step("grandchild 2", () => {
throw new Error("Fail.");
});
});
await t.step("child 2", () => {
throw new Error("Fail.");
});
});
Deno.test("parent 3", async (t) => {
await t.step("child 1", () => {});
await t.step("child 2", () => {});
});

36
tests/testdata/test/pass.junit.out vendored
View file

@ -1,38 +1,38 @@
Check [WILDCARD]/testdata/test/pass.ts
Check file:///[WILDCARD]/test/pass.ts
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="deno test" tests="16" failures="0" errors="0" time="[WILDCARD]">
<testsuite name="[WILDCARD]/testdata/test/pass.ts" tests="16" disabled="0" errors="0" failures="0">
<testcase name="test 0" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="1" col="6">
<testsuite name="./test/pass.ts" tests="16" disabled="0" errors="0" failures="0">
<testcase name="test 0" classname="./test/pass.ts" time="[WILDCARD]" line="1" col="6">
</testcase>
<testcase name="test 1" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="2" col="6">
<testcase name="test 1" classname="./test/pass.ts" time="[WILDCARD]" line="2" col="6">
</testcase>
<testcase name="test 2" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="3" col="6">
<testcase name="test 2" classname="./test/pass.ts" time="[WILDCARD]" line="3" col="6">
</testcase>
<testcase name="test 3" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="4" col="6">
<testcase name="test 3" classname="./test/pass.ts" time="[WILDCARD]" line="4" col="6">
</testcase>
<testcase name="test 4" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="5" col="6">
<testcase name="test 4" classname="./test/pass.ts" time="[WILDCARD]" line="5" col="6">
</testcase>
<testcase name="test 5" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="6" col="6">
<testcase name="test 5" classname="./test/pass.ts" time="[WILDCARD]" line="6" col="6">
</testcase>
<testcase name="test 6" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="7" col="6">
<testcase name="test 6" classname="./test/pass.ts" time="[WILDCARD]" line="7" col="6">
</testcase>
<testcase name="test 7" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="8" col="6">
<testcase name="test 7" classname="./test/pass.ts" time="[WILDCARD]" line="8" col="6">
</testcase>
<testcase name="test 8" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="9" col="6">
<testcase name="test 8" classname="./test/pass.ts" time="[WILDCARD]" line="9" col="6">
</testcase>
<testcase name="test 9" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="12" col="6">
<testcase name="test 9" classname="./test/pass.ts" time="[WILDCARD]" line="12" col="6">
</testcase>
<testcase name="test\b" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="16" col="6">
<testcase name="test\b" classname="./test/pass.ts" time="[WILDCARD]" line="16" col="6">
</testcase>
<testcase name="test\f" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="19" col="6">
<testcase name="test\f" classname="./test/pass.ts" time="[WILDCARD]" line="19" col="6">
</testcase>
<testcase name="test\t" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="23" col="6">
<testcase name="test\t" classname="./test/pass.ts" time="[WILDCARD]" line="23" col="6">
</testcase>
<testcase name="test\n" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="27" col="6">
<testcase name="test\n" classname="./test/pass.ts" time="[WILDCARD]" line="27" col="6">
</testcase>
<testcase name="test\r" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="31" col="6">
<testcase name="test\r" classname="./test/pass.ts" time="[WILDCARD]" line="31" col="6">
</testcase>
<testcase name="test\v" time="[WILDCARD]" filename="[WILDCARD]/testdata/test/pass.ts" line="35" col="6">
<testcase name="test\v" classname="./test/pass.ts" time="[WILDCARD]" line="35" col="6">
</testcase>
</testsuite>
</testsuites>