The following are definitions of some concepts related to testing in general and some other specific to SRFI 64 and Guile Probá.
This is the report that is displayed on screen when a test suite is run using the default test runner (see test-runner-uno). Contrary to the complete report (see Complete report), this report lists only the results of tests that need attention: FAIL, SKIP, XFAIL and XPASS (see Kinds of test results).
The following is an example of this kind of report:
------------------------------------------------------------ TEST SUITE ------------------------------------------------------------ ⚑ XPASS: Make a chopper. Group: [procedure] make-bike File: bogus-tests/test-things.scm:37 Expected value: "🚁" Actual value: "🚁" ❖ SKIP: Make bicycle. Group: [procedure] make-bike File: bogus-tests/test-things.scm:48 Expected value: #t Actual value: #f ⚑ XFAIL: Make breakfast. Group: [procedure] make-food File: bogus-tests/test-things.scm:58 Expected value: "🥙" Actual value: "🥞" ✖ FAIL: Make a shiny new robot. Group: [procedure] make-robot File: bogus-tests/test-things.scm:94 Expected value: "🤖 Roboto" Actual value: "💩 Roboto" ------------------------------------------------------------ TOTAL TESTS: 9 PASSED: 5 FAILED: 1 SKIPPED: 1 XPASSED: 1 XFAILED: 1 ------------------------------------------------------------ (Full log in "/tmp/guile-proba-test-suite.log")
This is similar to the brief report (see Brief report). The only difference is that this report is written to a file and it lists all kinds of results (see Kinds of test results) for every single test case. The location of the file is indicated in the brief report.
The following is an example of this kind of report:
✔ PASS: Make a baby with a given name. Group: [procedure] make-baby File: bogus-tests/test-things.scm:22 Expected value: "👶 Jane" Actual value: "👶 Jane" ✔ PASS: Make a scooter. Group: [procedure] make-bike File: bogus-tests/test-things.scm:31 Expected value: "🛵" Actual value: "🛵" ⚑ XPASS: Make a chopper. Group: [procedure] make-bike File: bogus-tests/test-things.scm:37 Expected value: "🚁" Actual value: "🚁" ❖ SKIP: Make bicycle. Group: [procedure] make-bike File: bogus-tests/test-things.scm:48 Expected value: #t Actual value: #f ⚑ XFAIL: Make breakfast. Group: [procedure] make-food File: bogus-tests/test-things.scm:58 Expected value: "🥙" Actual value: "🥞" ✔ PASS: Make lunch. Group: [procedure] make-food File: bogus-tests/test-things.scm:63 Expected value: "🍛" Actual value: "🍛" ✔ PASS: Make dinner. Group: [procedure] make-food File: bogus-tests/test-things.scm:68 Expected value: "🍜" Actual value: "🍜" ✔ PASS: Raise exception when the kind of food is unkown. Group: [procedure] make-food File: bogus-tests/test-things.scm:73 Expected value: #t Actual value: #f Actual error: (%exception #<&unknown-food-error>) ✖ FAIL: Make a shiny new robot. Group: [procedure] make-robot File: bogus-tests/test-things.scm:94 Expected value: "🤖 Roboto" Actual value: "💩 Roboto" ------------------------------------------------------------ TOTAL TESTS: 9 PASSED: 5 FAILED: 1 SKIPPED: 1 XPASSED: 1 XFAILED: 1
The following have the same meanings as their corresponding SRFI 64 result kinds.
The test passed, as expected.
The test failed but was not expected to.
Using test-skip, the authors of the test suite explicitly told
the test runner to skip the execution of this test. They will decide
what to do about it in another time.
Using test-expect-fail, the authors of the test suite
explicitly told the test runner that this test should not be counted
as a PASS. They know that the test passes, when it shoudn’t, and will
decide what to do about it in another time.
Using test-expect-fail, the authors of the test suite
explicitly told the test runner that this test should not be counted
as a FAIL. They know that the test fails, when it shouldn’t, and will
decide what to do about it in another time.