The parameter names were previously "lhs" and "rhs", which is incredibly
abstract for a testing framework. One of the parameters will tend to be
a baseline value to compare against while the other is the result of
some action we want to test. Thus I suggest they be renamed "computed"
and "baseline".
This way we can, hopefully, retain the semantic that the 'left'/first
argument is the computed ('actual' in QCOMPARE) value while the
'right'/second argument is the baseline ('expected' in QCOMPARE.)
Change-Id: I3e0fdce2a3f1faca06fdf7184ef6e0eb9724d990
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
The regex used a greedy .+ to match an actual dot and a space,
preceding an open parenthesis, with the result that if the message
contained any parentheses that .+ swallowed everything up to the last
of them. Correct the regex so that recently-added tests' error
messages show up correctly.
Change-Id: I6e52c9b2a6e7959335fcddbb4266f65b589eba68
Reviewed-by: Jason McDonald <macadder1@gmail.com>
The TAP test logger will now correctly print comparison types such
as QCOMPARE_NE or QCOMPARE_LT, and also provide a proper expected/wanted
value (by adding proper arithmetical operators in front of the value).
Sample output:
type: QCOMPARE_GE
message: Left value is expected to be greater than or equal to right value, but is not
wanted: >= 1 (rhs)
found: 0 (lhs)
expected: >= 1 (rhs)
actual: 0 (lhs)
at: tst_ExtendedCompare::compareUnregistereEnum() (tst_extendedcompare.cpp:232)
file: tst_extendedcompare.cpp
line: 232
As a drive-by: make some variables const.
Task-number: QTBUG-98873
Change-Id: Idb54eaabcb937b42d3fc844f30041aab82d73f69
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Replace the current license disclaimer in files by
a SPDX-License-Identifier.
Files that have to be modified by hand are modified.
License files are organized under LICENSES directory.
Task-number: QTBUG-67283
Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
Remove unneeded \fn qdoc lines as a drive-by.
Task-number: QTBUG-98434
Change-Id: Id93ddbb38b97a8f5a6734bfbc82686ccb3a87aa6
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Required for porting away from QLatin1Char/QLatin1String in scope of
QTBUG-98434.
Change-Id: I3debae5f481037958bfb65caabca97a0d4681eb9
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
It had some repetition that could be refactored out, some long lines
and a bool it worked out the hard way, when it had previously
determined the answer in passing without recording it.
Change-Id: I9e53ed087dfbe8067686b27b6cf9ac32040fbf19
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Previously, only the first B?XFAIL would be reported, all others would
be discarded. Furthermore, if a B?XFAIL had happened, B?PASS was also
not reported, since the B?XFAIL served as test line. However, if the
B?XFAIL was followed by a SKIP, B?XPASS or B?FAIL, these were reported
as normal, producing exactly the kind of duplicated test line that the
skipping of B?PASS was meant to supply.
So change B?XFAIL to be reported among the messages, but retain the
TODO annotation of the first on the test line of a subsequent B?PASS,
if nothing more drastic happens in the mean time. So now more than one
B?XFAIL can be reported, the test is still marked as a TODO and we
don't get duplicate test lines for a subsequent non-passing result.
This replaces the bool m_wasExpectedFail member with a QTestCharBuffer
m_firstExpectedFail that records the first XFAIL's TODO line (so its
isEmpty() fully replaces m_wasExpectedFail).
Previously, the at/file/line information for a B?XFail would be
supplied as top-level keys in the YAML block for a "Pass" reported as
not ok due to the XFail, as this location information is now part of
the B?XFail's message in the extensions/messages block. Duplicating
the first B?XFail's location at top level would add complexity and is
arguably misleading, as the test result is really a pass (after
ignoring known issues), and the location of the pass is indeterminate
(nominally the end of the test function, but actually also after the
cleanup() call for this test, when relevant), which is why a Pass has
no location information.
Task-number: QTBUG-96844
Change-Id: Ib3f24f56266ff49bf3bc9759ac8263ac69a62130
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
In the process, split it from its comments block and don't bother with
a YAML block if it would only have contained comments.
Task-number: QTBUG-96844
Change-Id: I08c20f796252bb270ba9caa4c055cdcc0843a88b
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
When producing a diagnostics block, include the file and line
information, if we have it, to describe it. This presently only adds
this information for skip, but could in principle do the same for a
B?XPass.
Task-number: QTBUG-96844
Change-Id: I6cc375d98e2369eba262010f9c2dfbcba931a6f1
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Our TAP output was delivering messages as comments before the test
line, where TAP clearly expects the details of a test to follow its
test line. Version 13 provides a YAML block to deliver diagnostics and
encourages use of this, so accumulate our messages in a
QTestCharBuffer instead of emitting them one by one.
However, messages produced after a test has produced its test line
belong to that test, but are too late to be included in its
diagnostics block, so should be emitted immediately as before, albeit
now with a type prefix. This at least separates such messages, from
the end of one test, from messages produced early in the next.
In the process, add a type-prefix to each, to make clear what type of
message it was. Since the Yamlish supported by TAP consumers doesn't
support a way to have many messages, use the extensions: top-level
hash tag with a messages: sub-tag to gather our messages as a list.
(This expands at least one expected output file significantly and
substantially rewrites some others.)
Add methods to QTestCharBuffer, and a helper function, to support this.
Task-number: QTBUG-96844
Change-Id: If44a33da5879ed1670ef0980042599afd516f9d2
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
... along with how QtTest soon shall be using it.
Change-Id: I9c723ebdb8ee5be1c4152aa911ae7c846ea61b5c
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
We only need to look at the first byte to determine whether the length
is > 0; no need to count how many more bytes after it aren't '\0'.
Change-Id: Id2db2a9a993086c0aee02c99396fcb346b1a348e
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
When a B?Fail's description doesn't match the QVERIFY/QCOMPARE
regexes, it got output as a comment in the YAML block after the test
line had already reported it as a TODO comment. An empty description
would also have lead to an empty comment in the YAML block. Condition
this fallback output case on there being a description that hasn't yet
been reported in the test line.
Task-number: QTBUG-96844
Change-Id: Id7fe81d26ddb01da3d8003ada8fa590a5e1a166f
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
In all places but one they were treated the same; fix that last one to
match the rest. This removes one line from the YAML block for each
blacklisted XFail test; in each case, this message duplicates the one
on its "not ok ... # TODO..." test line.
Change-Id: Iff2028afccd979db7f2c84e85d1b78541900008e
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Provide basic (internal) documentation of each logging class and link
the command-line documentation to pages relevant to the formats not
defined by Qt.
Change-Id: I3251dd1304203c6ab87dfe1f2dec0e9787ab69f8
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
After we've checked m_wasExpectedFail, we might as well set it
promptly, since one of our earlier checks is on the same
condition. Explain what that check is about, fix a typo in another
comment and make the local ok variable const, since it can be.
Change-Id: Ie04c0ad4248e47b116d3e22046116d39cf2ac6df
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
If an incident has an empty description we produced output with a
dangling space. Avoid triggering the inanity 'bot with those.
Reorganise the code to save some duplication.
Change-Id: I1dc29fa8ad4449a4584f11dddcf002e405cd9238
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The rationale for skipping pass after XFail is "to emit a single test
point for" the test; emitting several XFails violates that aim.
Task-number: QTBUG-96844
Change-Id: Ia8626dfc2dded234b3aa530fc2dc2324f1e28400
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Skip ends the test (albeit inconclusively). Rearrange the enums in
the abstract logger, move code to handle skip between relevant
function and tidy up various things that became simpler as a result.
Also reorder the message enum, and its switches, to separate testlib's
internals from the usual Qt messages, and put each group in ascending
order of severity.
Task-number: QTBUG-96844
Change-Id: I2c7a634b9f849830d64eafa750155e66e244b729
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
- Remove virtual from functions declared as override
- Use " = default" for trivial constructors/destructors
- Remove all special functions from QTestLog
Apply Fixits by Qt Creator with some amendments.
Task-number: QTBUG-69413
Change-Id: I812b8116e5b4c927e4e5cee44e63bc705385d866
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
That's an undocumented Qt 4/3/2 remnant, start remove usages.
Fix incorrect include header in qclass_lib_map.h as a drive-by.
Change-Id: I939be2621bc03e5c75f7e3f152546d3af6d37b91
Reviewed-by: Marc Mutz <marc.mutz@kdab.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Prioritize blacklisting over QEXPECT_FAIL so that a test that is
blacklisted no longer fails if QEXPECT_FAIL returns true unexpectedly. To
reflect this state properly, the two values of BXPASS and BXFAIL were
added to testlib's output.
[ChangeLog][Important Behavior Changes][QtTestLib] Blacklisting of tests
will be taken into account for XPASS and XFAIL. A blacklisted test that
causes an XPASS will no longer be a fail.
Task-number: QTBUG-72928
Change-Id: Ia2232fdc714d405fa3fd9aea6c89eb2836bc5950
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The Test Anything Protocol (TAP), was originally Perl's simple text-based
interface between testing modules and test harnesses, but has since been
adopted by a large number of producers and consumers in many different
languages, which allows colorizing and summarizing test results.
The format is very simple:
TAP version 13
ok 1 - test description
not ok 2 - test description
---
message: 'Failure message'
severity: fail
expected: 123
actual: 456
...
ok 3 - test description # SKIP
1..3
The specification [1] is very brief, so the implementation has been
based on how typical consumers behave, especially when it comes to
the undefined diagnostics block.
[1] http://testanything.org/tap-version-13-specification.html
Change-Id: I616e802ea380165c678510e940ddc6607d39c92d
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
Mårten Nordheim
Edward Welbourne
Ivan Solovev
Lucie Gérard
Sona Kurazyan
Tor Arne Vestbø
Friedemann Kleint
Giuseppe D'Angelo
Oliver Wolff
Mikhail Svetkin