Present behavior is to continue or abort, according to the mode,
regardless of whether the condition marked does fail.
Change-Id: I2eb5eefbbf173326101d08f3bbb378d214e93e71
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Paul Wicking <paul.wicking@qt.io>
Use \note instead of \b{Note:}, don't refer to functions as macros,
and don't lie. The data-table set-up functions can delegate their
calls to addColumn(), addRow() and newRow() to functions not called
directly by the test framework.
Change-Id: I3bef22cf1a45df91dc822ac04f07e7d15053825f
Reviewed-by: Paul Wicking <paul.wicking@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The "documentation" has also been removed for all types, as it didn't
add anything, and maintaining it to be in sync with the values just
adds overhead.
Pick-to: 6.2
Change-Id: Iab1dd7b9804c05559b4674342d6059b68ecf3ecf
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
None of the loggers rely on this at the moment, but one could imagine
one that does. Plus, doing any sort of debug logging in leaveTestFunction
will at the moment be attributed to UnknownTestFunc().
Pick-to: 6.2
Change-Id: I284b2785a276e028b9f57c26357679fd9e045ca7
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The separation between <system-out> and <system-err> is sufficient, and
we can't expect consumers to interpret our custom comment format.
The type of the text node has been renamed to more accurately identify
its purpose.
Pick-to: 6.2
Change-Id: I63c8ff17529fc087e1b695698350a6711eb5e68d
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
It's used to handle qCritical() messages, and corresponds to QtMsgType's
QtCriticalMsg, so change its name to reflect what it really is.
In the process, reclassify the -maxwarnings overflow message as a warning,
rather than a critical message.
Pick-to: 6.2
Change-Id: I87626117a547ae4498d5dc352b93bd6db8bfb332
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The original Ant JUnit reporter only writes <system-err> and <system-out>
to the <testsuite>, but more modern reporters such as Maven Surefire
scopes output to each individual <testcase>.
This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.:
https://github.com/jenkinsci/junit-plugin/commit/145eb5c98
Pick-to: 6.2
Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Test errors represents unanticipated problems, e.g., an unhandled
exception, or a problem with the implementation of the test.
Pick-to: 6.2
Change-Id: I87219e7ffdea56862278f005de44526ad97545f0
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
We don't produce <failure> elements for them, so we shouldn't include
them in the total count of failures.
In the future we might produce <skipped> elements instead, but for now
remove the incorrect increment.
Pick-to: 6.2
Change-Id: Ia89a5cdaf79f6aa2f92be53180112c40ad6da2c6
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The data itself is not indented, as consumers may read it as verbatim
data.
Pick-to: 6.2
Change-Id: Ia934616cea273feadc3a45d7c74726d4f804f0dc
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The Apache Ant and Surefire Maven specs document a <skipped> element that
can be used to signify skipped test, with a corresponding total skipped
test attribute on the <testsuite>.
The element includes an optional message attribute, documented in the
Surefire spec, and in the Ant source code, but not yet documented in
the reverse-engineered Ant spec:
https://github.com/windyroad/JUnit-Schema/pull/11
Pick-to: 6.2
Task-number: QTBUG-95424
Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The errors attribute on the <testsuite> element represents the number of
<error> elements, but we do not produce any at the moment.
Pick-to: 6.2
Task-number: QTBUG-95424
Change-Id: I7196d622a9a6bbb7e79ed2c2886984d539abb1da
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
539553a572 renamed the LET_Error element
enum to LET_Message, without renaming the corresponding "error" element
name. This was not an issue in practice, since we never write the actual
element in QTestJUnitStreamer, but the two should be in sync to avoid
any confusion.
Pick-to: 6.2
Change-Id: I6c29f5303393b5f36b2f9877940bf3f6eaf3b7d2
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The original Ant JUnit reporter produced test durations via Double.toString(),
supporting arbitrary precisions, and the de-facto schema declared them
as xs:decimal.
Sadly, the now popular Maven Surefire reporter limited the duration to
millisecond precision, and hard-coded this into its schema as SUREFIRE_TIME:
https://issues.apache.org/jira/browse/SUREFIRE-1533
Unfortunately this definition spread into tools such as the Jenkins xUnit
plugin, which relies on the schema provided by Maven Surefire:
https://issues.jenkins.io/browse/JENKINS-52152
As a result, anything that produces higher precision results will not
validate in the Jenkins xUnit plugin.
Other test frameworks have bitten the bullet and reduced their precision
correspondingly, e.g.:
https://github.com/catchorg/Catch2/issues/2221https://github.com/catchorg/Catch2/commit/581c46249acf8389e9
We follow suit, and our JUnit XML output now validates against both
the Jenkins JUnit and xUnit plugins, as well as the original Apache
Ant de-facto schema.
Pick-to: 6.2
Task-number: QTBUG-95424
Change-Id: I3097d10c03c2a29709960372301b29055d224e10
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The JUnit test framework did not initially have any XML reporting
facilities built in. Instead, the XML report was generated by the
Apache Ant JUnit task:
https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java
Many users interacted with these reports via the Jenkins JUnit plugin,
which provided graphical visualization of the test results:
https://plugins.jenkins.io/junit/
Due to the lack of an official XML schema for the Apache Ant JUnit
report there was some confusion about what the actual format was.
People started documenting the de-facto format, both as produced
by Ant, and as consumed by Jenkins:
https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsdhttps://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd
The XML produced by the Qt Test JUnit reporter was far from these
schemas, causing issues when importing results into tools such
as Jenkins, Allure2, or Test Center.
The following changes have been made to improve conformance:
- The 'timestamp' attribute on <testsuite> is is now in ISO
8601 local time, without any time zone specified
- The 'hostname' attribute on <testsuite> is now included
- The 'classname' attribute on <testcase> is now included
- The non-standard 'result' attribute on <testcase> has
been removed
- The non-standard 'result' attribute on <failure> has
been renamed to 'type'
- The <system-out> element on <testsuite> is always included,
even when empty
- The non-standard 'tag' attribute on <failure> has been
removed. Data-driven tests are now represented as individual
<testcase> elements, e.g.:
<testcase name="someTest(someData X)" ...>
<testcase name="someTest(someData Y)" ...>
<testcase name="someTest(someData Z)" ...>
The resulting XML validates against both the de-facto Apache Ant
'JUnit 4' schema and the Jenkins JUnit plugin schema.
Task-number: QTBUG-95424
Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
- Use the right name for the attribute (AI_Message),
rather than fixing it up in QTestJUnitStreamer.
- Don't pretend that we're adding line and file information,
only to discard it in QTestJUnitStreamer.
- Don't pretend to add benchmark information,
only to discard it in QTestJUnitStreamer.
Pick-to: 6.2
Change-Id: Ib6eadc12300157216fe9c6e8bcfebd7eb8a3ea68
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
It makes no sense for the event loop of a test to keep running after a
test has failed. This lets test code simply use the usual testlib
macros to compare and verify values in asynchronous tests that would
otherwise need to hand-test values and send a signal on failure (that
the main test can connect to an event-loops quit() or equivalent).
For example, QLocalSocket's benchmark simply uses the usual macros,
without doing anything to stop its event loop if they fail, with the
sad result that, when a test fails, it does so repeatedly and then
times out, causing the test program to be killed without running later
tests. With this change, that test code (once converted to use
QTestEventLoop) is able to exit gracefully on the first failure.
[ChangeLog][QtTest][QTestEventLoop] The QTestEventLoop new exits its
event loop as soon as the test is known to be failing.
Task-number: QTBUG-91713
Change-Id: If0d455741668722034906763025dda496d2afbb4
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
We try our best to pass on the file location of a failure, including for
fatal errors, but the reporting or logging machinery should not assume
there is one.
By passing on nullptr for the file location we allow the logging backends
to decide how to handle the situation, e.g. by not emitting extra fields
for failure location.
This effectively reverts c25687fa0b,
in favor of relying on the backends to cope with null filename,
which they already did.
As qFatal uses QMessageLogger, which by default disables file/line
information in release builds, we need to explicitly enable this in
our self-tests, to get uniform test results. Similarly, we disable
file/line info from testlib itself, as reporting Qt internal file
and line information for user diagnostics is less useful. The odd
one out there is qtestdata.cpp, which still ends up in test output
due to using QTEST_ASSERT instead of qFatal for its diagnostics.
Cleaning up that, and unifying how we report testlib issues to the
user, is left for another day.
Pick-to: 6.2
Change-Id: Ib9451b8eed86fe3ade4a4dcaf0037e1a3450321c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The QtTest best practices documentations recommends using output
mechanisms such as qDebug() and qWarning() for diagnostic messages,
and this is also what most of our own tests do.
The QWARN() macro and corresponding internal QTest::qWarn() function
was added when QtTest was first implemented, but was likely meant as
an internal implementation detail, like its cousin QTestLog::info(),
which does not have any corresponding macro.
This theory is backed by our own QtTest self-test (tst_silent)
describing the output from QWARN() as "an internal testlib warning".
The only difference between QWARN() and qWarning(), besides the much
richer feature set of the latter, is that qWarning() will not pass
on file and line number information in release mode, but QWARN() will.
This is an acceptable loss of functionality, considering that the user
can override this behavior by defining QT_MESSAGELOGCONTEXT.
[ChangeLog][QtTest] QWARN() has been deprecated in favor of qWarning()
Pick-to: 6.2
Change-Id: I5a2431ce48c47392244560dd520953b9fc735c85
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The key part of FailureReportingMode::QtTest is that it will result
in a test failure (via QTest::qVerify); not that it's using the
QtTest logging machinery.
Pick-to: 6.2
Change-Id: I18472fba173ce1323748f1f87301a4defa87020e
Reviewed-by: Giuseppe D'Angelo <giuseppe.dangelo@kdab.com>
The format was renamed in 27db9e458c.
Pick-to: 6.2
Change-Id: I53975c7467d8768dc9dc9ac2d89c42eefa12e22f
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Before this change, next() was the only way to advance the iterator,
whether the caller was ultimately interested in just the filePath()
(good) or not (bad luck, had to call .fileInfo()).
Add a new function, nextFileInfo(), with returns fileInfo() instead.
Incidentally, the returned object has already been constructed as part
of advance()ing the iterator, so the new function is faster than
next() even if the result is ignored, because we're not calculating a
QString result the caller may not be interested in.
Use the new function around the code.
Fix a couple of cases of next(); fileInfo().filePath() (just use
next()'s return value) as a drive-by.
[ChangeLog][QtCore][QDirIterator] Added nextFileInfo(), which is like
next(), but returns fileInfo() instead of filePath().
Change-Id: I601220575961169b44139fc55b9eae6c3197afb4
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Take two. f20edffcc8 only disabled the
backtrace preamble.
Pick-to: 6.2
Change-Id: I972d007af59d13ec6eb11da44fc2b5e25c36e46e
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
It now uses QtPrivate::condition_variable, it pulls in the correct
header for that, so #include <condition_variable> is no longer needed.
Separate opening braces of function bodies onto next line.
Change-Id: I08f721c4d52756932bb9409e34e51dcbb3eda104
Reviewed-by: Marc Mutz <marc.mutz@kdab.com>
char16_t, as returned from QStringView::utf16(), is guaranteed to be
unsigned, so there's no need to reinterpret_cast to ushort.
Change-Id: I7432dd26b4814c61fe70cc33ed307097ef46ce4e
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
... so don't try to call it.
The call resolves to a fall-back implementation of toString() that
just returns nullptr, so we might as well pass nullptr directly, like
three lines above, and not confuse readers of the code.
Change-Id: I85d196aae7a0317050cfe16b206252df3fdffa2d
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Says GCC:
In function ‘char* QTest::toString(QPair<T1, T2>&) [with T1 = QWidget*; T2 = QEvent::Type]’,
warning: ‘%s’ directive argument is null [-Wformat-overflow=]
Fix by re-using formatString(), once introduced for std::tuple.
As a side-effect, this gets rid of the funny double-quotes around the
output.
Change-Id: I2dd5f10fa2b3a392370bf487c1b7e98f3d190978
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Replace the hardcoded cpp exports with a generated one where it's
applicable.
Task-number: QTBUG-90492
Change-Id: Idc160b594987b2c765e75bd669aae851b4366282
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
MetaMethods is a std::vector, so we should avoid copying it. Add
strategic std::move()s and pass to the TestFunctions ctor by value,
not cref.
Change-Id: Iaa2879ae427b9603a7abaab0e3015556d9c247fc
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The compiler generated ones are broken. The RO5 exists for a reason...
Change-Id: Ibaa5322a7a1ad4ad06bd00e9d1c21bbbea6eb08c
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Remove the configure.json and configure.pri files that
were used for the qmake-based configure.
Remove the .prev_*.cmake files that were a by-product of
configurejson2cmake.py.
Pick-to: 6.2
Task-number: QTBUG-89536
Change-Id: Ie827562f7fd2513d59f69234d77b8b93124ea78e
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Previously, it went direct to QTestResults::addFailure() without going
via the checking for expected failure. Add QTestResults::fail() to
take care of this checking, as for verify() and compare().
Tidied up the code implementing expected failure and QFAIL(), while I
was about it. Adjusted an existing test to verify that expecting a
QFAIL() works, by using QFAIL() instead of QVERIFY(false).
Remove the QVERIFY(false) whose comment brought this to my attention.
[ChangeLog][QtTestLib][QFAIL] QEXPECT_FAIL() now correctly anticipates
a subsequent QFAIL(). Previously QFAIL() counted as a fail regardless.
Change-Id: Icc28cf70e5ff3006363791ea03aa01f2f591eb71
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Clazy warns about the fromUtf8() call with a constant argument, for
every use of the macro, so hide the stuff behind a compiler firewall.
Also fix the format injection error by using QLatin1String::arg()
instead of QString::arg() chaining.
Change-Id: I4bb4d4af56443540efc0c38c75819aa152a441fc
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Fixes
warning C5046: 'testObject::MyTestObject::toString': Symbol involving type with internal linkage not defined
Change-Id: I9925eb15e262f29e636c019c87311a6ea2c47505
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Paul Wicking <paul.wicking@qt.io>
Output object name and class in QCOMPARE(). This should help
to debug flaky QWidget tests that for example check on focusWidget().
[ChangeLog][QtTestLib] QCOMPARE() now reports QObject * values by class and objectName().
Task-number: QTBUG-64446
Change-Id: Ife04e89bba04fc78d077c8f0f07af17a17c9cf8c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The trick we use by spawning lldb to print the backtrace doesn't work
for some reason, and just results in:
Process 91619 stopped
* thread #1, stop reason = signal SIGSTOP
frame #0: 0xffffffffffffffff
Target 0: (No executable module.) stopped.
Architecture set to: .
(lldb) bt all
* thread #1, stop reason = signal SIGSTOP
frame #0: 0xffffffffffffffff
thread #2
frame #0: 0xffffffffffffffff
thread #3
frame #0: 0xffffffffffffffff
thread #4
frame #0: 0xffffffffffffffff
Debugging the stopped process manually with lldb works fine.
Change-Id: If5582803a11f1c5c84c31b42d1ae52bdd245bbbb
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
The list persists events, which reference objects. Those objects might
get deleted by the time we investigate the objects. We cannot change the
event to store a QPointer for BIC reasons, and for normal usage of the
events, that doesn't make sense either.
Instead, connect the objects destroyed signal to a lambda which clears
the events' object member.
In order to access the private member, we befriend the test class.
Change-Id: I036be7053dccde4bdf862173789564e89d729ee1
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Shawn Rutledge <shawn.rutledge@qt.io>
After 1ed8a7bff5,
volatile needs to be specified for toString(), else it is not
used by compare_ptr_helper(). Add an overload.
Pick-to: 6.1 5.15
Change-Id: I3c335f324df346233623272d1014c8360ca33160
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Not 100% sure if this is a bug in tsan, but turning expected into an
atomic variable will avoid the warnings.
Change-Id: I6d6a5d58f90258d201ae3880473228638de0a215
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Andrei Golubev <andrei.golubev@qt.io>
Change-Id: I4f5a2ca983902beccf967e983083cc7a315b8ef6
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Assam Boudjelthia
Alexandru Croitor
Edward Welbourne
Tor Arne Vestbø
Nico Vertriest
Marc Mutz
Alexey Edelev
Giuseppe D'Angelo
Joerg Bornemann
Kai Köhne
Lorn Potter
Friedemann Kleint
Fabian Kosmale
Lars Knoll
Juha Vuolle
Morten Sørvig