This removes one middle-man and a 512-byte variable in favor of a simple
32-bit enum. This was done in a way so we can extend to use either gdb
or lldb in any OS.
I've renamed the debuggerPresent() function to make its meaning clearer.
Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eba561628ff89b
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Instead of piping stuff via the shell into them, just use batch mode.
And also take the opportunity to tell them not to read their user-
provided configuration files.
Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eba21e71afaefa
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
If a previous handler was already installed, ensure it is called,
because there may be a reason why it was there. For example, the Android
ART adds a signal action to every fatal signal for logging purposes. We
do that by restoring the signal handler we had and re-raising the
signal.
If our handler was overridden by something else, then that handler was
already called, but will get uninstalled after our code runs. It won't
be a problem, because the application is exiting anyway.
[ChangeLog][QtTest][Behavior Change] On Unix, the QtTest code to
handle Unix/POSIX fatal signals will now call back to the original
handler that was installed, if there was one. This allows logging
frameworks (such as Android ART's), for example, to log the crash too.
Additionally, if there was no handler, the application should exit with
the correct signal instead of SIGABRT.
Fixes: QTBUG-97652
Pick-to: 6.3
Change-Id: Ifc4fca159b490d8d0b614d736e46caefcb903a4c
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
And only restore those signals, instead of iterating over all possible
signals, instead of attempting to restore to SIG_DFL. Also, as a
consequence, we will install our handler even if there was already a
handler installed for the signal.
Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb756685f4e8b8
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
And remove the unreachable code after std::abort() that was meant to
mimic that in INTEGRITY. The next commit will fix this properly.
Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb772018add694
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The #if around the function declaration was ugly.
Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb7540f5da080f
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
We get stack space reserved instead of using the heap.
Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb73fff0174150
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
They're very different, so there's no point in having them even in the
same class body. I've renamed the Windows one because Windows does not
report crashes via signals anyway. If you have an IDE that can scan both
branches of the #if, it will help you find the Windows-specific code
too.
Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb73ba196cfb74
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
The Qt environment handling functions lock a mutex. That's a big no-no
in signal handlers.
[ChangeLog][QtTest][Behavior Change] QtTest will now check the value of
the environment variable QTEST_PAUSE_ON_CRASH in QTest::qRun(), so if a
test wants to modify this variable, it must do so from the main() or
initMain() functions, not in the test itself (including initTestCase()).
Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb78867cd8f54e
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
This should work so long as there's no async-unsafe pthread_atfork()-
registered callback.
Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb72e4e313bc19
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
I don't get it on openSUSE Tumbleweed, but this is not the only place I
don't get a warn_unused_result warning and it happens elsewhere.
Change-Id: I7e305799c8594ebab255fffd16ee12ff30df6d4d
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
This patch enables usage of QAbstractItemModelTester on
QFileSystemModel. QAbstractItemModelTester called fetchMore()
on all items. QFileSystemModel represents the whole file system.
This led to very long test runs. To avoid this, this patch
introduces a new feature in QAbstractItemModelTester, namely
to disable calling of fetchMore().
Change-Id: Ie5d2e22fa4c143be7c080d9f79632cd2cbe07aac
Reviewed-by: David Faure <david.faure@kdab.com>
Replace the current license disclaimer in files by
a SPDX-License-Identifier.
Files that have to be modified by hand are modified.
License files are organized under LICENSES directory.
Task-number: QTBUG-67283
Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
All module header files should be listed in the corresponding sections
of modules SOURCEs to be accessible in CMake routines.
Task-number: QTBUG-103196
Change-Id: Ieb77ae70557e35e546a5b00387e1e0aa40338239
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
It's not async-signal-safe, so don't use it. Instead, let's use writev()
(which should write atomically and we won't need to have a buffer) and
std::to_chars to format numbers where available (where not, we roll out
our own implementation with divisions).
Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb70a158f44864
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
As if QIODevice::readLine() and QIODevice::Text open-mode aren't slow
enough, the old code took the line QByteArray, converted it to a
QString so it could apply a trivial regex to it: '^summary: (\d+)'.
We can, of course, use QByteArray::startsWith("summary: ") followed by
std::from_chars(), these days, to get the same effect, without having
to JIT-compile an RX and convert every line into UTF-16 first.
Change-Id: I20d80ffb469329d3c3efd08c71fcf5abf9f486d3
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Instead of trying to guess which OSes have this functionality.
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16ebc20fe7eca346
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Since each .arg() call starts from scratch, a file name containing a
suitable %n would mess up the formatting of the following .arg() call.
Fix by using multiArg(), which requires to pre-format the line into a
QString, but which performs only a single-pass, so doesn't suffer from
the placeholder injection problem that plagues .arg()-chaining.
Pick-to: 6.3 6.2 5.15
Change-Id: I549527643da657fca0bea63d5e3becadac529d4b
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Sona Kurazyan <sona.kurazyan@qt.io>
... instead of a pair of val and valSeen variables.
More elegant.
Pick-to: 6.3
Change-Id: I0fcf9a3b5611e30fb81d92619993a7828496cd1b
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Remove unneeded \fn qdoc lines as a drive-by.
Task-number: QTBUG-98434
Change-Id: Id93ddbb38b97a8f5a6734bfbc82686ccb3a87aa6
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Required for porting away from QLatin1Char/QLatin1String in scope of
QTBUG-98434.
Change-Id: I3debae5f481037958bfb65caabca97a0d4681eb9
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Including moc files directly into their classes' TU tends to improve
codegen and enables extended compiler warnings, e.g. about unused
private functions or fields.
Pick-to: 6.3 6.2 5.15
Task-number: QTBUG-102886
Change-Id: I9378ddcabbfa0ec5ca895f6ce55ab5f1ec2cade3
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
In the same ways as for blacklisting. In the process, move reporting
of an unrecognised tag outside the global-data loop, as the tag might
be, or start with, a global data tag; and also report the global data
alongside the test-specific data. Indent the lines reporting tags, as
this makes the structure of the lists more evident. Converted the tag
argument to QLatin1String() to facilitate searching and matching.
Updated documentation, including expanding on the option of specifying
the data tags on the command-line. Drive-by: fixed a link in the
documentation, removed a snippet that was nowhere used.
[ChangeLog][QtTest] When specifying what tests to run on a QtTest
command-line, it is now possible to include a global data tag in the
same way as a function-specific data tag or in combination with it.
Thus if the test reports itself as tst_Class::function(global:local),
command-line option function:global:local will select that specific
test and function:global will run all data-rows for the function on
the given global data tag.
Fixes: QTBUG-102269
Change-Id: I7c86d6026933b05f7994bfc3663a2ea6a47f5f38
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Separated from other clean-ups to avoid inanity 'bot complaints.
Change-Id: Ia33f87da224dba7c50bbf14beb642825dc46fe90
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Now that QL1S has its own arg(), we can use that directly instead of
going via QString before calling arg(). Only works in most cases, as
the QL1S::arg() overload set is more limited.
Change-Id: Ie4c566c5998538b1b8a953b1ba4481e0f1663bf7
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
We already have similar logic in the QWidget variant.
Pick-to: 6.3 6.2
Task-number: QTBUG-102253
Change-Id: Idf1e3474f3189909f1cba7d5dedc2cde12f1338e
Reviewed-by: Michal Klocek <michal.klocek@qt.io>
It had some repetition that could be refactored out, some long lines
and a bool it worked out the hard way, when it had previously
determined the answer in passing without recording it.
Change-Id: I9e53ed087dfbe8067686b27b6cf9ac32040fbf19
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Previously, only the first B?XFAIL would be reported, all others would
be discarded. Furthermore, if a B?XFAIL had happened, B?PASS was also
not reported, since the B?XFAIL served as test line. However, if the
B?XFAIL was followed by a SKIP, B?XPASS or B?FAIL, these were reported
as normal, producing exactly the kind of duplicated test line that the
skipping of B?PASS was meant to supply.
So change B?XFAIL to be reported among the messages, but retain the
TODO annotation of the first on the test line of a subsequent B?PASS,
if nothing more drastic happens in the mean time. So now more than one
B?XFAIL can be reported, the test is still marked as a TODO and we
don't get duplicate test lines for a subsequent non-passing result.
This replaces the bool m_wasExpectedFail member with a QTestCharBuffer
m_firstExpectedFail that records the first XFAIL's TODO line (so its
isEmpty() fully replaces m_wasExpectedFail).
Previously, the at/file/line information for a B?XFail would be
supplied as top-level keys in the YAML block for a "Pass" reported as
not ok due to the XFail, as this location information is now part of
the B?XFail's message in the extensions/messages block. Duplicating
the first B?XFail's location at top level would add complexity and is
arguably misleading, as the test result is really a pass (after
ignoring known issues), and the location of the pass is indeterminate
(nominally the end of the test function, but actually also after the
cleanup() call for this test, when relevant), which is why a Pass has
no location information.
Task-number: QTBUG-96844
Change-Id: Ib3f24f56266ff49bf3bc9759ac8263ac69a62130
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
In the process, split it from its comments block and don't bother with
a YAML block if it would only have contained comments.
Task-number: QTBUG-96844
Change-Id: I08c20f796252bb270ba9caa4c055cdcc0843a88b
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
This function should be used in QWidget-related test cases instead of
QWidget::show() when you need to move or resize the widget and then
compare its position or geometry with the expected value.
The reason for introducing it is that on Android QWidget::show() is
showing the widget maximized by default, so its position and size
can't be changed.
Task-number: QTBUG-87668
Pick-to: 6.3 6.2
Change-Id: I8ec5c07b73968b98d924a41e5d41ddb4d7be1ced
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
When producing a diagnostics block, include the file and line
information, if we have it, to describe it. This presently only adds
this information for skip, but could in principle do the same for a
B?XPass.
Task-number: QTBUG-96844
Change-Id: I6cc375d98e2369eba262010f9c2dfbcba931a6f1
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Our TAP output was delivering messages as comments before the test
line, where TAP clearly expects the details of a test to follow its
test line. Version 13 provides a YAML block to deliver diagnostics and
encourages use of this, so accumulate our messages in a
QTestCharBuffer instead of emitting them one by one.
However, messages produced after a test has produced its test line
belong to that test, but are too late to be included in its
diagnostics block, so should be emitted immediately as before, albeit
now with a type prefix. This at least separates such messages, from
the end of one test, from messages produced early in the next.
In the process, add a type-prefix to each, to make clear what type of
message it was. Since the Yamlish supported by TAP consumers doesn't
support a way to have many messages, use the extensions: top-level
hash tag with a messages: sub-tag to gather our messages as a list.
(This expands at least one expected output file significantly and
substantially rewrites some others.)
Add methods to QTestCharBuffer, and a helper function, to support this.
Task-number: QTBUG-96844
Change-Id: If44a33da5879ed1670ef0980042599afd516f9d2
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Global functions need an associated header file/class/namespace to make
the documentation visible.
Remove QDoc comment tag (/*!) for an internal function.
Pick-to: 6.3
Task-number: QTBUG-99578
Change-Id: Id390d433d7e081fc90fa57a02097b6c37fa13bee
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Reviewed-by: Kai Koehne <kai.koehne@qt.io>
Models are expected to return Qt::Alignment and Qt::CheckState
when asked for the text alignment / check state roles, respectively.
However, for legacy reasons, models may return `int` for either
possibility. Extend the tester to cover this.
Task-number: QTBUG-75172
Change-Id: I64bfc898466e8bb4d4a8a35726986ae610a8ffbe
Pick-to: 6.2 6.3
Reviewed-by: Richard Moe Gustavsen <richard.gustavsen@qt.io>
... along with how QtTest soon shall be using it.
Change-Id: I9c723ebdb8ee5be1c4152aa911ae7c846ea61b5c
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
We only need to look at the first byte to determine whether the length
is > 0; no need to count how many more bytes after it aren't '\0'.
Change-Id: Id2db2a9a993086c0aee02c99396fcb346b1a348e
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
When a B?Fail's description doesn't match the QVERIFY/QCOMPARE
regexes, it got output as a comment in the YAML block after the test
line had already reported it as a TODO comment. An empty description
would also have lead to an empty comment in the YAML block. Condition
this fallback output case on there being a description that hasn't yet
been reported in the test line.
Task-number: QTBUG-96844
Change-Id: Id7fe81d26ddb01da3d8003ada8fa590a5e1a166f
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
In all places but one they were treated the same; fix that last one to
match the rest. This removes one line from the YAML block for each
blacklisted XFail test; in each case, this message duplicates the one
on its "not ok ... # TODO..." test line.
Change-Id: Iff2028afccd979db7f2c84e85d1b78541900008e
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reorder the different formats to match the order in the help output
and include missing entries from the list of formats supported by the
-o filename,format variant.
Change-Id: I884f9facc7cfde7cec0bee75944efdd72d66a83e
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Provide basic (internal) documentation of each logging class and link
the command-line documentation to pages relevant to the formats not
defined by Qt.
Change-Id: I3251dd1304203c6ab87dfe1f2dec0e9787ab69f8
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
All callers do in fact pass members of the enum, so just making the
defualt value be LET_Undefined rather than -1 eliminates any need for
it to be an int.
Change-Id: Ic1c9ae1522363666d1208aba60d4b4df5eff6ce0
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Previously the blacklisting file format only worked with slot:data or
plain slot names for the items to blacklist. However, tests with
global data report themselves with the global data-row tag in the same
way as function-specific ones do; and tests which have both join the
two as slot(global:data) in the test output name, so the reader is apt
to mistake global:data for a data tag. In any case, it is potentially
desirable to be able to blacklist a function with either or both of
global and local data-row tags specified. Add support for that and
remove a blacklisting that was only needed due to the lack of this
support.
For now, make the new parameter to checkBlackLists() optional, so
that qtdeclarative's qmltest framework can adapt to this change.
Fixes: QTBUG-100870
Change-Id: I9125811ebdab75d3fb462ba8b60561f003426502
Reviewed-by: Pasi Petäjäjärvi <pasi.petajajarvi@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
By changing it to unique_ptr.
Pick-to: 6.2 6.3
Change-Id: I91abb69445b537d4c95983ae735341882352b29d
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Also include the comprehensive tests for bool cast compilation which I
originally wrote to confirm that the QTEST_ASSERT() change should be™
correct.
Pick-to: 6.3
Task-number: QTBUG-101406
Change-Id: I9a2871bfd4be9999b7a720bec775bba7aeffbe24
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Giuseppe D'Angelo <giuseppe.dangelo@kdab.com>
In preparation for adding it to headersclean.
Some remarks:
* QStandardItemModel builds just fine (QFlags has comparison operators
against literal zero); the warning we however get is about 0 converted
to a null pointer constant. There's nothing we can do about that one
(even <compare> gives such a warning).
* Several code was depending on flags->int conversions. Add toInt(),
but also cast again to the expected type to avoid warnings in case
toInt() returns unsigned int.
* Ported to explicit casts to bool rather than test(Any)Flag to minimize
confusion for people unfamiliar with the test*Flag methods.
Change-Id: I5be280ac33a0b38e2680096f0e79129fd55ba241
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
QT_TYPESAFE_FLAGS allows explicit casts only to QFlags<T>::Int, which
is either int or unsigned int. The cast to the resp. other type fails.
To fix, first convert to QFlags<T>::Int with toInt(), and only then
cast to int or unsigned int.
Fixes: QTBUG-101399
Pick-to: 6.3
Change-Id: Ie74d53adc601cdf19708265b040092780676058f
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Giuseppe D'Angelo <giuseppe.dangelo@kdab.com>
We don't currently have a use for the extra parameters, but Samuel
Mira's fix to ensure we don't miss fatal signals on Android needs to
be able to call the prior handler, which may need these parameters.
In the process, check returns from sigaction() and use its nullptr
semantics to query first when detecting whether to restore the default
when done, rather than setting and then restoring if it turns out
we've been replaced in the meantime.
Task-number: QTBUG-97652
Pick-to: 6.2 6.3
Change-Id: If30a0db35946c3dceb409ae3f41aa437149472e6
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
See script in qtbase/util/includeprivate for the rules.
Since these files are being touched anyway, I also ran the
updatecopyright.pl script too.
Change-Id: Ib056b47dde3341ef9a52ffff13ef677e471674b6
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
This patch adds checks to QAbstractItemModelTester checking
that only one change is in flight at a time.
Change-Id: I15fe1ad1274649aa652ed6489a889df72c8a8292
Reviewed-by: David Faure <david.faure@kdab.com>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Needed to prevent qasciikey.cpp from failing an assertion when
Qt::Key_Select is used in tests.
Change-Id: I56aa517e8f6f332636b3df106d7265677d1b33ee
Reviewed-by: Richard Moe Gustavsen <richard.gustavsen@qt.io>
Even QtCore alone cannot be built without the properties feature since
Qt 5.5. While fixing this is easy, other modules like dbus,
networking are also using QObject::property() and friends liberally.
All in all I doubt that anybody will miss the feature (otherwise it
would have been fixed in the last decade).
Change-Id: Iaf3cc20bda54ee2ff3b809fac8fa82b94ecc88c0
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
If the CSR_ALLOW_UNRESTRICTED_FS bit of System Integrity Protection (SIP)
is enabled lldb will fail to print a valid stack trace when launched from
the crashed process, and might also resulting in hanging or crashing the
parent process (Terminal e.g.):
https://github.com/llvm/llvm-project/issues/53254
We detect this situation and avoid printing a stack trace if so.
Pick-to: 6.3 6.2
Change-Id: Iad8cab5fcdc545d810ca4d4e985aefc0988d0234
Reviewed-by: Timur Pocheptsov <timur.pocheptsov@qt.io>
Rename the `failureLocation` variable to `messageLocation`, as
it does not necessarily contain information about a failure.
Change-Id: I9e78dc76c8111c78e34c36f6623e9a8cc9e4764e
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
When running a test under increased verbosity levels, QTestLib would
print "failure location" for every QTest function (QVERIFY, QCOMPARE,
...), even if there's no failure at all.
Keep the code centralized, but split the formatting of failures and
non-failures (other messages).
Task-number: QTBUG-96844
Change-Id: I3c508653176b68579dc0eb0cffcc153a52da2e2c
Pick-to: 5.15 6.2 6.3
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Remove a couple of unnecessary string literals that only coped with an
impossible situation (default return after a switch that cover all the
enumerators).
Add a Q_UNREACHABLE() for good measure.
Change-Id: I8065218554edf98408d9c0823b8c301932681c59
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The recommended way to register defaults on macOS is via registerDefaults:
which puts the key/value into the volatile NSRegistrationDomain.
This stops testlib from dumping preference files into ~/Library/Preferences:
❯ plutil -p tst_qwidget.plist
{
"ApplePersistenceIgnoreState" => 1
}
Pick-to: 6.2 6.3 5.15
Change-Id: I4ac812014f9ff97bc446806eaf2108cd1c8b89c3
Reviewed-by: Timur Pocheptsov <timur.pocheptsov@qt.io>
Added tests for repeated skips and failures (from within void lambdas,
to simulate skips and failures from within event handlers). These
exhibit yet more ways to count more than one outcome for a test. The
new QTest::failOnWarning() can also provoke more than one failure from
a single test, and several existing selftests exhibited various ways
for the Totals line's counts to add up to more than the number of
actual tests run.
Fixed counting so that only the first decisive incident is counted.
Tests can still report later failure or skipping, but only the first
is counted.
Added a currentTestState in qtestlog.cpp, by which it keeps track of
whether the test has resolved to a result, and clearCurrentTestState()
by which other code can reset that at the end of each test. This
brought to light various places where test-end clean-up was not being
handled - due to failure or skipping in a *_data() method or init, or
a skip in cleanup.
Fixes: QTBUG-95661
Change-Id: I5d24a37a53d3db225fa602649d8aad8f5ed6c1ad
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
...fixing tst_QLocale:
QDEBUG : tst_QLocale::formatDate() expected data of type 'QString', got 'char8_t [64]' for element 1 of data with tag 'unicode in format string'
QFATAL : tst_QLocale::formatDate() ASSERT: false in file /home/marc/Qt/qtbase/src/testlib/qtestdata.cpp, line 105
In C++20, the type of u8 string literals changed from const char[] to
const char8_t[].
[ChangeLog][QtTest][QTestData] Fixed streaming of u8 string literals
in C++20 mode.
Pick-to: 6.3 6.2 5.15
Task-number: QTBUG-99480
Task-number: QTBUG-99481
Change-Id: I53bd73522a4ebce307f360e225f25590f981d44a
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Added support for blacklisting autotests for webOS platform
since it looks like we might have to blacklist some of those.
Task-number: QTQAINFRA-4706
Change-Id: I212be15161bd042f1511000ced2c8c022c3206e1
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Also regenerate the expected output for tst_selftests, to match new
output. Changed one line source code in tst_seftests for the
same purpose.
Change-Id: I930ba4bb290568d6f67a8910a781725f01f08bf1
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Using REQUIRED as a prefix instead of suffix works better with
OPTIONAL_COMPONENTS, and is also the order in the CMake manual.
Task-number: QTBUG-98867
Pick-to: 6.2
Change-Id: I1ab68408b95d8edf06272a3b9fceccd8d8e597fc
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
No need to wrap the QLatin1String in a QString() to get at arg now.
Change-Id: Iac90e052b88077b9c2bd3e7eca6bc31935baa4b8
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
After we've checked m_wasExpectedFail, we might as well set it
promptly, since one of our earlier checks is on the same
condition. Explain what that check is about, fix a typo in another
comment and make the local ok variable const, since it can be.
Change-Id: Ie04c0ad4248e47b116d3e22046116d39cf2ac6df
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
If an incident has an empty description we produced output with a
dangling space. Avoid triggering the inanity 'bot with those.
Reorganise the code to save some duplication.
Change-Id: I1dc29fa8ad4449a4584f11dddcf002e405cd9238
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The subclass's code can refer to the base's public enums without
needing the (long) class prefix.
Change-Id: I4824440d216695aee9345776faba2e510130425e
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The rationale for skipping pass after XFail is "to emit a single test
point for" the test; emitting several XFails violates that aim.
Task-number: QTBUG-96844
Change-Id: Ia8626dfc2dded234b3aa530fc2dc2324f1e28400
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
It makes no sense to include output for a blacklisted test that would
have been excluded had it not been blacklisted.
Task-number: QTBUG-96844
Change-Id: I164e0c26a921101cbf8986904851260d181be00f
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Skip ends the test (albeit inconclusively). Rearrange the enums in
the abstract logger, move code to handle skip between relevant
function and tidy up various things that became simpler as a result.
Also reorder the message enum, and its switches, to separate testlib's
internals from the usual Qt messages, and put each group in ascending
order of severity.
Task-number: QTBUG-96844
Change-Id: I2c7a634b9f849830d64eafa750155e66e244b729
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
A comment on QTRY_IMPL() helpfully mentioned the bug number of the
issue that prevented us from using qWaitFor; and MSVC < 2017 are no
longer supported so it looked possible to make the change suggested.
However, the implementation has, in the mean time, grown some features
not (currently) provided by qWaitFor(), so update the comment.
In the process, fix one minor error in QTRY_TIMEOUT_DEBUG_IMPL() where
parentheses enclosed the wrong text; (2 * timeoutValue) would not
produce the intended result if timeoutValue were 1 + 500, for
example. The macro parameter should be enclosed, not the expression
using it as parameter to a further macro.
Task-number: QTBUG-59096
Change-Id: Ie16e297def9d1f983cd5964bc628b2c3c0aaecda
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The various variants duplicated some rather complex code around
varying setup in the middle. Rework in terms of a macro that defines
main() and takes the setup code as a parameter. That setup code also
had some common structure, so package that in a setup macro that takes
the class to be used.
Reworked various testlib selftests that were using QTEST_MAIN_IMPL();
change to use the new QTEST_MAIN_WRAPPER() and TEST_MAIN_SETUP().
These might be better dealt with by supporting a second form of the
initMain() test-setup function in the test classes, that takes
references for argc and argv, to let a test massage its command-line
options.
Change-Id: I7fb16b38d51c80ba2f5c9c82f3b7a37ffc636795
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Invert a condition to turn a nested if (that was missing one of its
layers of braces) into an else-if chain, split some long strings to
limit line length.
Change-Id: I10d90487a09affe981aa11c3588281aeb3666df5
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
When recursively checking children in QAbstractItemModelTester,
the currentDepth of recursion was incorrectly increased.
This patch fixes it.
Pick-to: 6.2 5.15
Change-Id: Ia79508f358e2fcb6b90780a79baec053522e871c
Reviewed-by: David Faure <david.faure@kdab.com>
When running a test using -callgrind, we recurse into a child process,
run under valgrind, to which we pass -callgrindchild; and we only want
the output from the child process, since the parent won't actually be
running any tests. We also won't be using the global data table for
the test in the parent process. So bypass the set-up and tear-down of
both logging and the global data table in the parent process.
Prior to commit 3ee6d8d336, these parts
of the set-up and tear-down were skipped in the callgrind parent
process, but that refactoring split qExec() up into qInit(), qRun()
and qCleanup() to enable QtQuick to recombine them to implement an
equivalent of qExec(), calling qRun() once for each built-in style. It
needs these pieces of set-up to happen in qInit(), and of tear-down in
qCleanup(), to avoid repeating them for each style. Leave a comment in
qExec() that might help future readers to understand why it's done the
way it is.
Change-Id: Ieaca9a125c713b8fcf8dec8f9be0c024a798d504
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Using partially-converted text would lead to invalid XML, so don't use
the buffer contents if the return is zero. As a result,
QTestJUnitStreamer::formatEnd() needs to return some indication of
whether *it* succeeded, so change it to forward their int returns;
and, as it's in fact only used internally by the streamer, make it
private.
Make these functions [[nodiscard]] so that further uses of them will
be discouraged from ignoring the possibility of failure. Make the
public versions return bool so that they can correctly succeed on
empty input. Assert various conditions we can infer to save work we
don't need to do.
Change-Id: I899bad23d1dfbd05fc725de269def4ce213dbc5a
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Extract Method QTest::qCaught() to take the string handling out of the
header. This should help a bit in speeding up compilation of large
unit test files (provided they use QVERIFY_THROWS_EXCEPTION), although
I have no data to support that.
Since we changed the error message, update the selftest accordingly.
Change-Id: Id4a3c8c34d5df8d0c7a861106d269097f4a6de5c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This solves the long-standing problem of not being able to easily
fail a test when a certain warning is output.
[ChangeLog][QtTest] Added QTest::failOnWarning. When called in a test
function, any warning that matches the given pattern will cause a test
failure. The test will continue execution when a failure is added.
All patterns are cleared at the end of each test function.
Fixes: QTBUG-70029
Change-Id: I5763f8d4acf1cee8178be43a503619fbfb0f4f36
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Use variable args macros to swallow any extra commas in the
expression. To use this, the type of the exception has to be first.
Use Eddy's suggestion for a new name to avoid breaking the old macro.
[ChangeLog][QtTest] Added QVERIFY_THROWS_EXCEPTION, replacing
QVERIFY_EXCEPTION_THROWN, which has therefore been deprecated.
Change-Id: I16825c35bae0631c5fad5a9a3ace4d6edc067f83
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Swallowing unknown exceptions is dangerous business, as the exception
might be a pthread cancellation token, the swallowing of which would
terminate the program.
Instead of returning from the catch-all-clause, therefore, re-throw
the unknown exception.
Fix tst_verifyexceptionthrown failure cases that use
non-std::exception-derived true negative exceptions to not let the
exception escape from the test function.
As a drive-by, pretty up the macro's docs.
[ChangeLog][QtTest][QVERIFY_EXCEPTION_THROWN] Now re-throws unknown
exceptions (= not derived from std::exception) (was: swallowed them
and returned from the test function), in order to play nice with
pthread cancellation.
Pick-to: 6.2 5.15
Change-Id: Ic036d4a9ed4b7683fa67e27af8bcbae0eefdd0da
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Unit tests should not fail, so a call to qFail() is by definition
exceptional. Therefore, mark the function as cold. It probably doesn't
matter in the grand scheme of things, but it also doesn't cost much.
Change-Id: I0cafcfa65ff285812155d0687deded8d26cf4efd
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The stack needs to be at least SIGSTKSZ, which isn't constexpr, so we
can't allocate it at compile time. However, we can resize(SIGSTKSZ) a
QVarLengthArray that's probably big enough anyway. At the same time,
increase the compile-time size to 32k, to match what our Catch2
harness for the self-test uses.
Change-Id: I3a34ece73901dd402672cd6fe4da66923f1932c8
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Shrinks the const array by one zero-terminator and makes the code
accessing it read more gracefully.
Change-Id: I4034116a83ff3cd05ea0feb0ce8a4340c54a9faa
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
If we fail to install our own action, the contents of oldact are noise
anyway, so we can't use them to decide anything; and, in any case, we
didn't replace the prior action, so have no duty to restore it.
In the process, restructure and comment the code to make it easier to
follow what's going on and why. Package a conditional in a lambda to
make the #if-ery less problematic. This should also make it easier to
hack any other complications into the condition, should they arise.
Change-Id: I712335ee27f68a8594dc2fe6441a91f686967da2
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Previously, the loggers would leak if the application failed to call
stopLogging(). Now they are owned by the global static which will delete
them in that case.
Also, since we have to adapt loggerCount() to the fact that std::vector
uses size_t, recognize that we only ever want to know whether the number
of loggers is 0. Change the method to only provide that information
rather than the actual number.
Change-Id: Ieb2e185048d573ec7f36373ad49bb2a0ca391ce3
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Don't rely on transitive include from qobject.h, which will go away.
Change-Id: I99dd97ff4fb1d0632d040daab0bffa2d7b85d3ae
Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
We cannot assume that calling QCursor::setCursor results in a mouse move
event, and we can definitely not assume that the previously pressed
button (via QTest::mousePress) will be included in such an event.
Instead, follow the mechanism used in the QWindow-overload to keep track
of the mouse buttons pressed, and make those the buttons pressed in the
mouse move event we generate.
[ChangeLog][QTestLib] QTest::mouseMove no longer moves the mouse cursor
via QCursor::setPos, but instead generates a QEvent::MouseMove. Testing
of code that relies on QCursor::pos needs to be done with explicit calls
to QCursor::setPos.
Change-Id: Ia643bcc999498a0dc93479b77e107b989dfe202d
Reviewed-by: Jason McDonald <macadder1@gmail.com>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This should restrict changes that
introduce new warnings.
Change-Id: I7e4b5d9d5d84b7c336509c380bc7e6d86e360f4a
Reviewed-by: Topi Reiniö <topi.reinio@qt.io>
The various macros that implement a test's main() call a test object
initMain() static function, except for QTEST_APPLESS_MAIN. Add it to it.
[ChangeLog][QtTest] initMain() is now also supported when using
QTEST_APPLESS_MAIN.
Change-Id: I46b1eae5ce3bb8c9a5aca525f193c9250e520c6d
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The comment against the use of this claimed qvsnprintf() lacks 64-bit
support, which is probably untrue by now but certainly irrelevant as
the value being formatted is a qreal, not a 64-bit integral type.
Since %g wants a double, make the value passed explicitly double by
casting its denominator. Casting its numerator to qreal was
superfluous as that's its type already.
Change-Id: I5ff885fbeb9b638b2b0507061e0a19e3b8522143
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Make the overloads taking a size private (they're only called by the
implementation of the others) and have them take qsizetype for the
size, rather than size_t. Make the still-public ones consistently
return 0 on failure and move the documentation to them.
Change some spurious checks (QTestCharBuffer::size() is necessarily
positive) to assertions, use '\0' for the nul character to make it
visually distinct from the integer 0. Use a do..while loop instead of
break-ing out of the tail of a for (;;) loop. Remove spurious extra
nul-termination - the function taking size alread did that, so the
wrapper to size-expand needn't. Minor style cleanups.
Change-Id: I7440f3d9be5e3b2dd88c2aad62bcd1af9fc02278
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The extern declaration for filter_unprintable() was nowhere used and
no such function is anywhere defined.
The for (;;) loop with lots of reasons to break; was better structured
as a do {...} while loop. A comment on -1 as return from qvsprintf()
was misplaced. The '\0'-termination after calling qvsprintf() was
redundant as vsprintf() reliably '\0'-terminates anyway. Turned a
static const into a constexpr. Assert size (a QTestCharBuffer
necessarily has size >= 512).
Change-Id: I5b7729b9bd66fea0ee7ce3e7cfdde6770f10b36c
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Task-number: QTBUG-96844
Change-Id: I9af60ef0cf3def699e58b3477b647f2e071afa40
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Paul Wicking <paul.wicking@qt.io>
Put stars after spaces, not before; indent cases level with switch.
Change-Id: I1dc29d0729d5be449b8123e15510177d814d42da
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Provide the information an implementor needs in order to be able to
implement an logging backend.
Change-Id: I7bb522de57149dad843e01a8afa9f8891d011d29
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The one place that actually called it with file and line number was
internal to QtTest, so could call QtTestLog::warn().
Change-Id: I8be10fc81996cc3de4d2eea3ecedc59671ffe759
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
It's the recommended replacement when file and line aren't known, and
simplifies the code in any case, thanks to taking format and varargs.
It incidentally also enables tests to tell QtTest's custom message
logger to anticipate its warnings, when they arise.
Fixes: QTBUG-67702
Change-Id: I10c883f4dd0b69dca6e59e0dbd55ab318f1938f4
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
Reviewed-by: Shawn Rutledge <shawn.rutledge@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Calls to QTestResult::setCurrentTestData() are handled via an RAII
class, QTestDataSetter, that ensures this final call passing nullptr
was redundant.
Change-Id: I7788e3ce898e5628d5590ef9d34182f05e85fe50
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The indirect include is not available everywhere
Amends b8191f41c6
Pick-to: 6.2
Change-Id: If0abf3b0186594ff5381dab847cbdf13e4fcf448
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The custom linked list implementation was implemented using
recursion, and as a result didn't handle long lists of test
cases, exhausting the stack on e.g. Windows where the default
stack is only 1MB. This was the case with e.g. the tst_QChar
test that produces 20K test cases.
Replacing with a std::vector should do nicely for our use-case.
No attempt has been made at further reducing the complexity
of QTestElement/QTestCoreElement/QTestElementAttribute.
Pick-to: 6.2
Change-Id: Ie295f7cf937ec6abdc4606b6120818551ad285c7
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
customFilters defined in .qdocconf are not supported anymore by Qt
Assistant since Qt 5.13. Therefore remove them from all .qdocconf files,
also to avoid cargo-culting them to new help modules.
Task-number: QTBUG-95987
Change-Id: I664391460637d2e859348da0338e1a4a3ee9f570
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Paul Wicking <paul.wicking@qt.io>
This reverts commit 904617dfb8 and makes
clear to future readers why that wouldn't be a sensible change.
Update the test's data, eliminating a case of duplicate counting that
was caused by the reverted commit.
Task-number: QTBUG-95661
Pick-to: 5.12 5.15 6.1 6.2
Change-Id: Ice6d3ab06ca171e0d6eb0fac757f1ab774e229f0
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Present behavior is to continue or abort, according to the mode,
regardless of whether the condition marked does fail.
Change-Id: I2eb5eefbbf173326101d08f3bbb378d214e93e71
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Paul Wicking <paul.wicking@qt.io>
Use \note instead of \b{Note:}, don't refer to functions as macros,
and don't lie. The data-table set-up functions can delegate their
calls to addColumn(), addRow() and newRow() to functions not called
directly by the test framework.
Change-Id: I3bef22cf1a45df91dc822ac04f07e7d15053825f
Reviewed-by: Paul Wicking <paul.wicking@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
The "documentation" has also been removed for all types, as it didn't
add anything, and maintaining it to be in sync with the values just
adds overhead.
Pick-to: 6.2
Change-Id: Iab1dd7b9804c05559b4674342d6059b68ecf3ecf
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
None of the loggers rely on this at the moment, but one could imagine
one that does. Plus, doing any sort of debug logging in leaveTestFunction
will at the moment be attributed to UnknownTestFunc().
Pick-to: 6.2
Change-Id: I284b2785a276e028b9f57c26357679fd9e045ca7
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The separation between <system-out> and <system-err> is sufficient, and
we can't expect consumers to interpret our custom comment format.
The type of the text node has been renamed to more accurately identify
its purpose.
Pick-to: 6.2
Change-Id: I63c8ff17529fc087e1b695698350a6711eb5e68d
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
It's used to handle qCritical() messages, and corresponds to QtMsgType's
QtCriticalMsg, so change its name to reflect what it really is.
In the process, reclassify the -maxwarnings overflow message as a warning,
rather than a critical message.
Pick-to: 6.2
Change-Id: I87626117a547ae4498d5dc352b93bd6db8bfb332
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The original Ant JUnit reporter only writes <system-err> and <system-out>
to the <testsuite>, but more modern reporters such as Maven Surefire
scopes output to each individual <testcase>.
This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.:
https://github.com/jenkinsci/junit-plugin/commit/145eb5c98
Pick-to: 6.2
Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Test errors represents unanticipated problems, e.g., an unhandled
exception, or a problem with the implementation of the test.
Pick-to: 6.2
Change-Id: I87219e7ffdea56862278f005de44526ad97545f0
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
We don't produce <failure> elements for them, so we shouldn't include
them in the total count of failures.
In the future we might produce <skipped> elements instead, but for now
remove the incorrect increment.
Pick-to: 6.2
Change-Id: Ia89a5cdaf79f6aa2f92be53180112c40ad6da2c6
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The data itself is not indented, as consumers may read it as verbatim
data.
Pick-to: 6.2
Change-Id: Ia934616cea273feadc3a45d7c74726d4f804f0dc
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The Apache Ant and Surefire Maven specs document a <skipped> element that
can be used to signify skipped test, with a corresponding total skipped
test attribute on the <testsuite>.
The element includes an optional message attribute, documented in the
Surefire spec, and in the Ant source code, but not yet documented in
the reverse-engineered Ant spec:
https://github.com/windyroad/JUnit-Schema/pull/11
Pick-to: 6.2
Task-number: QTBUG-95424
Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The errors attribute on the <testsuite> element represents the number of
<error> elements, but we do not produce any at the moment.
Pick-to: 6.2
Task-number: QTBUG-95424
Change-Id: I7196d622a9a6bbb7e79ed2c2886984d539abb1da
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
539553a572 renamed the LET_Error element
enum to LET_Message, without renaming the corresponding "error" element
name. This was not an issue in practice, since we never write the actual
element in QTestJUnitStreamer, but the two should be in sync to avoid
any confusion.
Pick-to: 6.2
Change-Id: I6c29f5303393b5f36b2f9877940bf3f6eaf3b7d2
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The original Ant JUnit reporter produced test durations via Double.toString(),
supporting arbitrary precisions, and the de-facto schema declared them
as xs:decimal.
Sadly, the now popular Maven Surefire reporter limited the duration to
millisecond precision, and hard-coded this into its schema as SUREFIRE_TIME:
https://issues.apache.org/jira/browse/SUREFIRE-1533
Unfortunately this definition spread into tools such as the Jenkins xUnit
plugin, which relies on the schema provided by Maven Surefire:
https://issues.jenkins.io/browse/JENKINS-52152
As a result, anything that produces higher precision results will not
validate in the Jenkins xUnit plugin.
Other test frameworks have bitten the bullet and reduced their precision
correspondingly, e.g.:
https://github.com/catchorg/Catch2/issues/2221https://github.com/catchorg/Catch2/commit/581c46249acf8389e9
We follow suit, and our JUnit XML output now validates against both
the Jenkins JUnit and xUnit plugins, as well as the original Apache
Ant de-facto schema.
Pick-to: 6.2
Task-number: QTBUG-95424
Change-Id: I3097d10c03c2a29709960372301b29055d224e10
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The JUnit test framework did not initially have any XML reporting
facilities built in. Instead, the XML report was generated by the
Apache Ant JUnit task:
https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java
Many users interacted with these reports via the Jenkins JUnit plugin,
which provided graphical visualization of the test results:
https://plugins.jenkins.io/junit/
Due to the lack of an official XML schema for the Apache Ant JUnit
report there was some confusion about what the actual format was.
People started documenting the de-facto format, both as produced
by Ant, and as consumed by Jenkins:
https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsdhttps://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd
The XML produced by the Qt Test JUnit reporter was far from these
schemas, causing issues when importing results into tools such
as Jenkins, Allure2, or Test Center.
The following changes have been made to improve conformance:
- The 'timestamp' attribute on <testsuite> is is now in ISO
8601 local time, without any time zone specified
- The 'hostname' attribute on <testsuite> is now included
- The 'classname' attribute on <testcase> is now included
- The non-standard 'result' attribute on <testcase> has
been removed
- The non-standard 'result' attribute on <failure> has
been renamed to 'type'
- The <system-out> element on <testsuite> is always included,
even when empty
- The non-standard 'tag' attribute on <failure> has been
removed. Data-driven tests are now represented as individual
<testcase> elements, e.g.:
<testcase name="someTest(someData X)" ...>
<testcase name="someTest(someData Y)" ...>
<testcase name="someTest(someData Z)" ...>
The resulting XML validates against both the de-facto Apache Ant
'JUnit 4' schema and the Jenkins JUnit plugin schema.
Task-number: QTBUG-95424
Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
- Use the right name for the attribute (AI_Message),
rather than fixing it up in QTestJUnitStreamer.
- Don't pretend that we're adding line and file information,
only to discard it in QTestJUnitStreamer.
- Don't pretend to add benchmark information,
only to discard it in QTestJUnitStreamer.
Pick-to: 6.2
Change-Id: Ib6eadc12300157216fe9c6e8bcfebd7eb8a3ea68
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
It makes no sense for the event loop of a test to keep running after a
test has failed. This lets test code simply use the usual testlib
macros to compare and verify values in asynchronous tests that would
otherwise need to hand-test values and send a signal on failure (that
the main test can connect to an event-loops quit() or equivalent).
For example, QLocalSocket's benchmark simply uses the usual macros,
without doing anything to stop its event loop if they fail, with the
sad result that, when a test fails, it does so repeatedly and then
times out, causing the test program to be killed without running later
tests. With this change, that test code (once converted to use
QTestEventLoop) is able to exit gracefully on the first failure.
[ChangeLog][QtTest][QTestEventLoop] The QTestEventLoop new exits its
event loop as soon as the test is known to be failing.
Task-number: QTBUG-91713
Change-Id: If0d455741668722034906763025dda496d2afbb4
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
We try our best to pass on the file location of a failure, including for
fatal errors, but the reporting or logging machinery should not assume
there is one.
By passing on nullptr for the file location we allow the logging backends
to decide how to handle the situation, e.g. by not emitting extra fields
for failure location.
This effectively reverts c25687fa0b,
in favor of relying on the backends to cope with null filename,
which they already did.
As qFatal uses QMessageLogger, which by default disables file/line
information in release builds, we need to explicitly enable this in
our self-tests, to get uniform test results. Similarly, we disable
file/line info from testlib itself, as reporting Qt internal file
and line information for user diagnostics is less useful. The odd
one out there is qtestdata.cpp, which still ends up in test output
due to using QTEST_ASSERT instead of qFatal for its diagnostics.
Cleaning up that, and unifying how we report testlib issues to the
user, is left for another day.
Pick-to: 6.2
Change-Id: Ib9451b8eed86fe3ade4a4dcaf0037e1a3450321c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The QtTest best practices documentations recommends using output
mechanisms such as qDebug() and qWarning() for diagnostic messages,
and this is also what most of our own tests do.
The QWARN() macro and corresponding internal QTest::qWarn() function
was added when QtTest was first implemented, but was likely meant as
an internal implementation detail, like its cousin QTestLog::info(),
which does not have any corresponding macro.
This theory is backed by our own QtTest self-test (tst_silent)
describing the output from QWARN() as "an internal testlib warning".
The only difference between QWARN() and qWarning(), besides the much
richer feature set of the latter, is that qWarning() will not pass
on file and line number information in release mode, but QWARN() will.
This is an acceptable loss of functionality, considering that the user
can override this behavior by defining QT_MESSAGELOGCONTEXT.
[ChangeLog][QtTest] QWARN() has been deprecated in favor of qWarning()
Pick-to: 6.2
Change-Id: I5a2431ce48c47392244560dd520953b9fc735c85
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The key part of FailureReportingMode::QtTest is that it will result
in a test failure (via QTest::qVerify); not that it's using the
QtTest logging machinery.
Pick-to: 6.2
Change-Id: I18472fba173ce1323748f1f87301a4defa87020e
Reviewed-by: Giuseppe D'Angelo <giuseppe.dangelo@kdab.com>
The format was renamed in 27db9e458c.
Pick-to: 6.2
Change-Id: I53975c7467d8768dc9dc9ac2d89c42eefa12e22f
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Before this change, next() was the only way to advance the iterator,
whether the caller was ultimately interested in just the filePath()
(good) or not (bad luck, had to call .fileInfo()).
Add a new function, nextFileInfo(), with returns fileInfo() instead.
Incidentally, the returned object has already been constructed as part
of advance()ing the iterator, so the new function is faster than
next() even if the result is ignored, because we're not calculating a
QString result the caller may not be interested in.
Use the new function around the code.
Fix a couple of cases of next(); fileInfo().filePath() (just use
next()'s return value) as a drive-by.
[ChangeLog][QtCore][QDirIterator] Added nextFileInfo(), which is like
next(), but returns fileInfo() instead of filePath().
Change-Id: I601220575961169b44139fc55b9eae6c3197afb4
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Take two. f20edffcc8 only disabled the
backtrace preamble.
Pick-to: 6.2
Change-Id: I972d007af59d13ec6eb11da44fc2b5e25c36e46e
Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
Thiago Macieira
Andreas Buhr
Lucie Gérard
Alexey Edelev
Marc Mutz
Sona Kurazyan
Florian Bruhin
Edward Welbourne
Allan Sandfeld Jensen
Ivan Solovev
Topi Reinio
Giuseppe D'Angelo
Mårten Nordheim
Samuel Mira
Noah Davis
Kai Köhne
Tor Arne Vestbø
Janne Juntunen
Dimitrios Apostolou
Mitch Curtis
Marc Mutz
Ulf Hermann
Volker Hilsheimer
Venugopal Shivashankar
Assam Boudjelthia
Alexandru Croitor
Nico Vertriest