Commit Graph

428 Commits (8d728a0ed9c1fb366c64babc1f753b5ea77c2cdf)

Author SHA1 Message Date
Mikolaj Boc 8d728a0ed9 Implement the batch_tests feature
An approach of test batching (joining multiple tests into a single
binary) has been taken, due to long linking times/binary size on certain
platforms, including WASM. This change adds a new feature
'batch_test_support' in Qt testlib. Based on the value of the feature,
test batching may become enabled with the -batch-tests switch.

Batching works for every target added via qt_internal_add_test. When
first such target is being processed, a new combined target for all of
the future test sources is created under the name of 'test_batch'.
CMake attempts to merge the parameters of each of the tests, and some
basic checks are run for parameter differences that are impossible to
reconcile.

On the C++ level, convenience macros instantiating the tests are
redefined when batch_tests is on. The new, changed behavior triggered
by the changes in the macros registers the tests in a central test
registry, where they are available for execution based solely on their
test name. The test name is interoperable with the names CMake is aware
of, so CTest is able to run the tests one by one in the combined binary.

Task-number: QTBUG-105273
Change-Id: I2b6071d58be16979bd967eab2d405249f5a4e658
Reviewed-by: Topi Reiniö <topi.reinio@qt.io>
2022-08-24 02:46:37 +02:00
Edward Welbourne f9b58b5c14 Include current test name in crash reports
Previously a crashing test's output would end with the last completed
test's output followed by a crash report, leading readers
(understandably enough) to conclude that the last-named test is the
one that crashed. In fact the crashing test is typically the next one
in the class definition.

Include the current test function's name (when non-null) in the output
accompanying crash logs. This always goes to stderr so does not show
up in the expected output.

Pick-to: 6.4
Change-Id: Icab0ccd1fe434827ee92459ab0c97f9dc034754e
Reviewed-by: Jason McDonald <macadder1@gmail.com>
2022-08-12 20:32:30 +02:00
Thiago Macieira 37aa9a9cef WindowsFaultHandler: print the backtrace to stderr, not stdout
Otherwise the contents that vary per architecture and build will make it
impossible to self-check (tst_selftests)

Change-Id: Ibcde9b9795ad42ac9978fffd16f2cbc352c3503c
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-07-27 08:43:00 +02:00
Edward Welbourne 281cada9c1 Replace 0 with \nullptr in QTest::currentDataTag()'s doc
The function's return type is a pointer, not an integer.

Change-Id: Iab6686ac7e4e24b9b1bd0127346c5854cf593a57
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
2022-07-11 21:41:51 +02:00
Edward Welbourne 4d88c80fbd Improve formatting of QTest message on missing function
If there were no matches to the name given on the command line, the
message reported included a "Possible matches:" preamble for a list of
functions containing the requested function name. This looked
incongruous when no actual functions matched.

Turn that preamble into an optional parameter to qPrintTestSlots(),
that it'll output before the first match if it finds one, rework
qPrintTestSlots() to package its matching condition in a lambda and
return true if it found any matches.  Change this caller to output a
newline in place of the preamble (which ended in a newline), if no
match was found.

Change-Id: I9716ffa29c3c46e3c7e7fcf25a676c0356dab91c
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Dimitrios Apostolou <jimis@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-07-11 14:42:27 +02:00
Dimitrios Apostolou d23dcc0542 qtestlib: exit with error if a test is invoked with unknown data tag
Previously trying to execute a test function with an unknown data tag
would print an error message but exit with 0.

This patch stores a test failure, and continues trying to execute the
rest of the command line arguments, if any. In the end the process exits
with the usual exit code (number of failed tests) which is now !=0.

Pick-to: 6.4 6.3 6.2
Fixes: QTBUG-24240
Change-Id: Id4d422035f173e01e77ca88028dfd94dc0f9085c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-06-28 18:06:33 +02:00
Thiago Macieira 5e67e7efaa FatalSignalHandler: print some more information from siginfo_t
The siginfo_t parameter allows us to show what process sent a signal or
the crashing address. Additionally, it allows us to determine if the
crashing signal was indeed sent due to a crash.

The selftest tst_crashes produces now:

$ QTEST_DISABLE_STACK_DUMP=1 ./crashes
********* Start testing of tst_Crashes *********
Config: Using QtTest library 6.4.0, Qt 6.4.0 (x86_64-little_endian-lp64 shared (dynamic) debug build; by GCC 11.2.1 20220420 [revision 691af15031e00227ba6d5935c1d737026cda4129]), opensuse-tumbleweed 20220428
PASS   : tst_Crashes::initTestCase()
Received signal 11 (SIGSEGV), code 1, for address 0x0000000000000004
         Function time: 0ms, total time: 0ms
[1]    201995 segmentation fault (core dumped)  QTEST_DISABLE_STACK_DUMP=1 ./crashes

The last line comes from the shell. The code isn't decoded, but on Linux
it's a SEGV_MAPERR. macOS prints exactly the same thing.

I've updated one of the expected_crashes_*.txt output that doesn't seem
possible (the "Received a fatal error" message does not appear in Qt
anywhere).

Pick-to: 6.4
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16ebc8391234f0e2
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2022-06-28 00:39:35 +00:00
Mårten Nordheim 44927b801a QTest: switch some text-conversion functions to use qsizetype
To avoid potential narrowing.

Pick-to: 6.4
Task-number: QTBUG-104125
Change-Id: I37bfc5c49e7c919f5204a76a905758a92527d864
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
2022-06-28 00:39:35 +00:00
Kai Köhne 9d2cc4dd76 Fix typos in docs and comments
Found by codespell

Pick-to: 6.4
Change-Id: Ie3e301a23830c773a2e9aff487c702a223d246eb
Reviewed-by: Nicholas Bennett <nicholas.bennett@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-06-15 21:31:02 +02:00
Edward Welbourne 37bad1f43b Don't exit(1) on unrecognised test function name, just report a failure
This way, if you name several test functions on the command-line,
you'll at least get the ones that do exist run (and you'll be told all
of the ones that don't exist, rather than only the first).

Pick-to: 6.4 6.3 6.2
Change-Id: I14a515fcfacb6ca49e0470b236c05475b25db4f2
Reviewed-by: Dimitrios Apostolou <jimis@qt.io>
2022-06-15 20:23:06 +02:00
Thiago Macieira 17454bf9c6 FatalSignalHandler: print the signal name on crash
It's easier to remember what "SIGSEGV" means instead of "11".

GNU libc has offered sigabbrev_np() (non-portable) since 2.32; for older
libcs, we'll be happy with a hardcoded list.

Selftest updated to match... though it didn't seem to be necessary.

Pick-to: 6.4
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16ebc66ecf6e9465
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-06-08 20:39:18 +00:00
Mårten Nordheim 4d2ef82089 QTest: Remove pre-Qt6 code
It's disabled now, so can be deleted

Pick-to: 6.4 6.3 6.2
Change-Id: I0d548327e7ef42bbca9ed88556bf9f8456038cc7
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-06-08 19:08:49 +02:00
Mikolaj Boc 3df23b1fe4 Define out unused functions on WASM build
The struct iovec conversion functions that are needed on Unix are
unused on WASM build. This makes the build fail with -Werror on Mac
since the WASM build is treated as a variant of Unix. Cross-compilation
with clang:
Apple clang version 13.0.0 (clang-1300.0.27.3)
Target: arm64-apple-darwin21.3.0

Fixes: QTBUG-103974
Pick-to: 6.3 6.4
Change-Id: I34c65a18832ceedb9064a98f5729e45667749461
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2022-06-08 08:30:45 +00:00
Ivan Solovev 78c677376a QTestLib: improve documentation
Add missing '\since' version numbers.

This commit amends cc6d984390
and 0681a2dd5a

Pick-to: 6.4
Change-Id: Ia10b991c996fb58f08a17e485c4dfcbfbe8eba0a
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-06-08 10:30:20 +02:00
Thiago Macieira 8b26233573 qtestcase.cpp: create a common function to print the test runtime
The time was getting printed twice for the stack trace, but zero for the
watch dog if the stack trace was disabled.

Selftest appears to pass (though I thought it shouldn't).

Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16ebc4ec99e7fc5a
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
2022-06-03 16:52:41 -07:00
Ivan Solovev 0681a2dd5a QTestLib: rework QTest::compare_helper()
[ChangeLog][QTestLib] QCOMPARE now evaluates toString() on its
arguments lazily, speeding up the general case where the comparison
doesn't fail. This is true for the QCOMPARE functionality provided
by Qt. If you specialized qCompare() for your own types, then you
need to change its implementation in line with Qt's own qCompare()
specializations in order to enable this feature.

[ChangeLog][QTestLib] QCOMPARE calls with nullptr argument(s) will
now print the actual and expected values upon failure.

Previously it was not like that because of the compareHelper()
overload in qtestresult.cpp that treated the presence of
nullptr-arguments as a reason to ignore formatFailMessage() call.
New implementation does not have this check, and correctly
executes formatFailMessage() for all arguments.

Note that the qCompare() overloads that call QTestResult::compare()
internally were not affected by this patch, because they already
defer toString() invocation until the comparison fails.

Some numbers, collected against shared release developer build.
I checked how this change affects the test execution. The idea was
to pick some tests for types that do not have a specific
QTestResult::compare overload, so I picked a couple of QByteArray
tests.
The comparison is done by running a test 10 times and taking the
average execution duration, as reported in the log.

tst_qbytearrayapisymmetry:
 Before: 15.6 ms
 After:  14.2 ms

tst_qbytearray:
 Before: 41 ms
 After:  36 ms

The benefit is around 9% and 12% respectively.

Fixes: QTBUG-98874
Change-Id: I7d59ddc760168b15974e7720930f629fb34efa13
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
2022-06-03 20:37:51 +00:00
Ivan Solovev 343e0ff485 Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}()
[ChangeLog][QTestLib] Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}()
macros. These new macros behave similarly to QVERIFY(a op b),
where 'op' is ==, !=, <, <=, >, >= respectively, but print
a formatted error message with argument values in case of failure.
The formatting is done lazily, which means that the strings will
be generated only when the comparison fails.

Also add a new test for tst_selftest and generate expected output
for it.

Fixes: QTBUG-98873
Task-number: QTBUG-98874
Change-Id: Ic8074798901d7a469b1f58d5cd28bbf49a3da1db
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-06-03 22:37:50 +02:00
Thiago Macieira dface58142 QtTest: move the disabling of core dumps from load-time to qInit()
There's no reason it has to be done THAT early. It was added in commit
aec85a53df to have the selftests (which do
crash) not leave lots of core files around. I could replace it with a
QProcess::setChildProcessModifier() function, but the variable is now
documented in the QtTest manual, so it would be no gain.

Function renamed to reflect its implementation.

Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16ebc25df97a98e3
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-06-01 22:12:17 -07:00
Thiago Macieira 92046dde55 FatalSignalHandler: handle SIGABRT too
Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb77e9c77d10ba
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-05-30 21:27:03 -07:00
Thiago Macieira 7ad8a2e8fe FatalSignalHandler: expand to more Unix OSes than just Linux and macOS
At least for my FreeBSD this makes sense. The default debugger search is
gdb first then lldb on Linux-not-Android and for QNX (qcc is GCC after
all), but lldb first everywhere else.

With LLVM14 from Ports, I get:

$ tests/auto/corelib/tools/qhashseed/tst_qhashseed | head -1
********* Start testing of tst_QHashSeed *********
Config: Using QtTest library 6.4.0, Qt 6.4.0 (x86_64-little_endian-lp64 shared (dynamic) debug build; by Clang 14.0.0), freebsd 13.0
=== Received signal at function time: 1ms, total time: 3ms, dumping stack ===
(lldb) process attach --pid 1782
Process 1782 stopped

Executable module set to "/usr/home/tjmaciei/obj/qt/qt6/qtbase/tests/auto/corelib/tools/qhashseed/tst_qhashseed".
Architecture set to: x86_64--freebsd13.0.
(lldb) bt all
* thread #1, name = 'tst_qhashseed'
  * frame #0: 0x0000000800f227c8 libc.so.7`_wait4 at _wait4.S:4
    frame #1: 0x00000008003243bc libthr.so.3`__thr_wait4(pid=<unavailable>, status=<unavailable>, options=<unavailable>, rusage=<unavailable>) at thr_syscalls.c:581:8
    frame #2: 0x00000008002b9c73 libQt6Test.t.so.6`(anonymous namespace)::StackTraceHandler::generate() at qtestcase.cpp:393:9
[...]
=== End of stack trace ===
Received signal 13
         Function time: 1ms Total time: 3ms

Support for Windows left as an exercise for later. The
WindowsFaultHandler code doesn't even call generateStackTrace().

Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eba471f58ff3cb
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
2022-05-30 15:56:38 -07:00
Thiago Macieira d5f4f91c3c FatalSignalHandler: use mmap() to create the alternate stack
So we can mark the bottom page as inaccessible, thus be able to catch a
stack overflow in the handler itself. Our code shouldn't cause
overflows, but it's possible that a chained handler does more work than
expected.

Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb83a294ab7958
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
2022-05-30 15:56:26 -07:00
Thiago Macieira 9c0c1be122 StackTraceHandler: call the debugger directly, instead of via /bin/sh
This removes one middle-man and a 512-byte variable in favor of a simple
32-bit enum. This was done in a way so we can extend to use either gdb
or lldb in any OS.

I've renamed the debuggerPresent() function to make its meaning clearer.

Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eba561628ff89b
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2022-05-23 14:53:18 -07:00
Thiago Macieira 4ea5e39b27 StackTraceHandler: simplify the gdb and lldb commands
Instead of piping stuff via the shell into them, just use batch mode.
And also take the opportunity to tell them not to read their user-
provided configuration files.

Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eba21e71afaefa
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-05-23 14:53:18 -07:00
Thiago Macieira 3d9e56aa6c FatalSignalHandler: chain back to the original crash handler
If a previous handler was already installed, ensure it is called,
because there may be a reason why it was there. For example, the Android
ART adds a signal action to every fatal signal for logging purposes. We
do that by restoring the signal handler we had and re-raising the
signal.

If our handler was overridden by something else, then that handler was
already called, but will get uninstalled after our code runs. It won't
be a problem, because the application is exiting anyway.

[ChangeLog][QtTest][Behavior Change] On Unix, the QtTest code to
handle Unix/POSIX fatal signals will now call back to the original
handler that was installed, if there was one. This allows logging
frameworks (such as Android ART's), for example, to log the crash too.
Additionally, if there was no handler, the application should exit with
the correct signal instead of SIGABRT.

Fixes: QTBUG-97652
Pick-to: 6.3
Change-Id: Ifc4fca159b490d8d0b614d736e46caefcb903a4c
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-05-20 12:01:38 -07:00
Thiago Macieira 29dd43d5ab FatalSignalHandler: remember the previous signal's handler
And only restore those signals, instead of iterating over all possible
signals, instead of attempting to restore to SIG_DFL. Also, as a
consequence, we will install our handler even if there was already a
handler installed for the signal.

Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb756685f4e8b8
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
2022-05-20 12:01:37 -07:00
Thiago Macieira c4600b2e25 FatalSignalHandler: simplify SA_RESETHAND code
And remove the unreachable code after std::abort() that was meant to
mimic that in INTEGRITY. The next commit will fix this properly.

Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb772018add694
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-05-20 12:01:36 -07:00
Thiago Macieira 3e596da488 FatalSignalHandler: simplify the SA_SIGINFO handling
The #if around the function declaration was ugly.

Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb7540f5da080f
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-05-20 12:01:35 -07:00
Thiago Macieira 65ff731b62 FatalSignalHandler: use std::optional instead of QScopedPointer
We get stack space reserved instead of using the heap.

Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb73fff0174150
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-05-20 12:01:34 -07:00
Thiago Macieira 3bd1aa09b6 FatalSignalHandler: split the Windows and Unix contents
They're very different, so there's no point in having them even in the
same class body. I've renamed the Windows one because Windows does not
report crashes via signals anyway. If you have an IDE that can scan both
branches of the #if, it will help you find the Windows-specific code
too.

Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb73ba196cfb74
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
2022-05-20 12:01:33 -07:00
Thiago Macieira 4e8da8f1ec Put DebugSymbolResolver and FatalSignalHandler in an unnamed namespace
Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb735315b337b6
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-05-20 12:01:32 -07:00
Thiago Macieira 55a4d35dc3 FatalSignalHandler: remove call to qEnvironmentXxx from handler code
The Qt environment handling functions lock a mutex. That's a big no-no
in signal handlers.

[ChangeLog][QtTest][Behavior Change] QtTest will now check the value of
the environment variable QTEST_PAUSE_ON_CRASH in QTest::qRun(), so if a
test wants to modify this variable, it must do so from the main() or
initMain() functions, not in the test itself (including initTestCase()).

Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb78867cd8f54e
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
2022-05-20 12:01:31 -07:00
Thiago Macieira c8d9b7291a qtestcase: make the stack trace generation more signal-safe
This should work so long as there's no async-unsafe pthread_atfork()-
registered callback.

Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb72e4e313bc19
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-05-20 12:01:30 -07:00
Thiago Macieira d75c595eac qtestcase.cpp: fix warning about ignoring the result of writev()
I don't get it on openSUSE Tumbleweed, but this is not the only place I
don't get a warn_unused_result warning and it happens elsewhere.

Change-Id: I7e305799c8594ebab255fffd16ee12ff30df6d4d
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
2022-05-19 14:00:23 -07:00
Lucie Gérard 05fc3aef53 Use SPDX license identifiers
Replace the current license disclaimer in files by
a SPDX-License-Identifier.
Files that have to be modified by hand are modified.
License files are organized under LICENSES directory.

Task-number: QTBUG-67283
Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
2022-05-16 16:37:38 +02:00
Thiago Macieira 046bc971c3 qtestcase/Unix: remove fprintf from signal handler
It's not async-signal-safe, so don't use it. Instead, let's use writev()
(which should write atomically and we won't need to have a buffer) and
std::to_chars to format numbers where available (where not, we roll out
our own implementation with divisions).

Pick-to: 6.3
Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb70a158f44864
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-05-05 20:35:42 -07:00
Thiago Macieira 30067d102c qtestcase.cpp: use #ifdef RLIMIT_CORE to guard RLIMIT_CORE usage
Instead of trying to guess which OSes have this functionality.

Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16ebc20fe7eca346
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-05-04 20:15:36 -07:00
Sona Kurazyan 89225c951e QtTestLib: replace remaining uses of QL1String with QL1StringView
Remove unneeded \fn qdoc lines as a drive-by.

Task-number: QTBUG-98434
Change-Id: Id93ddbb38b97a8f5a6734bfbc82686ccb3a87aa6
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-05-02 20:06:20 +00:00
Sona Kurazyan 7baa06fe0c QtTestLib: replace QLatin1String uses with _L1/_s/QStringLiteral
Task-number: QTBUG-98434
Change-Id: Ie327fd4af1880002e5a1e09b43384f2b709625e7
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-05-02 20:06:20 +00:00
Sona Kurazyan 79cf895c70 QtTestLib: stop using QLatin1Char constructor for creating char literals
Required for porting away from QLatin1Char/QLatin1String in scope of
QTBUG-98434.

Change-Id: I3debae5f481037958bfb65caabca97a0d4681eb9
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
2022-04-28 11:15:11 +02:00
Edward Welbourne 383368a321 Support test-case selection based on global data tag
In the same ways as for blacklisting. In the process, move reporting
of an unrecognised tag outside the global-data loop, as the tag might
be, or start with, a global data tag; and also report the global data
alongside the test-specific data. Indent the lines reporting tags, as
this makes the structure of the lists more evident. Converted the tag
argument to QLatin1String() to facilitate searching and matching.

Updated documentation, including expanding on the option of specifying
the data tags on the command-line. Drive-by: fixed a link in the
documentation, removed a snippet that was nowhere used.

[ChangeLog][QtTest] When specifying what tests to run on a QtTest
command-line, it is now possible to include a global data tag in the
same way as a function-specific data tag or in combination with it.
Thus if the test reports itself as tst_Class::function(global:local),
command-line option function:global:local will select that specific
test and function:global will run all data-rows for the function on
the given global data tag.

Fixes: QTBUG-102269
Change-Id: I7c86d6026933b05f7994bfc3663a2ea6a47f5f38
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2022-04-25 17:47:21 +02:00
Edward Welbourne ec59ae6189 Spacing-only change: fix indents and spacing in qtestcase.cpp
Separated from other clean-ups to avoid inanity 'bot complaints.

Change-Id: Ia33f87da224dba7c50bbf14beb642825dc46fe90
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
2022-04-14 16:58:17 +02:00
Edward Welbourne f3184770e2 qtestcase.cpp: s/QString::fromLatin1/QLatin1String/ where viable
Now that QL1S has its own arg(), we can use that directly instead of
going via QString before calling arg(). Only works in most cases, as
the QL1S::arg() overload set is more limited.

Change-Id: Ie4c566c5998538b1b8a953b1ba4481e0f1663bf7
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
2022-04-14 16:58:17 +02:00
Edward Welbourne 479c95729a Support global data tags in blacklisting identification of test-cases
Previously the blacklisting file format only worked with slot:data or
plain slot names for the items to blacklist. However, tests with
global data report themselves with the global data-row tag in the same
way as function-specific ones do; and tests which have both join the
two as slot(global:data) in the test output name, so the reader is apt
to mistake global:data for a data tag. In any case, it is potentially
desirable to be able to blacklist a function with either or both of
global and local data-row tags specified. Add support for that and
remove a blacklisting that was only needed due to the lack of this
support.

For now, make the new parameter to checkBlackLists() optional, so
that qtdeclarative's qmltest framework can adapt to this change.

Fixes: QTBUG-100870
Change-Id: I9125811ebdab75d3fb462ba8b60561f003426502
Reviewed-by: Pasi Petäjäjärvi <pasi.petajajarvi@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-03-14 20:16:35 +01:00
Mårten Nordheim 034d8898f8 Fix deprecated uses of QScopedPointer
By changing it to unique_ptr.

Pick-to: 6.2 6.3
Change-Id: I91abb69445b537d4c95983ae735341882352b29d
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2022-03-08 15:44:17 +01:00
Samuel Mira 38a86afcc4 Tidy up QTest::FatalSignalHandler
Rename some variables: avoid abbreviated names.
(Don't bother with renaming variables an imminent commit removes, though.)

Pick-to: 6.2 6.3
Change-Id: I3bd11f37ca687047555d43dff83c3c2f935135d8
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2022-03-02 14:58:10 +01:00
Edward Welbourne fcecaf53f5 Adapt QTest::FatalSignalHandler() to support sa_sigaction when available
We don't currently have a use for the extra parameters, but Samuel
Mira's fix to ensure we don't miss fatal signals on Android needs to
be able to call the prior handler, which may need these parameters.

In the process, check returns from sigaction() and use its nullptr
semantics to query first when detecting whether to restore the default
when done, rather than setting and then restoring if it turns out
we've been replaced in the meantime.

Task-number: QTBUG-97652
Pick-to: 6.2 6.3
Change-Id: If30a0db35946c3dceb409ae3f41aa437149472e6
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2022-03-02 14:58:10 +01:00
Tor Arne Vestbø 985f0e9ca5 macOS: Don't print stack trace via lldb on test failure if SIP prevents it
If the CSR_ALLOW_UNRESTRICTED_FS bit of System Integrity Protection (SIP)
is enabled lldb will fail to print a valid stack trace when launched from
the crashed process, and might also resulting in hanging or crashing the
parent process (Terminal e.g.):

  https://github.com/llvm/llvm-project/issues/53254

We detect this situation and avoid printing a stack trace if so.

Pick-to: 6.3 6.2
Change-Id: Iad8cab5fcdc545d810ca4d4e985aefc0988d0234
Reviewed-by: Timur Pocheptsov <timur.pocheptsov@qt.io>
2022-02-09 18:19:57 +01:00
Edward Welbourne 13d2e13290 Make counts of various types of test result add up correctly
Added tests for repeated skips and failures (from within void lambdas,
to simulate skips and failures from within event handlers). These
exhibit yet more ways to count more than one outcome for a test.  The
new QTest::failOnWarning() can also provoke more than one failure from
a single test, and several existing selftests exhibited various ways
for the Totals line's counts to add up to more than the number of
actual tests run.

Fixed counting so that only the first decisive incident is counted.
Tests can still report later failure or skipping, but only the first
is counted.

Added a currentTestState in qtestlog.cpp, by which it keeps track of
whether the test has resolved to a result, and clearCurrentTestState()
by which other code can reset that at the end of each test. This
brought to light various places where test-end clean-up was not being
handled - due to failure or skipping in a *_data() method or init, or
a skip in cleanup.

Fixes: QTBUG-95661
Change-Id: I5d24a37a53d3db225fa602649d8aad8f5ed6c1ad
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
2022-01-13 19:47:22 +01:00
Edward Welbourne b67c367e0e Tidy up testlib's parsing of -callgrind parameter
Invert a condition to turn a nested if (that was missing one of its
layers of braces) into an else-if chain, split some long strings to
limit line length.

Change-Id: I10d90487a09affe981aa11c3588281aeb3666df5
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2021-12-09 20:54:25 +01:00
Edward Welbourne be63c3011b Suppress test set-up and tear-down in callgrind parent process
When running a test using -callgrind, we recurse into a child process,
run under valgrind, to which we pass -callgrindchild; and we only want
the output from the child process, since the parent won't actually be
running any tests. We also won't be using the global data table for
the test in the parent process. So bypass the set-up and tear-down of
both logging and the global data table in the parent process.

Prior to commit 3ee6d8d336, these parts
of the set-up and tear-down were skipped in the callgrind parent
process, but that refactoring split qExec() up into qInit(), qRun()
and qCleanup() to enable QtQuick to recombine them to implement an
equivalent of qExec(), calling qRun() once for each built-in style. It
needs these pieces of set-up to happen in qInit(), and of tear-down in
qCleanup(), to avoid repeating them for each style. Leave a comment in
qExec() that might help future readers to understand why it's done the
way it is.

Change-Id: Ieaca9a125c713b8fcf8dec8f9be0c024a798d504
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2021-12-06 19:06:30 +01:00