As we did for convertToUnicode. To support more than 2Gi input, we
need to handle the input in chunks because of the `int` parameter in the
Windows API. Testing also revealed some corner cases we also need to
handle, which is mostly happening when there is an incomplete surrogate
pair at the end of the current input window.
The test takes between 3 (plain MinGW) and 8 (MSVC with ASAN) seconds
to run on my machine.
Pick-to: 6.7 6.6 6.5
Fixes: QTBUG-105105
Change-Id: I4fb0420b88ca41dfa8b561a35c6d96659bd81468
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
To properly support more than 2Gi input we have to support being asked
to resize more than once. Previously we would only have to resize the
one time because we went from our 4K stack buffer to the final size
heap buffer. But now, since our input size can only be specified in
int, we have to deal with looping over the input and resizing the buffer
as needed.
We also have to deal with trailing data at the end of our sliding window
potentially causing issues for the encoding. So we try to shrink our
window when it causes issues, or store the trailing data for the next
call.
The >2Gi test takes about 6-8 seconds on my machine.
Pick-to: 6.7 6.6 6.5
Task-number: QTBUG-105105
Change-Id: I9a44b8f379bf2c2c58183f961544ed2f4c8c7215
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Qt currently has custom implementation for Indic grapheme breaks.
QTBUG-121907 was created to evaluate whether it is better to use
Unicode implementation instead.
Remove BLACKLIST. The remaining tests pass.
Fixes: QTBUG-121529
Task-number: QTBUG-121907
Change-Id: Ifd8fd76aeeba51d08c9f23b867abbf39750fc7d9
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Add enumerator for the new Unicode version to QChar::UnicodeVersion.
Remap new line breaking classes to their Unicode 15.0 values:
* AK, AP and AS to AL,
* VI and VF to CM.
These are classes for new line breaking support for Indic scripts
that require more work.
Blacklist failing tests for now:
* tst_QUrlUts46::idnaTestV2
* tst_QTextBoundaryFinder::lineBoundariesDefault
* tst_QTextBoundaryFinder::graphemeBoundariesDefault
Regenerate the source files.
Task-number: QTBUG-121529
Change-Id: I869cc9fbaa53765d8ae6265c22cdbef9f19d05bf
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
According to QUIP-18 [1], all tests file should be
LicenseRef-Qt-Commercial OR GPL-3.0-only
[1]: https://contribute.qt-project.org/quips/18
Pick-to: 6.7
Task-number: QTBUG-121787
Change-Id: I9657df5d660820e56c96d511ea49d321c54682e8
Reviewed-by: Christian Ehrlicher <ch.ehrlicher@gmx.de>
QCOMPARE() can report enum values by name just fine, no need to
laboriously convert them to strings. While comparing all tags in one
go did allow a more comprehensive report, it's enough to know we
failed; this is testing cross-platform code, so a debugger can tell us
all those extra details if we get a failure.
Testing qHash() doesn't distinguish equal things is fairly low value;
at least avoid duplicating the construction of the reference value.
Replace a bunch of other QVERIFY()s with the new cousins of QCOMPARE()
for ordered and different comparisons.
In the process, mark some of the QLocale objects as const.
Change-Id: Ic93b8ed60c6f2cc846fbba428983778896d61291
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
They were expanding as simple blocks, so their uses didn't end in
semicolon, which looks wrong when reading the code.
Pick-to: 6.7 6.6 6.5
Change-Id: Ibea7b01ac165045604b6eb7a838765b2061c368a
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
(This turns out to be identical to v44, for our purposes.)
The CLDR license has been revised at v44 to "UNICODE LICENSE V3",
which is now included (as LICENSES/UNICODE-3.0.txt) in addition to the
old license (still in use, presumably, by UCD - at least until its
next update). Some new QLocale::Language entries are needed. There is
no change to the time-zone data.
Some tests needed changes:
* Various Arabic locales now use U+0623 (Arabic letter aleph with
hamza above) in exponent separator, replacing plain U+0627 (Arabic
letter aleph); it is still followed by U+0633 (Arabic letter seen).
* Where likely sub-tags used to fill in world, 001, as territory for a
language, they now (e.g. for Prussian and Yiddish) give specific
countries.
* Tamil locales now have something of a mix of inherited and localized
forms for AM/PM, which looks a lot like a mistake in CLDR.
* New likely sub-tag rules fix ctor(und_US) and ctor(und_GB), which
previously failed.
[ChangeLog][Third-Party Code] Updated QLocale's data extracted from
the Unicode Common Locale Data Repository (CLDR) to v44.1. The license
changed to Unicode License V3.
Pick-to: 6.7 6.6 6.5
Fixes: QTBUG-121485
Task-number: QTBUG-121325
Change-Id: Ide1a68016129526d7a5aa3fc67f1a674858696bc
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Upgrading emscripten to 3.1.50 breaks this test, so we disable the time
for time being. After emscripten update this test is to be enabled
and fixed.
Change-Id: Ic48d81e2285ed8f7639bf20c6c29b2b9e402a591
Reviewed-by: Morten Johan Sørvig <morten.sorvig@qt.io>
These exhibit the problem described by a recent bug report.
Task-number: QTBUG-121485
Change-Id: Ia09acfa22e687ba096091a73f30df1ffd22a6e32
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
There were three things going on in this test (itself a sufficient
reason to split it up):
* Some reporting and checking of the default locale; the reporting is
duplicated in defaulted_ctor() and the check fitted more naturally
there.
* Checks that various combinations of language, script and territory
got resolved according to likely-subtag rules. These were handled
via a macro and natural candidates to become data-driven.
* A test that territory is preserved when it's the only given tag
(with a few known exceptions); broken out as a steparate test.
In the process, give the data-rows of the likely-subtag parts names
that let me extend their testing to also test construction from
string. The territory-only cases can't support that, as QLocale
doesn't support und_* forms of tags for unspecified language.
Change-Id: Id9f0fc46f30eb887b47931bad1619255acb44266
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Replace various QFETCH()/QCOMPARE() pairs with QTEST().
Just because it's terser.
Change-Id: I8496a293e3634991dcb33b8c7939f1c3028a63c5
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Actually triggered by investigating QTBUG-121485, where the issue
turns out to be at construction time, but the complete lack of any
testing of endonyms clearly needed addressed in any case. In the
process, break up the long list of private slots in the test-class
declaration.
Task-number: QTBUG-121485
Change-Id: I49021f78d3bea2e1e55b2755a45943ab3fc23722
Reviewed-by: Øystein Heskestad <oystein.heskestad@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Every test row was following its locale-specific test with tests for
Irish and Greek day names, repeated regardless of the locale-specific
data. Express these tests as rows in their own right and shorten the
per-row testing, so we only do each once.
Change-Id: I8f919b50ac54423bacab6e5a9d34254b7db59a55
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
In the process, change it to use QCOMPARE(A, B) rather than QVERIFY(A == B).
Change-Id: I26c64100b08aef43c56f5266c0de71f5cde12816
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
A null QString / QStringView has a null begin pointer stored as its
array beginning (something we hide a little in the QString::data()
function, but not in QStringView::data()). We've been passing a null
pointer to memcpy() every time someone passed a null QStringView for
QString's single-argument arg() call, though not the multi-string arg()
version (which is the only one QStringView offers).
Commit f5021835df made this worse by
making QStringViews created from null QStrings retain the nullness (as
was intended).
Fixes: QTBUG-120624
Pick-to: 6.2 6.5 6.6 6.7
Change-Id: I6e2677aad2ab45759db2fffd17a870639b19340b
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Commit ba5db13c8d ("QStringView: add
internal qToStringViewIgnoringNull()") said:
As long as a null QString still returns non-null data(), QStringView's
QString constructor needs to call isNull(). That's a branch we often
can do without, so add an internal function that bypasses this
correctness check.
It's internal, since we might have a Q6String that returns nullptr
data() when null, which will remove the need for this function.
For Qt 6, we made QString and QByteArray be able to return nullptr from
data() when null... but that's not enabled by default yet. However, the
begin() functions do return nullptr, so we can avoid the extra branch
the commit was talking about.
Task-number: QTBUG-119750
Change-Id: Ica7a43f6147b49c187ccfffd179e4cf032bc8565
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
A default-constructed QLocale gets initialized with the
currently-active default locale, and apparently retains that setting
henceforth. That is why the `prior` member is not explicitly
initialized, which is confusing at face value.
Explain the mechanism better, and explicitly default-initialize the
member, so the next reader of the code saves the time to research
this.
Amends 76dfda1ad1.
Pick-to: 6.6 6.5 6.2 5.15
Change-Id: I1d1171f8564c70a971938b92b809f63ba5637d3a
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The (internal) docs say that - and _ are ignored, and they're ignored
everywhere, except as suffixes. If the old code only ignored them as
infixes, fine, that would make some sense, but it ignored infixes and
prefixes, so there's no reason for it to not ignore suffixes, too.
Fix by continuing the loop until both input ranges are exhausted, or a
mismatch was found.
[ChangeLog][QtCore][QStringConverter] Fixed a bug where
encodingForName() failed due to trailing characters (`_`, `-`) that
ought to have been ignored.
Pick-to: 6.6 6.5
Change-Id: Iec21489d988eda7d33c744c170f88cd665b73f34
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
Add more tests and DRY the code using the usual row() lambda trick.
Pick-to: 6.6 6.5
Change-Id: I810fba0ab2a96e740e67392155f9507675fe57ae
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
The nameMatch() function has an implicit precondition that neither
argument is nullptr: it immediately dereferences both arguments.
Prevent the crash by checking for name == nullptr early, before
passing to nameMatch().
Add tests for null and empty.
As a drive-by, make variables in the test const (needed for the
QByteArray to avoid detaching, peer pressure for the others).
Amends a639bcda1e.
Pick-to: 6.6 6.5 6.2
Change-Id: I4a30f6c130310eb701ba7c7251168294489c34db
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
By just setting state to nullptr.
Pick-to: 6.6 6.5
Task-number: QTBUG-105105
Change-Id: I6b4f8fe39f1ba51dcfaf98ce7e42c2acd4c4cf98
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
The win32 API doesn't give us much choice. _Some_ code pages have
support for returning some error if we pass a specific flag, but not
all of them.
Anyway, since the code pages might not support all that UTF-16 provides,
we can't reasonably make it error out on characters that cannot be
converted.
So, the most reasonable thing we can handle is a unpaired high surrogate
at the end of a string, assume that the rest of the string was fine, and
that the low surrogate will be provided in the next call.
Pick-to: 6.6 6.5
Fixes: QTBUG-118185
Task-number: QTBUG-105105
Change-Id: I1f193c9d8e04bec769d885d32440c759d9dff0c2
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Both to store and to restore.
Without this a 3 or more octet sequence would cause errors or wrong
output. This can be seen with GB 18030.
Pick-to: 6.6 6.5
Fixes: QTBUG-118318
Task-number: QTBUG-105105
Change-Id: Id1f7f5f2fba4633b9f888add2186f4d8d21b7293
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
A few code pages do not support this flag[0]. It's also deprecated[1]
and is what Windows prefers to generate by default. So let's drop it.
[0] https://learn.microsoft.com/en-us/windows/win32/api/stringapiset/nf-stringapiset-multibytetowidechar
See note at the end for the dwFlags parameter.
[1] It's mentioned in the header files, but not online...
Pick-to: 6.6 6.5
Task-number: QTBUG-118185
Task-number: QTBUG-105105
Change-Id: I798c387170c73a953be874de139868543b2d775e
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The actual formatting of date-time strings is handled by the calendar
backend, but the code's in qlocale.cpp as it uses some of its tools.
When feature timezone is unavailable, we're stuck (as before) with
using QDateTime::timeZoneAbbreviation(), but when it's available we
can use QTimeZone::displayName() to get the localized form of the
abbreviation and offset string.
Make matching changes in QDTP so that it recognizes these localized
abbreviations. We now have another candidate for what local time might
be called, to add to those that must be checked.
This naturally implied some changes to tests. It turns out ICU
believes en_US uses GMT+1/GMT+2 for CET/CEST. Replace some MS
QEXPECT_FAIL()s by including the non-abbreviations we do in fact use
on MS in the lists of "abbreviations" to accept.
[ChangeLog][QtCore][QLocale] When a datetime format includes the
timezone (or offset), the appropriately localised form is (to the
extent the timezone backend in use supports this) used where,
previously, a haphazard choice of system and C locale was used. This
applies to both serialization and parsing.
Task-number: QTBUG-115158
Change-Id: I04f9c1055c3b9008320bb8b758490287fd8be5cd
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Since the TZ backend resorts to ICU for display names, we got
inconsistent results if they disagree about when transitions happened.
Also, ICU uses the current (or only recent history) names for the
zone, so one currently not doing DST (Africa/Tripoli) doesn't get a
report of its DST name at a historical time when it did DST (but ICU
doesn't know about it). Since the ICU backend, in any case, doesn't
override the displayName(qint64, ...), we can simply use the QTZP base
version instead of overloading in the TZ back-end, so we only delegate
to ICU when it might actually help. This also saves duplicating some
locking and lazy-initialization code.
In the process, turn a unique lock into a scoped lock within a
suitable scope. Also, make a comment more grammatical and less
verbose.
Change-Id: Iaaed1fb6a380be8b4258c43cbc4bacd5784345fb
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
The first/last/sliced API may be what we suggest users use, but the vast
majority of the installed codebase uses left/mid/right because they've
been available since time immemorial.
An additional benefit of this is to make left() and right() available as
inline methods.
Change-Id: Ifeb6206a9fa04424964bfffd1788383817ed906c
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Those ought to have been the original implementation, when they were
added in commit 38096a3d70, for Qt 6.0.
Because these classes are exported, we need to provide the previous only
implementations for MSVC. All other compilers would provide inline or
emit local, out-of-line copies.
Change-Id: Ifeb6206a9fa04424964bfffd178836a2ae56157d
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
The test that needs this bool is using 2013, so test that year for a
match. (Africa/Tunis toyed with DST in 1990, the year used before, but
thought better of it.) In the process, move the initialization to the
member-initialization of the class and make the member const.
Change-Id: Ib87636cdb0b038fad0cdef9fbe49e96f7bf79d1f
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Then we can easily test how fromLocal8Bit() and toLocal8Bit() behave
with different code-pages.
Pick-to: 6.6 6.5
Task-number: QTBUG-118318
Task-number: QTBUG-118185
Task-number: QTBUG-105105
Change-Id: Ib1cd3bccd27d598f4c80915557e332befcd96354
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
[ChangeLog][QtCore][QStringList] Added lastIndexOf() overloads that take
a QString/QStringView/QLatin1StringView and a Qt::CaseSenitivity
parameters. Prior to this calling lastIndexOf() would call the methods
inherited from the base class. This change is source compatible and
existing code should continue to work.
Task-number: QTBUG-116918
Change-Id: Ia50c884c00021bf581c23c12e0e0c22700dae446
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
[ChangeLog][QtCore][QStringList] Added filter(QLatin1StringView)
overload, which is more optimized when searching for a Latin-1 string
literal as no conversion to QString is necessary.
Task-number: QTBUG-116918
Change-Id: Ieb92f4cfd545b070258dbc5c701ddfb2e6f3fc64
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
[ChangeLog][QtCore][QStringList] Added indexOf() overloads that take
QString/QStringView/QLatin1StringView, and a Qt::CaseSensitivity
parameter. Prior to this using QStringList::indexOf() called the methods
inherited from the base class.
Task-number: QTBUG-116918
Change-Id: Ibc42130b6509f6ecfe7de0d6be378f226ae61982
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Now that users can pass a QStringMatcher to do the matching, change the
existing overload to not use QStringMatcher.
Thanks to Giuseppe D'Angelo for the idea of passing a QStringMatcher to
filter instead of using a magic number to decide whether to use
QStringMatcher or not.
Results of running filter() and filter_stringMatcher, times are in msecs
and this was compiled with gcc -O3:
Without With QStringMatcher
list10 0.00022 0.000089
list20 0.00040 0.00014
list30 0.00058 0.00018
list40 0.000770 0.00023
list50 0.00094 0.00027
list70 0.0012 0.00037
list80 0.0014 0.00041
list100 0.0018 0.00050
list300 0.0054 0.0014
list500 0.0091 0.0023
list700 0.012 0.0032
list900 0.016 0.0041
list10000 0.17 0.045
Drive-by change: optimize tst_QStringList::populateList().
[ChangeLog][QtCore][QStringList] Added filter(const QStringMatcher &)
overload, which may be faster for large lists and/or lists with very
long strings.
[ChangeLog][Possible Performance Changes][QtCore][QStringList] Changed
the implementation of filter(QStringView) overload to not use
QStringMatcher by default. Using QStringMatcher adds overhead, so it is
beneficial/faster when searching for a pattern in large lists and/or
lists with long strings, otherwise using plain string comparison is
faster. If using QStringMatcher makes a difference in your code, you can
use the newly added filter(QStringMatcher) overload.
Change-Id: I7bb1262706d673f0ce0d9b7699f03c995ce28677
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The existing pattern() method always returns a QString, which means that
if the matcher was constructed using a QStringView, pattern() would
uncoditionally convert it to a QString.
This is useful to check if a match is exact:
auto pattern = matcher.patternView();
if (pattern.size() == needle.size() && matcher.indexIn(needle) == 0)
....
This may be needed for a later change in QStringList::contains();
regardless of that, this change makes sense on its own.
Change-Id: I49018551dd22a8f88cf6b9f878a5166902a26f58
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
- Initialize QStringList with an initializer_list instead of old style
operator <<()
- Use Qt::StringLiterals more, better readability
- Test CaseSensitivity
Change-Id: If7dde14333d54b8c2f682036634ad94d5f9f9c74
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Implement the missing overload to handle UTF-8 specific data types,
including char8_t (C++20), char, uchar and signed char.
Introduce the helper function 'assign_helper_char8' which handles the
non-contiguous_iterator case. The contiguous_iterator case is already
handled by the QAnyStringView overload.
Include 'qstringconverter.h' at the end of the file, since it can't
be included at the top due to diamond dependency conflicts.
QStringDecoder is an implementation detail we don't want users to
depend on when using assign(it, it). It would be unnatural to not
be able to use a function just because we didn't include an
apparently unrelated header.
[ChangeLog][QtCore][QString] Enabled assign() for UTF-8 data types.
Fixes: QTBUG-114208
Change-Id: Ia39bbb70ca105a6bbf1a131b2533f29a919ff66d
Reviewed-by: Marc Mutz <marc.mutz@qt.io>
Not only are we subject to Q and P defines, we're also included in the
unnamed namespace now.
Amends df030e06a8.
Pick-to: 6.6
Change-Id: Ie2f4c9f45d9845d8a26140e0e1214e87b615ff02
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
By first checking if the list has any matches before potentially making
it detach.
Change-Id: I7a42c2910ef6efc45033e562573414a3a9ef972e
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
QStaticLatin1StringMatcher is a static templated Latin-1 Boyer-Moore
string matcher which can be case sensitive or not. It should be used
when the needle is known at compile time so there is no run-time
overhead when generating the skip table.
The convenience functions qMakeStaticCaseSensitiveLatin1StringMatcher
and qMakeStaticCaseInsensitiveLatin1StringMatcher should be used to
construct the matcher objects.
Green Hills Optimizing Compilers are currently not supported.
[ChangeLog][QtCore] Added QStaticLatin1StringMatcher, which can be used
to create a static constexpr string matcher for Latin-1 content.
Task-number: QTBUG-100236
Change-Id: I8b8eed1e88e152f29cbf8d36d83e410fafc5ca2c
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Mårten Nordheim
Ievgenii Meshcheriakov
Tasuku Suzuki
Lucie Gérard
Edward Welbourne
Piotr Wierciński
Thiago Macieira
Kai Köhne
Marc Mutz
Anton Kudryavtsev
Ahmad Samir
Dennis Oberst
Øystein Heskestad