They added some new scripts.
There were a few changes to the line break algorithm,
most notably there is more rules that require more context than before.
While not major, there was some shuffling and additions to our
implementation to match the new rules.
IDNA test data now disallows the trailing dot/empty root label,
technically to be toggled off by an option that controls a few things,
but we don't have options. For test-data they changed the format a
little - "" is used to mean empty string, while a blank segment is
null/no string, update the parser to read this.
Changes in this cherry-pick:
- Reran tool to resolve conflicts due to
emoji-data not being extracted in this branch
[ChangeLog][Third-Party Code] Updated the Unicode Character Database to
UCD revision 34/Unicode 16.
Fixes: QTBUG-132902
Task-number: QTBUG-132851
Pick-to: 6.5
Change-Id: I4569703659f6fd0f20943110a03301c1cf8cc1ed
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit 85899ff181984a1310cd1ad10cdb0824f1ca5118)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
(cherry picked from commit 5985c90d37a096f35b68546f916bec29a218e112)
On MSVC the values stored end up as negative.
Conflicts resolved in cherry-pick:
- The emoji-data code didn't exist in this branch
Task-number: QTBUG-132902
Pick-to: 6.5
Change-Id: I963c57c34479041911c1364a1100d04998bdfaed
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
(cherry picked from commit 62685375a2d71fa34015faac76f03e709e4217c7)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
(cherry picked from commit 09b06a740dd5e2ea08b49f76a2c384e6eb6fa6c6)
This is both more accurate and also more stable: The suffix-list
update usually takes several days to go through all branches; using
the commit date avoids using newer fetch dates in older branches.
Pick-to: 6.5 6.2 5.15
Change-Id: Ifefd7158e4104c7bbdfe6a2d8a505d8acf7275b4
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
(cherry picked from commit cbea57f80b43a969d2e15d53b759abb9924d52c1)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
(cherry picked from commit 0ce4cdddb46c90acd7c037d764616925663212f4)
Also add a comment to publicsuffix-list's entry that the update is
done by a script that should be updated whenever this file's contents
change.
Amends 36dca3c04f759449f74008a3e79021a179b0f35e.
Pick-to: 6.5
Change-Id: I1a0e51568e6d4300197c09f759ad3d82277a74d1
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
(cherry picked from commit b9f74f9a0b97588fc07d75fa6a427f5c8d6a2c6c)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
(cherry picked from commit 94e4f3c53214d65f6049d44064c348e95682ef8b)
New languages added with v46
- Kara-Kalpak
- Swampy Cree
Several new Chinese-language locales have been added, including one
using Latin script, which invalidated some prior QLocale tests, which
have been adjusted to fit.
Some obsolete time-zone identifiers are now treated as deprecated
aliases. These have lost their AnyTerritory association, implying
changes to QTimeZone tests.
Many redundant likely sub-tag rules for unspecified language have been
dropped, in favor of simpler rules.
[ChangeLog][Third-Party Code] Updated CLDR data, used by QLocale, to
v46.
Task-number: QTBUG-130877
Change-Id: I92cf210422c7759dd829a7ca2f845d20e263d25b
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit e316276b76b9c3768ca4e19a04d03308ef21fe12)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
(cherry picked from commit 9413c19cc1f394bc39a9f46d7d12a71fb42c8d1a)
The lack of this was hidden by other rules (redundant with it) until
CLDR v45, but v46 prunes the redundant rules, breaking this. So
include the missing rule and tweak the code that assumed likely
sub-tag rules preserved language, since this one doesn't. Rework the
tail of withLikelySubtagsAdded() to correctly use this rule, now that
we have it. (The prior comment about there being no match-all was
wrong: CLDR did have it, but our data skipped it.) Amended one test
affected by it (when system locale wasn't en_US).
On picking to 6.8, uiLanguages() needed some coaxing to avoid
duplicate C locale entries in tests of qualified C locale.
Task-number: QTBUG-130877
Change-Id: I2a415b67af4bc8aa6a766bcc1e349ee5bda9f174
Reviewed-by: Mate Barany <mate.barany@qt.io>
(cherry picked from commit 303863170c3ea7d1ee1b7188f507ad432ed3f860)
Also fix the annotation of englishNaming in cldr.py. Spotted it while
annotating __enumTable.
Task-number: QTBUG-129564
Change-Id: I93f698b4cf1b5ae90c21fe77330e4f167143a9f3
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit 3045a08e5edd5d8bc7c94f425ea60a7f48786ac1)
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Amends commit 3b541daff7 to deal with
likely subtags being handled as tuples of enum names, rather than as
tuples of their numeric values. The change from names to value went to
6.9 (commit bd5bb70b7c980d737d03db7a9d00b864fa144de0) but isn't picked
back to 6.8 or earlier. However, the type annotations done on 6.9
after that were picked back to 6.8 and I missed this twist in review.
Task-number: QTBUG-129613
Change-Id: I35eb21d26db1ff0537ecaf5e727431a16c6443c4
Reviewed-by: Mate Barany <mate.barany@qt.io>
Use QMultiHash explicitly, and build list of values in a QSet via the
range constructor.
Task-number: QTBUG-131842
Change-Id: I9cbcddeada0bfd88b11515262f5476e5d59e0fad
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit e1217cc52bd367b7e040edeb86700dc923df39a8)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
The AnyTerritory entries in the zoneDataTable are derived from
territory="ZZ" entries in the upstream CLDR data; the World ones from
territory="001". The latter give the default IANA ID for each MS ID,
the former give an (often legacy) IANA ID for the MS ID, that is not
based on geography. Some of these are being removed at CLDR v46.
The documentation said the ZZ entries have "no known territorial
association", hinting that there may be some (unknown) territorial
association; however, CLDR's inclusion of them is as entries with a
known non-territorial association, so revise the phrasing to reflect
this.
Also document that windowsIdToDefaultIanaId() returns empty when
there is no territory-specific value, and callers can use the
territory-neutral call to get a suitable value in that case. (They
may, however, wish to distinguish this case, to treat it differently,
so I decided not to just return that in place of empty in any case.)
The upstream CLDR tables do have entries for territory 001, so we
should report these if asked for World as territory. Amend the
available zone ID lookup and mapping from MS to IANA functions that
take a territory to duly handle World via the default-data that was
derived from 001 data in CLDR, instead of from the territory-varying
table, from which those were effectively filtered out when generating
the two tables. Update docs to mention this handling of World, for
contrast with that of AnyTerritory.
In the process remove a spurious split-on-space from the MS to default
IANA lookup, asserting there is no space (in a field now stored in the
table for single IANA ID entries, instead of the one for space-joined
lists of them in which it used to be stored, before I noticed it's
always only one ID). There is a matching assertion in the cldr.py code
that extracts the data. Added an assertion to this last, that each
default IANA ID given by CLDR's MS data does in fact also appear as
one of the IANA IDs for at least one territory (potentially ZZ), and
comment in C++ code on why this means we don't need to scan the
windowsDataTable in a few places, where it would just produce
duplicate entries.
On picking to 6.8, removed the timezone_locale addition, only relevant
on 6.9 and later.
[ChangeLog][QtCore][QTimeZone] Corrected handling of QLocale::World
and clarified in docs how QLocale::AnyTerritory is handled when
QTimeZone selects zones by territory.
Task-number: QTBUG-130877
Change-Id: I861c777c68b0cb73a194138fe23fbff839df49e6
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
(cherry picked from commit e23dc7c420297fb62db9834a17c59bbf5992dad7)
Reviewed-by: Mate Barany <mate.barany@qt.io>
The change adds CPE and PURL keys to all qt_attribution.json files in
the repo.
In case if no sensible CPE or PURL exists, a "Comment" field is added
with the text "no relevant CPE or PURL found". If only one of them
does not exist, it is written as such in the Comment field.
This allows filtering for files that haven't had the information added
yet vs those that were looked up but no relevant information was
found.
For sources that are not hosted on github, a generic PURL is used with
a download_url fragment pointing either to the exact location where
the sources can be downloaded, or to the homepage of the project.
The generic package name was chosen based on the 'Id' key of the
attribution entry where it was present, and is not authoritative.
For PURL github packages, the 'git tag' name was specified into the
'version' part of the PURL, rather than the 'version number', because
SBOM processing tooling handle that better than the version number.
For example for the freetype package, we specify the string
'VER-2-13-3' rather than the tag name '2.13.3'.
We might revisit this in the future.
[ChangeLog][Third-Party Code] Added PURL and CPE information to the
attribution files of 3rd party sources.
Task-number: QTBUG-122899
Task-number: QTBUG-129602
Change-Id: Iad126242cafc3ea0b678c5c36b26f857039b1dbd
Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
(cherry picked from commit 36dca3c04f759449f74008a3e79021a179b0f35e)
Add some type annotatons to cldr2qlocalexml.py as well. Based on the
default arguments the constructor of CldrReader was expecting callables
that return None, but in reality we are passing in functions that
return integers.
Task-number: QTBUG-129613
Change-Id: I06832240956ea635ca0cc0ec45c466a3b2539ff7
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit 812f79e75f2d6f2675c4b032f3a98610721e98de)
This should keep gcc from complaining that they're unused const
variables. Pick to 6.8 required conflict resolutions in non-generated
code and qlocalexml2cpp.py due to dev's updates to handle time zone
name L10n; regeneration of data took care of the other conflicts.
Fixes: QTBUG-128930
Change-Id: I4d0bccc5f158dfc72e4cf38a085b90cb562dc185
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
(cherry picked from commit 08ef4d232df82c056d4f5fda636b62b65fc054ad)
Those files are read by reuse to complement or override the copyright
and licensing information found in file.
The use of REUSE.toml files was introduced in REUSE version 3.1.0a1.
This reuse version is compatible with reuse specification
version 3.2 [1].
With this commit's files,
* The SPDX document generated by reuse spdx conforms to SPDX 2.3,
* The reuse lint command reports that the Qt project is reuse compliant.
[1]: https://reuse.software/spec-3.2/
Task-number: QTBUG-124453
Task-number: QTBUG-125211
Change-Id: I01023e862607777a5e710669ccd28bbf56091097
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
(cherry picked from commit 4c83657448f66ac0ee46900aff2e5a230eda49b0)
Amends commit 52909797ac, whose conflict
resolutions inadvertently incorporated the movement of IANA IDs from
the list table to the IANA table, a change only done on dev (6.9), not
picked back to 6.8, but tagged with a # TODO comment in 6.8 that, none
the less, is not to be done in 6.8.
Restore writing to the correct table.
Change-Id: I70fd133eb54526efecae189306b3ae07c1c638db
Reviewed-by: Mate Barany <mate.barany@qt.io>
A missing update of a "last" variable meant the loop inevitably did
nothing useful. Include type-annotation for last, while doing this.
Thankfully the check still doesn't find any duplications, now that
I've fixed it so that actually would, were any present.
Pick-to: 6.5
Change-Id: I672e6570359a3ff102a364d8af98c5c8c0bdc4d9
Reviewed-by: Mate Barany <mate.barany@qt.io>
(cherry picked from commit 98db7a35d2ee56f5da11b4a8e745f2ee6a965077)
Found these while adding type annotations.
Task-number: QTBUG-129566
Change-Id: I51c8e5676f958094946c0e6f396b98c083fd9de0
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit ba9d6b261b04119b15d27f3f57dcfe0aadad5012)
Add some type hints to unicode2hex as well, it is used by
ByteArrayData.
Task-number: QTBUG-128634
Change-Id: I86b7ce8567483bf8a4d4db78c9585652526cb90d
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit c1ffb6e7515621a953b5ebf19c384b27df785d73)
The lookup into it is done case-insensitively (because user-supplied
names of zones might not have the right case) but I forgot to make the
sorting of the data table case-insensitive in the aliases. Regenerate
data: only the qtimezone*_data_p.h are changed by the reindexing of
zone aliases.
On picking to 6.8, drop the change to QTZlocale_data_p.h, as that's
not active until 6.9; and back-date the QTZprivate_data_p.h to match
the change on dev four days ago, as that might just reduce the risk of
later conflicts.
Change-Id: Id5e95c245c7ca421a77298f23baefe6b7021a396
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
(cherry picked from commit ff58e09c8d5b3f505d5d3fe9a205e35e96de922d)
Reviewed-by: Mate Barany <mate.barany@qt.io>
Intel CPUs have had this since 2013 (Ivy Bridge), but some older
Bulldozer AMD CPUs appear to be missing it. This creates a mismatch
between when the __haswell__ macro gets declared in qsimd_p.h and the
runtime check using the CpuArchHaswell value. That in turn creates a
condition where qInitDrawhelperFunctions() in qdrawhelper.cpp leaves the
memfill pointers set to null.
#elif defined(__SSE2__)
# ifndef __haswell__
qt_memfill32 = qt_memfill32_sse2;
qt_memfill64 = qt_memfill64_sse2;
# endif
...
#if defined(QT_COMPILER_SUPPORTS_AVX2)
if (qCpuHasFeature(ArchHaswell)) {
qt_memfill32 = qt_memfill32_avx2;
qt_memfill64 = qt_memfill64_avx2;
It does this so the qt_memfillXX_sse2 functions don't have to be defined
anywhere, so the QtGui build won't carry unnecessary dead code.
This is old code (from Qt 4.x) and several improvements I've made for
QtCore are not applied yet. My work for qSimdDispatcher[1] isn't
complete: it might have avoided this problem here, but it would also
have required major work for the draw helpers to work in the first
place.
[1] https://codereview.qt-project.org/c/qt/qtbase/+/537384
Pick-to: 6.7 6.5 6.2
Fixes: QTBUG-129193
Change-Id: Ia427a9e502b0fb46b2bdfffda8e2131b7091c9e9
Reviewed-by: Allan Sandfeld Jensen <allan.jensen@qt.io>
(cherry picked from commit 54c24313fe1e4ed58b3260189cb623a7c852ab1d)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
On Unix, if target exists and it is a file, rename silently replaces it
if the user has permission. However, on Windows, if the target exists,
FileExistError will be raised.
With replace, if target points to an existing file or empty directory,
it will be unconditionally replaced.
Change-Id: I2774152fec78a00c4ca6c9d1b927e503df2f2e84
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit eff8e6b885dfa3f26523ca39164725b4932c31bd)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
Using the default system encoding cldr2qlocalexml.py and
qlocalexml2cpp.py may terminate with encoding errors on Windows.
Warn the user to set the PYTHONUTF8 environment variable to 1 before
running those scripts to avoid encoding errors.
Change-Id: I315a45072cb6ea516d3e9bb7613c6f251792ec59
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
(cherry picked from commit b3ef2fdd8b99c6920a382fbbc417209bf436c3f3)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
The variable ianalist is not really used for anything, it was probably
meant to be ianaList.
Change-Id: Ie9f42bf9716da28ee0017319dda96389c415ef4f
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit 67ec126168b81adc2adc9fdb346c0a913c3c501d)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
This amends commit 880d1aef99 and
extends it to cover the testlocales program under util/.
Pick-to: 6.7 6.5
Task-number: QTBUG-121653
Change-Id: I3efadc69ce08810876f8e20aa4636c7624728153
Reviewed-by: Lucie Gerard <lucie.gerard@qt.io>
(cherry picked from commit 00644f4b6d69d82a8d27f3b0619de88c0e4fdd6f)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
This was in fact present in v44, but we overlooked it somehow. The new
version also fixes some inconsistencies in the data, that I reported
against v44.1; in particular, Tamil no longer claims to override the
root AM/PM markers (probably because it uses 24-hour time so doesn't
need them).
Add the test-file under util to the list of files containing generated
content.
Conflict at 6.8 resolved by regenerating the data; this only changed
the date of generation, not the data. Then hand-edited the date to
match the picked upstream commit, to avoid future conflicts.
[ChangeLog][Third-Party Code] Updated CLDR data, used by QLocale, to
v45.
Task-number: QTBUG-126060
Pick-to: 6.7 6.5 6.2
Change-Id: I81a5bcca49519b55091fc541de6b73b606661bb4
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
(cherry picked from commit f79548e268a496698d77d0e78365334d0e507212)
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Move to construction time, instead of passing to each append() call;
the table's field sizes are, after all, the same for all entries.
Add support for larger tables by allowing more than 16-bit indices.
Task-number: QTBUG-115158
Change-Id: I8f1113482e80838c512da6353fa17b9f365f956a
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Reviewed-by: Mate Barany <mate.barany@qt.io>
It was setting *_code='0' for the Any* forms of language, script and
territory; this is wrong, the codes for these are all empty or other
special tokens (like 'und', 'Zzzz', 'ZZ'). The IDs for them are zero,
as an int not a string, but were omitted. Also add the variant
details, for all that they're currently unused, for consistency.
This makes no difference to the generated data.
Task-number: QTBUG-115158
Change-Id: I339d1b201e50e2bbc510758ffbbaae0fa02277d4
Reviewed-by: Mate Barany <mate.barany@qt.io>
The qlocalexml.py Locale.C() had to replicate a whole lot of data that
isn't really relevant to how C differs from en_US and every addition
to what we support required further additions to it. So pass the en_US
Locale object to the pseudoconstructor so that C can inherit from it
and only override the parts where we care about the difference.
Hand-code shortening for short Jalali month names, to match Soroush's
original contribution, and include the narrow forms in the hard-coded
data to keep the generated data unchanged (for now). Note some of the
departures from CLDR; we may want to drop these overrides later.
In the process, convert the mapping from keys to locales to
consistently use IDs for all members of the key, instead of using the
(empty) code value for (as yet unused) variant; it now gets ID 0 and
is consistent with returns from codesToIdNames(). This makes life
easier for the code that now has to construct an en_US key.
Task-number: QTBUG-115158
Change-Id: I3d7acb6a4059daec1bba341fcf015c39c7a6803b
Reviewed-by: Kai Köhne <kai.koehne@qt.io>
Omit parentheses round what python will form into a tuple anyway.
Include trailing commas on last entries of tuples so adding future
entries don't drag the existing line into their diffs.
Let the writer's tag-opener handle attributes, if supplied.
Clean up spacing in some doc-strings.
This is all preparation for further changes, to limit their diffs.
Change-Id: I989ae28bbd235b2af9c1d72467d4741c4f1f20ae
Reviewed-by: Mate Barany <mate.barany@qt.io>
Future work shall need the timezone alias data to be synchronized
between the (expanded) locale-independent timezone data and the
(coming) locale-dependent timezone data. The latter shall need to come
via QLocaleXml, hence the former now needs to, too.
This makes no change to the generated data, aside from changing the
regeneration instructions for qtimezoneprivate_data_p.h, to use the
same scripts as locale data, instead of cldr2qtimezone.py, which is
now removed.
Task-number: QTBUG-115158
Change-Id: I47ddd95f6af1855cbb1f601e9074c13f213cd61c
Reviewed-by: Mate Barany <mate.barany@qt.io>
It's trivial to do - and done when generating our compiled data
tables, so makes no difference to users - but makes the offset list
table simpler. Reformat the list so that the fragment-of-hour offsets
are clearly distinguished from the whole-hour ones.
Change-Id: I6e0ea23dc317542b3256e88492e4073faedef1d7
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
It was originally (without any comment to this effect, either in the
code or the commit message) just the list of offset-zones
corresponding to known Windows zones' offsets, augmented to include
each whole hour offset out to ±14 hours. Absent documentation, of
course, this was not maintained.
Added the four offset zones implied by that, that hadn't been added
when new entries joined the Windows IDs with novel offsets. Check,
after scanning CLDR for Windows data, that this has been kept up to
date. Updated the generated data.
Change-Id: I3cf3932c320876f7f2f74840d8c3951be49cfe70
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The CLDR's "IANA" IDs may (for the sake of stability) date back to
before IANA's own naming has been updated. As a result, the "IANA" IDs
we were using were in some cases out of date. CLDR does provide a
mapping from its stable IDs to all aliases and the current IANA name
for each (which I shall soon be needing in other work), so use that to
map the CLDR IDs to contemporary IANA ones.
Revise the documentation of CldrAccess.readWindowsTimeZones() to take
this into account, pass it the alias mapping from the table, use that
to map IDs internally and, in passing, rename a variable. Update
cldr2qtimezone.py to match the new CldrAccess methods and regenerate
the data.
Change-Id: I23d8a7d048d76392099d125376b544a41faf7eb3
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Mate Barany <mate.barany@qt.io>
Mårten Nordheim
Marc Mutz
Jani Heikkinen
Mate Barany
Edward Welbourne
Volker Hilsheimer
Alexandru Croitor
Lucie Gérard
Thiago Macieira