New languages added with v46
- Kara-Kalpak
- Swampy Cree
Several new Chinese-language locales have been added, including one
using Latin script, which invalidated some prior QLocale tests, which
have been adjusted to fit.
Some obsolete time-zone identifiers are now treated as deprecated
aliases. These have lost their AnyTerritory association, implying
changes to QTimeZone tests.
Many redundant likely sub-tag rules for unspecified language have been
dropped, in favor of simpler rules.
[ChangeLog][Third-Party Code] Updated CLDR data, used by QLocale, to
v46.
Task-number: QTBUG-130877
Change-Id: I92cf210422c7759dd829a7ca2f845d20e263d25b
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit e316276b76b9c3768ca4e19a04d03308ef21fe12)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
(cherry picked from commit 9413c19cc1f394bc39a9f46d7d12a71fb42c8d1a)
The lack of this was hidden by other rules (redundant with it) until
CLDR v45, but v46 prunes the redundant rules, breaking this. So
include the missing rule and tweak the code that assumed likely
sub-tag rules preserved language, since this one doesn't. Rework the
tail of withLikelySubtagsAdded() to correctly use this rule, now that
we have it. (The prior comment about there being no match-all was
wrong: CLDR did have it, but our data skipped it.) Amended one test
affected by it (when system locale wasn't en_US).
On picking to 6.8, uiLanguages() needed some coaxing to avoid
duplicate C locale entries in tests of qualified C locale.
Task-number: QTBUG-130877
Change-Id: I2a415b67af4bc8aa6a766bcc1e349ee5bda9f174
Reviewed-by: Mate Barany <mate.barany@qt.io>
(cherry picked from commit 303863170c3ea7d1ee1b7188f507ad432ed3f860)
Also fix the annotation of englishNaming in cldr.py. Spotted it while
annotating __enumTable.
Task-number: QTBUG-129564
Change-Id: I93f698b4cf1b5ae90c21fe77330e4f167143a9f3
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit 3045a08e5edd5d8bc7c94f425ea60a7f48786ac1)
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Amends commit 3b541daff7 to deal with
likely subtags being handled as tuples of enum names, rather than as
tuples of their numeric values. The change from names to value went to
6.9 (commit bd5bb70b7c980d737d03db7a9d00b864fa144de0) but isn't picked
back to 6.8 or earlier. However, the type annotations done on 6.9
after that were picked back to 6.8 and I missed this twist in review.
Task-number: QTBUG-129613
Change-Id: I35eb21d26db1ff0537ecaf5e727431a16c6443c4
Reviewed-by: Mate Barany <mate.barany@qt.io>
The AnyTerritory entries in the zoneDataTable are derived from
territory="ZZ" entries in the upstream CLDR data; the World ones from
territory="001". The latter give the default IANA ID for each MS ID,
the former give an (often legacy) IANA ID for the MS ID, that is not
based on geography. Some of these are being removed at CLDR v46.
The documentation said the ZZ entries have "no known territorial
association", hinting that there may be some (unknown) territorial
association; however, CLDR's inclusion of them is as entries with a
known non-territorial association, so revise the phrasing to reflect
this.
Also document that windowsIdToDefaultIanaId() returns empty when
there is no territory-specific value, and callers can use the
territory-neutral call to get a suitable value in that case. (They
may, however, wish to distinguish this case, to treat it differently,
so I decided not to just return that in place of empty in any case.)
The upstream CLDR tables do have entries for territory 001, so we
should report these if asked for World as territory. Amend the
available zone ID lookup and mapping from MS to IANA functions that
take a territory to duly handle World via the default-data that was
derived from 001 data in CLDR, instead of from the territory-varying
table, from which those were effectively filtered out when generating
the two tables. Update docs to mention this handling of World, for
contrast with that of AnyTerritory.
In the process remove a spurious split-on-space from the MS to default
IANA lookup, asserting there is no space (in a field now stored in the
table for single IANA ID entries, instead of the one for space-joined
lists of them in which it used to be stored, before I noticed it's
always only one ID). There is a matching assertion in the cldr.py code
that extracts the data. Added an assertion to this last, that each
default IANA ID given by CLDR's MS data does in fact also appear as
one of the IANA IDs for at least one territory (potentially ZZ), and
comment in C++ code on why this means we don't need to scan the
windowsDataTable in a few places, where it would just produce
duplicate entries.
On picking to 6.8, removed the timezone_locale addition, only relevant
on 6.9 and later.
[ChangeLog][QtCore][QTimeZone] Corrected handling of QLocale::World
and clarified in docs how QLocale::AnyTerritory is handled when
QTimeZone selects zones by territory.
Task-number: QTBUG-130877
Change-Id: I861c777c68b0cb73a194138fe23fbff839df49e6
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
(cherry picked from commit e23dc7c420297fb62db9834a17c59bbf5992dad7)
Reviewed-by: Mate Barany <mate.barany@qt.io>
Add some type annotatons to cldr2qlocalexml.py as well. Based on the
default arguments the constructor of CldrReader was expecting callables
that return None, but in reality we are passing in functions that
return integers.
Task-number: QTBUG-129613
Change-Id: I06832240956ea635ca0cc0ec45c466a3b2539ff7
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit 812f79e75f2d6f2675c4b032f3a98610721e98de)
This should keep gcc from complaining that they're unused const
variables. Pick to 6.8 required conflict resolutions in non-generated
code and qlocalexml2cpp.py due to dev's updates to handle time zone
name L10n; regeneration of data took care of the other conflicts.
Fixes: QTBUG-128930
Change-Id: I4d0bccc5f158dfc72e4cf38a085b90cb562dc185
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
(cherry picked from commit 08ef4d232df82c056d4f5fda636b62b65fc054ad)
Amends commit 52909797ac, whose conflict
resolutions inadvertently incorporated the movement of IANA IDs from
the list table to the IANA table, a change only done on dev (6.9), not
picked back to 6.8, but tagged with a # TODO comment in 6.8 that, none
the less, is not to be done in 6.8.
Restore writing to the correct table.
Change-Id: I70fd133eb54526efecae189306b3ae07c1c638db
Reviewed-by: Mate Barany <mate.barany@qt.io>
A missing update of a "last" variable meant the loop inevitably did
nothing useful. Include type-annotation for last, while doing this.
Thankfully the check still doesn't find any duplications, now that
I've fixed it so that actually would, were any present.
Pick-to: 6.5
Change-Id: I672e6570359a3ff102a364d8af98c5c8c0bdc4d9
Reviewed-by: Mate Barany <mate.barany@qt.io>
(cherry picked from commit 98db7a35d2ee56f5da11b4a8e745f2ee6a965077)
Found these while adding type annotations.
Task-number: QTBUG-129566
Change-Id: I51c8e5676f958094946c0e6f396b98c083fd9de0
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit ba9d6b261b04119b15d27f3f57dcfe0aadad5012)
Add some type hints to unicode2hex as well, it is used by
ByteArrayData.
Task-number: QTBUG-128634
Change-Id: I86b7ce8567483bf8a4d4db78c9585652526cb90d
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit c1ffb6e7515621a953b5ebf19c384b27df785d73)
The lookup into it is done case-insensitively (because user-supplied
names of zones might not have the right case) but I forgot to make the
sorting of the data table case-insensitive in the aliases. Regenerate
data: only the qtimezone*_data_p.h are changed by the reindexing of
zone aliases.
On picking to 6.8, drop the change to QTZlocale_data_p.h, as that's
not active until 6.9; and back-date the QTZprivate_data_p.h to match
the change on dev four days ago, as that might just reduce the risk of
later conflicts.
Change-Id: Id5e95c245c7ca421a77298f23baefe6b7021a396
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
(cherry picked from commit ff58e09c8d5b3f505d5d3fe9a205e35e96de922d)
Reviewed-by: Mate Barany <mate.barany@qt.io>
On Unix, if target exists and it is a file, rename silently replaces it
if the user has permission. However, on Windows, if the target exists,
FileExistError will be raised.
With replace, if target points to an existing file or empty directory,
it will be unconditionally replaced.
Change-Id: I2774152fec78a00c4ca6c9d1b927e503df2f2e84
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit eff8e6b885dfa3f26523ca39164725b4932c31bd)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
Using the default system encoding cldr2qlocalexml.py and
qlocalexml2cpp.py may terminate with encoding errors on Windows.
Warn the user to set the PYTHONUTF8 environment variable to 1 before
running those scripts to avoid encoding errors.
Change-Id: I315a45072cb6ea516d3e9bb7613c6f251792ec59
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
(cherry picked from commit b3ef2fdd8b99c6920a382fbbc417209bf436c3f3)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
The variable ianalist is not really used for anything, it was probably
meant to be ianaList.
Change-Id: Ie9f42bf9716da28ee0017319dda96389c415ef4f
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit 67ec126168b81adc2adc9fdb346c0a913c3c501d)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
This amends commit 880d1aef99 and
extends it to cover the testlocales program under util/.
Pick-to: 6.7 6.5
Task-number: QTBUG-121653
Change-Id: I3efadc69ce08810876f8e20aa4636c7624728153
Reviewed-by: Lucie Gerard <lucie.gerard@qt.io>
(cherry picked from commit 00644f4b6d69d82a8d27f3b0619de88c0e4fdd6f)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
This was in fact present in v44, but we overlooked it somehow. The new
version also fixes some inconsistencies in the data, that I reported
against v44.1; in particular, Tamil no longer claims to override the
root AM/PM markers (probably because it uses 24-hour time so doesn't
need them).
Add the test-file under util to the list of files containing generated
content.
Conflict at 6.8 resolved by regenerating the data; this only changed
the date of generation, not the data. Then hand-edited the date to
match the picked upstream commit, to avoid future conflicts.
[ChangeLog][Third-Party Code] Updated CLDR data, used by QLocale, to
v45.
Task-number: QTBUG-126060
Pick-to: 6.7 6.5 6.2
Change-Id: I81a5bcca49519b55091fc541de6b73b606661bb4
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
(cherry picked from commit f79548e268a496698d77d0e78365334d0e507212)
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Move to construction time, instead of passing to each append() call;
the table's field sizes are, after all, the same for all entries.
Add support for larger tables by allowing more than 16-bit indices.
Task-number: QTBUG-115158
Change-Id: I8f1113482e80838c512da6353fa17b9f365f956a
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Reviewed-by: Mate Barany <mate.barany@qt.io>
It was setting *_code='0' for the Any* forms of language, script and
territory; this is wrong, the codes for these are all empty or other
special tokens (like 'und', 'Zzzz', 'ZZ'). The IDs for them are zero,
as an int not a string, but were omitted. Also add the variant
details, for all that they're currently unused, for consistency.
This makes no difference to the generated data.
Task-number: QTBUG-115158
Change-Id: I339d1b201e50e2bbc510758ffbbaae0fa02277d4
Reviewed-by: Mate Barany <mate.barany@qt.io>
The qlocalexml.py Locale.C() had to replicate a whole lot of data that
isn't really relevant to how C differs from en_US and every addition
to what we support required further additions to it. So pass the en_US
Locale object to the pseudoconstructor so that C can inherit from it
and only override the parts where we care about the difference.
Hand-code shortening for short Jalali month names, to match Soroush's
original contribution, and include the narrow forms in the hard-coded
data to keep the generated data unchanged (for now). Note some of the
departures from CLDR; we may want to drop these overrides later.
In the process, convert the mapping from keys to locales to
consistently use IDs for all members of the key, instead of using the
(empty) code value for (as yet unused) variant; it now gets ID 0 and
is consistent with returns from codesToIdNames(). This makes life
easier for the code that now has to construct an en_US key.
Task-number: QTBUG-115158
Change-Id: I3d7acb6a4059daec1bba341fcf015c39c7a6803b
Reviewed-by: Kai Köhne <kai.koehne@qt.io>
Omit parentheses round what python will form into a tuple anyway.
Include trailing commas on last entries of tuples so adding future
entries don't drag the existing line into their diffs.
Let the writer's tag-opener handle attributes, if supplied.
Clean up spacing in some doc-strings.
This is all preparation for further changes, to limit their diffs.
Change-Id: I989ae28bbd235b2af9c1d72467d4741c4f1f20ae
Reviewed-by: Mate Barany <mate.barany@qt.io>
Future work shall need the timezone alias data to be synchronized
between the (expanded) locale-independent timezone data and the
(coming) locale-dependent timezone data. The latter shall need to come
via QLocaleXml, hence the former now needs to, too.
This makes no change to the generated data, aside from changing the
regeneration instructions for qtimezoneprivate_data_p.h, to use the
same scripts as locale data, instead of cldr2qtimezone.py, which is
now removed.
Task-number: QTBUG-115158
Change-Id: I47ddd95f6af1855cbb1f601e9074c13f213cd61c
Reviewed-by: Mate Barany <mate.barany@qt.io>
It's trivial to do - and done when generating our compiled data
tables, so makes no difference to users - but makes the offset list
table simpler. Reformat the list so that the fragment-of-hour offsets
are clearly distinguished from the whole-hour ones.
Change-Id: I6e0ea23dc317542b3256e88492e4073faedef1d7
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
It was originally (without any comment to this effect, either in the
code or the commit message) just the list of offset-zones
corresponding to known Windows zones' offsets, augmented to include
each whole hour offset out to ±14 hours. Absent documentation, of
course, this was not maintained.
Added the four offset zones implied by that, that hadn't been added
when new entries joined the Windows IDs with novel offsets. Check,
after scanning CLDR for Windows data, that this has been kept up to
date. Updated the generated data.
Change-Id: I3cf3932c320876f7f2f74840d8c3951be49cfe70
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
The CLDR's "IANA" IDs may (for the sake of stability) date back to
before IANA's own naming has been updated. As a result, the "IANA" IDs
we were using were in some cases out of date. CLDR does provide a
mapping from its stable IDs to all aliases and the current IANA name
for each (which I shall soon be needing in other work), so use that to
map the CLDR IDs to contemporary IANA ones.
Revise the documentation of CldrAccess.readWindowsTimeZones() to take
this into account, pass it the alias mapping from the table, use that
to map IDs internally and, in passing, rename a variable. Update
cldr2qtimezone.py to match the new CldrAccess methods and regenerate
the data.
Change-Id: I23d8a7d048d76392099d125376b544a41faf7eb3
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Mate Barany <mate.barany@qt.io>
There are various legacy IANA IDs that we should recognize as aliases
for their contemporary equivalents. Later work shall also take these
into account in the Windows IDs. Scan CLDR's data about these aliases
and use it when constructing QTimeZone. This adds aliasMappingTable
and aliasIdData arrays to QTZP_data_p.h and an AliasData type to its
QtTimeZoneCldr namespace.
Change-Id: I1bbfce62959a7e1b7a0bc4a320c32f5a174a2ff2
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
This separates the large slabs of data (and their documentation) from
the code that mixes them with CLDR-derived data and generates the data
we actually use. In the process, put the shorter table before the
longer one, to make it less likely that folk shall fail to notice it's
even there at all.
Change-Id: I8457741911657dac0dad53c2e65b977821bb4e71
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
The QLocale XML reader was passing datetime formats through a format
conversion despite the data being converted at the point where we read
it from CLDR. It turns out this was needed because the long date and
time formats in our hard-coded data for the C Locale object used CLDR
format strings, unlike all other Locale objects. Fix those two formats
in the C locale and remove the redundant processing step. This, in
turn, enables the parser to include the date and time formats in its
general handling of most fields that it reads.
This does not result in any change to the generated data QLocale uses
(although it does change the intermediate QLocale XML file).
Task-number: QTBUG-115158
Change-Id: Iaf9da206158043dda2e9e5a3790f009b100e46b4
Reviewed-by: Mate Barany <mate.barany@qt.io>
Include documentation in both, using common phrasing. Take sys.argv as
a parameter, along with sys.stdout and sys.stderr, so that we can
invoke them from python when importing into a python session to debug
or test. Supply the script name to the argument parser as prog, so it
can correctly report it and forward the rest of argv to parse_args().
Remove comments anticipating one of the several calendars we don't yet
support; the existing entries suffice to make clear what shall be
needed when we get round to adding more.
Change-Id: I2cebd385679e3c84d4ccf899e60091ac823ad10d
Reviewed-by: Mate Barany <mate.barany@qt.io>
After several years unused, it had bit-rotted to the point of not
compiling and failing an assertion. It also appears to have always had
a bad free() on exit, due to passing the address of a static object to
a function that took ownership and later deleted it.
Change-Id: I91856258c3fedf820bf151b5d205d257876a8e13
Reviewed-by: Jason McDonald <macadder1@gmail.com>
This old test program has bitrotted due to not being autogenerated as
part of CLDR updates. Amend qlocalexml2cpp.py to regenerate it and do
such an update. It was still using Qt5's QLocale enum numeric values,
many of which have changed in Qt6. Actually fixing the code so that it
compiles and runs can wait for a later commit.
Inspired by a patch supplied by Kizito Birabwa.
Task-number: QTBUG-124200
Change-Id: I33811313976a4860aad6d7b5b88a40c5b111a4fe
Reviewed-by: Mate Barany <mate.barany@qt.io>
It has many grumbles about spacing, but at least this code is
currently consistent about its departure from PEP8's spacing rules
(and closer to Qt's) for the present. We can review whether to do a
drastic spacing revolution later.
Change-Id: Ife4e8a5b02b63434bd9c7ac7ba4cbc11b6311f9f
Reviewed-by: Mate Barany <mate.barany@qt.io>
They're a bit more readable than calling dict on a generator.
Change-Id: I3177e31b1f617b80d1cf5d5f83df7036fc0c4c01
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Although the code does not, in fact, know about them, it's more
pertinent to say that they're unsupported than to say that the variant
in question is unknown.
Change-Id: I411d792dc91f2d7af58a4b7919c952a005b3417e
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
I neglected to update the CLDR dateconverter code when I expanded the
range of forms we support for display of a timezone. Even that
expanded range doesn't cover all the cases CLDR does, but we can at
least approximate each of CLDR's options by the closest we do support.
Make matching changes to how the Darwin backend for the system locale
maps its ICU-derived formats to ours.
This in practice changes all locales previously using t (abbreviation)
as zone format to use tttt (IANA ID) instead. Test data updated to
match.
[ChangeLog][QtCore][QLocale] Date-time formats now more faithfully
follow the CLDR data in handling timezones. In most cases this means
the IANA ID is used in place of the abbreviation.
Change-Id: I0276843085839ba9a7855a78922cffe285174643
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Mate Barany
Edward Welbourne