Commit Graph

157 Commits (152153cb111db1bbd42e94f27712bbc39accf310)

Author SHA1 Message Date
Mate Barany d8128041e0 Add type annotations to LocaleKeySorter
Task-number: QTBUG-128634
Change-Id: I9a4261746cac029b0abf26fbd03b1915a0035147
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
(cherry picked from commit bd475ddf47e33535c2b79367c8850cd611089e3c)
2024-10-17 19:30:34 +00:00
Edward Welbourne c5e38fd385 Fix sorting of timezone alias table to be in case-insensitive order
The lookup into it is done case-insensitively (because user-supplied
names of zones might not have the right case) but I forgot to make the
sorting of the data table case-insensitive in the aliases. Regenerate
data: only the qtimezone*_data_p.h are changed by the reindexing of
zone aliases.

On picking to 6.8, drop the change to QTZlocale_data_p.h, as that's
not active until 6.9; and back-date the QTZprivate_data_p.h to match
the change on dev four days ago, as that might just reduce the risk of
later conflicts.

Change-Id: Id5e95c245c7ca421a77298f23baefe6b7021a396
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
(cherry picked from commit ff58e09c8d5b3f505d5d3fe9a205e35e96de922d)
Reviewed-by: Mate Barany <mate.barany@qt.io>
2024-10-17 21:30:33 +02:00
Mate Barany 9709c4f842 Use replace instead of rename in localetools.py
On Unix, if target exists and it is a file, rename silently replaces it
if the user has permission. However, on Windows, if the target exists,
FileExistError will be raised.

With replace, if target points to an existing file or empty directory,
it will be unconditionally replaced.

Change-Id: I2774152fec78a00c4ca6c9d1b927e503df2f2e84
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit eff8e6b885dfa3f26523ca39164725b4932c31bd)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
2024-09-18 09:51:00 +00:00
Mate Barany d8e1416977 Add a note to README about encoding errors on windows
Using the default system encoding cldr2qlocalexml.py and
qlocalexml2cpp.py may terminate with encoding errors on Windows.

Warn the user to set the PYTHONUTF8 environment variable to 1 before
running those scripts to avoid encoding errors.

Change-Id: I315a45072cb6ea516d3e9bb7613c6f251792ec59
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
(cherry picked from commit b3ef2fdd8b99c6920a382fbbc417209bf436c3f3)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
2024-09-18 09:50:58 +00:00
Mate Barany b0e2abfe9a Fix typo in cldr.py
The variable ianalist is not really used for anything, it was probably
meant to be ianaList.

Change-Id: Ie9f42bf9716da28ee0017319dda96389c415ef4f
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit 67ec126168b81adc2adc9fdb346c0a913c3c501d)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
2024-09-18 09:50:55 +00:00
Edward Welbourne bcd33276d1 Include relevant Unicode Inc. copyright line in generated data files
This amends commit 880d1aef99 and
extends it to cover the testlocales program under util/.

Pick-to: 6.7 6.5
Task-number: QTBUG-121653
Change-Id: I3efadc69ce08810876f8e20aa4636c7624728153
Reviewed-by: Lucie Gerard <lucie.gerard@qt.io>
(cherry picked from commit 00644f4b6d69d82a8d27f3b0619de88c0e4fdd6f)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
2024-08-31 08:56:42 +00:00
Edward Welbourne 6a0f00ac4e Update CLDR to v45, adding language Kuvi
This was in fact present in v44, but we overlooked it somehow. The new
version also fixes some inconsistencies in the data, that I reported
against v44.1; in particular, Tamil no longer claims to override the
root AM/PM markers (probably because it uses 24-hour time so doesn't
need them).

Add the test-file under util to the list of files containing generated
content.

Conflict at 6.8 resolved by regenerating the data; this only changed
the date of generation, not the data. Then hand-edited the date to
match the picked upstream commit, to avoid future conflicts.

[ChangeLog][Third-Party Code] Updated CLDR data, used by QLocale, to
v45.

Task-number: QTBUG-126060
Pick-to: 6.7 6.5 6.2
Change-Id: I81a5bcca49519b55091fc541de6b73b606661bb4
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
(cherry picked from commit f79548e268a496698d77d0e78365334d0e507212)
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2024-07-17 12:57:30 +02:00
Edward Welbourne 877e77eced qlocalexml2cpp.py: rework StringData handling of bit-sizes
Move to construction time, instead of passing to each append() call;
the table's field sizes are, after all, the same for all entries.
Add support for larger tables by allowing more than 16-bit indices.

Task-number: QTBUG-115158
Change-Id: I8f1113482e80838c512da6353fa17b9f365f956a
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Reviewed-by: Mate Barany <mate.barany@qt.io>
2024-06-02 15:26:17 +02:00
Edward Welbourne 69aefa4edf Update C Locale constructor to match others on ids and codes
It was setting *_code='0' for the Any* forms of language, script and
territory; this is wrong, the codes for these are all empty or other
special tokens (like 'und', 'Zzzz', 'ZZ'). The IDs for them are zero,
as an int not a string, but were omitted. Also add the variant
details, for all that they're currently unused, for consistency.

This makes no difference to the generated data.

Task-number: QTBUG-115158
Change-Id: I339d1b201e50e2bbc510758ffbbaae0fa02277d4
Reviewed-by: Mate Barany <mate.barany@qt.io>
2024-06-02 15:26:05 +02:00
Edward Welbourne 0c809fc3b5 Derive C locale data from en_US, overriding minor details
The qlocalexml.py Locale.C() had to replicate a whole lot of data that
isn't really relevant to how C differs from en_US and every addition
to what we support required further additions to it. So pass the en_US
Locale object to the pseudoconstructor so that C can inherit from it
and only override the parts where we care about the difference.

Hand-code shortening for short Jalali month names, to match Soroush's
original contribution, and include the narrow forms in the hard-coded
data to keep the generated data unchanged (for now). Note some of the
departures from CLDR; we may want to drop these overrides later.

In the process, convert the mapping from keys to locales to
consistently use IDs for all members of the key, instead of using the
(empty) code value for (as yet unused) variant; it now gets ID 0 and
is consistent with returns from codesToIdNames(). This makes life
easier for the code that now has to construct an en_US key.

Task-number: QTBUG-115158
Change-Id: I3d7acb6a4059daec1bba341fcf015c39c7a6803b
Reviewed-by: Kai Köhne <kai.koehne@qt.io>
2024-06-02 15:25:52 +02:00
Edward Welbourne 5641b17e2f qlocalexml2cpp.py: Make clear that ByteArrayData is always ASCII
The container would be unsuitable otherwise.

Change-Id: I0b0aa22625fbd638bf8409c5ee257f62332d8e05
Reviewed-by: Mate Barany <mate.barany@qt.io>
2024-06-02 15:25:45 +02:00
Edward Welbourne 72a7dddc25 QLocaleXML: Improve documentation, tidy up a bit
Omit parentheses round what python will form into a tuple anyway.
Include trailing commas on last entries of tuples so adding future
entries don't drag the existing line into their diffs.
Let the writer's tag-opener handle attributes, if supplied.
Clean up spacing in some doc-strings.
This is all preparation for further changes, to limit their diffs.

Change-Id: I989ae28bbd235b2af9c1d72467d4741c4f1f20ae
Reviewed-by: Mate Barany <mate.barany@qt.io>
2024-06-02 15:25:36 +02:00
Edward Welbourne 9534341654 Integrate timezone data into the CLDR-via-QLocaleXml pipeline
Future work shall need the timezone alias data to be synchronized
between the (expanded) locale-independent timezone data and the
(coming) locale-dependent timezone data. The latter shall need to come
via QLocaleXml, hence the former now needs to, too.

This makes no change to the generated data, aside from changing the
regeneration instructions for qtimezoneprivate_data_p.h, to use the
same scripts as locale data, instead of cldr2qtimezone.py, which is
now removed.

Task-number: QTBUG-115158
Change-Id: I47ddd95f6af1855cbb1f601e9074c13f213cd61c
Reviewed-by: Mate Barany <mate.barany@qt.io>
2024-06-02 15:25:27 +02:00
Edward Welbourne 4e23dbb742 Add assorted notes and suggestions in util/locale_database/
Change-Id: I22534943f2c9710d501235672811a861a5fd3aea
Reviewed-by: Øystein Heskestad <oystein.heskestad@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2024-06-02 15:25:21 +02:00
Edward Welbourne b48b4f4d3b Simplify UTC offset ID data by computing the offsets
It's trivial to do - and done when generating our compiled data
tables, so makes no difference to users - but makes the offset list
table simpler. Reformat the list so that the fragment-of-hour offsets
are clearly distinguished from the whole-hour ones.

Change-Id: I6e0ea23dc317542b3256e88492e4073faedef1d7
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
2024-06-02 15:25:13 +02:00
Edward Welbourne 08c877a703 Update the utcIdList (now that I've worked out where it came from)
It was originally (without any comment to this effect, either in the
code or the commit message) just the list of offset-zones
corresponding to known Windows zones' offsets, augmented to include
each whole hour offset out to ±14 hours. Absent documentation, of
course, this was not maintained.

Added the four offset zones implied by that, that hadn't been added
when new entries joined the Windows IDs with novel offsets. Check,
after scanning CLDR for Windows data, that this has been kept up to
date. Updated the generated data.

Change-Id: I3cf3932c320876f7f2f74840d8c3951be49cfe70
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2024-06-02 15:25:07 +02:00
Edward Welbourne 99475db542 Revise Windows time-zone mapping to use proper IANA IDs
The CLDR's "IANA" IDs may (for the sake of stability) date back to
before IANA's own naming has been updated. As a result, the "IANA" IDs
we were using were in some cases out of date. CLDR does provide a
mapping from its stable IDs to all aliases and the current IANA name
for each (which I shall soon be needing in other work), so use that to
map the CLDR IDs to contemporary IANA ones.

Revise the documentation of CldrAccess.readWindowsTimeZones() to take
this into account, pass it the alias mapping from the table, use that
to map IDs internally and, in passing, rename a variable.  Update
cldr2qtimezone.py to match the new CldrAccess methods and regenerate
the data.

Change-Id: I23d8a7d048d76392099d125376b544a41faf7eb3
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Mate Barany <mate.barany@qt.io>
2024-05-30 20:28:55 +02:00
Edward Welbourne bcadcb029e Use CLDR alias data to find canonical IANA IDs
There are various legacy IANA IDs that we should recognize as aliases
for their contemporary equivalents. Later work shall also take these
into account in the Windows IDs. Scan CLDR's data about these aliases
and use it when constructing QTimeZone. This adds aliasMappingTable
and aliasIdData arrays to QTZP_data_p.h and an AliasData type to its
QtTimeZoneCldr namespace.

Change-Id: I1bbfce62959a7e1b7a0bc4a320c32f5a174a2ff2
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2024-05-21 17:23:21 +02:00
Edward Welbourne 020a224a79 Break out timezone data from cldr2qtimezone.py
This separates the large slabs of data (and their documentation) from
the code that mixes them with CLDR-derived data and generates the data
we actually use. In the process, put the shorter table before the
longer one, to make it less likely that folk shall fail to notice it's
even there at all.

Change-Id: I8457741911657dac0dad53c2e65b977821bb4e71
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
2024-05-06 20:27:41 +02:00
Edward Welbourne 5f8dc8ea5f Purge an almost-redundant duplicate datetime format conversion
The QLocale XML reader was passing datetime formats through a format
conversion despite the data being converted at the point where we read
it from CLDR. It turns out this was needed because the long date and
time formats in our hard-coded data for the C Locale object used CLDR
format strings, unlike all other Locale objects. Fix those two formats
in the C locale and remove the redundant processing step. This, in
turn, enables the parser to include the date and time formats in its
general handling of most fields that it reads.

This does not result in any change to the generated data QLocale uses
(although it does change the intermediate QLocale XML file).

Task-number: QTBUG-115158
Change-Id: Iaf9da206158043dda2e9e5a3790f009b100e46b4
Reviewed-by: Mate Barany <mate.barany@qt.io>
2024-04-30 18:30:15 +02:00
Edward Welbourne f83206229e Apply a common style to the main()s of locale database programs
Include documentation in both, using common phrasing. Take sys.argv as
a parameter, along with sys.stdout and sys.stderr, so that we can
invoke them from python when importing into a python session to debug
or test. Supply the script name to the argument parser as prog, so it
can correctly report it and forward the rest of argv to parse_args().
Remove comments anticipating one of the several calendars we don't yet
support; the existing entries suffice to make clear what shall be
needed when we get round to adding more.

Change-Id: I2cebd385679e3c84d4ccf899e60091ac823ad10d
Reviewed-by: Mate Barany <mate.barany@qt.io>
2024-04-26 07:36:16 +02:00
Edward Welbourne 065548e7b4 Modernise testlocales/ program and make it compile
After several years unused, it had bit-rotted to the point of not
compiling and failing an assertion. It also appears to have always had
a bad free() on exit, due to passing the address of a static object to
a function that took ownership and later deleted it.

Change-Id: I91856258c3fedf820bf151b5d205d257876a8e13
Reviewed-by: Jason McDonald <macadder1@gmail.com>
2024-04-26 07:36:16 +02:00
Edward Welbourne 1d48bf34db Automate updating of list of locales for testlocales
This old test program has bitrotted due to not being autogenerated as
part of CLDR updates. Amend qlocalexml2cpp.py to regenerate it and do
such an update. It was still using Qt5's QLocale enum numeric values,
many of which have changed in Qt6. Actually fixing the code so that it
compiles and runs can wait for a later commit.

Inspired by a patch supplied by Kizito Birabwa.

Task-number: QTBUG-124200
Change-Id: I33811313976a4860aad6d7b5b88a40c5b111a4fe
Reviewed-by: Mate Barany <mate.barany@qt.io>
2024-04-26 05:36:16 +00:00
Edward Welbourne e08ca2c9c8 Fix spacing inconsistencies brought to light by flake8
It has many grumbles about spacing, but at least this code is
currently consistent about its departure from PEP8's spacing rules
(and closer to Qt's) for the present. We can review whether to do a
drastic spacing revolution later.

Change-Id: Ife4e8a5b02b63434bd9c7ac7ba4cbc11b6311f9f
Reviewed-by: Mate Barany <mate.barany@qt.io>
2024-04-23 20:51:19 +02:00
Edward Welbourne cf0ebc9ad3 Fix typo in doc comment for QLocaleXmlWriter.close()
Change-Id: I128ed5e0ebd01a7ed1f3a3049d2b63f1df042562
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2024-04-22 18:56:20 +02:00
Edward Welbourne f2a2379de8 Use dict comprehensions more in cldr.py and qlocalexml.py
They're a bit more readable than calling dict on a generator.

Change-Id: I3177e31b1f617b80d1cf5d5f83df7036fc0c4c01
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2024-04-22 18:56:20 +02:00
Edward Welbourne d935a89d25 Tweak the message for variants
Although the code does not, in fact, know about them, it's more
pertinent to say that they're unsupported than to say that the variant
in question is unknown.

Change-Id: I411d792dc91f2d7af58a4b7919c952a005b3417e
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2024-04-22 17:22:12 +02:00
Edward Welbourne dd56558ecd Improve fidelity of approximation to CLDR zone representations
I neglected to update the CLDR dateconverter code when I expanded the
range of forms we support for display of a timezone. Even that
expanded range doesn't cover all the cases CLDR does, but we can at
least approximate each of CLDR's options by the closest we do support.
Make matching changes to how the Darwin backend for the system locale
maps its ICU-derived formats to ours.

This in practice changes all locales previously using t (abbreviation)
as zone format to use tttt (IANA ID) instead. Test data updated to
match.

[ChangeLog][QtCore][QLocale] Date-time formats now more faithfully
follow the CLDR data in handling timezones. In most cases this means
the IANA ID is used in place of the abbreviation.

Change-Id: I0276843085839ba9a7855a78922cffe285174643
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2024-04-22 11:58:25 +02:00
Edward Welbourne b87db4296c Correct handling of 'u' in CLDR date format strings
It explicitly excludes having a two-digit special case like 'yy'.
Correct that in qlocale_mac.mm, add support in dateconverter.py
No current locale actually uses the 'u' format, so this makes no
change to data.

Change-Id: I16dfed2d3a7d2054b4b86f9a246bff297df9fc0a
Reviewed-by: Dennis Oberst <dennis.oberst@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2024-04-19 13:39:41 +02:00
Edward Welbourne 42e4e1816a Fix handling of am/pm indicators in mapping from CLDR to Qt formats
Both qlocale_mac.mm and dateconverter.py were mapping the CLDR am/pm
indicator, 'a', to the Qt format token 'AP', forcing the indicator to
uppercase. The LDML spec [0] says:

    May be upper or lowercase depending on the locale and other
    options.

[0] https://www.unicode.org/reports/tr35/tr35-68/tr35-dates.html#Date_Field_Symbol_Table
We don't support the "other options" mentioned, but we can at least
(since 6.3) preserve the the locale-appropriate case, instead of
forcing upper-case. As such, this change is a follow-up to
commit 4641ff0f6a

Changes locale data, as expected, to use "Ap" in place of "AP" in
various formats in the time_format_data[] array.

[ChangeLog][QtCore][QLocale] Where CLDR specifies an am/pm indicator,
the case of the CLDR-supplied indicator is used, where previously
QLocale forced it to upper-case.

Change-Id: Iee7d55e6f3c78372659668b9798c8e24a1fa8982
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2024-04-19 13:39:41 +02:00
Edward Welbourne d778d17b9f Cope with CLDR's "day period" format specifiers
The LDML spec includes a 'b' pattern character which is like the 'a'
pattern, for AM and PM, but would rather use noon and midnight
indicators for those specific times. We don't support those and using
am/pm will be right enough of the time to be better than simply
discarding this option, if it ever gets used (which it currently
isn't), so treat as an alias for 'a'. No locale in CLDR currently uses
this.

CLDR also has a 'B' specifiers for "flexible day periods", including
things like "at night" and "in the day". At present only zh_Hant uses
'B'. As a result, this change only affects zh_Hant's formats for time
and datetime, which only zh_Hant_TW uses - zh_Hant_HK overrides them
to use am/pm markers and zh_Hant_MO inherits that from
zh_Hant_HK. Based on this and user feed-back, I've opted to treat 'B'
as another synonym of 'a'.

This removes an entry from the time_format_data[] table (it happened
to occupy one whole twelve-character row), causing many other locales'
offsets into that table to be shifted by 12. Only zh_Hant_TW has an
actual change to which entry in the table it uses.
Added a test-case.

[ChangeLog][QtCore][QLocale] CLDR's 'B' (flexible day period, e.g. "at
night" &c.) field, not currently supported, is now handled as a
synonym for the AM/PM field 'a', instead of leaving the B as literal
text. Only affects zh_TW at present.

Fixes: QTBUG-123872
Change-Id: I6ba008c0a048190bf7af8c7df7629a885b05804f
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2024-04-19 13:39:40 +02:00
Edward Welbourne ea806fa3f1 Rewrite CLDR-ingestion's date-time format conversion
Rework the somewhat ad-hoc handling of format blocks. Instead of
converting one character at a time, then coming back to map contiguous
chunks of various lengths to Qt's best match, use the first
non-separator character to select a function that looks ahead to see
what to consume with it. Quoted text can be handled the same way, with
a look-ahead. This potentially allows for more flexible parsing in
future.

In the process, matching qlocale_mac.mm, treat all unquoted letters as
reserved. The LDML spec says:

  Currently, A..Z and a..z are reserved for use as pattern characters
  (unless they are quoted, see next item).

and its description of literal text explcitly says these reserved
characters are not to be understood as literals. Document the letters
we do know about as unsupported pattern characters, but don't do
anything specific to handle them. This transiently changes zh_TW's
"Bh" hour fields to plain "h" but an imminent commit will change that
again and there is no other change to data, so the locale data is not
regenerated in this commit, to save churn.

This makes the parsing front-end function more straightforward and
makes it easier to document the quirks of the different format letters
and the impedance mismatches between CLDR's and Qt's. In the process,
recognize C, like j and J, as special magic to ignore and harmonize
with what qlocale_mac.cpp's macToQtFormat() does, where it's right and
dateconverter.py differed. Document the need to stay in sync with this
last.

Task-number: QTBUG-123872
Change-Id: I490d395b37751c9b8d6f3ee5ed4edbc0d405db5b
Reviewed-by: Mate Barany <mate.barany@qt.io>
2024-04-19 13:39:40 +02:00
Edward Welbourne 43cb4136a9 Move LocaleScanner's INHERIT check from find upstream to __find
When digesting CLDR v44.1's github form, some data (e.g. pt_BR's
language endonym) were None that had perfectly sensible values in the
zip-file form.  Letting __find() yield INHERIT entries lead to find()
sometimes returning None, where __find() should have tried harder or
raised an Error.

This further amends commit bcdd51cfae
(after commit 0f770b0b34 isolated its
magic value).

Pick-to: 6.7
Task-number: QTBUG-115158
Change-Id: I1af92a687cd50b8fd026c25f068c804a3516ef95
Reviewed-by: Mate Barany <mate.barany@qt.io>
2024-04-19 13:39:40 +02:00
Edward Welbourne a643a956d4 Rework enumdata.py's comments
Turn the large comment at the start into a doc-string and add some
more details to it. Fix the Ivory Coast comment's indent and a typo in
it.

Change-Id: I36b4e5094d3c3d5c5b91809424b424bcac5daafa
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
2024-03-18 14:59:26 +01:00
Edward Welbourne 693bf76306 Minor tidy-up of CldrAccess.__enumMap: revise comment, modernize code
A comment dated from when variables misleadingly named language_list,
script_list and country_list actually held mappings not lists; they've
been renamed to s/list/map/ a while back, so rephrase.

Use a dict-comprehension rather than the somewhat lisp-ier invocation
of the dict constructor on an iterator over pairs.

Change-Id: Ibcb97122434122dbb1dcb0f621aae02b25a4e1fa
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2024-02-13 15:58:43 +01:00
Edward Welbourne b0f4bd7f23 Move QTimeZone's CLDR-derived data into a namespace
Introduce namespace QtTimeZoneCldr instead of having a Q prefix on
each class name used for the data.

Change-Id: Icb22a91340b67f9cc93173b77374a70f69f81bbe
Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
2024-02-08 15:40:04 +01:00
Edward Welbourne a736e613fc Document LocaleScanner's constructor
I needed to know in order to make recent changes.
Save the need to work it out again next time.

Change-Id: Ibc606cbe2e6af16e6820fd753a643331a03cdfb3
Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
2024-02-08 15:40:04 +01:00
Edward Welbourne 063026cc50 Update QLocale and calendar data to CLDR v44.1
(This turns out to be identical to v44, for our purposes.)

The CLDR license has been revised at v44 to "UNICODE LICENSE V3",
which is now included (as LICENSES/UNICODE-3.0.txt) in addition to the
old license (still in use, presumably, by UCD - at least until its
next update). Some new QLocale::Language entries are needed. There is
no change to the time-zone data.

Some tests needed changes:
* Various Arabic locales now use U+0623 (Arabic letter aleph with
  hamza above) in exponent separator, replacing plain U+0627 (Arabic
  letter aleph); it is still followed by U+0633 (Arabic letter seen).
* Where likely sub-tags used to fill in world, 001, as territory for a
  language, they now (e.g. for Prussian and Yiddish) give specific
  countries.
* Tamil locales now have something of a mix of inherited and localized
  forms for AM/PM, which looks a lot like a mistake in CLDR.
* New likely sub-tag rules fix ctor(und_US) and ctor(und_GB), which
  previously failed.

[ChangeLog][Third-Party Code] Updated QLocale's data extracted from
the Unicode Common Locale Data Repository (CLDR) to v44.1. The license
changed to Unicode License V3.

Pick-to: 6.7 6.6 6.5
Fixes: QTBUG-121485
Task-number: QTBUG-121325
Change-Id: Ide1a68016129526d7a5aa3fc67f1a674858696bc
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
2024-02-02 08:26:03 +01:00
Edward Welbourne 4b2eb26bf9 Fix ordering of Windows timezones
The list is meant to be sorted in increasing order, requiring
"<anything> (Mexico)" to appear after "<anything>" but in two out of
four cases such pairs were in the wrong order. China sorts after
Chatham Island and lexical sorting of numbers doesn't match sorting by
numeric value.

Assert the expected ordering. (The more important check needs a
QBAV::compare(), which isn't constexpr, so we can't static_assert.)
Later commits shall use binary chop exploiting this ordering. The
assertion failed without the rest of this change.

Also improve the comments describing the data tables these searches
check and the types of their entries. Some were inaccurate, others
merely unclear. Likewise, comment the sorting expectations in the
python code that generates the tables.

Change-Id: I640a3cca8f820c5fd5939a2fe5feb96b04407335
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2024-02-01 21:50:50 +01:00
Edward Welbourne 0f770b0b34 Move special-case LDML value to a module global
Giving it a symbolic name is clearer (and saves me the need to
duplicate the comment when I add some more references to it).
This amends commit bcdd51cfae

Task-number: QTBUG-115158
Change-Id: I7577e0cde783fcda840009c7aea46934964c6e4c
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2024-01-29 15:14:43 +01:00
Edward Welbourne c8b70f4e51 ldml.LocaleScanner.__find(): only Error if no matches found
The existing caller returns early on finding a match, so never ran off
the end of the iteration unless there were no matches. I'll soon be
adding a new caller that wants to iterate all matches, so will run off
the end even when there are some. So only raise the Error if we found
nothing.

Task-number: QTBUG-115158
Change-Id: I1cae4674eb5e83c433554c15ecc4441b756f20eb
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2024-01-29 15:14:43 +01:00
Edward Welbourne 82f8afe6ba Package DOM attributes for Node objects
The Supplement type did the needed mapping (using nodeValue when the
value wasn't a string) and it turns out to be useful to do the same
for the DOM object packaged by a Node, too. Pull out into a helper
function, use dict-comprehension and expose as a method of Node.

Change-Id: Ice6737a54a33372b45cf42152e3fdbf5f2da7ba4
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
2024-01-29 15:14:43 +01:00
Edward Welbourne bcdd51cfae Prepare to support taking CLDR data from its github upstream
We've previously used the zip-file form, but that's not been published
for CLDR v44.1 - the advice on the list was to use github
instead. That, however, has ↑↑↑ as a special value for fields, meaning
to inherit from a prent locale. So special-case that value. I have
verified that v44 from the zip file produces identical results to v44
from github, with this minor fix. As it happens v44.1 also produces
identical results.

Pick-to: 6.7 6.5
Change-Id: I6eb0aedda7556753cdc83bb9d76652fbb68dc669
Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
2024-01-19 15:38:25 +01:00
Edward Welbourne e45d05dfc0 Convert UTC offset table look-ups to binary chop
The table was almost sorted by offset - its UTC entry, with offset 0,
was at the front rather than first among the offset 0 entries. The
lookups in it were being done as if the IDs were in space-joined lists
(as for the IANA IDs in the Windows table), splitting on space,
despite the fact that it had separate entries for different IDs at the
same offset (this only arose for offset 0). So actually massage the
input table in python to combine IDs with the same offset using space,
placing UTC first among the offset 0 entries, and ensure the C++ table
is sorted. Regenerated the CLDR data tables using the updated script.

In the process, fix an off-by-one error in the iteration over
space-joined IDs, where the search only advanced to the space, rather
than to just after it. That wasn't a problem before, but now would be.

Change-Id: Ib49c27bac269b557166fa10738c3e396d58456c0
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2023-11-03 18:27:13 +01:00
Edward Welbourne 2636258b29 Add note on -00:00 as offset suffix and UTC-00:00 as zone
The former is overtly forbidden by ISO 8601 but should be recognized
due to RFC 3339's use of it. The "Serialising Extended Data About
Times and Events Working Group" (sedate WG) has established how to
resolve this, so document that conclusion and note the consequent
inadvisability of using UTC-00:00 as a zone ID (although it should
still be accepted).

Change-Id: Ib9fbbe6765117bfa9a84e726d0e75f7397a4c813
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2023-10-27 20:06:06 +02:00
Christian Ehrlicher 4e8b54eb81 Preparations to deprecate QItemDelegate
QItemDelegate was superseded since Qt4 by QStyledItemDelegate but it
took until Qt6.7 to remove the last occurrences in qtbase.
 - remove unused includes / replace with qabstractitemdelegate.h
 - replace references in the documentation with QStyledItemDelegate
 - adjust the examples and tests to use QStyledItemDelegate

Pick-to: 6.5 6.6
Change-Id: I246755004ce2d01192a726ca0972106c237df0cc
Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
Reviewed-by: Paul Wicking <paul.wicking@qt.io>
2023-10-05 21:08:45 +02:00
Edward Welbourne 2ec8acf611 Change enumdata.py names so comments read more naturally
Now that the "and" is only seen in enumdata.py and comments, we can
s/And/and/ in all the various territory names that used it.

Change-Id: Ic376d5904b6f5ab54931f96230c1dd5b7f357b8d
Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
2023-08-09 17:53:45 +02:00
Edward Welbourne 1ae24f8b50 Use CLDR's names in QLocale::*ToName() for language, script, territory
Various comments need to continue using the enumdata.py names, as they
associate data with particular enum members, but we can now correctly
use the en.xml versions of their names when we report them, rather
than the enum-friendly names we use in the code. Since this now means
the data may stray outside plain ASCII - it'll be UTF-8-encoded - this
implies replacing the QLatin1StringView()s of the code that formerly
read this data with QString::fromUtf8().

Fixes: QTBUG-94460
Change-Id: Id3b08875a46af58c0555c3e303b0e15a19441509
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2023-08-09 17:53:42 +02:00
Edward Welbourne afd7d68244 Revise enumdata.py's names to more closely match CLDR's
We could already use dashes in some, rather than spaces, and now no
longer need to capitalize each word. This changes the *_name_list[]
entries for affected languages to more closely match what CLDR gives
as their names. It also amends various comments. Added tests for the
QLocale::*ToString() functions to cover the entries changed.

Task-number: QTBUG-94460
Change-Id: I0163795cb282881f15a97be00a5311c1936c3a09
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
2023-08-09 17:53:36 +02:00
Edward Welbourne 743ceb7cc2 Move enum-name-munging from LocaleHeaderWriter to QLocaleXmlReader
The former needed the latter's .dupes to do the job, so can now just
take a method as a tool to do the job instead, letting .dupes become
private. In the process refine the munging to free enumdata.py from
having to capitalize each word in its names. This will, in due course,
let us use more natural forms in various comments. This causes no
change to generted data.

Update enumdata.py's introduction doc, mainly to reflect this but also
fixing the out-of-date names (old *_list have long been *_map) and
adding some details to other paragraphs.

Task-number: QTBUG-94460
Change-Id: If195b2e94a53a495fc4f1f216bed07a910439fa7
Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
2023-08-09 17:53:26 +02:00