They added some new scripts.
There were a few changes to the line break algorithm,
most notably there is more rules that require more context than before.
While not major, there was some shuffling and additions to our
implementation to match the new rules.
IDNA test data now disallows the trailing dot/empty root label,
technically to be toggled off by an option that controls a few things,
but we don't have options. For test-data they changed the format a
little - "" is used to mean empty string, while a blank segment is
null/no string, update the parser to read this.
Changes in this cherry-pick:
- Reran tool to resolve conflicts due to
emoji-data not being extracted in this branch
[ChangeLog][Third-Party Code] Updated the Unicode Character Database to
UCD revision 34/Unicode 16.
Fixes: QTBUG-132902
Task-number: QTBUG-132851
Pick-to: 6.5
Change-Id: I4569703659f6fd0f20943110a03301c1cf8cc1ed
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
(cherry picked from commit 85899ff181984a1310cd1ad10cdb0824f1ca5118)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
(cherry picked from commit 5985c90d37a096f35b68546f916bec29a218e112)
Those files are read by reuse to complement or override the copyright
and licensing information found in file.
The use of REUSE.toml files was introduced in REUSE version 3.1.0a1.
This reuse version is compatible with reuse specification
version 3.2 [1].
With this commit's files,
* The SPDX document generated by reuse spdx conforms to SPDX 2.3,
* The reuse lint command reports that the Qt project is reuse compliant.
[1]: https://reuse.software/spec-3.2/
Task-number: QTBUG-124453
Task-number: QTBUG-125211
Change-Id: I01023e862607777a5e710669ccd28bbf56091097
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
(cherry picked from commit 4c83657448f66ac0ee46900aff2e5a230eda49b0)
Add enumerator for the new Unicode version to QChar::UnicodeVersion.
Remap new line breaking classes to their Unicode 15.0 values:
* AK, AP and AS to AL,
* VI and VF to CM.
These are classes for new line breaking support for Indic scripts
that require more work.
Blacklist failing tests for now:
* tst_QUrlUts46::idnaTestV2
* tst_QTextBoundaryFinder::lineBoundariesDefault
* tst_QTextBoundaryFinder::graphemeBoundariesDefault
Regenerate the source files.
Task-number: QTBUG-121529
Change-Id: I869cc9fbaa53765d8ae6265c22cdbef9f19d05bf
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This corresponds to Unicode version 15.0.0.
Added the following scripts:
* Kawi
* Nag Mundari
Full support of these scripts requires harfbuzz version 5.2.0,
this version adds support for Unicode 15.0:
https://github.com/harfbuzz/harfbuzz/releases/tag/5.2.0
Fixes: QTBUG-106810
Change-Id: Ib06c526e49b0f01ef9f21123bcf875c6b19f2601
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
There are no commented out test cases remaining, so the normal
test vectors are identical to full test vectors.
Fixes: QTBUG-97537
Pick-to: 6.3
Change-Id: I987f178f192e1c8e2d998d36499fdce84f237e77
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
UAX #14, revision 45 (Unicode 13) has changed rule LB30 to only
trigger if the open parentheses is non-wide:
(AL | HL | NU) × [OP-[\p{ea=F}\p{ea=W}\p{ea=H}]]
This fixes the remaining 24 line break tests.
Task-number: QTBUG-97537
Pick-to: 6.3
Change-Id: I9870588c04bf0f6ae0a98289739bef8490f67f69
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Implement part of LB30b introduced by UAX #14, revision 47
(Unicode 14.0.0):
[\p{Extended_Pictographic}&\p{Cn}] × EM
This fixes one line breaking test.
Task-number: QTBUG-97537
Pick-to: 6.3
Change-Id: I3fd2372a057b7391d8846e9c146f69a54686ea61
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Word breaking rule WB3d should not be affected by WB4.
This fixes the remaining word break test.
Task-number: QTBUG-97537
Pick-to: 6.2 6.3
Change-Id: I99aee831d7c54fafcd2a9d526a3e078b12c5bfad
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Disable break between sequences of WSegSpace characters (rule WB3d,
introduced in UAX #29, version 33, Unicode 11.0.0). Also disable breaks
between WSegSpace and (Extend | Format | ZWJ) due to rule WB4.
Adjust "words4" test to take the above changes into account (space
character belongs to WSegSpace).
Mention the full class name in a comment inside the word break table.
This fixes 34 word break tests.
Task-number: QTBUG-97537
Pick-to: 6.2 6.3
Change-Id: I7dfe8367e45c86913bb7d7fe2adb053711978487
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This rule was simplified in version UTS #14 version 45 (Unicode 13.0.0)
to read:
× IN
Re-enabled 28 fixed line break tests.
Task-number: QTBUG-97537
Pick-to: 6.2 6.3
Change-Id: I1c5565a8c1633428c22379917215d4e424ff0055
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Adjust implementation of rule LB8a of UAX #14. The rule was changed
in version 41 (corresponding to Unicode 11.0.0):
ZWJ × (ID | EB | EM) ⇒ ZWJ ×
Fixing this rule fixes 9 line break tests. Those are re-enabled.
Task-number: QTBUG-97537
Pick-to: 6.2 6.3
Change-Id: I1570719590a46ae28c98ed7d5053e72b12915db7
Reviewed-by: Øystein Heskestad <oystein.heskestad@qt.io>
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
This corresponds to Unicode version 14.0.0.
Added the following scripts:
* CyproMinoan
* OldUyghur
* Tangsa
* Toto
* Vithkuqi
Full support of these scripts requires harfbuzz version 3.0.0,
this version adds support for Unicode 14.0:
https://github.com/harfbuzz/harfbuzz/releases/tag/3.0.0
With this release 10 test cases in tst_qurluts46 were fixed, one
additional test case is failing in tst_qtextboundaryfinder and
is commented out. In total 62 line break test cases and 44 word
break test cases are failing.
A comment in src/corelib/text/qt_attribution.json was updated to
include the URL of the page containing UCD version number.
Fixes: QTBUG-94359
Change-Id: Iefc9ff13f3df279f91cbdb1246d56f75b20ecb35
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
UAX #29 in Unicode 11 changed the EGC algorithm to its current form.
Although Qt has upgraded the Unicode tables all the way up to
Unicode 13, the algorithm has never been adapted; in other words,
it has been working by chance for years. Luckily, MOST
of the cases were dealt with correctly, but emoji handling
actually manages to break it.
This commit:
* Adds parsing of emoji-data.txt into the unicode table generator.
That is necessary to extract the Extended_Pictographic property,
which is used by the EGC algorithm.
* Regenerates the tables.
* Removes some obsoleted grapheme cluster break properties, and
adds the ones added in the meanwhile.
* Rewrites the EGC algorithm according to Unicode 13. This is
done by simplifying a lot the lookup table. Some rules (GB11,
GB12, GB13) can't be done by the table alone so some hand-rolled
code is necessary in that case.
* Thanks to these fixes, the complete upstream GraphemeBreakTest
now passes. Remove the "edited" version that ignored some rows
(because they were failing).
Change-Id: Iaa07cb2e6d0ab9deac28397f46d9af189d2edf8b
Pick-to: 6.1 6.0 5.15
Fixes: QTBUG-92822
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
Include WordBreakTest.html, since a test uses sample strings from it,
albeit without actually reading the file.
Had to comment out more of the new tests, as at Revision 24, pending
an update to harfbuzz and the text boundary detection code.
Task-number: QTBUG-79631
Task-number: QTBUG-79418
Task-number: QTBUG-82747
Change-Id: I0082294b09d67ffdc6a9b5c15acf77ad3b86f65f
Reviewed-by: Lars Knoll <lars.knoll@qt.io>
Had to teach the update program to accept category Lm as for
Joining_Transparent, for the sake of a new ArabicShaping.txt entry.
Added three new Unicode versions, several new scripts and a new
word-break class.
Updated UCD's test data for tst_QTextBoundaryFinder. This left 57
tests failing; I have commented out the data rows for those tests,
pending someone with more knowledge addressing this.
Task-number: QTBUG-79631
Task-number: QTBUG-79418
Change-Id: Ic33d3b3551195d47a84d98e84020f57a68f0b201
Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@qt.io>
Mårten Nordheim
Lucie Gérard
Ievgenii Meshcheriakov
Giuseppe D'Angelo
Edward Welbourne