...where the values are not aliased to Common script.
The old QUnicodeTables::Script enum was retained for compatibility reasons
until Qt internals are updated to use QChar::script().
Using QChar::Script instead of QUnicodeTables::Script would improve both
the text analysis (itemization, boundary finding) and the text shaping quality.
This also a required step for switching to Hurfbuzz-NG.
/* This adds 6668 more .rodata bytes */
Change-Id: I5aa3d12c550528d0052542436990f8d0779ea8e5
Reviewed-by: Eskil Abrahamsen Blomfeldt <eskil.abrahamsen-blomfeldt@digia.com>
Version 6.2 of the Unicode Standard is a special release
dedicated to the early publication of the newly encoded Turkish lira sign.
In addition, there are some significant changes to the Unicode algorithms
for text segmentation and line breaking to improve breaking for emoji symbols.
For more details, see http://www.unicode.org/versions/Unicode6.2.0/
Change-Id: I21cfd4f307e41b41a19d36cce87f7a44c2661bc2
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Change copyrights and license headers from Nokia to Digia
Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
Chars that have a case conversion that converts
them into several characters can't be handled
by QChar::toUpper() etc and should get ignored. The code
didn't do that correctly.
Change-Id: I281d122e90bf49187b6449088d2fccef2ef75e86
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Make them call exactly their Latin 1 counterparts.
For the QString functions that take a single char, also use fromAscii
directly.
Change-Id: I87645aba6ab9cde34c1df3cbc3a979fbd9e91f9d
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
+ QChar::LastValidCodePoint enum value that supercede the UNICODE_LAST_CODEPOINT macro
replace uses of hardcoded values with the new API; remove leftovers
Change-Id: I1395c9840b85fcb6b08e241b131794a98773c952
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
inline all non-static members to a static ones (declared with QT_FASTCALL),
ushort converts automatically to uint and the conversion cost is minimal.
Task-Number: QTBUG-13052
Change-Id: I189a6f205736766adcd3de2d61cee71f30cc64f3
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
in Unicode 5.1, it was clarified that unassigned codepoints normally
should not be excluded from the process; moreover, unassigned codepoints
assumed to be starters in decomposition by default
Change-Id: Ic4a61ec3759ee62b6843c81045e1d611d0684ead
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
> http://www.unicode.org/versions/Unicode5.2.0/
D. Character Additions:
There are three new characters in the newly-encoded Kaithi script that will
require changes in implementations which make hard-coded assumptions about
composition during normalization. Most new characters added to the standard
with decompositions cannot be generated by the operations toNFC() or toNFKC),
but these three can. Implementers should check their code carefully
to ensure that it handles these three characters correctly.
U+1109A KAITHI LETTER DDDHA
U+1109C KAITHI LETTER RHA
U+110AB KAITHI LETTER VA
UCD 6.1 adds two more of them:
U+1112E CHAKMA VOWEL SIGN O
U+1112F CHAKMA VOWEL SIGN AU
Change-Id: I781a26848078d8b83a182b0fd4e681be2a6d9a27
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Those functions are about to be deprecated and shouldn't be referred
to in the main documentation. Since they were temporarily changed to
mean UTF-8, this is not a behaviour change.
The next commit will update the code to match the documentation.
Change-Id: Ia8c2843c7f2b478f5691fe0224d5e631d94b1af6
Reviewed-by: Casper van Donderen <casper.vandonderen@nokia.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
* by not using QString::fromUtf16() as we know for sure that
the data is 'raw' UCS-2;
* it's safe to avoid a check for > UNICODE_LAST_CODEPOINT
as GET_DECOMPOSITION_INDEX macro already does a similar check
Change-Id: Ifb660efc51c664d06733ac8ed46d54278520da06
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
there are two such codepoints were added in the Unicode 5.1:
U+1D79 LATIN SMALL LETTER INSULAR G
U+A77D LATIN CAPITAL LETTER INSULAR G
two more of them were added in the Unicode 6.0:
U+0265 LATIN SMALL LETTER TURNED H
U+A78D LATIN CAPITAL LETTER TURNED H
and two more were added in the Unicode 6.1:
U+0266 LATIN SMALL LETTER H WITH HOOK
U+A7AA LATIN CAPITAL LETTER H WITH HOOK
we map them like special cases with length == 1
(note: all are in BMP which is checked explicitly in the generator)
Change-Id: I8a34164eb3ee2e575b7799cc12d4b96ad5bcd9c6
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
using templates for toLower()/toUpper()/toTitleCase()/toCaseFolded()
makes the upcoming patches smaller and thus easier to review
Change-Id: Ideb23e8669dbc2fe9ea3f129bf0137e1805ece11
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
if there is no need to swap codepoints A and B,
then we proceeding with advance to the next codepoint B
that becomes A, and some next codepoint C becomes B;
in such case we can easily skip the re-calculations for A
by using previously calculated results for B
Change-Id: I5c63589c274acaddf0f6a4cb1e0608d352a0c1b3
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
results are now equals to results of ICU's u_isprint() for the entire set
of the Unicode code points
Change-Id: I763f4b37cccd285eb01543d486f25bd7ea011241
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
by reordering and regrouping conditions so that they lead to result earlier
in most-common usecases (l.letters, spaces and puncts, u.letters, other);
there are no title cased letters in range [0..127] -> use this in isTitleCase();
test for 0xa0 (nbsp) early in isSpace() as it is quite common in HTML, etc.;
add early test to isNumber().
Change-Id: Ib415f34cb1212d9ccf8753de2d1beaece1aa2243
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@nokia.com>
This setting is extremely harmful, as code cannot know whether or not to expect
it. It also made the behaviour of QString::fromAscii and ::toAscii unintuitive,
and caused a lot of people to make mistakes with it.
Change-Id: I2f429fa7ef93bd75bb93a7f64c56db15b7283388
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
As in the past, to avoid rewriting various autotests that contain
line-number information, an extra blank line has been inserted at the
end of the license text to ensure that this commit does not change the
total number of lines in the license header.
Change-Id: I311e001373776812699d6efc045b5f742890c689
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
Make it inline; add fast checks for typical spaces;
add fallback function that uses the fastcall calling
convention.
On ia32, this change makes isSpace ~340x faster for
ascii spaces, ~170x faster for non-space ascii
characters, and ~1.3x faster for non-ascii characters.
Note that this change is NOT binary compatible.
Also add an autotest with expected results from
before the optimization, to ensure that the behavior
is the same.
Change-Id: I9438d0ad3c9ba2e80560c4bee7eed05115265798
Reviewed-on: http://codereview.qt-project.org/4905
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Make it inline; add fast checks for ascii letters
and digits; add fallback function that uses the
fastcall calling convention.
On ia32, this change makes isLetterOrNumber ~120x
faster for ascii letters and digits, ~150x faster
for non-letter/digit ascii characters, and ~1.3x
faster for non-ascii characters.
Note that this change is NOT binary compatible.
Also add an autotest with expected results from
before the optimization, to ensure that the
behavior is the same.
Change-Id: Ia4e13692f4dd79f6aa0b96da29449e0487971b0e
Reviewed-on: http://codereview.qt-project.org/4904
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Make it inline; add fast checks for ascii letters;
add fallback function that uses the fastcall calling
convention.
On ia32, this change makes isLetter ~370x faster for
ascii letters, ~250x faster for non-letter ascii
characters, and ~1.5x faster for non-ascii characters.
Note that this change is NOT binary compatible.
Also add an autotest with expected results from
before the optimization, to ensure that the
behavior is the same.
Change-Id: I06f8d3d43114537cee5567e670898cef6494c20a
Reviewed-on: http://codereview.qt-project.org/4903
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
Make it inline; add fast checks for ascii digits;
add fallback function that uses the fastcall calling
convention.
On ia32, this change makes isDigit ~370x faster for
ascii digit characters, ~250x faster for non-digit
ascii characters, and ~1.5x faster for non-ascii
characters.
Note that this change is NOT binary compatible.
Also add an autotest with expected results from
before the optimization, to ensure that the
behavior is the same.
Change-Id: I718fadecda3f591d6f4c22374d8e476f4724fd83
Reviewed-on: http://codereview.qt-project.org/4902
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
http://www.unicode.org/reports/tr15/#Guaranteeing_Process_Stability:
> handle any code points that were not defined in the earlier version
> as if they were unassigned: such code points will not decompose or compose,
> and their Canonical_Combining_Class value will be zero.
since QChar::Unicode_Unassigned value is 0, it's less than any other
QChar::UnicodeVersion value and must ba handled explicitly
Change-Id: I6df025b4173d407660adae77ec5eeb98d15cb8ce
Reviewed-on: http://codereview.qt.nokia.com/4084
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
there is no such category in the Unicode specs. the QChar::NoCategory
was a subject of bugs since it was introduced. int 4.6 it's meaning was
limited to mention ucs4 > UNICODE_LAST_CODEPOINT only (which is useless anyways)
in order to preserve the old (wrong) behavior.
fix it now for qtbase
Change-Id: I630534824e071090b39772881e747c1fdb758719
Reviewed-on: http://codereview.qt.nokia.com/1584
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
When we removed the useless "const" in the return type, we broke
compatibility with a few compilers that include the return type in the
mangling. We don't need that anymore in Qt 5.
This change should have had a ### Qt5 mark everywhere, not just in a
comment in qstring.cpp.
Change-Id: I8839f8dc540b34e57a3efdb160a1c015f7328422
Merge-request: 13
Reviewed-by: Olivier Goffart <olivier.goffart@nokia.com>
Reviewed-on: http://codereview.qt.nokia.com/1385
Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com>
This is the beginning of revision history for this module. If you
want to look at revision history older than this, please refer to the
Qt Git wiki for how to use Git history grafting. At the time of
writing, this wiki is located here:
http://qt.gitorious.org/qt/pages/GitIntroductionWithQt
If you have already performed the grafting and you don't see any
history beyond this commit, try running "git log" with the "--follow"
argument.
Branched from the monolithic repo, Qt master branch, at commit
896db169ea224deb96c59ce8af800d019de63f12
Frederik Gladhorn
Sergio Ahumada
Konstantin Ritt
Iikka Eklund
Thiago Macieira
Lars Knoll
Robin Burchell
Jason McDonald
Casper van Donderen
Kent Hansen
David Boddie
Thiago Macieira
Jyri Tahtela
Qt by Nokia