8dd47e34b9 removed the handling of the
BOMs but did not document it. This brings the behavior back and adds a
unit test so we don't break it again.
Discussed-on: http://lists.qt-project.org/pipermail/development/2014-April/016532.html
Change-Id: Ifb7a9a6e5a494622f46b8ab435e1d168b862d952
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
The bit scan function returns the index of the last non-ASCII
character. The next ASCII is the one after this. This means all the
benchmarks were made while reentering the SIMD loop uselessly...
Change-Id: If7de485a63428bfa36d413049d9239ddda1986aa
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
The code didn't check for malformed surrogate pairs. That means that
- high surrogates followed by *anything* were decoded as they formed
a valid surrogate pair;
- stray low surrogates were returned as-is.
We can't return surrogate values in UCS-4, so properly detect these
cases and return U+FFFD instead.
[ChangeLog][QtCore][QTextCodec] Encoding a QString in UTF-32 will now
replace malformed UTF-16 subsequences in the string with the Unicode
replacement character (U+FFFD).
Change-Id: I5cd771d6aa21ffeff4dd9d9e5a7961cf692dc457
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Konstantin Ritt <ritt.ks@gmail.com>
Decoding from UTF-8 is easy: if the high bit is set, we fall back to
the byte-by-byte decoding. Encoding to UTF-8 requires a little bit
more work: to detect anything between 0x0080 and 0xffff, we have
several options but none as easy as above. Multiple alternatives are
in the benchmark code.
In both loops, we do two things once we run into a non-ASCII
character: first, we continue the loop for the remainder of ASCII
characters in the buffer (which we can tell by checking the bits set
in the mask), then we find the last non-ASCII character in that
16-character group, so we don't reenter the SSE code too soon.
For the UTF-8 encoding, I have chosen the alternative that results in
the best performance. It's closely tied to the alternative running the
PMIN instruction, but that requires SSE 4.1. It's not worth the
complexity. And quite counter-intuitively, the dedicated string
instruction from SSE 4.2 performs most poorly of all solutions. This
begs re-visiting the performance of the toLatin1 encoder.
The best of 10 benchmark runs of this code were measured on my
SandyBridge CPU @ 2.66 GHz (turbo @ 3.3 GHz), both as CPU cycles and
as CPU ticks:
Compared to: ICU Qt 4.7 non-SSE Qt 5.3
Data set fromUtf8 toUtf8 fromUtf8 toUtf8 fromUtf8 toUtf8
ASCII only 7.50x 6.22x 6.94x 7.60x 4.45x 4.90x
2-char UTF-8 1.17x 1.33x 1.64x 1.56x 1.01x 1.02x
3-char UTF-8 1.08x 1.18x 1.48x 1.33x 0.97x 0.92x
4-char UTF-8 1.05x 1.19x 1.20x 1.21x 0.97x 0.97x
Creator data 3.62x 2.16x 2.60x 1.25x 1.78x 1.23x
As shown by the numbers, the SSE-based code is slightly worse than the
non-SSE code for dense non-ASCII strings. However, as evident in the
Qt Creator data, most strings manipulated by applications are either
pure ASCII or mostly so, so there's a net gain.
Done-with: H. Peter Anvin <hpa@linux.intel.com>
Change-Id: Ia74fbdfdcd7b088f6cba5048c03a153c01f5dbc1
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Like before, this is taken from the existing QUrl code and is optimized for
ASCII handling (for the same reasons). And like previously, make
QString::fromUtf8 use a stateless version of the codec, which is faster.
There's a small change in behavior in the decoding: we insert a U+FFFD for
each byte that cannot be decoded properly. Previously, it would "eat" all bad
high-bit bytes and replace them all with one single U+FFFD. Either behavior is
allowed by the UTF-8 specifications, even though this new behavior will cause
misalignment in the Bradley Kuhn sample UTF-8 text.
Change-Id: Ib1b1f0b4291293bab345acaf376e00204ed87565
Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
This is a new and faster UTF-8 encoder, based on the code from QUrl. This code
specializes for ASCII, which is the most common case anyway, especially since
QString's "ascii" mode is actually UTF-8 now.
In addition, make QString::toUtf8 use a stateless encoder. Stateless means that
the function doesn't handle state between calls in the form of
QTextCodec::ConverterState. This allows it to be faster than otherwise.
The new code is in the form of a template so that it can be used from
QJsonDocument and QUrl, which have small modifications to how the
encoding is handled.
Change-Id: I305ee0fd8523cc4ec74c2678cb9ea88b75bac7ac
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Changed the processing of non-character code handling in the UTF8 codec.
Non-character codes are now accepted in QStrings, QUrls and QJson strings.
Unit tests were adapted accordingly.
For more info about non-character codes,
see: http://www.unicode.org/versions/corrigendum9.html
[ChangeLog][QtCore][QUtf8]
UTF-8 now accepts non-character unicode points; these are not replaced
by the replacement character anymore
[ChangeLog][QtCore][QUrl]
QUrl now fully accepts non-character unicode points; they are encoded as
percent characters; they can also be pretty decoded
[ChangeLog][QtCore][QJson]
The Writer and the Parser now fully accept non-character unicode points.
Change-Id: I77cf4f0e6210741eac8082912a0b6118eced4f77
Task-number: QTBUG-33229
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Change copyrights and license headers from Nokia to Digia
Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e
Reviewed-by: Lars Knoll <lars.knoll@digia.com>
Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
+ QChar::LastValidCodePoint enum value that supercede the UNICODE_LAST_CODEPOINT macro
replace uses of hardcoded values with the new API; remove leftovers
Change-Id: I1395c9840b85fcb6b08e241b131794a98773c952
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
in order to reduce code duplication and prepare the ground for upcoming changes
Change-Id: I980244149f65384c9484bbec4682de8b7b848b08
Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
0xfdef-0xfdd0 is definitely 31 and not 15 :)
also fix all copy-pastes of this code (greping for '0xfdd0' helps ;)
Change-Id: I8f3bd4fd9d85f9de066f0f5df378b9188c12bd48
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Denis Dzyubenko <denis.dzyubenko@nokia.com>
As in the past, to avoid rewriting various autotests that contain
line-number information, an extra blank line has been inserted at the
end of the license text to ensure that this commit does not change the
total number of lines in the license header.
Change-Id: I311e001373776812699d6efc045b5f742890c689
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
This is the beginning of revision history for this module. If you
want to look at revision history older than this, please refer to the
Qt Git wiki for how to use Git history grafting. At the time of
writing, this wiki is located here:
http://qt.gitorious.org/qt/pages/GitIntroductionWithQt
If you have already performed the grafting and you don't see any
history beyond this commit, try running "git log" with the "--follow"
argument.
Branched from the monolithic repo, Qt master branch, at commit
896db169ea224deb96c59ce8af800d019de63f12
Thiago Macieira
Giuseppe D'Angelo
Kurt Pattyn
Sergio Ahumada
Iikka Eklund
Konstantin Ritt
Jason McDonald
Kent Hansen
suzuki toshiya
Jyri Tahtela
Qt by Nokia