whytofly
/
qt6-bb10
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Refactor HSTS cache implementation 

The original monstrosity is not needed at all. It was born only to implement
RFC6797's description of the host matching algorithm (starting from superdomains
and moving to subdomains). Actually, it does not really matter how we find
known host - it can be a congruent match first instead, and then we proceed
with superdomains. This way I can use QMap and my tests so far show it actually
works faster (both insertion and lookup), also the code is cleaner now.

Also, introduce the new class QHstsPolicy that essentially allows to mark
a host as known host and conveniently encapsulates host name/expiration date/
subdomains policy.

Add a public API providing access to HSTS policies, so that client code
can pre-set or read back discovered known hosts (to implement persistent
HSTS storage, for example).

We support server-driven HSTS - this means client code is allowed to provide
policies as hints to QNetworkAccessManager, but these policies can be
overridden by HTTP responses with 'Strict-Transport-Security' headers.

Change-Id: I64d250b6dc78bcb01003fadeded5302471d1389e
Reviewed-by: Timur Pocheptsov <timur.pocheptsov@qt.io>
bb10
Timur Pocheptsov 2017-01-23 12:26:55 +01:00
parent bd78f57463
commit d2758b2f1d
7 changed files with 456 additions and 153 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
src/network/access/access.pri
View File

@ -40,7 +40,8 @@ HEADERS += \
access/qhttpmultipart_p.h \
access/qnetworkfile_p.h \
access/qhttp2protocolhandler_p.h \
access/qhsts_p.h
access/qhsts_p.h \
access/qhstspolicy.h
SOURCES += \
access/qftp.cpp \
@ -74,7 +75,8 @@ SOURCES += \
access/qhttpmultipart.cpp \
access/qnetworkfile.cpp \
access/qhttp2protocolhandler.cpp \
access/qhsts.cpp
access/qhsts.cpp \
access/qhstspolicy.cpp
mac: LIBS_PRIVATE += -framework Security

178
src/network/access/qhsts.cpp
View File

@ -45,17 +45,8 @@
QT_BEGIN_NAMESPACE
static bool expired_policy(const QDateTime &expires)
static bool is_valid_domain_name(const QString &host)
{
return !expires.isValid() || expires <= QDateTime::currentDateTimeUtc();
}
static bool has_valid_domain_name(const QUrl &url)
{
if (!url.isValid())
return false;
const QString host(url.host());
if (!host.size())
return false;
@ -82,117 +73,106 @@ static bool has_valid_domain_name(const QUrl &url)
return true;
}
QHstsCache::QHstsCache()
{
// Top-level domain without any label.
children.push_back(Domain());
}
void QHstsCache::updateFromHeaders(const QList<QPair<QByteArray, QByteArray>> &headers,
const QUrl &url)
{
if (!has_valid_domain_name(url))
if (!url.isValid())
return;
QHstsHeaderParser parser;
if (parser.parse(headers))
updateKnownHost(url, parser.expirationDate(), parser.includeSubDomains());
updateKnownHost(url.host(), parser.expirationDate(), parser.includeSubDomains());
}
void QHstsCache::updateKnownHost(const QUrl &originalUrl, const QDateTime &expires,
void QHstsCache::updateFromPolicies(const QList<QHstsPolicy> &policies)
{
for (const auto &policy : policies)
updateKnownHost(policy.host(), policy.expiry(), policy.includesSubDomains());
}
void QHstsCache::updateKnownHost(const QUrl &url, const QDateTime &expires,
bool includeSubDomains)
{
if (!has_valid_domain_name(originalUrl))
if (!url.isValid())
return;
updateKnownHost(url.host(), expires, includeSubDomains);
}
void QHstsCache::updateKnownHost(const QString &host, const QDateTime &expires,
bool includeSubDomains)
{
if (!is_valid_domain_name(host))
return;
// HSTS is a per-host policy, regardless of protocol, port or any of the other
// details in an URL; so we only want the host part. We still package this as
// a QUrl since this handles IDNA 2003 (RFC3490) for us, as required by
// HSTS (RFC6797, section 10).
QUrl url;
url.setHost(originalUrl.host());
// 1. Update our hosts:
QStringList labels(url.host().split(QLatin1Char('.')));
std::reverse(labels.begin(), labels.end());
size_type domainIndex = 0;
for (int i = 0, e = labels.size(); i < e; ++i) {
Q_ASSERT(domainIndex < children.size());
auto &subDomains = children[domainIndex].labels;
const auto &label = labels[i];
auto pos = std::lower_bound(subDomains.begin(), subDomains.end(), label);
if (pos == subDomains.end() || pos->label != label) {
// A new, previously unknown host.
if (expired_policy(expires)) {
// Nothing to do at all - we did not know this host previously,
// we do not have to - since its policy expired.
return;
}
pos = subDomains.insert(pos, label);
domainIndex = children.size();
pos->domainIndex = domainIndex;
children.resize(children.size() + (e - i));
for (int j = i + 1; j < e; ++j) {
auto &newDomain = children[domainIndex];
newDomain.labels.push_back(labels[j]);
newDomain.labels.back().domainIndex = ++domainIndex;
}
break;
// details in an URL; so we only want the host part. QUrl::host handles
// IDNA 2003 (RFC3490) for us, as required by HSTS (RFC6797, section 10).
const HostName hostName(host);
const auto pos = knownHosts.find(hostName);
const QHstsPolicy newPolicy(expires, includeSubDomains, hostName.name);
if (pos == knownHosts.end()) {
// A new, previously unknown host.
if (newPolicy.isExpired()) {
// Nothing to do at all - we did not know this host previously,
// we do not have to - since its policy expired.
return;
}
domainIndex = pos->domainIndex;
knownHosts.insert(pos, hostName, newPolicy);
return;
}
Q_ASSERT(domainIndex > 0 && domainIndex < children.size());
children[domainIndex].setHostPolicy(expires, includeSubDomains);
if (newPolicy.isExpired())
knownHosts.erase(pos);
else
*pos = std::move(newPolicy);
}
bool QHstsCache::isKnownHost(const QUrl &originalUrl) const
bool QHstsCache::isKnownHost(const QUrl &url) const
{
if (!has_valid_domain_name(originalUrl))
if (!url.isValid() || !is_valid_domain_name(url.host()))
return false;
QUrl url;
url.setHost(originalUrl.host());
/*
RFC6797, 8.2. Known HSTS Host Domain Name Matching
QStringList labels(url.host().split(QLatin1Char('.')));
std::reverse(labels.begin(), labels.end());
* Superdomain Match
If a label-for-label match between an entire Known HSTS Host's
domain name and a right-hand portion of the given domain name
is found, then this Known HSTS Host's domain name is a
superdomain match for the given domain name. There could be
multiple superdomain matches for a given domain name.
* Congruent Match
If a label-for-label match between a Known HSTS Host's domain
name and the given domain name is found -- i.e., there are no
further labels to compare -- then the given domain name
congruently matches this Known HSTS Host.
Q_ASSERT(children.size());
size_type domainIndex = 0;
for (int i = 0, e = labels.size(); i < e; ++i) {
Q_ASSERT(domainIndex < children.size());
const auto &subDomains = children[domainIndex].labels;
auto pos = std::lower_bound(subDomains.begin(), subDomains.end(), labels[i]);
if (pos == subDomains.end() || pos->label != labels[i])
return false;
We start from the congruent match, and then chop labels and dots and
proceed with superdomain match. While RFC6797 recommends to start from
superdomain, the result is the same - some valid policy will make a host
known.
*/
Q_ASSERT(pos->domainIndex < children.size());
domainIndex = pos->domainIndex;
auto &domain = children[domainIndex];
if (domain.validateHostPolicy() && (i + 1 == e || domain.includeSubDomains)) {
/*
RFC6797, 8.2. Known HSTS Host Domain Name Matching
* Superdomain Match
If a label-for-label match between an entire Known HSTS Host's
domain name and a right-hand portion of the given domain name
is found, then this Known HSTS Host's domain name is a
superdomain match for the given domain name. There could be
multiple superdomain matches for a given domain name.
* Congruent Match
If a label-for-label match between a Known HSTS Host's domain
name and the given domain name is found -- i.e., there are no
further labels to compare -- then the given domain name
congruently matches this Known HSTS Host.
*/
return true;
bool superDomainMatch = false;
const QString hostNameAsString(url.host());
HostName nameToTest(static_cast<QStringRef>(&hostNameAsString));
while (nameToTest.fragment.size()) {
auto const pos = knownHosts.find(nameToTest);
if (pos != knownHosts.end()) {
if (pos.value().isExpired())
knownHosts.erase(pos);
else if (!superDomainMatch || pos.value().includesSubDomains())
return true;
}
const int dot = nameToTest.fragment.indexOf(QLatin1Char('.'));
if (dot == -1)
break;
nameToTest.fragment = nameToTest.fragment.mid(dot + 1);
superDomainMatch = true;
}
return false;
@ -200,10 +180,12 @@ bool QHstsCache::isKnownHost(const QUrl &originalUrl) const
void QHstsCache::clear()
{
children.resize(1);
children[0].labels.clear();
// Top-level is never known:
Q_ASSERT(!children[0].isKnownHost);
knownHosts.clear();
}
QList<QHstsPolicy> QHstsCache::policies() const
{
return knownHosts.values();
}
// The parser is quite simple: 'nextToken' knowns exactly what kind of tokens

81
src/network/access/qhsts_p.h
View File

@ -51,17 +51,16 @@
// We mean it.
//
#include <QtNetwork/qhstspolicy.h>
#include <QtCore/qbytearray.h>
#include <QtCore/qdatetime.h>
#include <QtCore/qstring.h>
#include <QtCore/qglobal.h>
#include <QtCore/qvector.h>
#include <QtCore/qlist.h>
#include <QtCore/qpair.h>
#include <QtCore/qurl.h>
#include <algorithm>
#include <vector>
#include <QtCore/qmap.h>
QT_BEGIN_NAMESPACE
@ -69,72 +68,48 @@ class Q_AUTOTEST_EXPORT QHstsCache
{
public:
QHstsCache();
void updateFromHeaders(const QList<QPair<QByteArray, QByteArray>> &headers,
const QUrl &url);
void updateFromPolicies(const QList<QHstsPolicy> &hosts);
void updateKnownHost(const QUrl &url, const QDateTime &expires,
bool includeSubDomains);
bool isKnownHost(const QUrl &url) const;
void clear();
QList<QHstsPolicy> policies() const;
private:
using size_type = std::vector<int>::size_type;
void updateKnownHost(const QString &hostName, const QDateTime &expires,
bool includeSubDomains);
struct DomainLabel
struct HostName
{
DomainLabel(const QString &name = QString()) : label(name), domainIndex(0) {}
explicit HostName(const QString &n) : name(n) { }
explicit HostName(const QStringRef &r) : fragment(r) { }
bool operator < (const DomainLabel &rhs) const
{ return label < rhs.label; }
QString label;
size_type domainIndex;
};
struct Domain
{
void setHostPolicy(const QDateTime &expiration, bool subs)
bool operator < (const HostName &rhs) const
{
expirationTime = expiration;
isKnownHost = expirationTime.isValid()
&& expirationTime > QDateTime::currentDateTimeUtc();
includeSubDomains = subs;
if (fragment.size()) {
if (rhs.fragment.size())
return fragment < rhs.fragment;
return fragment < QStringRef(&rhs.name);
}
if (rhs.fragment.size())
return QStringRef(&name) < rhs.fragment;
return name < rhs.name;
}
bool validateHostPolicy()
{
if (!isKnownHost)
return false;
if (expirationTime > QDateTime::currentDateTimeUtc())
return true;
isKnownHost = false;
includeSubDomains = false;
return false;
}
bool isKnownHost = false;
bool includeSubDomains = false;
QDateTime expirationTime;
std::vector<DomainLabel> labels;
// We use 'name' for a HostName object contained in our dictionary;
// we use 'fragment' only during lookup, when chopping the complete host
// name, removing subdomain names (such HostName object is 'transient', it
// must not outlive the original QString object.
QString name;
QStringRef fragment;
};
/*
Each Domain represents a DNS name or prefix thereof; each entry in its
std::vector<DomainLabel> labels pairs the next fragment of a DNS name
with the index into 'children' at which to find another Domain object.
The root Domain, children[0], has top-level-domain DomainLabel entries,
such as "com", "org" and "net"; the entry in 'children' at the index it
pairs with "com" is the Domain entry for .com; if that has "example" in
its labels, it'll be paired with the index of the entry in 'children'
that represents example.com; from which, in turn, we can find the
Domain object for www.example.com, and so on.
*/
mutable std::vector<Domain> children;
mutable QMap<HostName, QHstsPolicy> knownHosts;
};
class Q_AUTOTEST_EXPORT QHstsHeaderParser

218
src/network/access/qhstspolicy.cpp Normal file
View File

@ -0,0 +1,218 @@
/****************************************************************************
**
** Copyright (C) 2017 The Qt Company Ltd.
** Contact: https://www.qt.io/licensing/
**
** This file is part of the QtNetwork module of the Qt Toolkit.
**
** $QT_BEGIN_LICENSE:LGPL$
** Commercial License Usage
** Licensees holding valid commercial Qt licenses may use this file in
** accordance with the commercial license agreement provided with the
** Software or, alternatively, in accordance with the terms contained in
** a written agreement between you and The Qt Company. For licensing terms
** and conditions see https://www.qt.io/terms-conditions. For further
** information use the contact form at https://www.qt.io/contact-us.
**
** GNU Lesser General Public License Usage
** Alternatively, this file may be used under the terms of the GNU Lesser
** General Public License version 3 as published by the Free Software
** Foundation and appearing in the file LICENSE.LGPL3 included in the
** packaging of this file. Please review the following information to
** ensure the GNU Lesser General Public License version 3 requirements
** will be met: https://www.gnu.org/licenses/lgpl-3.0.html.
**
** GNU General Public License Usage
** Alternatively, this file may be used under the terms of the GNU
** General Public License version 2.0 or (at your option) the GNU General
** Public license version 3 or any later version approved by the KDE Free
** Qt Foundation. The licenses are as published by the Free Software
** Foundation and appearing in the file LICENSE.GPL2 and LICENSE.GPL3
** included in the packaging of this file. Please review the following
** information to ensure the GNU General Public License requirements will
** be met: https://www.gnu.org/licenses/gpl-2.0.html and
** https://www.gnu.org/licenses/gpl-3.0.html.
**
** $QT_END_LICENSE$
**
****************************************************************************/
#include "qhstspolicy.h"
#include <QtCore/qdatetime.h>
#include <QtCore/qstring.h>
QT_BEGIN_NAMESPACE
/*!
\class QHstsPolicy
\brief The QHstsPolicy class specifies that a host supports HTTP Strict Transport
Security policy (HSTS).
\since 5.9
\ingroup network
\inmodule QtNetwork
HSTS policy defines a period of time during which QNetworkAccessManager
should only access a host in a secure fashion. HSTS policy is defined by
RFC6797.
You can set expiry time and host name for this policy, and control whether it
applies to subdomains, either in the constructor or by calling setExpiry(),
setHost() and setIncludesSubdomains().
\sa QNetworkAccessManager::enableStrictTransportSecurity()
*/
class QHstsPolicyPrivate
{
public:
QUrl url;
QDateTime expiry;
bool includeSubDomains = false;
bool operator == (const QHstsPolicyPrivate &other) const
{
return url.host() == other.url.host() && expiry == other.expiry
&& includeSubDomains == other.includeSubDomains;
}
};
/*!
Constructs an invalid (expired) policy with empty host name and subdomains
not included.
*/
QHstsPolicy::QHstsPolicy() : d(new QHstsPolicyPrivate)
{
}
/*!
Constructs QHstsPolicy with \a expiry (in UTC); \a includeSubDomains parameter
defines if this policy must also include subdomains, \a host data is interpreted
according to \a mode.
\sa QUrl::setHost(), QUrl::ParsingMode
*/
QHstsPolicy::QHstsPolicy(const QDateTime &expiry, bool includeSubDomains, const QString &host,
QUrl::ParsingMode mode)
: d(new QHstsPolicyPrivate)
{
d->url.setHost(host, mode);
d->expiry = expiry;
d->includeSubDomains = includeSubDomains;
}
/*!
Creates a copy of \a other object.
*/
QHstsPolicy::QHstsPolicy(const QHstsPolicy &other)
: d(new QHstsPolicyPrivate(*other.d))
{
}
/*!
Destructor.
*/
QHstsPolicy::~QHstsPolicy()
{
}
/*!
Copy-assignment operator, makes a copy of \a other.
*/
QHstsPolicy &QHstsPolicy::operator=(const QHstsPolicy &other)
{
*d = *other.d;
return *this;
}
/*!
Move-assignment operator.
*/
QHstsPolicy &QHstsPolicy::operator=(QHstsPolicy &&other) Q_DECL_NOTHROW
{
qSwap(d, other.d);
return *this;
}
/*!
Sets a host, \a host data is interpreted according to \a mode parameter.
\sa host(), QUrl::setHost(), QUrl::ParsingMode
*/
void QHstsPolicy::setHost(const QString &host, QUrl::ParsingMode mode)
{
d->url.setHost(host, mode);
}
/*!
Returns a host for a given policy, formatted according to \a options.
\sa setHost(), QUrl::host(), QUrl::ComponentFormattingOptions
*/
QString QHstsPolicy::host(QUrl::ComponentFormattingOptions options) const
{
return d->url.host(options);
}
/*!
Sets the expiration date for the policy (in UTC).
\sa expiry()
*/
void QHstsPolicy::setExpiry(const QDateTime &expiry)
{
d->expiry = expiry;
}
/*!
Returns the expiration date for the policy (in UTC).
\sa setExpiry()
*/
QDateTime QHstsPolicy::expiry() const
{
return d->expiry;
}
/*!
Includes or excludes subdomains for this policy.
\sa includeSubdomains()
*/
void QHstsPolicy::setIncludesSubDomains(bool include)
{
d->includeSubDomains = include;
}
/*!
Returns \c true if this policy also includes subdomains.
\sa setIncludesSubDomains()
*/
bool QHstsPolicy::includesSubDomains() const
{
return d->includeSubDomains;
}
/*!
Returns \c true if the two policies have the same host and expriration date
while agreeing on whether to include or exclude subdomains.
*/
bool QHstsPolicy::operator==(const QHstsPolicy &other) const
{
return *d == *other.d;
}
/*!
Return \c true if this policy has a valid expiration date and this date
is greater than QDateTime::currentGetDateTimeUtc().
\sa setExpiry(), expiry()
*/
bool QHstsPolicy::isExpired() const
{
return !d->expiry.isValid() || d->expiry <= QDateTime::currentDateTimeUtc();
}
QT_END_NAMESPACE

82
src/network/access/qhstspolicy.h Normal file
View File

@ -0,0 +1,82 @@
/****************************************************************************
**
** Copyright (C) 2017 The Qt Company Ltd.
** Contact: https://www.qt.io/licensing/
**
** This file is part of the QtNetwork module of the Qt Toolkit.
**
** $QT_BEGIN_LICENSE:LGPL$
** Commercial License Usage
** Licensees holding valid commercial Qt licenses may use this file in
** accordance with the commercial license agreement provided with the
** Software or, alternatively, in accordance with the terms contained in
** a written agreement between you and The Qt Company. For licensing terms
** and conditions see https://www.qt.io/terms-conditions. For further
** information use the contact form at https://www.qt.io/contact-us.
**
** GNU Lesser General Public License Usage
** Alternatively, this file may be used under the terms of the GNU Lesser
** General Public License version 3 as published by the Free Software
** Foundation and appearing in the file LICENSE.LGPL3 included in the
** packaging of this file. Please review the following information to
** ensure the GNU Lesser General Public License version 3 requirements
** will be met: https://www.gnu.org/licenses/lgpl-3.0.html.
**
** GNU General Public License Usage
** Alternatively, this file may be used under the terms of the GNU
** General Public License version 2.0 or (at your option) the GNU General
** Public license version 3 or any later version approved by the KDE Free
** Qt Foundation. The licenses are as published by the Free Software
** Foundation and appearing in the file LICENSE.GPL2 and LICENSE.GPL3
** included in the packaging of this file. Please review the following
** information to ensure the GNU General Public License requirements will
** be met: https://www.gnu.org/licenses/gpl-2.0.html and
** https://www.gnu.org/licenses/gpl-3.0.html.
**
** $QT_END_LICENSE$
**
****************************************************************************/
#ifndef QHSTSPOLICY_H
#define QHSTSPOLICY_H
#include <QtNetwork/qtnetworkglobal.h>
#include <QtCore/qscopedpointer.h>
#include <QtCore/qurl.h>
QT_BEGIN_NAMESPACE
class QHstsPolicyPrivate;
class QDateTime;
class QString;
class Q_NETWORK_EXPORT QHstsPolicy
{
public:
QHstsPolicy();
QHstsPolicy(const QDateTime &expiry, bool includeSubDomains, const QString &host,
QUrl::ParsingMode mode = QUrl::DecodedMode);
QHstsPolicy(const QHstsPolicy &rhs);
QHstsPolicy &operator=(const QHstsPolicy &rhs);
QHstsPolicy &operator=(QHstsPolicy &&rhs) Q_DECL_NOTHROW;
~QHstsPolicy();
void setHost(const QString &host, QUrl::ParsingMode mode = QUrl::DecodedMode);
QString host(QUrl::ComponentFormattingOptions options = QUrl::FullyDecoded) const;
void setExpiry(const QDateTime &expiry);
QDateTime expiry() const;
void setIncludesSubDomains(bool include);
bool includesSubDomains() const;
bool operator==(const QHstsPolicy &rhs) const;
bool isExpired() const;
private:
QScopedPointer<QHstsPolicyPrivate> d;
};
QT_END_NAMESPACE
#endif // QHSTSPOLICY_H

41
src/network/access/qnetworkaccessmanager.cpp
View File

@ -45,6 +45,8 @@
#include "qnetworkcookie.h"
#include "qnetworkcookiejar.h"
#include "qabstractnetworkcache.h"
#include "qhstspolicy.h"
#include "qhsts_p.h"
#include "QtNetwork/qnetworksession.h"
#include "QtNetwork/private/qsharednetworksession_p.h"
@ -741,6 +743,45 @@ bool QNetworkAccessManager::strictTransportSecurityEnabled() const
return d->stsEnabled;
}
/*!
\since 5.9
Adds HTTP Strict Transport Security policies into HSTS cache.
\note An expired policy will remove a known host from the cache, if previously
present.
\note While processing HTTP responses, QNetworkAccessManager can also update
the HSTS cache, removing or updating exitsting policies or introducing new
known hosts. The current implementation thus is server-driven, client code
can provide QNetworkAccessManager with previously known or discovered
policies, but this information can be overridden by "Strict-Transport-Security"
response headers.
\sa addStrictTransportSecurityHosts(), QHstsPolicy
*/
void QNetworkAccessManager::addStrictTransportSecurityHosts(const QList<QHstsPolicy> &knownHosts)
{
Q_D(QNetworkAccessManager);
d->stsCache.updateFromPolicies(knownHosts);
}
/*!
\since 5.9
Returns the list of HTTP Strict Transport Security policies. This list can
differ from what was initially set via addStrictTransportSecurityHosts() if
HSTS cache was updated from a "Strict-Transport-Security" response header.
\sa addStrictTransportSecurityHosts(), QHstsPolicy
*/
QList<QHstsPolicy> QNetworkAccessManager::strictTransportSecurityHosts() const
{
Q_D(const QNetworkAccessManager);
return d->stsCache.policies();
}
/*!
Posts a request to obtain the network headers for \a request
and returns a new QNetworkReply object which will contain such headers.

3
src/network/access/qnetworkaccessmanager.h
View File

@ -61,6 +61,7 @@ class QNetworkReply;
class QNetworkProxy;
class QNetworkProxyFactory;
class QSslError;
class QHstsPolicy;
#ifndef QT_NO_BEARERMANAGEMENT
class QNetworkConfiguration;
#endif
@ -123,6 +124,8 @@ public:
void enableStrictTransportSecurity();
void disableStrictTransportSecurity();
bool strictTransportSecurityEnabled() const;
void addStrictTransportSecurityHosts(const QList<QHstsPolicy> &knownHosts);
QList<QHstsPolicy> strictTransportSecurityHosts() const;
QNetworkReply *head(const QNetworkRequest &request);
QNetworkReply *get(const QNetworkRequest &request);