State of the internet

Methodology

Which domains

We check every domain in the list of .gov.uk domain names published by the Government Digital Service and Central Digital and Data Office on GOV.UK, used under the Open Government Licence v3.0. We fetch the list again before each monthly scan and keep the copy we used with the report: the newest CSV file on that page.

The list contains domains registered by central government, local councils, parish councils, the NHS, schools and other public bodies. Many are small sites, old campaign domains or domains that only exist to send mail, so the figures describe the whole register, not just the best-known services. Names that are not valid host names (such as "_dmarc.gov.uk") are skipped.

We looked at adding a list of popular websites (such as Tranco) as a second group. Tranco publishes no licence of its own and is built from sources with different terms, some of them non-commercial, so we left it out.

What we check, and how

• DNS goes through our own iterative resolver: it asks the root, the uk and gov.uk servers and then each domain's own nameservers, with no public resolver in between. A domain that answers NXDOMAIN, or has no nameservers, counts as "does not exist"; one whose nameservers give no usable answer counts as "DNS failed".
• DMARC: the TXT record at _dmarc.domain; its p= policy and whether it asks for aggregate reports (rua=mailto:). Each .gov.uk domain is its own organisational domain, so there is nothing to inherit from gov.uk.
• SPF: the v=spf1 TXT record, how it ends (-all, ~all, ?all, +all) and how many DNS lookups it needs, following include: and redirect= the way receivers do (RFC 7208 allows 10). Several SPF records count as published but not as ending in -all or ~all. We do not look for DKIM keys: selectors can have any name, so their absence cannot be measured from outside.
• MTA-STS and TLS-RPT: the TXT records at _mta-sts.domain and _smtp._tls.domain, counted for domains that receive mail (with an MX record that is not a null MX). We do not fetch the MTA-STS policy file.
• IPv6 for mail: whether at least one of the domain's first five MX hosts has an AAAA record.
• DNSSEC: a DS record for the domain in the gov.uk zone (the chain of trust), and separately whether the zone publishes DNSKEY records. We do not validate signatures, so a broken signature still counts as DNSSEC; use the DNSSEC check for that.
• CAA: a CAA record on the domain itself. Certificate authorities would also honour one on gov.uk, but at the time of writing gov.uk has none.
• IPv6 for the website: an AAAA record on the domain or on www.
• HTTPS: we connect to the domain (or to www when the domain itself has no IPv4 address) on port 443 and complete a TLS handshake that checks the certificate is trusted and valid for the name. If that fails we connect once more without checking, to record why (expired, wrong name, self-signed, untrusted chain). Our client offers TLS 1.3, so "TLS 1.3 negotiated" means the server supports it.
• HTTP to HTTPS: we request http://host/ and follow up to three redirects that stay on plain HTTP. If one of them sends us to an https:// address, the site redirects.
• HSTS: the Strict-Transport-Security header on https://host/ with a max-age above 0, and whether it includes preload. If the domain sends visitors straight on to https://www. without the header, we read it there instead. Only counted over a trusted certificate, as browsers ignore it otherwise.
• Certificate authority: the organisation in the issuer of the certificate, grouped by brand (for example, RapidSSL and GeoTrust certificates are issued by DigiCert).

From where, and how politely

The scan runs from our web server in London, over IPv4 only (our hosting has no IPv6, so IPv6 figures come from DNS records, not from connections). It runs in small batches from the 1st of each month: a few dozen domains at a time, with only a handful of domains' DNS lookups in flight together and one web connection at a time. Domains are visited in a shuffled order so no single organisation or hosting provider sees a burst. Each domain gets at most two TLS handshakes and a few HTTP requests that stop as soon as the headers arrive; no page content is downloaded.

Our web requests identify themselves with a User-Agent that links to our contact page, which explains how to opt a domain out. Opted-out domains are not scanned and are not shown in the domain lookup.

Limitations

• One snapshot per domain per month, from one place. A site that was down, slow (we wait up to 6 seconds) or blocking us at that moment counts as it looked then.
• Web checks look at one host per domain (the domain itself, or www when the domain has no IPv4 address). Other services on other names are not covered.
• Percentages only count domains where a check gave an answer, and each measure has its own base (shown with every figure). Bases change from month to month as domains appear, disappear or break.
• Registering a domain does not mean it is in use: domains that exist only to protect a name pull adoption figures down. That is part of the picture too: an unused domain without SPF and DMARC can still be used to spoof mail.
• Certificate trust is checked against the CA bundle of our server's operating system, which may differ slightly from browsers'.

Publishing per-domain results

Everything we measure is public: anyone can look it up for any domain. We publish the monthly totals and let anyone look up a single domain's latest result, so the people who run it can see what we saw and fix it. We do not publish a full per-domain list or ranking: a month-old table of weaknesses is more useful to someone looking for an easy target than to the organisations in it.

Privacy

The scan only looks at public DNS records and web servers of organisations. It collects no personal data and sets no cookies.