docs/idn.md - chromium/src - Git at Google

 # Internationalized Domain Names (IDN) in Google Chrome

 ## Background

 Many years ago, domains could only consist of the Latin letters A to Z, digits,
 and a few other characters. [Internationalized Domain Names
 (IDNs)](https://en.wikipedia.org/wiki/Internationalized_domain_name) were
 created to better support non-Latin alphabets for web users around the globe.

 Different characters from different (or even the same!) languages can look very
 similar. We’ve seen
 [reports](https://bugs.chromium.org/p/chromium/issues/detail?id=683314) of
 proof-of-concept attacks. These are called [homograph
 attacks](https://en.wikipedia.org/wiki/IDN_homograph_attack). For example, the
 Latin "a" looks a lot like the Cyrillic "а", so someone could register
 `http://ebаy.com` (using Cyrillic "`а`"), which could be confused for
 `http://ebay.com`. This is a limitation of how URLs are displayed in browsers in
 general, not a specific bug in Chrome.

 In a perfect world, domain registrars would not allow these confusable domain
 names to be registered. Some domain registrars do exactly that, mostly by
 restricting the characters allowed, but many do not. To better protect against
 these attacks, browsers display some domains in
 [punycode](https://en.wikipedia.org/wiki/Punycode) (looks like `xn--...`)
 instead of the original IDN, according to their own IDN policies.

 This is a challenging problem space. Chrome has a global user base of billions
 of people around the world, many of whom are not viewing URLs with Latin
 letters. We want to prevent confusion, while ensuring that users across
 languages have a great experience in Chrome. Displaying either punycode or a
 visible security warning on too wide of a set of URLs would hurt web usability
 for people around the world.

 Chrome and other browsers try to balance these needs by implementing IDN
 policies in a way that allows IDN to be shown for valid domains, but protects
 against confusable homograph attacks.

 Chrome's IDN policy is one of several tools that aim to protect users.
 [Google Safe Browsing](https://safebrowsing.google.com/) continues to help
 protect over two billion devices every day by showing warnings to users when
 they attempt to navigate to dangerous or deceptive sites or download dangerous
 files. Password managers continue to remember which domain password logins are
 for, and won’t automatically fill a password into a domain that is not the
 exactly correct one.

 ## How IDN works

 IDNs were devised to support arbitrary Unicode characters in hostnames in a
 backward-compatible way. This works by having user agents transform hostnames
 containing non-ASCII Unicode characters into an ASCII-only hostname, which can
 then be sent on to DNS servers. This is done by encoding each domain label into
 its punycode representation. This representation includes a four-character
 prefix (`xn--`) and then the unicode translated to ASCII Compatible Encoding
 (ACE). For example, `http://öbb.at` is transformed to `http://xn--bb-eka.at`.

 ## Google Chrome's IDN policy

 Since Chrome 51, Chrome uses an IDN display policy that does not take into
 account the language settings (the Accept-Language list) of the browser. A
 [similar strategy](https://wiki.mozilla.org/IDN_Display_Algorithm#Algorithm) is
 used by Firefox.

 Google Chrome decides if it should show Unicode or punycode for each domain
 label (component) of a hostname separately. To decide if a component should be
 shown in Unicode, Google Chrome uses the following algorithm:
 1. Convert each component stored in the ACE to Unicode per [UTS 46 transitional
    processing](http://unicode.org/reports/tr46/#Processing) (`ToUnicode`).

 2. If there is an error in `ToUnicode` conversion (e.g. contains [disallowed
    characters](http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%3Auts46%3Ddisallowed%3A%5D&abb=on&g=&i=),
    [starts with a combining
    mark](https://unicode-org.github.io/icu-docs/apidoc/released/icu4c/uidna_8h.html#a0411cd49bb5b71852cecd93bcbf0ca2da390a6b3d9844a1dcc1f99fb1ae478ecf),
    or [violates BiDi
    rules](https://unicode-org.github.io/icu-docs/apidoc/released/icu4c/uidna_8h.html#a0411cd49bb5b71852cecd93bcbf0ca2da8a9311811fb0f3db1644ac1a88056370)),
    show punycode.

 3. If there is a character in a label not belonging to [Characters allowed in
    identifiers](http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%3AIdentifierStatus%3DAllowed%3A&abb=on&g=&i=)
    per [Unicode Technical Standard 39 (UTS
    39)](http://www.unicode.org/reports/tr39/#Identifier_Status_and_Type), show
    punycode.

 4. If any character in a label belongs to [the disallowed
    list](https://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%5Cu01CD-%5Cu01DC%5D+%5B%5Cu1c80-%5Cu1c8f%5D++%5B%5Cu1e90-%5Cu1e9b%5D++%5B%5Cu1f00-%5Cu1fff%5D++%5B%5Cua640-%5Cua69f%5D-%5B%5Cua720-%5Cua72f%5D+%5B%5Cu0338+%5Cu058a+%5Cu2010+%5Cu2019+%5Cu2027+%5Cu30a0+%5Cu02bb+%5Cu02bc+%5D&abb=on&g=&i=),
    show punycode.

 5. If the component uses characters drawn from multiple scripts, it is subject
 to a script mixing check based on ["Highly Restrictive" profile of UTS
 39](http://www.unicode.org/reports/tr39/#Restriction_Level_Detection) with an
 additional restriction on Latin. If the component fails the check, show the
 component in punycode.
   - Latin, Cyrillic or Greek characters cannot be mixed with each other
   - Latin characters in the ASCII range can be mixed ONLY with Chinese (Han,
     Bopomofo), Japanese (Kanji, Katakana, Hiragana), or Korean (Hangul, Hanja)
   - Han (CJK Ideographs) can be mixed with Bopomofo
   - Han can be mixed with Hiragana and Katakana
   - Han can be mixed with Korean Hangul

 6. If two or more numbering systems (e.g. European digits + Bengali digits) are
 mixed, show punycode.

 7. If there are any invisible characters (e.g. a sequence of the same combining
 mark or a sequence of Kana combining marks), show punycode.

 8. If there are any characters used in an unusual way, show punycode. E.g.
 [`LATIN MIDDLE DOT (·)`](https://unicode.org/cldr/utility/character.jsp?a=00B7)
 used outside [ela geminada](https://en.wiktionary.org/wiki/ela_geminada).

 9. Test the label for [mixed script confusable per UTS
 39](http://unicode.org/reports/tr39/#Mixed_Script_Confusables). If mixed script
 confusable is detected, show punycode.

 10. Test the label for [whole script
 confusables](http://unicode.org/reports/tr39/#Whole_Script_Confusables): If all
 the letters in a given label belong to a set of whole-script-confusable letters
 in one of the [whole-script-confusable
 scripts](https://cs.chromium.org/chromium/src/components/url_formatter/spoof_checks/idn_spoof_checker.cc?type=cs&q=kWholeScriptConfusables&sq=package:chromium)
 and if the hostname doesn't have a corresponding
 [allowed top-level-domain](https://cs.chromium.org/chromium/src/components/url_formatter/spoof_checks/idn_spoof_checker.h?type=cs&q=allowed_tlds)
 for that script, show punycode.
 **Example for Cyrillic:**
 The first label in hostname `аррӏе.com` (`xn--80ak6aa92e.com`) is all [Cyrillic
 letters that look like Latin letters](http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%D0%B0%D1%81%D4%81%D0%B5%D2%BB%D1%96%D1%98%D3%8F%D0%BE%D1%80%D4%9B%D1%95%D4%9D%D1%85%D1%83%D1%8A%D0%AC%D2%BD%D0%BF%D0%B3%D1%B5%D1%A1%5D&g=gc&i=)
 **AND** the TLD (`com`) is not Cyrillic **AND** the TLD is not one of the TLDs
 known to host a large number of Cyrillic domains (e.g. `ru`, `su`, `pyc`, `ua`).
 Show it in punycode.

 11. If the label contains only [digits and digit
 spoofs](https://cs.chromium.org/chromium/src/components/url_formatter/spoof_checks/idn_spoof_checker.cc?type=cs&q=IsDigitLookalike),
 show punycode.

 12. If the label matches a [dangerous
 pattern](https://cs.chromium.org/chromium/src/components/url_formatter/spoof_checks/idn_spoof_checker.cc?type=cs&g=0&l=422),
 show punycode.

 13. If the [skeleton](http://unicode.org/reports/tr39/#def-skeleton) of the
 registrable part of a hostname is identical to one of the top domains after
 removing diacritic marks and mapping each character to its spoofing skeleton
 (e.g. `www.googlé.com` with `é` in place of `e`), show punycode.

 Otherwise, show Unicode.

 This is implemented by `IDNToUnicodeOneComponent()` and `IsIDNComponentSafe()`
 in
 [`components/url_formatter/url_formatter.cc`](https://cs.chromium.org/search/?q=components/url_formatter/url_formatter.cc)
 and `IDNSpoofChecker` class in
 [`components/url_formatter/spoof_checks/idn_spoof_checker.cc`](https://cs.chromium.org/chromium/src/components/url_formatter/spoof_checks/idn_spoof_checker.cc).

 ## Additional Protections

 In addition to the spoof checks above, Chrome also implements a full page
 security warning to protect against lookalike URLs. You can find an example of
 this warning at `chrome://interstitials/lookalike`. This warning blocks main
 frame navigations that involve lookalike URLs, either as a direct navigation or
 as part of a redirect.

 The algorithm to show this warning is as follows:

 1. If the scheme of the navigation is not `http` or `https`, allow
 the navigation.

 2. If the navigation is a redirect, check the redirect chain. If the redirect
 chain is safe, allow the navigation. (See Defensive Registrations section for
 details).

 3. If the hostname of the navigation has at least a medium site engagement
 score, allow the navigation. Site engagement score is assigned to sites by the
 [Site Engagement
 Service](https://www.chromium.org/developers/design-documents/site-engagement).

 4. If the hostname of the navigation is in
 [`domains.list`](https://cs.chromium.org/chromium/src/components/url_formatter/spoof_checks/top_domains/domains.list),
 allow the navigation.

 5. If the user previously allowed the hostname of the navigation by clicking
 "Ignore" in the warning, allow the navigation. Currently, user decisions are
 stored per tab, so navigating to the same site in a new tab may show the
 warning.

 6. If the hostname has the same skeleton as a recently engaged site or a top 500
 domain, block the navigation and show the warning.

 All of these checks are done locally on the client side.

 ### Defensive Registrations

 Domain owners can sometimes register multiple versions of their domains, such
 as the ASCII and IDN versions, to improve user experience and prevent potential
 spoofs. We call these supplementary domains defensive registrations.

 In some cases, Chrome's lookalike warning may flag and block navigations to
 these domains:
  - If one of the sites is in `domains.list` but the other isn't, the latter will
 be blocked.
  - If the user engaged with one of the sites but not the other, the latter will
 be blocked.

 ### Avoiding a lookalike warning on your site

 **Domain owners can avoid the "Did you mean" warning by redirecting their
 defensive registrations to their canonical domain.**

 **Example**: If you own both `example.com` and `éxample.com` and the majority of
 your traffic is to `example.com`, you can fix the warning by redirecting
 `éxample.com` to `example.com`. The lookalike warning logic considers this a
 safe redirect and allows the navigation. If you must also redirect `http`
 navigations to `https`, do this in a single redirect such as
 `http://éxample.com -> https://example.com`. Use HTTP 301 or HTTP 302
 redirects, the lookalike warning ignores meta redirects.

 ## Reporting Security Bugs

 We reward certain cases of IDN spoofs according to [Chrome's Vulnerability
 Reward Program](https://www.google.com/about/appsecurity/chrome-rewards/index.html)
 policies. Please see [this
 document]( https://docs.google.com/document/d/1_xJz3J9kkAPwk3pma6K3X12SyPTyyaJDSCxTfF8Y5sU/edit?usp=sharing)
 before reporting a security bug.
	# Internationalized Domain Names (IDN) in Google Chrome

	## Background

	Many years ago, domains could only consist of the Latin letters A to Z, digits,
	and a few other characters. [Internationalized Domain Names
	(IDNs)](https://en.wikipedia.org/wiki/Internationalized_domain_name) were
	created to better support non-Latin alphabets for web users around the globe.

	Different characters from different (or even the same!) languages can look very
	similar. We’ve seen
	[reports](https://bugs.chromium.org/p/chromium/issues/detail?id=683314) of
	proof-of-concept attacks. These are called [homograph
	attacks](https://en.wikipedia.org/wiki/IDN_homograph_attack). For example, the
	Latin "a" looks a lot like the Cyrillic "а", so someone could register
	`http://ebаy.com` (using Cyrillic "`а`"), which could be confused for
	`http://ebay.com`. This is a limitation of how URLs are displayed in browsers in
	general, not a specific bug in Chrome.

	In a perfect world, domain registrars would not allow these confusable domain
	names to be registered. Some domain registrars do exactly that, mostly by
	restricting the characters allowed, but many do not. To better protect against
	these attacks, browsers display some domains in
	[punycode](https://en.wikipedia.org/wiki/Punycode) (looks like `xn--...`)
	instead of the original IDN, according to their own IDN policies.

	This is a challenging problem space. Chrome has a global user base of billions
	of people around the world, many of whom are not viewing URLs with Latin
	letters. We want to prevent confusion, while ensuring that users across
	languages have a great experience in Chrome. Displaying either punycode or a
	visible security warning on too wide of a set of URLs would hurt web usability
	for people around the world.

	Chrome and other browsers try to balance these needs by implementing IDN
	policies in a way that allows IDN to be shown for valid domains, but protects
	against confusable homograph attacks.

	Chrome's IDN policy is one of several tools that aim to protect users.
	[Google Safe Browsing](https://safebrowsing.google.com/) continues to help
	protect over two billion devices every day by showing warnings to users when
	they attempt to navigate to dangerous or deceptive sites or download dangerous
	files. Password managers continue to remember which domain password logins are
	for, and won’t automatically fill a password into a domain that is not the
	exactly correct one.

	## How IDN works

	IDNs were devised to support arbitrary Unicode characters in hostnames in a
	backward-compatible way. This works by having user agents transform hostnames
	containing non-ASCII Unicode characters into an ASCII-only hostname, which can
	then be sent on to DNS servers. This is done by encoding each domain label into
	its punycode representation. This representation includes a four-character
	prefix (`xn--`) and then the unicode translated to ASCII Compatible Encoding
	(ACE). For example, `http://öbb.at` is transformed to `http://xn--bb-eka.at`.

	## Google Chrome's IDN policy

	Since Chrome 51, Chrome uses an IDN display policy that does not take into
	account the language settings (the Accept-Language list) of the browser. A
	[similar strategy](https://wiki.mozilla.org/IDN_Display_Algorithm#Algorithm) is
	used by Firefox.

	Google Chrome decides if it should show Unicode or punycode for each domain
	label (component) of a hostname separately. To decide if a component should be
	shown in Unicode, Google Chrome uses the following algorithm:
	1. Convert each component stored in the ACE to Unicode per [UTS 46 transitional
	processing](http://unicode.org/reports/tr46/#Processing) (`ToUnicode`).

	2. If there is an error in `ToUnicode` conversion (e.g. contains [disallowed
	characters](http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%3Auts46%3Ddisallowed%3A%5D&abb=on&g=&i=),
	[starts with a combining
	mark](https://unicode-org.github.io/icu-docs/apidoc/released/icu4c/uidna_8h.html#a0411cd49bb5b71852cecd93bcbf0ca2da390a6b3d9844a1dcc1f99fb1ae478ecf),
	or [violates BiDi
	rules](https://unicode-org.github.io/icu-docs/apidoc/released/icu4c/uidna_8h.html#a0411cd49bb5b71852cecd93bcbf0ca2da8a9311811fb0f3db1644ac1a88056370)),
	show punycode.

	3. If there is a character in a label not belonging to [Characters allowed in
	identifiers](http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%3AIdentifierStatus%3DAllowed%3A&abb=on&g=&i=)
	per [Unicode Technical Standard 39 (UTS
	39)](http://www.unicode.org/reports/tr39/#Identifier_Status_and_Type), show
	punycode.

	4. If any character in a label belongs to [the disallowed
	list](https://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%5Cu01CD-%5Cu01DC%5D+%5B%5Cu1c80-%5Cu1c8f%5D++%5B%5Cu1e90-%5Cu1e9b%5D++%5B%5Cu1f00-%5Cu1fff%5D++%5B%5Cua640-%5Cua69f%5D-%5B%5Cua720-%5Cua72f%5D+%5B%5Cu0338+%5Cu058a+%5Cu2010+%5Cu2019+%5Cu2027+%5Cu30a0+%5Cu02bb+%5Cu02bc+%5D&abb=on&g=&i=),
	show punycode.

	5. If the component uses characters drawn from multiple scripts, it is subject
	to a script mixing check based on ["Highly Restrictive" profile of UTS
	39](http://www.unicode.org/reports/tr39/#Restriction_Level_Detection) with an
	additional restriction on Latin. If the component fails the check, show the
	component in punycode.
	- Latin, Cyrillic or Greek characters cannot be mixed with each other
	- Latin characters in the ASCII range can be mixed ONLY with Chinese (Han,
	Bopomofo), Japanese (Kanji, Katakana, Hiragana), or Korean (Hangul, Hanja)
	- Han (CJK Ideographs) can be mixed with Bopomofo
	- Han can be mixed with Hiragana and Katakana
	- Han can be mixed with Korean Hangul

	6. If two or more numbering systems (e.g. European digits + Bengali digits) are
	mixed, show punycode.

	7. If there are any invisible characters (e.g. a sequence of the same combining
	mark or a sequence of Kana combining marks), show punycode.

	8. If there are any characters used in an unusual way, show punycode. E.g.
	[`LATIN MIDDLE DOT (·)`](https://unicode.org/cldr/utility/character.jsp?a=00B7)
	used outside [ela geminada](https://en.wiktionary.org/wiki/ela_geminada).

	9. Test the label for [mixed script confusable per UTS
	39](http://unicode.org/reports/tr39/#Mixed_Script_Confusables). If mixed script
	confusable is detected, show punycode.

	10. Test the label for [whole script
	confusables](http://unicode.org/reports/tr39/#Whole_Script_Confusables): If all
	the letters in a given label belong to a set of whole-script-confusable letters
	in one of the [whole-script-confusable
	scripts](https://cs.chromium.org/chromium/src/components/url_formatter/spoof_checks/idn_spoof_checker.cc?type=cs&q=kWholeScriptConfusables&sq=package:chromium)
	and if the hostname doesn't have a corresponding
	[allowed top-level-domain](https://cs.chromium.org/chromium/src/components/url_formatter/spoof_checks/idn_spoof_checker.h?type=cs&q=allowed_tlds)
	for that script, show punycode.
	Example for Cyrillic:
	The first label in hostname `аррӏе.com` (`xn--80ak6aa92e.com`) is all [Cyrillic
	letters that look like Latin letters](http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%D0%B0%D1%81%D4%81%D0%B5%D2%BB%D1%96%D1%98%D3%8F%D0%BE%D1%80%D4%9B%D1%95%D4%9D%D1%85%D1%83%D1%8A%D0%AC%D2%BD%D0%BF%D0%B3%D1%B5%D1%A1%5D&g=gc&i=)
	AND the TLD (`com`) is not Cyrillic AND the TLD is not one of the TLDs
	known to host a large number of Cyrillic domains (e.g. `ru`, `su`, `pyc`, `ua`).
	Show it in punycode.

	11. If the label contains only [digits and digit
	spoofs](https://cs.chromium.org/chromium/src/components/url_formatter/spoof_checks/idn_spoof_checker.cc?type=cs&q=IsDigitLookalike),
	show punycode.

	12. If the label matches a [dangerous
	pattern](https://cs.chromium.org/chromium/src/components/url_formatter/spoof_checks/idn_spoof_checker.cc?type=cs&g=0&l=422),
	show punycode.

	13. If the [skeleton](http://unicode.org/reports/tr39/#def-skeleton) of the
	registrable part of a hostname is identical to one of the top domains after
	removing diacritic marks and mapping each character to its spoofing skeleton
	(e.g. `www.googlé.com` with `é` in place of `e`), show punycode.

	Otherwise, show Unicode.

	This is implemented by `IDNToUnicodeOneComponent()` and `IsIDNComponentSafe()`
	in
	[`components/url_formatter/url_formatter.cc`](https://cs.chromium.org/search/?q=components/url_formatter/url_formatter.cc)
	and `IDNSpoofChecker` class in
	[`components/url_formatter/spoof_checks/idn_spoof_checker.cc`](https://cs.chromium.org/chromium/src/components/url_formatter/spoof_checks/idn_spoof_checker.cc).

	## Additional Protections

	In addition to the spoof checks above, Chrome also implements a full page
	security warning to protect against lookalike URLs. You can find an example of
	this warning at `chrome://interstitials/lookalike`. This warning blocks main
	frame navigations that involve lookalike URLs, either as a direct navigation or
	as part of a redirect.

	The algorithm to show this warning is as follows:

	1. If the scheme of the navigation is not `http` or `https`, allow
	the navigation.

	2. If the navigation is a redirect, check the redirect chain. If the redirect
	chain is safe, allow the navigation. (See Defensive Registrations section for
	details).

	3. If the hostname of the navigation has at least a medium site engagement
	score, allow the navigation. Site engagement score is assigned to sites by the
	[Site Engagement
	Service](https://www.chromium.org/developers/design-documents/site-engagement).

	4. If the hostname of the navigation is in
	[`domains.list`](https://cs.chromium.org/chromium/src/components/url_formatter/spoof_checks/top_domains/domains.list),
	allow the navigation.

	5. If the user previously allowed the hostname of the navigation by clicking
	"Ignore" in the warning, allow the navigation. Currently, user decisions are
	stored per tab, so navigating to the same site in a new tab may show the
	warning.

	6. If the hostname has the same skeleton as a recently engaged site or a top 500
	domain, block the navigation and show the warning.

	All of these checks are done locally on the client side.

	### Defensive Registrations

	Domain owners can sometimes register multiple versions of their domains, such
	as the ASCII and IDN versions, to improve user experience and prevent potential
	spoofs. We call these supplementary domains defensive registrations.

	In some cases, Chrome's lookalike warning may flag and block navigations to
	these domains:
	- If one of the sites is in `domains.list` but the other isn't, the latter will
	be blocked.
	- If the user engaged with one of the sites but not the other, the latter will
	be blocked.

	### Avoiding a lookalike warning on your site

	**Domain owners can avoid the "Did you mean" warning by redirecting their
	defensive registrations to their canonical domain.**

	Example: If you own both `example.com` and `éxample.com` and the majority of
	your traffic is to `example.com`, you can fix the warning by redirecting
	`éxample.com` to `example.com`. The lookalike warning logic considers this a
	safe redirect and allows the navigation. If you must also redirect `http`
	navigations to `https`, do this in a single redirect such as
	`http://éxample.com -> https://example.com`. Use HTTP 301 or HTTP 302
	redirects, the lookalike warning ignores meta redirects.

	## Reporting Security Bugs

	We reward certain cases of IDN spoofs according to [Chrome's Vulnerability
	Reward Program](https://www.google.com/about/appsecurity/chrome-rewards/index.html)
	policies. Please see [this
	document]( https://docs.google.com/document/d/1_xJz3J9kkAPwk3pma6K3X12SyPTyyaJDSCxTfF8Y5sU/edit?usp=sharing)
	before reporting a security bug.