* rndunix.c (start_gatherer) [ENABLE_SELINUX_HACKS]: Don't allow
[gnupg.git] / contrib / UTF-8-test.txt
1 UTF-8 decoder capability and stress test                                      |
2 ----------------------------------------                                      |
3                                                                               |
4 Markus Kuhn <mkuhn@acm.org> - 1999-11-07                                      |
5                                                                               |
6 This test text examines, how UTF-8 decoders handle various types of           |
7 corrupted or otherwise interesting UTF-8 sequences.                           |
8                                                                               |
9 According to ISO 10646-1, sections R.7 and 2.3c, a device receiving           |
10 UTF-8 shall interpret a "malformed sequence in the same way that it           |
11 interprets a character that is outside the adopted subset". This means        |
12 usually that the malformed UTF-8 sequence is replaced by a replacement        |
13 character (U+FFFD), which looks a bit like an inverted question mark,         |
14 or a similar symbol. It might be a good idea to visually distinguish a        |
15 malformed UTF-8 sequence from a correctly encoded Unicode character           |
16 that is just not available in the current font but otherwise fully            |
17 legal. For both cases, a clearly recognisable symbol should be used.          |
18 Just ignoring malformed sequences or unavailable characters will make         |
19 debugging more difficult and can lead to user confusion.                      |
20                                                                               |
21 Test sequences:                                                               |
22                                                                               |
23 Check, whether a malformed UTF-8 sequence is (1) represented at all,          |
24 (2) represented by exactly one replacement character (or equivalent           |
25 signal), and (3) the following quotation mark after an illegal UTF-8          |
26 sequence is correctly displayed, i.e. proper resynchronization takes          |
27 place. This file says "THE END" in the last line, so if you don't see         |
28 that, your decoder crashed somehow before, which is also not nice.            |
29                                                                               |
30 All lines in this file are exactly 79 characters long (plus the line          |
31 feed). In addition, all lines end with "|", except for the two test           |
32 lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls             |
33 U+0000 and U+007F. If you display this file with a fixed-width font,          |
34 these "|" characters should all line up in column 79 (right margin).          |
35 This allows you to test quickly, whether your UTF-8 decoder finds the         |
36 correct number of characters in every line.                                   |
37                                                                               |
38 Here come the tests:                                                          |
39                                                                               |
40 1  Some correct UTF-8 text                                                    |
41                                                                               |
42 You should see the Greek word 'kosme':       "κόσμε"                          |
43                                                                               |
44 2  Minimum and maximum valid values for every UTF-8 sequence length           |
45                                                                               |
46 You should see a correctly encoded character each time (which is not          |
47 necessarily an existing character).                                           |
48                                                                               |
49 2.1  First possible sequence of a certain length                              |
50                                                                               |
51 2.1.1  1 byte  (U-00000000):        "\0"                                        
52 2.1.2  2 bytes (U-00000080):        "\80"                                       |
53 2.1.3  3 bytes (U-00000800):        "ࠀ"                                       |
54 2.1.4  4 bytes (U-00010000):        "𐀀"                                       |
55 2.1.5  5 bytes (U-00200000):        ""                                       |
56 2.1.6  6 bytes (U-04000000):        ""                                       |
57                                                                               |
58 2.2  Last possible sequence of a certain length                               |
59                                                                               |
60 2.2.1  1 byte  (U-0000007f):        "\7f"                                        
61 2.2.2  2 bytes (U-000007ff):        "߿"                                       |
62 2.2.3  3 bytes (U-0000ffff):        "￿"                                       |
63 2.2.4  4 bytes (U-001fffff):        ""                                       |
64 2.2.5  5 bytes (U-03ffffff):        ""                                       |
65 2.2.6  6 bytes (U-7fffffff):        ""                                       |
66                                                                               |
67 3  Malformed sequences                                                        |
68                                                                               |
69 3.1  Unexpected continuation bytes                                            |
70                                                                               |
71 Each unexpected continuation byte should be separately signalled as a         |
72 malformed sequence of its own.                                                |
73                                                                               |
74 3.1.1  First continuation byte 0x80: "\80"                                      |
75 3.1.2  Last  continuation byte 0xbf: "¿"                                      |
76                                                                               |
77 3.1.3  2 continuation bytes: "\80¿"                                             |
78 3.1.4  3 continuation bytes: "\80¿\80"                                            |
79 3.1.5  4 continuation bytes: "\80¿\80¿"                                           |
80 3.1.6  5 continuation bytes: "\80¿\80¿\80"                                          |
81 3.1.7  6 continuation bytes: "\80¿\80¿\80¿"                                         |
82 3.1.8  7 continuation bytes: "\80¿\80¿\80¿\80"                                        |
83                                                                               |
84 3.1.9  Sequence of all 64 possible continuation bytes (0x80-0xbf):            |
85                                                                               |
86    "\80\81\82\83\84\85\86\87\88\89\8a\8b\8c\8d\8e\8f                                                          |
87     \90\91\92\93\94\95\96\97\98\99\9a\9b\9c\9d\9e\9f                                                          |
88      ¡¢£¤¥¦§¨©ª«¬­®¯                                                          |
89     °±²³´µ¶·¸¹º»¼½¾¿"                                                         |
90                                                                               |
91 3.2  Lonely start characters                                                  |
92                                                                               |
93 3.2.1  All 32 first bytes of 2-byte sequences (0xc0-0xdf),                    |
94        each followed by a space character:                                    |
95                                                                               |
96    "À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï                                           |
97     Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Þ ß "                                         |
98                                                                               |
99 3.2.2  All 16 first bytes of 3-byte sequences (0xe0-0xef),                    |
100        each followed by a space character:                                    |
101                                                                               |
102    "à á â ã ä å æ ç è é ê ë ì í î ï "                                         |
103                                                                               |
104 3.2.3  All 8 first bytes of 4-byte sequences (0xf0-0xf7),                     |
105        each followed by a space character:                                    |
106                                                                               |
107    "ð ñ ò ó ô õ ö ÷ "                                                         |
108                                                                               |
109 3.2.4  All 4 first bytes of 5-byte sequences (0xf8-0xfb),                     |
110        each followed by a space character:                                    |
111                                                                               |
112    "ø ù ú û "                                                                 |
113                                                                               |
114 3.2.5  All 2 first bytes of 6-byte sequences (0xfc-0xfd),                     |
115        each followed by a space character:                                    |
116                                                                               |
117    "ü ý "                                                                     |
118                                                                               |
119 3.3  Sequences with last continuation byte missing                            |
120                                                                               |
121 All bytes of an incomplete sequence should be signalled as a single           |
122 malformed sequence, i.e., you should see only a single replacement            |
123 characters in each of the next 10 tests. (Characters as in section 2)         |
124                                                                               |
125 3.3.1  2-byte sequence with last byte missing (U+0000):     "À"               |
126 3.3.2  3-byte sequence with last byte missing (U+0000):     "à\80"               |
127 3.3.3  4-byte sequence with last byte missing (U+0000):     "ð\80\80"               |
128 3.3.4  5-byte sequence with last byte missing (U+0000):     "ø\80\80\80"               |
129 3.3.5  6-byte sequence with last byte missing (U+0000):     "ü\80\80\80\80"               |
130 3.3.6  2-byte sequence with last byte missing (U-000007ff): "ß"               |
131 3.3.7  3-byte sequence with last byte missing (U-0000ffff): "ï¿"               |
132 3.3.8  4-byte sequence with last byte missing (U-001fffff): "÷¿¿"               |
133 3.3.9  5-byte sequence with last byte missing (U-03ffffff): "û¿¿¿"               |
134 3.3.10 6-byte sequence with last byte missing (U-7fffffff): "ý¿¿¿¿"               |
135                                                                               |
136 3.4  Concatenation of incomplete sequences                                    |
137                                                                               |
138 All the 10 sequences of 3.3 concatenated, you should see 10 malformed         |
139 sequences being signalled:                                                    |
140                                                                               |
141    "Àà\80ð\80\80ø\80\80\80ü\80\80\80\80ßï¿÷¿¿û¿¿¿ý¿¿¿¿"                                                               |
142                                                                               |
143 3.5  Impossible bytes                                                         |
144                                                                               |
145 The following two bytes cannot appear in a correct UTF-8 string               |
146                                                                               |
147 3.5.1  fe = "þ"                                                               |
148 3.5.2  ff = "ÿ"                                                               |
149 3.5.3  fe fe ff ff = "þþÿÿ"                                                   |
150                                                                               |
151 4  Overlong sequences                                                         |
152                                                                               |
153 The following sequences are not malformed according to the letter of          |
154 the Unicode 2.0 standard. However, they are longer then necessary and         |
155 a correct UTF-8 encoder is not allowed to produce them. A "safe UTF-8         |
156 decoder" should reject them just like malformed sequences for two             |
157 reasons: (1) It helps to debug applications if overlong sequences are         |
158 not treated as valid representations of characters, because this helps        |
159 to spot problems more quickly. (2) Overlong sequences provide                 |
160 alternative representations of characters, that could maliciously be          |
161 used to bypass filters that check only for ASCII characters. For              |
162 instance, a 2-byte encoded line feed (LF) would not be caught by a            |
163 line counter that counts only 0x0a bytes, but it would still be               |
164 processed as a line feed by an unsafe UTF-8 decoder later in the              |
165 pipeline. From a security point of view, ASCII compatibility of UTF-8         |
166 sequences means also, that ASCII characters are *only* allowed to be          |
167 represented by ASCII bytes in the range 0x00-0x7f. To ensure this             |
168 aspect of ASCII compatibility, use only "safe UTF-8 decoders" that            |
169 reject overlong UTF-8 sequences for which a shorter encoding exists.          |
170                                                                               |
171 4.1  Examples of an overlong ASCII character                                  |
172                                                                               |
173 With a safe UTF-8 decoder, all of the following five overlong                 |
174 representations of the ASCII character slash ("/") should be rejected         |
175 like a malformed UTF-8 sequence, for instance by substituting it with         |
176 a replacement character. If you see a slash below, you do not have a          |
177 safe UTF-8 decoder!                                                           |
178                                                                               |
179 4.1.1 U+002f = c0 af             = "À¯"                                        |
180 4.1.2 U+002f = e0 80 af          = "à\80¯"                                        |
181 4.1.3 U+002f = f0 80 80 af       = "ð\80\80¯"                                        |
182 4.1.4 U+002f = f8 80 80 80 af    = "ø\80\80\80¯"                                        |
183 4.1.5 U+002f = fc 80 80 80 80 af = "ü\80\80\80\80¯"                                        |
184                                                                               |
185 4.2  Maximum overlong sequences                                               |
186                                                                               |
187 Below you see the highest Unicode value that is still resulting in an         |
188 overlong sequence if represented with the given number of bytes. This         |
189 is a boundary test for safe UTF-8 decoders. All five characters should        |
190 be rejected like malformed UTF-8 sequences.                                   |
191                                                                               |
192 4.2.1  U-0000007f = c1 bf             = "Á¿"                                   |
193 4.2.2  U-000007ff = e0 9f bf          = "à\9f¿"                                   |
194 4.2.3  U-0000ffff = f0 8f bf bf       = "ð\8f¿¿"                                   |
195 4.2.4  U-001fffff = f8 87 bf bf bf    = "ø\87¿¿¿"                                   |
196 4.2.5  U-03ffffff = fc 83 bf bf bf bf = "ü\83¿¿¿¿"                                   |
197                                                                               |
198 4.3  Overlong representation of the NUL character                             |
199                                                                               |
200 The following five sequences should also be rejected like malformed           |
201 UTF-8 sequences and should not be treated like the ASCII NUL                  |
202 character.                                                                    |
203                                                                               |
204 4.3.1  U+0000 = c0 80             = "À\80"                                       |
205 4.3.2  U+0000 = e0 80 80          = "à\80\80"                                       |
206 4.3.3  U+0000 = f0 80 80 80       = "ð\80\80\80"                                       |
207 4.3.4  U+0000 = f8 80 80 80 80    = "ø\80\80\80\80"                                       |
208 4.3.5  U+0000 = fc 80 80 80 80 80 = "ü\80\80\80\80\80"                                       |
209                                                                               |
210 THE END                                                                       |