Show Lecture.Unicode as a slide show.

CS253 Unicode

Unicode (ISO-10646)

• First published 1991
• Version 14.0, published September 2021, has 144,697 characters.
• Incorporates ASCII as code points 0–127, no change.
• Incorporates ISO-8859-1 (Latin-1) as code points 0–255, no change.
• Code points, not encoding (patience)
  • U+0041 A LATIN CAPITAL LETTER A
  • U+2FC2 ⿂ KANGXI RADICAL FISH
  • U+1F355 🍕 SLICE OF PIZZA
• Meaning, not pictures

U+ notation

By convention, Unicode code points are represented as U+ followed by four (five if needed) upper-case hexadecimal digits.

RIGHT SQUARE BRACKET is written U+005D, not U+5D. You could also call it Unicode character number 93, but don’t.

What’s in Unicode

All Unicode “blocks”: https://unicode.org/Public/UNIDATA/Blocks.txt

Code Points

Initially, Unicode is all about mapping integers to characters:

Now, do that for 144,000+ more characters. These integers (written as U+hexadecimal ) are code points.

History

• Of course, the code points aren’t arbitrary.
• Code points 0–127 are ASCII.
• Code points 128–255 are compatible with the popular Latin-1 encoding, and Microsoft’s Windows-1252 (foolishly called “ANSI”)
• Huge swaths of code points are taken from other character sets, e.g., emojis from phone manufacturers.
• Other character sets, such as IBM’s EBCDIC, and HP’s Roman-8, didn’t make the cut.
• I can only imagine the negotiations regarding Shift-JIS, Big5, or EUC-KR.

Encoding

Fine, we’ve defined this mapping. How do we actually represent those code points in a computer, in memory, or in a file on a disk? That’s the job of an encoding. An encoding is a mapping of the bits in an integer code point to bytes.

code point:

encoding:

16-bit Encodings

• UCS-2 (obsolete):
  • Fixed-length 16-bit.
  • Each character is two 8-bit bytes, whether in memory, or on a disk.
  • Certainly is straightforward.
  • Inadequate for modern Unicode, which has many more than 2¹⁶ characters. Can’t even represent U+1F369 🍩 DOUGHNUT.
  • Unicode originally had a much more modest scope, only living languages, so that might have worked.

     ····J···· ····a···· ····c···· ····k····
    ┌────┬────┬────┬────┬────┬────┬────┬────┐
    │ 00 │ 4A │ 00 │ 61 │ 00 │ 63 │ 00 │ 6B │
    └────┴────┴────┴────┴────┴────┴────┴────┘
      0    1    2    3    4    5    6    7

Endian

union {
    int n;
    char p[4] = {1,2,3,4};
};
cout << hex << n << ": ";
switch (n) {
  case 0x01020304:  cout << "Big-endian";    break;
  case 0x04030201:  cout << "Little-endian"; break;
  default:          cout << "Weird-endian";  break;
}

4030201: Little-endian

• Big-endian: the first byte has the most significant bits of the value.
• Little-endian: the first byte has the least significant bits of the value.
• big: 2024‒06‒30 🇨🇦 🇨🇳, little: 30‒06‒2024 🇲🇽 🇬🇧, middle: 06/30/2024 🇺🇸

16-bit Encodings

UTF-16:

• Slightly variable-length: values ≤ U+FFFF take two bytes, other values take four bytes.
• Consider U+203D ‽ INTERROBANG:
  • UTF-16BE (big-endian): bytes are 20 3D
  • UTF-16LE (little-endian): bytes are 3D 20
• For values ≥ U+10000 and < U+10FFFF:
  • Subtract out 0x10000, yielding a 20-bit number
  • Emit U+D800 plus the top ten bits.
  • Emit U+DC00 plus the lower ten bits.
  • There are no valid code points U+D800…U+DFFF.
• 100% overhead for ASCII text.

32-bit Encodings

UTF-32:

• straightforward rendering of the code point in binary, with the same problems about byte order:
  • UTF-32BE: big-endian version
  • UTF-32LE: little-endian version
• 300% overhead for ASCII text.
  • Sure, disk space is cheap, but, c’mon.

  ········J········   ········a········   ········c········   ········k········
┌────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┐
│ 00 │ 00 │ 00 │ 4A │ 00 │ 00 │ 00 │ 61 │ 00 │ 00 │ 00 │ 63 │ 00 │ 00 │ 00 │ 6B │
└────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┘
   0    1    2    3    4    5    6    7    8    9   10   11   12   13   14   15

False Positives

Hey, there’s a slash in this string! No, wait, there isn’t.

• U+002F / SOLIDUS
• U+262F ☯ YIN YANG

When using UTF-16 or UTF-32 encoding, a naïve algorithm will falsely detect a slash (oh, excuse me, a solidus) in one of the bytes of U+262F.

Similarly, a C-string cannot hold a UTF-16 or UTF-32 string, because of the embedded zero bytes.

One encoding to rule them all

• Then, a miracle occurred—UTF-8 variable-length encoding was born.
• Naïve programs can still think that they’re only dealing in ASCII.
• This is where the world is going, and it can’t be stopped.
• This slide alone contains:
  • U+2014 — EM DASH
  • U+00EF ï LATIN SMALL LETTER I WITH DIAERESIS
  • U+2019 ’ RIGHT SINGLE QUOTATION MARK
  • U+1F9DF 🧟 ZOMBIE
• Microsoft Windows, Linux, and Mac OSX use UTF-8.
• Jooooooooooiiiiiiiiiin uuuuuuuusssssssss! 🧟🧟🧟

UTF-8 Variable-Length Encoding

• ASCII only appears when intended; NUL and slash aren’t in other byte sequences. Therefore, most programs never notice that they’re processing UTF-8 instead of ASCII.
• Find previous/next char are fast operations.
• 0% overhead for ASCII text.

    J    a    c    k
  ┌────┬────┬────┬────┐
  │ 4A │ 61 │ 63 │ 6B │
  └────┴────┴────┴────┘
    0    1    2    3
  

Illustration of Various Encodings

Example

• Consider U+1F42E 🐮
  • 1F42E₁₆ = 1 1111 0100 0010 1110₂ (17 bits)
  • Need 21 bits, add leading zeroes: 0 0001 1111 0100 0010 1110
  • Grouped properly: 000 011111 010000 101110
  • Byte #1: 11110xxx, use first three bits, 11110 000
  • Byte #2: 10xxxxxx, use the next six bits, 10 011111
  • Byte #3: 10xxxxxx, use the next six bits, 10 010000
  • Byte #4: 10xxxxxx, use the next six bits, 10 101110
    • 11110 000  10 011111  10 010000  10 101110 as flag & data bits
    • 11110000   10011111   10010000   10101110   as 8-bit bytes
    • 1111 0000  1001 1111  1001 0000  1010 1110 as 4-bit nybbles
    • F0 9F 90 AE  as hex digits

Byte Order Mark

Often, files contain a “magic number”—initial bytes that indicate file type.

The character U+FEFF ZERO WIDTH NO BREAK SPACE, is also used as a Byte Order Mark, or BOM. When used as the first bytes of a data file, indicates the encoding (assuming that you’re limited to Unicode).

If not processed as a BOM, then ZERO WIDTH NO BREAK SPACE is mostly harmless. It has no width and does not cause a line break.

Programming

• It’s all about bytes vs. characters.
• In the bad old days, we assumed that everything was ASCII, or, at the very best, an eight-bit encoding such as Latin-1.
• Therefore, each character fit into an eight-bit byte.
• Nowadays, a UTF-8 character can require several bytes.

Programming

Too many languages have no byte type, so programmers use char instead. Trouble! The language has no idea whether you’re processing text, which should be treated as Unicode, or bytes of data, which would happen if a program were parsing a JPEG file.

cout << hex << 'Jack';

c.cc:1: warning: multi-character character constant
4a61636b

cout << hex << "ñ";
cout << hex << 'ñ';

c.cc:2: warning: multi-character character constant
ñc3b1

for (unsigned char c : "abcñ")
    cout << hex << setw(2) << setfill('0') << int(c) << ' ';

61 62 63 c3 b1 00 

Bash Commands

echo -e \u: up to four digits; echo -e \U: up to eight digits

% echo -e "\uf1"
ñ
% echo -e "\U1f435"
🐵

wc -c counts bytes; wc -m counts characters

% echo "🐵" | wc -c
5
% echo "🐵" | wc -m
2

Viewing Files

View with xxd or od:

% echo "ABC" | xxd
00000000: 4142 430a                                ABC.
% echo "🐵" | xxd
00000000: f09f 90b5 0a                             .....
% echo -e "\U1f435" | xxd
00000000: f09f 90b5 0a                             .....

% echo "ABC" | od -t x1
0000000 41 42 43 0a
0000004
% echo "🐵" | od -t x1
0000000 f0 9f 90 b5 0a
0000005
% echo -e "\U1f435" | od -t x1
0000000 f0 9f 90 b5 0a
0000005