Show Lecture.Unicode as a slide show.

CS253 Unicode

Unicode (ISO-10646)

• First published 1991
• Version 13.0, published March 2020, has 143,859 characters.
• Incorporates ASCII as code points 0–127, no change.
• Incorporates ISO-8859-1 (Latin-1) as code points 0–255, no change.
• Code points, not encoding (patience)
  • U+0041 A LATIN CAPITAL LETTER A
  • U+2FC2 ⿂ KANGXI RADICAL FISH
  • U+1F355 🍕 SLICE OF PIZZA
• Meaning, not pictures

U+ notation

By convention, Unicode code points are represented as U+ followed by four (five if needed) upper-case hexadecimal digits.

RIGHT SQUARE BRACKET is written U+005D, not U+5D. You could also call it Unicode character #93, but don’t.

What’s in Unicode

All Unicode “blocks”: https://unicode.org/Public/UNIDATA/Blocks.txt

Code Points

Initially, Unicode is all about mapping integers to characters:

Now, do that for 143,000+ more characters. These integers (usually represented as U+hexadecimal ) are code points.

History

• Of course, the code points aren’t arbitrary.
• Code points 0–127 are ASCII.
• Code points 128–255 are compatible with the popular Latin-1 encoding, and Microsoft’s Windows-1252 (foolishly called “ANSI”)
• Huge swaths of code points are taken from other character sets, e.g., emojis from phone manufacturers.
• Other character sets, such as IBM’s EBCDIC, and HP’s Roman-8, didn’t make the cut.
• I can only imagine the negotiations regarding Shift-JIS, Big5, or EUC-KR.

Encoding

Fine, we’ve defined this mapping. How do we actually represent those in a computer, in memory, or in a file on a disk? That’s the job of an encoding. An encoding is a mapping of the bits in an integer code point to bytes.

code point:

encoding:

16-bit Encodings

• UCS-2:
  • Fixed-length 16-bit.
  • Each character is two 8-bit bytes, whether in memory, or on a disk.
  • Certainly is straightforward.
  • Inadequate for modern Unicode, which has many more than 2¹⁶ characters. Can’t even represent U+1F369 🍩 DOUGHNUT.
  • Unicode originally had a much more modest scope, only living languages, so that might have worked.

     ····J···· ····a···· ····c···· ····k····
    ┌────┬────┬────┬────┬────┬────┬────┬────┐
    │ 00 │ 4A │ 00 │ 61 │ 00 │ 63 │ 00 │ 6B │
    └────┴────┴────┴────┴────┴────┴────┴────┘
      0    1    2    3    4    5    6    7

Endian

int n = 0x01020304;
auto p = reinterpret_cast<char *>(&n);
switch (p[0]) {
  case 1:  cout << "Big-endian";    break;
  case 4:  cout << "Little-endian"; break;
  default: cout << "Weird-endian";  break;
}
for (int i=0; i<4; i++)
    cout << ' ' << int(p[i]);

Little-endian 4 3 2 1

• Big-endian: the first byte has the most significant bits of the value.
• Little-endian: the first byte has the least significant bits of the value.
• big: 2024‒11‒21 🇨🇦 🇨🇳, little: 21‒11‒2024 🇲🇽 🇬🇧, middle: 11/21/2024 🇺🇸

16-bit Encodings

UTF-16:

• Slightly variable-length: values ≤ U+FFFF take two bytes, other values take four bytes.
• Consider U+203D ‽ INTERROBANG:
  • UTF-16BE (big-endian): bytes are 20 3D
  • UTF-16LE (little-endian): bytes are 3D 20
• For values ≥ U+10000 and < U+10FFFF:
  • Subtract out 0x10000, yielding a 20-bit number
  • Emit U+D800 plus the top ten bits.
  • Emit U+DC00 plus the lower ten bits.
  • There are no valid code points U+D800…U+DFFF.
• 100% overhead for ASCII text.

32-bit Encodings

UTF-32:

• straightforward rendering of the code point in binary, with the same problems about byte order:
  • UTF-32BE: big-endian version
  • UTF-32LE: little-endian version
• 300% overhead for ASCII text.
  • Sure, disk space is cheap, but, c’mon.

  ········J········   ········a········   ········c········   ········k········
┌────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┐
│ 00 │ 00 │ 00 │ 4A │ 00 │ 00 │ 00 │ 61 │ 00 │ 00 │ 00 │ 63 │ 00 │ 00 │ 00 │ 6B │
└────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┘
   0    1    2    3    4    5    6    7    8    9   10   11   12   13   14   15

False Positives

Hey, there’s a slash in this string! No, wait, there isn’t.

• U+002F / SOLIDUS
• U+262F ☯ YIN YANG

When using UTF-16 or UTF-32 encoding, a naïve algorithm will falsely detect a slash (oh, excuse me, a solidus) in one of the bytes of U+262F.

Similarly, a C-string cannot hold a UTF-16 or UTF-32 string, because of the embedded zero bytes.

One encoding to rule them all

• Then, a miracle occurred—UTF-8 variable-length encoding was born.
• Naïve programs can still think that they’re only dealing in ASCII.
• This is where the world is going, and it can’t be stopped.
• This slide alone contains:
  • U+2014 — EM DASH
  • U+00EF ï LATIN SMALL LETTER I WITH DIAERESIS
  • U+2019 ’ RIGHT SINGLE QUOTATION MARK
  • U+1F9DF 🧟 ZOMBIE
• Microsoft Windows, Linux, and Mac OSX use UTF-8.
• Jooooooooooiiiiiiiiiin uuuuuuuusssssssss! 🧟🧟🧟

UTF-8 Variable-Length Encoding

• ASCII only appears when intended; NUL and slash aren’t in other byte sequences. Therefore, most programs never notice that they’re processing UTF-8 instead of ASCII.
• Find previous/next char are fast operations.
• 0% overhead for ASCII text.

    J    a    c    k
  ┌────┬────┬────┬────┐
  │ 4A │ 61 │ 63 │ 6B │
  └────┴────┴────┴────┘
    0    1    2    3
  

Illustration of Various Encodings

Example

Byte Order Mark

Often, files contain a “magic number”—initial bytes that indicate file type.

The character U+FEFF ZERO WIDTH NO BREAK SPACE, is also used as a Byte Order Mark, or BOM. When used as the first bytes of a data file, indicates the encoding (assuming that you’re limited to Unicode).

If not processed as a BOM, then ZERO WIDTH NO BREAK SPACE is mostly harmless. It has no width and does not cause a line break.

Programming

• It’s all about bytes vs. characters.
• In the bad old days, we assumed that everything was ASCII, or, at the very least, an eight-bit encoding such as Latin-1.
• Therefore, each character fit into an eight-bit byte.
• Nowadays, a UTF-8 character can require several bytes.

Programming

Too many languages have no byte type, so programmers use char instead. Trouble! The language has no idea whether you’re processing text, which should be treated as Unicode, or bytes of data, which would happen if a program were parsing a JPEG file.

cout << hex << 'Jack';

c.cc:1: warning: multi-character character constant
4a61636b

cout << hex << "ñ" << 'ñ';

c.cc:1: warning: multi-character character constant
ñc3b1

for (unsigned char c : "abcñ")
    cout << hex << int(c) << ' ';

61 62 63 c3 b1 0 

Linux Commands

echo \u: up to four digits; \U: up to eight digits

    % echo -e '\uf1'
    ñ
    % echo -e '\U1f435'
    🐵

wc -c counts bytes; wc -m counts characters

    % echo -e '\U1f435' | wc -c
    5
    % echo -e '\U1f435' | wc -m
    2

Viewing Files

View with xxd or od:

    % echo -e 'ABC' | xxd
    00000000: 4142 430a				       ABC.
    % echo -e '\U1f435' | xxd
    00000000: f09f 90b5 0a			       .....

    % echo -e 'ABC' | od -t x1
    0000000 41 42 43 0a
    0000004
    % echo -e '\U1f435' | od -t x1
    0000000 f0 9f 90 b5 0a
    0000005