See this page as a slide show

Unicode

Unicode (ISO-10646)

• First published 1991
• Version 9.0 published June 2016.
• More than 128,000 characters
• Incorporates ASCII as code points 0–127 without change.
• Incorporates ISO-8859-1 (Latin-1) as code points 0–255 without change.
• Code points, not encoding (patience)
  • U+0041 A LATIN CAPITAL LETER A
  • U+2FC2 ⿂ KANGXI RADICAL FISH
  • U+1F355 🍕 SLICE OF PIZZA
• Meaning, not pictures

U+ notation

By convention, Unicode code points are represented as U+ followed by at four (more only if needed) upper-case hexadecimal digits.

RIGHT SQUARE BRACKET is written U+005D, not U+5D. You could also call it Unicode character #93, but don’t.

What’s in Unicode

All Unicode “blocks”: http://unicode.org/Public/UNIDATA/Blocks.txt

Code Points

Initially, Unicode is all about mapping integers to characters:

Now, do that for 128,000+ more characters.

Encoding

Fine, so we’ve defined this mapping. How do we actually represent those in a computer? That’s the job of an encoding. An encoding is a mapping of the bits in an integer to bytes.

16-bit Encodings

• UCS-2:
  • Fixed-length 16-bit.
  • Each character is two 8-bit bytes, whether in memory, or on a disk.
  • Certainly is straightforward.
  • Inadequate for modern Unicode, which has many more than 2¹⁶ characters. Can’t even represent U+1F554 🕔 CLOCK FACE FIVE OCLOCK.
  • Unicode originally had a much more modest scope, only living languages, so that might have worked.

 ····J···· ····a···· ····c···· ····k····
┌────┬────┬────┬────┬────┬────┬────┬────┐
│ 00 │ 4A │ 00 │ 61 │ 00 │ 63 │ 00 │ 6B │
└────┴────┴────┴────┴────┴────┴────┴────┘
  0    1    2    3    4    5    6    7

16-bit Encodings

UTF-16:

• Slightly variable-length: values ≤ U+FFFF take two bytes, other values take four bytes.
• Consider U+203D ‽ INTERROBANG:
  • UTF-16BE (big-endian): bytes are 20 3D
  • UTF-16LE (little-endian): bytes are 3D 20
• For values ≥ U+10000 and < U+10FFFF:
  • Subtract out 0x1000
  • Emit U+D800 plus the top ten bits.
  • Emit U+DC00 plus the lower ten bits.
  • There are no valid code points U+D800…U+DFFF.
• 100% overhead for ASCII text.

32-bit Encodings

UTF-32:

• straightforward rendering of the code point in binary, with the same problems about byte order:
  • UTF-32BE: big-endian version
  • UTF-32LE: little-endian version
• 300% overhead for ASCII text.
  • Sure, disk space is cheap, but, c’mon.

  ········J········   ········a········   ········c········   ········k········
┌────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┬────┐
│ 00 │ 00 │ 00 │ 4A │ 00 │ 00 │ 00 │ 61 │ 00 │ 00 │ 00 │ 63 │ 00 │ 00 │ 00 │ 6B │
└────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┴────┘
   0    1    2    3    4    5    6    7    8    9   10   11   12   13   14   15

False Positives

Hey, there’s a slash in this string! No, wait, there isn’t.

• U+002F / SOLIDUS
• U+262F ☯ YIN YANG

When using UTF-16 or UTF-32 encoding, a naïve algorithm will falsely detect a slash (oh, excuse me, a solidus) in one of the bytes of U+262F.

Similarly, a C-string cannot hold a UTF-16 or UTF-32 string, because of the embedded zero bytes.

UTF-8 Variable-Length Encoding

• Features:
  • ASCII never appears except as intended; NUL and slash never appear in other byte sequences.
  • Therefore, no kernel changes.
  • Find previous/next char are fast operations.
  • Self-synchronizing: if some bytes are damaged, it’s easy to find the beginning of the next/previous character.
  • 0% overhead for ASCII text.

  J    a    c    k
┌────┬────┬────┬────┐
│ 4A │ 61 │ 63 │ 6B │
└────┴────┴────┴────┘
  0    1    2    3

Illustration of Various Encodings

Example

Byte Order Mark

Often, files contain a “magic number”—initial bytes that indicate what sort of file it is.

The character U+FEFF ZERO WIDTH NO BREAK SPACE, is also used as a Byte Order Mark, or BOM. When used as the first bytes of a data file, indicates the encoding (assuming that you’re limited to Unicode).

If not processed as a BOM, then ZERO WIDTH NO BREAK SPACE is mostly harmless.

Programming

It’s all about bytes vs. characters. Too many languages have no byte type, so programmers use char instead. Trouble! The language has no idea whether you’re processing text, which should be treated as Unicode, or bytes of data, which would happen if a program were parsing a JPEG file.

Linux Commands

echo \u: up to four digits; \U: up to eight digits

    % echo -e '\uf1'
    ñ
    % echo -e '\U1f435'
    🐵

wc -c counts bytes; wc -m counts characters

    % echo -e '\U1f435' | wc -c
    5
    % echo -e '\U1f435' | wc -m
    2

Viewing Files

View with xxd or od:

    % echo -e 'ABC' | xxd
    00000000: 4142 430a                                ABC.
    % echo -e '\U1f435' | xxd
    00000000: f09f 90b5 0a                             .....

    % echo -e 'ABC' | od -t x1
    0000000 41 42 43 0a
    0000004
    % echo -e '\U1f435' | od -t x1
    0000000 f0 9f 90 b5 0a
    0000005