Unicode: What Is It? How It Replaced ASCII

One Number for Every Character

According to the Unicode Consortium’s FAQ, Unicode is the universal character-encoding standard, the basis on which text in any language gets processed, stored and exchanged. To every possible character — a letter, a punctuation mark, a symbol — the standard assigns a unique number, called a code point, written in U+ notation followed by hexadecimal digits. Inside, a computer only ever works with binary numbers: Unicode is the bridge that turns those numbers into characters a person can read.

Real-world example: open the same text file on a computer in Italy, one in Japan and one in Saudi Arabia, and see the same characters everywhere — accents, ideograms, right-to-left script — and that’s only possible because every system agrees on the same number for the same character. Before Unicode, each system kept its own table, and the same number could stand for a different letter depending on the machine.

Why ASCII Wasn’t Enough

ASCII, published in 1963 as standard ASA X3.4-1963, uses 7 bits per character: a value from 0 to 127, for a total of 128 code points, 95 of them printable and 33 control characters. That was plenty for English typed on an American keyboard, but according to Wikipedia the standard doesn’t directly support words or names such as résumé, jalapeño or naïve, since it has no code points for letters with diacritical marks. Later revisions tried stretching the space to 8 bits, with “relative” encodings of ASCII built for individual languages, but they stayed incompatible with one another: none covered more than one alphabet at a time.

How UTF-8 Works

Unicode assigns the numbers; UTF-8 is the most common way to turn them into bytes for storage or transmission. It’s specified by the IETF’s RFC 3629, published in November 2003: every character becomes a sequence of 1 to 4 bytes, depending on how high its code point sits, covering the entire U+0000-U+10FFFF range. The first byte carries a bit pattern that says how many bytes follow it; the following bytes always start with the same two bits, so a program can always tell where a new character begins, even opening the file partway through.

The feature that made UTF-8 the web’s default choice is backward compatibility: per RFC 3629, US-ASCII characters are encoded in a single octet carrying the normal US-ASCII value. A file written purely in ASCII, read as UTF-8, produces exactly the same text — no conversion, no byte changes. When text travels from a server to a browser, across the network that carries data from one end to the other, the declared encoding has to match the one the file was written in, or the characters arrive garbled.

A Short History

The idea of a universal encoding emerged, according to the Consortium’s history page, roughly four years before the organization was founded — so around 1987 — from discussions among engineers at Xerox (Joe Becker) and Apple (Lee Collins and Mark Davis). The Unicode Consortium was formally incorporated in January 1991, in California. Version 1.0 of the standard, Volume 1, was published a few months later, in October of that same year.

Since then the standard has grown version by version, adding scripts and new ranges of code points, including a private-use area of over 137,000 code points for anyone who needs custom characters outside the standard. Developers work with text already encoded in Unicode almost constantly, often through the UTF-8 algorithm described above, without even noticing: one of the developments covered in the history of computers.