ISO 8859 and Go

The ISO 8859 series are 8-bit character sets for different parts of Europe and some other areas. They all have the ASCII set common in the low part, but differ in the top part. According to Google, ISO 8859 codes account for about 20% of the web pages it sees.

The first code, ISO 8859-1 or Latin-1, has the first 256 characters in common with Unicode. The encoded value of the Latin-1 characters is the same in UTF-16 and in the default ISO 8859-1 encoding. But this doesn’t really help much, as UTF-16 is a 16-bit encoding and ISO 8859-1 is an 8-bit encoding. UTF-8 is a 8-bit encoding, but it uses the top bit to signal extra bytes, so only the ASCII subset overlaps for UTF-8 and ISO 8859-1. So UTF-8 doesn’t help much either.

But the ISO 8859 series don’t have any complex issues. To each character in each set corresponds a unique Unicode character. For example, in ISO 8859-2, the character “latin capital letter I with ogonek” has ISO 8859-2 code point 0xc7 (in hexadecimal) and corresponding Unicode code point of U+012E. Transforming either way between an ISO 8859 set and the corresponding Unicode characters is essentially just a table lookup.

The table from ISO 8859 code points to Unicode code points could be done as an array of 256 integers. But many of these will have the same value as the index. So we just use a map of the different ones, and those not in the map take the index value.

For ISO 8859-2 a portion of the map is

  1. var unicodeToISOMap = map[int] uint8 {
  2.     0x12e: 0xc7,
  3.     0x10c: 0xc8,
  4.     0x118: 0xca,
  5.     // plus more
  6. }

and a function to convert UTF-8 strings to an array of ISO 8859-2 bytes is

  1. /* Turn a UTF-8 string into an ISO 8859 encoded byte array
  2. */ 
  3. func unicodeStrToISO(str string) []byte {
  4.         // get the unicode code points
  5.     codePoints := []int(str)
  7.         // create a byte array of the same length
  8.     bytes := make([]byte, len(codePoints))
  10.     for n, v := range(codePoints) {
  11.                 // see if the point is in the exception map
  12.         iso, ok := unicodeToISOMap[v]
  13.         if !ok {
  14.                         // just use the value
  15.             iso = uint8(v)
  16.         }
  17.         bytes[n] = iso
  18.     }
  19.     return bytes
  20. }

In a similar way you can change an array of ISO 8859-2 bytes into a UTF-8 string:

  1. var isoToUnicodeMap = map[uint8] int {
  2.     0xc7: 0x12e, 
  3.     0xc8: 0x10c,
  4.     0xca: 0x118,
  5.     // and more
  6. }
  8. func isoBytesToUnicode(bytes []byte) string {
  9.     codePoints := make([]int, len(bytes))
  10.     for n, v := range(bytes) {
  11.         unicode, ok :=isoToUnicodeMap[v]
  12.         if !ok {
  13.             unicode = int(v)
  14.         }
  15.         codePoints[n] = unicode
  16.     }
  17.     return string(codePoints)
  18. }

These functions can be used to read and write UTF-8 strings as ISO 8859-2 bytes. By changing the mapping table, you can cover the other ISO 8859 codes. Latin-1, or ISO 8859-1, is a special case - the exception map is empty as the code points for Latin-1 are the same in Unicode. You could also use the same technique for other character sets based on a table mapping, such as Windows 1252.