Marcus Kracht: Linux Tricks

• Until there is a uniform standard for character encodings, there is a problem with compatibility. In Germany, many people use Latin-1 (ISO 8859-1) encoding. They do that without knowing, since the keyboard drivers insert the code into the file on key press and users do not care what is in the file. My specific problem was this: I get a list of students via the university's server, and some data by the teaching assistant. I try to automate the fusion of these files using a program. However, I realise that the search fails for names that contain umlaut characters (ä, ö and ü). This turns out to be consistent. So what to do? In both files, the characters look the same, yet underlyingly something is different. To see the difference, open a file with gvim and then convert it to HEX (using the command :%!xxd; to convert back, use :%!xxd-r). The character ö has the HEX code c3b6 if set to Unicode UTF-8, but f6 if set to Latin1. This is to say, among other things, that in Latin1 it spans one byte, in UTF-8 two bytes. Once you know that you can detect which encoding you have. Look at the HEX converted file and look for a known character such as ö and see how it is encoded. Second step: suppose that your file is in Latin1, how do you get UTF-8? Here is the trick in Unix: type

  iconv -t utf8 -f laint1 --output DESTINATION SOURCE

  (This means: convert to utf8 from latin1 into file DESTINATION.) For more information see man iconv.