I had a text file that had several weird characters in it and I was struggling to remove them using substitution in vi.
^X This is my test file ^Y ^S It^Xs full of problems ^\ like this ^]
First find the octal number for the problem characters. Use od -c on a sample problem text (seriously – not the whole file, it’s very difficult to read output)
me@home2 # od -c mel.txt 0000000 030 T h i s i s m y t e s 0000020 t f i l e 031 023 I t 030 s 0000040 f u l l o f p r o b l e m s 0000060 034 l i k e t h i s 035 \n 0000100
You can see from this my problem characters are octal 031030 023 034 035
From context I can guess what these characters should be – 030 and 031 are single quotes, 034 and 035 are double quotes and 023 is a -.
I’m going to use tr ‘<string1>’ ‘<string2’ What this format of the tr command does is when it encounters a character in <string1> it substitutes it with the character in the same position in <string2>. The only complication in my case is that I need to substitute in ‘ characters so I’m going to specify their octal value.
cat mel.txt | tr '2334353130' '-""4747' ' This is my test file ' - It's full of problems " like this "
I’ve been left with some padding spaces around my replaced characters, but these are pretty simple to remove in vi.