I am currently trying to write a script that iterates over an input file and checks data on a website. If it finds the new data, it prints out to the terminal that it passes, if it doesn't it tells me it fails. And vice versa for deleted data. It was working fine until the input file I was given contains the "™" character. Then when ruby gets to that line, it is spitting out an error:
PDAPWeb.rb:73:in `include?': incompatible character encodings: UTF-8 and IBM437
if browser.text.include? (program_name)
program_name = record.gsub("\n", '').force_encoding("utf-8").encode("IBM437", replace: nil)
program_name = record.gsub("\n", '').force_encoding("IBM437").encode("utf-8", replace: nil)
PDAPWeb.rb:48:in `encode': U+2122 from UTF-8 to IBM437 (Encoding::UndefinedConve
Here’s what it looks like is going on. Your input file contains a
™ character, and it is in UTF-8 encoding. However when you read it, since you don’t specify the encoding, Ruby assumes it is in your system’s default encoding of IBM437 (you must be on Windows).
This is basically the same as this:
>> input = "™" => "™" >> input.encoding => #<Encoding:UTF-8> >> input.force_encoding 'ibm437' => "\xE2\x84\xA2"
force_encoding doesn’t change the actual string, just the label associated with it. This is the same outcome as in your case, only you arrive here via a different route (by reading the file).
The web page also has a
™ symbol, and is also encoded as UTF-8, but in this case Ruby has the encoding correct (Watir probably uses the headers from the page):
>> web_page = '™' => "™" >> web_page.encoding => #<Encoding:UTF-8>
Now when you try to compare these two strings you get the compatibility error, because they have different encodings:
>> web_page.include? input Encoding::CompatibilityError: incompatible character encodings: UTF-8 and IBM437 from (irb):11:in `include?' from (irb):11 from /Users/matt/.rvm/rubies/ruby-2.2.1/bin/irb:11:in `<main>'
If either of the two strings only contained ASCII characters (i.e. code points less that 128) then this comparison would have worked. Both UTF-8 and IBM437 are both supersets of ASCII, and are only incompatible if they both contain characters outside of the ASCII range. This is why you only started seeing this behaviour when the input file had a
The fix is to inform Ruby what the actual encoding of the input file is. You can do this with the already loaded string:
>> input.force_encoding 'utf-8' => "™"
You can also do this when reading the file, e.g. (there are a few ways of reading files, they all should allow you to explicitly specify the encoding):
input = File.read("input_file.txt", :encoding => "utf-8") # now input will be in the correct encoding
Note in both of these the string isn’t being changed, it still contains the same bytes, but Ruby now knows its correct encoding.
Now the comparison should work okay:
>> web_page.include? input => true
There is no need to
encode the string. Here’s what happens if you do. First if you correct the encoding to UTF-8 then encode to IBM437:
>> input.force_encoding("utf-8").encode("IBM437", replace: nil) Encoding::UndefinedConversionError: U+2122 from UTF-8 to IBM437 from (irb):16:in `encode' from (irb):16 from /Users/matt/.rvm/rubies/ruby-2.2.1/bin/irb:11:in `<main>'
IBM437 doesn’t include the
™ character, so you can’t encode a string containing it to this encoding without losing data. By default Ruby raises an exception when this happens. You can force the encoding by using the
:undef option, but the symbol is lost:
>> input.force_encoding("utf-8").encode("IBM437", :undef => :replace) => "?"
If you go the other way, first using
force_encoding to IBM437 then encoding to UTF-8 you get the string
>> input.force_encoding("IBM437").encode("utf-8", replace: nil) => "Γäó"
The string is already in IBM437 encoding as far as Ruby is concerned, so
force_encoding doesn’t do anything. The UTF-8 representation of
™ is the three bytes
0xe2 0x84 0xa2, and when interpreted as IBM437 these bytes correspond to the three characters seen here which are then converted into their UTF-8 representations.
(These two outcomes are the other way round from what you describe in the question, hence my comment above. I’m assuming that this is just a copy-and-paste error.)