user2845840 user2845840 - 1 year ago 214
Perl Question

LWP::Simple::get changes encoding

I suppose I am using LWP::Simple::get incorrectly, but I am at my wit's end as to how to correct it. My first try was a simple

perl -e 'use LWP::Simple; print get("http://localhost/wtf.txt");'

, but that did not work.
contains a single UTF-8-encoded character
). Using
I made sure that the HTTP server sends the correct header line
Content-Type: text/plain; charset=utf-8
and that the content is as expected. But the above perl code instead returns
as ISO-8859-1-encoded.

I thought that this is a simple encoding issue with a simple fix, but digging deeper I found it to be not quite as straightforward as I had hoped. I created a second file
with the single UTF-8-encoded character
) and fetched both with the following perl code:

use LWP::Simple;
print $wtf;
print "\n";
print $wtf2;
print "\n$wtf\n$wtf2\n";
print (unpack("H*",$wtf)."\n");

When fetching
, this code writes 4 times
in its ISO-8859-1-encoded form, followed by
(its ISO-8859-1-encoded form in hex). Up to here, everything is as before. But when fetching
, this code writes
in its UTF-8-encoded form, followed by
) in ISO-8859-1,
in UTF-8,
in UTF-8,
(ISO-8859-1 of
in hex).

Given that
have nothing to do with each other apart from the latter being a bitmasked/truncated version of the former, I expect that not only re-encoding happens inside LWP::Simple, but also some truncating. I'm inclined to file a bug report to LWP::Simple, but I'm still hoping for a simple fix and/or an explanation.

By the way, none of the described issues occur if I replace the second and third line with
and simply read the files from
instead of fetching them via LWP::Simple::get.

I tested this using perl 5.14.2 and libwww 6.04 on Debian 7.

Answer Source

This is a bug in your code.

LWP::Simple::get doesn't return the original bytes (in some encoding), it returns decoded text (i.e. Unicode). (Which makes sense, because if it returned bytes, you wouldn't know how to decode them because get doesn't tell you the encoding.)

So get("http://localhost/wtf.txt") returns a string containing the codepoint U+00f6. print then writes some bytes to STDOUT. What are those bytes? That depends on the encoding layer currently set on the filehandle. By default this is a weird mix of Latin-1 and UTF-8 (it might even depend on the internal encoding of the string).

If you want to get UTF-8 output, do binmode STDOUT, ":encoding(UTF-8)"; first. That ensures all text written to STDOUT is encoded as UTF-8.

On the other hand, if you want to ignore encodings and just write the bytes that you received from the web server, then LWP::Simple is the wrong choice. Use LWP::UserAgent instead and call $response->content. (LWP::Simple::get uses $response->decoded_content internally.)

The truncation in your second example is probably due to pack/unpack, which don't make sense on Unicode strings (they're meant for byte strings, i.e. all codepoints <= 255).

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download