user2845840 user2845840 - 28 days ago 19
Perl Question

LWP::Simple::get changes encoding

I suppose I am using LWP::Simple::get incorrectly, but I am at my wit's end as to how to correct it. My first try was a simple

perl -e 'use LWP::Simple; print get("http://localhost/wtf.txt");'


, but that did not work.
wtf.txt
contains a single UTF-8-encoded character
u+00f6
(i.e.
ö
). Using
wget
and
xxd
I made sure that the HTTP server sends the correct header line
Content-Type: text/plain; charset=utf-8
and that the content is as expected. But the above perl code instead returns
u+00f6
as ISO-8859-1-encoded.

I thought that this is a simple encoding issue with a simple fix, but digging deeper I found it to be not quite as straightforward as I had hoped. I created a second file
wtf2.txt
with the single UTF-8-encoded character
u+30e4
(i.e.
) and fetched both with the following perl code:

#!/usr/bin/perl
use LWP::Simple;
$wtf=get("http://localhost/$ARGV[0]");
$wtf2=pack("H*",unpack("H*",$wtf));
print $wtf;
print "\n";
print $wtf2;
print "\n$wtf\n$wtf2\n";
print (unpack("H*",$wtf)."\n");


When fetching
wtf.txt
, this code writes 4 times
u+00f6
in its ISO-8859-1-encoded form, followed by
f6
(its ISO-8859-1-encoded form in hex). Up to here, everything is as before. But when fetching
wtf2.txt
, this code writes
u+30e4
in its UTF-8-encoded form, followed by
u+00e4
(i.e.
ä
) in ISO-8859-1,
u+30e4
in UTF-8,
u+00e4
in UTF-8,
e4
(ISO-8859-1 of
u+00e4
in hex).

Given that
u+30e4
and
u+00e4
have nothing to do with each other apart from the latter being a bitmasked/truncated version of the former, I expect that not only re-encoding happens inside LWP::Simple, but also some truncating. I'm inclined to file a bug report to LWP::Simple, but I'm still hoping for a simple fix and/or an explanation.

By the way, none of the described issues occur if I replace the second and third line with
$wtf=<>;
and simply read the files from
stdin
instead of fetching them via LWP::Simple::get.

I tested this using perl 5.14.2 and libwww 6.04 on Debian 7.

Answer

This is a bug in your code.

LWP::Simple::get doesn't return the original bytes (in some encoding), it returns decoded text (i.e. Unicode). (Which makes sense, because if it returned bytes, you wouldn't know how to decode them because get doesn't tell you the encoding.)

So get("http://localhost/wtf.txt") returns a string containing the codepoint U+00f6. print then writes some bytes to STDOUT. What are those bytes? That depends on the encoding layer currently set on the filehandle. By default this is a weird mix of Latin-1 and UTF-8 (it might even depend on the internal encoding of the string).

If you want to get UTF-8 output, do binmode STDOUT, ":encoding(UTF-8)"; first. That ensures all text written to STDOUT is encoded as UTF-8.

On the other hand, if you want to ignore encodings and just write the bytes that you received from the web server, then LWP::Simple is the wrong choice. Use LWP::UserAgent instead and call $response->content. (LWP::Simple::get uses $response->decoded_content internally.)

The truncation in your second example is probably due to pack/unpack, which don't make sense on Unicode strings (they're meant for byte strings, i.e. all codepoints <= 255).

Comments