I need to fetch data from a URL with non-ascii characters but urllib2.urlopen refuses to open the resource and raises:
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0131' in position 26: ordinal not in range(128)
Strictly speaking URIs can't contain non-ASCII characters; what you have there is an IRI.
To convert an IRI to a plain ASCII URI:
non-ASCII characters in the hostname part of the address have to be encoded using the Punycode-based IDNA algorithm;
non-ASCII characters in the path, and most of the other parts of the address have to be encoded using UTF-8 and %-encoding, as per Ignacio's answer.
import re, urlparse def urlEncodeNonAscii(b): return re.sub('[\x80-\xFF]', lambda c: '%%%02x' % ord(c.group(0)), b) def iriToUri(iri): parts= urlparse.urlparse(iri) return urlparse.urlunparse( part.encode('idna') if parti==1 else urlEncodeNonAscii(part.encode('utf-8')) for parti, part in enumerate(parts) ) >>> iriToUri(u'http://www.a\u0131b.com/a\u0131b') 'http://www.xn--ab-hpa.com/a%c4%b1b'
(Technically this still isn't quite good enough in the general case because
urlparse doesn't split away any
user:pass@ prefix or
:port suffix on the hostname. Only the hostname part should be IDNA encoded. It's easier to encode using normal
.encode('idna') at the time you're constructing a URL than to have to pull an IRI apart.)