sebulba sebulba - 1 year ago 160
C++ Question

C++ & Boost: encode/decode UTF-8

I'm trying to do a very simple task: take a unicode-aware

and convert it to a
, encoded as UTF8 bytes, and then the opposite way around: take a
containing UTF8 bytes and convert it to unicode-aware

The problem is, I need it cross-platform and I need it work with Boost... and I just can't seem to figure a way to make it work. I've been toying with

Trying to convert the code to use
instead of files of whatever, but nothing seems to work.

For instance, in Python it would look like so:

>>> u"שלום"
>>> u"שלום".encode("utf8")
>>> '\xd7\xa9\xd7\x9c\xd7\x95\xd7\x9d'.decode("utf8")

What I'm ultimately after is this:

wchar_t uchars[] = {0x5e9, 0x5dc, 0x5d5, 0x5dd, 0};
wstring ws(uchars);
string s = encode_utf8(ws);
// s now holds "\xd7\xa9\xd7\x9c\xd7\x95\xd7\x9d"
wstring ws2 = decode_utf8(s);
// ws2 now holds {0x5e9, 0x5dc, 0x5d5, 0x5dd}

I really don't want to add another dependency on the ICU or something in that spirit... but to my understanding, it should be possible with Boost.

Some sample code would greatly be appreciated! Thanks

Answer Source

Thanks everyone, but ultimately I resorted to -- it's a header-only library that's very lightweight and easy to use. I'm sharing a demo code here, should anyone find it useful:

inline void decode_utf8(const std::string& bytes, std::wstring& wstr)
    utf8::utf8to32(bytes.begin(), bytes.end(), std::back_inserter(wstr));
inline void encode_utf8(const std::wstring& wstr, std::string& bytes)
    utf8::utf32to8(wstr.begin(), wstr.end(), std::back_inserter(bytes));


wstring ws(L"\u05e9\u05dc\u05d5\u05dd");
string s;
encode_utf8(ws, s);
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download