pvgoddijn pvgoddijn - 5 months ago 84
Java Question

How to replace/remove 4(+)-byte characters from a UTF-8 string in Java?

Because MySQL 5.1 does not support 4 byte UTF-8 sequences, I need to replace/drop the 4 byte sequences in these strings.

I'm looking a clean way to replace these characters.

Apache libraries are replacing the characters with a question-mark is fine for this case, although ASCII equivalent would be nicer, of course.

N.B. The input is from external sources (e-mail names) and upgrading the database is not a solution at this point in time.


We ended up implementing the following method in Java for this problem. Basicaly replacing the characters with a higher codepoint then the last 3byte UTF-8 char.

The offset calculations are to make sure we stay on the unicode code points.

public static final String LAST_3_BYTE_UTF_CHAR = "\uFFFF";
public static final String REPLACEMENT_CHAR = "\uFFFD"; 

public static String toValid3ByteUTF8String(String s)  {
    final int length = s.length();
    StringBuilder b = new StringBuilder(length);
    for (int offset = 0; offset < length; ) {
       final int codepoint = s.codePointAt(offset);

       // do something with the codepoint
       if (codepoint > CharUtils.LAST_3_BYTE_UTF_CHAR.codePointAt(0)) {
       } else {
           if (Character.isValidCodePoint(codepoint)) {
           } else {
       offset += Character.charCount(codepoint);
    return b.toString();