Java Unicode encoding

You can handle them all if you're careful enough.

Java's char is a UTF-16 code unit. For characters with code-point > 0xFFFF it will be encoded with 2 chars (a surrogate pair).

See http://www.oracle.com/us/technologies/java/supplementary-142654.html for how to handle those characters in Java.

(BTW, in Unicode 5.2 there are 107,154 assigned characters out of 1,114,112 slots.)

You said:

A Java char is 2 bytes (max size of 65,536) but there are 95,221 Unicode characters.

Unicode grows

Actually, the inventory of characters defined in Unicode has grown dramatically. Unicode continues to grow — and not just because of emojis.

• 143,859 characters in Unicode 13 (Java 15, release notes)
• 137,994 characters in Unicode 12.1 (Java 13 & 14)
• 136,755 characters in Unicode 10 (Java 11 & 12)
• 120,737 characters in Unicode 8 (Java 9)
• 110,182 characters in Unicode 6.2 (Java 8)
• 109,449 characters in Unicode 6.0 (Java 7)
• 96,447 characters in Unicode 4.0 (Java 5 & 6)
• 49,259 characters in Unicode 3.0 (Java 1.4)
• 38,952 characters in Unicode 2.1 (Java 1.1.7)
• 38,950 characters in Unicode 2.0 (Java 1.1)
• 34,233 characters in Unicode 1.1.5 (Java 1.0)

char is legacy

The char type is long outmoded, now legacy.

Use code point numbers

Instead, you should be working with code point numbers.

---

You asked:

Does this mean that you can't handle certain Unicode characters in a Java application?

The char type can address less than half of today's Unicode characters.

To represent any Unicode character, use code point numbers. Never use char.

Every character in Unicode is assigned a code point number. These range over a million, from 0 to 1,114,112. Doing the math when comparing to the numbers listed above, this means most of the numbers in that range have not yet been assigned to a character yet. Some of those numbers are reserved as Private Use Areas and will never be assigned.

The String class has gained methods for working with code point numbers, as did the Character class.

Get the code point number for any character in a string, by zero-based index number. Here we get 97 for the letter a.

int codePoint = "Cat".codePointAt( 1 ) ; // 97 = 'a', hex U+0061, LATIN SMALL LETTER A.

For the more general CharSequence rather than String, use Character.codePointAt.

We can get the Unicode name for a code point number.

String name = Character.getName( 97 ) ; // letter `a`

LATIN SMALL LETTER A

We can get a stream of the code point numbers of all the characters in a string.

IntStream codePointsStream = "Cat".codePoints() ;

We can turn that into a List of Integer objects. See How do I convert a Java 8 IntStream to a List?.

List< Integer > codePointsList = codePointsStream.boxed().collect( Collectors.toList() ) ;

Any code point number can be changed into a String of a single character by calling Character.toString.

String s = Character.toString( 97 ) ; // 97 is `a`, LATIN SMALL LETTER A. 

a

We can produce a String object from an IntStream of code point numbers. See Make a string from an IntStream of code point numbers?.

IntStream intStream = IntStream.of( 67 , 97 , 116 , 32 , 128_008 ); // 32 = SPACE, 128,008 = CAT (emoji).

String output =
        intStream
                .collect(                                     // Collect the results of processing each code point.
                        StringBuilder :: new ,                // Supplier<R> supplier
                        StringBuilder :: appendCodePoint ,    // ObjIntConsumer<R> accumulator
                        StringBuilder :: append               // BiConsumer<R,​R> combiner
                )                                             // Returns a `CharSequence` object.
                .toString();                                  // If you would rather have a `String` than `CharSequence`, call `toString`. 

Cat 🐈

---

You asked:

Does this boil down to what character encoding you are using?

Internally, a String in Java is always using UTF-16.

You only use other character encoding when importing or exporting text in or out of Java strings.

So, to answer your question, no, character encoding is not directly related here. Once you get your text into a Java String, it is in UTF-16 encoding and can therefore contain any Unicode character. Of course, to see that character, you must be using a font with a glyph defined for that particular character.

When exporting text from Java strings, if you specify a legacy character encoding that cannot represent some of the Unicode characters used in your text, you will have a problem. So use a modern character encoding, which nowadays means UTF-8 as UTF-16 is now considered harmful.

Java uses UTF-16. A single Java char can only represent characters from the basic multilingual plane. Other characters have to be represented by a surrogate pair of two chars. This is reflected by API methods such as String.codePointAt().

And yes, this means that a lot of Java code will break in one way or another when used with characters outside the basic multilingual plane.

To add to the other answers, some points to remember:

• A Java char takes always 16 bits.

• A Unicode character, when encoded as UTF-16, takes "almost always" (not always) 16 bits: that's because there are more than 64K unicode characters. Hence, a Java char is NOT a Unicode character (though "almost always" is).

• "Almost always", above, means the 64K first code points of Unicode, range 0x0000 to 0xFFFF (BMP), which take 16 bits in the UTF-16 encoding.

• A non-BMP ("rare") Unicode character is represented as two Java chars (surrogate representation). This applies also to the literal representation as a string: For example, the character U+20000 is written as "\uD840\uDC00".

• Corolary: string.length() returns the number of java chars, not of Unicode chars. A string that has just one "rare" unicode character (eg U+20000) would return length() = 2 . Same consideration applies to any method that deals with char-sequences.

• Java has little intelligence for dealing with non-BMP unicode characters as a whole. There are some utility methods that treat characters as code-points, represented as ints eg: Character.isLetter(int ch). Those are the real fully-Unicode methods.