HTML : Form does not send UTF-8 format inputs

I added the meta tag : nothing changed.

It indeed doesn't have any effect when the page is served over HTTP instead of e.g. from local disk file system (i.e. the page's URL is http://... instead of e.g. file://...). In HTTP, the charset in HTTP response header will be used. You've already set it as below:

<%@page pageEncoding="UTF-8"%>

This will not only write out the HTTP response using UTF-8, but also set the charset attribute in the Content-Type response header.

This one will be used by the webbrowser to interpret the response and encode any HTML form params.

I added the accept-charset attribute in form : nothing changed.

It has only effect in Microsoft Internet Explorer browser. Even then it is doing it wrongly. Never use it. All real webbrowsers will instead use the charset attribute specified in the Content-Type header of the response. Even MSIE will do it the right way as long as you do not specify the accept-charset attribute. As said before, you have already properly set it via pageEncoding.

Get rid of both the meta tag and accept-charset attribute. They do not have any useful effect and they will only confuse yourself in long term and even make things worse when enduser uses MSIE. Just stick to pageEncoding. Instead of repeating the pageEncoding over all JSP pages, you could also set it globally in web.xml as below:

<jsp-config>
    <jsp-property-group>
        <url-pattern>*.jsp</url-pattern>
        <page-encoding>UTF-8</page-encoding>
    </jsp-property-group>
</jsp-config>

As said, this will tell the JSP engine to write HTTP response output using UTF-8 and set it in the HTTP response header too. The webbrowser will use the same charset to encode the HTTP request parameters before sending back to server.

Your only missing step is to tell the server that it must use UTF-8 to decode the HTTP request parameters before returning in getParameterXxx() calls. How to do that globally depends on the HTTP request method. Given that you're using POST method, this is relatively easy to achieve with the below servlet filter class which automatically hooks on all requests:

@WebFilter("/*")
public class CharacterEncodingFilter implements Filter {

    @Override
    public void init(FilterConfig config) throws ServletException {
        // NOOP.
    }

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
        request.setCharacterEncoding("UTF-8");
        chain.doFilter(request, response);
    }

    @Override
    public void destroy() {
        // NOOP.
    }
}

That's all. In Servlet 3.0+ (Tomcat 7 and newer) you don't need additional web.xml configuration.

You only need to keep in mind that it's very important that setCharacterEncoding() method is called before the POST request parameters are obtained for the first time using any of getParameterXxx() methods. This is because they are parsed only once on first access and then cached in server memory.

So e.g. below sequence is wrong:

String foo = request.getParameter("foo"); // Wrong encoding.
// ...
request.setCharacterEncoding("UTF-8"); // Attempt to set it.
String bar = request.getParameter("bar"); // STILL wrong encoding!

Doing the setCharacterEncoding() job in a servlet filter will guarantee that it runs timely (at least, before any servlet).

In case you'd like to instruct the server to decode GET (not POST) request parameters using UTF-8 too (those parameters you see after ? character in URL, you know), then you'd basically need to configure it in the server end. It's not possible to configure it via servlet API. In case you're using for example Tomcat as server, then it's a matter of adding URIEncoding="UTF-8" attribute in <Connector> element of Tomcat's own /conf/server.xml.

In case you're still seeing Mojibake in the console output of System.out.println() calls, then chances are big that the stdout itself is not configured to use UTF-8. How to do that depends on who's responsible for interpreting and presenting the stdout. In case you're using for example Eclipse as IDE, then it's a matter of setting Window > Preferences > General > Workspace > Text File Encoding to UTF-8.

See also:

  • Unicode - How to get the characters right?

Based on your posted output it seems that the parameter is sent as UTF8 and later the unicode bytes of the string are interpreted as ISO-8859-1.

Following snippet demonstrates your observed behavior

String eGrave = "\u00E8"; // the letter è
System.out.printf("letter UTF8      : %s%n", eGrave);
byte[] bytes = eGrave.getBytes(StandardCharsets.UTF_8);
System.out.printf("UTF-8 hex        : %X %X%n",
        bytes[0], bytes[1], bytes[0], bytes[1]
);
System.out.printf("letter ISO-8859-1: %s%n",
        new String(bytes, StandardCharsets.ISO_8859_1)
);

output

letter UTF8      : è
UTF-8 hex        : C3 A8
letter ISO-8859-1: è

For me the form send the correct UTF8 encoded data, but later this data is not treated as UTF8.

edit Some other points to try:

output the character encoding your request has

System.out.println(request.getCharacterEncoding())

force the usage of UTF-8 to retrieve the parameter (untested, only an idea)

request.setCharacterEncoding("UTF-8");
... request.getParameter(...);

Warm up

Let me start by saying the universal fact which we all know that computer doesn't understand anything but bits - 0's and 1's.

Now, when you are submitting a HTML form over HTTP and values travel over the wire to reach destination server then essentially a whole lot of bits - 0's and 1's are being passed over.

  • Before sending the data to the server, HTTP client (browser or curl etc.) will encode it using some encoding scheme and expects server to decode it using same scheme so that server knows exactly what client has sent.
  • Before sending response back to the client, server will encode it using some encoding scheme and expects client to decode it using same scheme so that client knows exactly what server has sent.

An analogy for this can be - I am sending a letter to you and telling you whether it is written in English or French or Dutch, so that you will get exact message as I intended to send you. And while replying to me you will also mention in which language I should read.

Important take away is that the fact that when data is leaving the client it will be encoded and same will be decoded at server side, and vice-versa. If you do not specify anything then content will be encoded as per application/x-www-form-urlencoded before leaving from client side to server side.

Core concept

Reading warm up is important. There are couple of things you need to make sure to get the expected results.

  • Having correct encoding set before sending data from client to server.
  • Having correct decoding and encoding set at server side to read request and write response back to client (this was the reason why you were not getting expected results)
  • Ensure that everywhere same encoding scheme is used, it should not happen that at client you are encoding using ISO-8859-1 and at server you are decoding using UTF-8, else there will be goof-up (from my analogy, I am writing you in English and you are reading in French)
  • Having correct encoding set for your logs viewer, if trying to verify using log using Windows command-line or Eclipse log viewer etc. (this was a contributing reason for your issue but it was not primary reason because in the first place your data read from request object was not correctly decoded. windows cmd or Eclipse log viewer encoding also matters, read here)

Having correct encoding set before sending data from client to server

To ensure this, there are several ways talked about but I will say use HTTP Accept-Charset request-header field. As per your provided code snippet you are already using and using it correctly so you are good from that front.

There are people who will say that do not use this or it is not implemented but I would very humbly disagree with them. Accept-Charset is part of HTTP 1.1 specification (I have provided link) and browser implementing HTTP 1.1 will implement the same. They may also argue that use Accept request-header field's "charset" attribute but

  • Really it is not present, check the Accept request-header field link I provided.
  • Check this

I am providing you all data and facts, not just words, but still if you are not satisfied then do following tests using different browsers.

  • Set accept-charset="ISO-8859-1" in your HTML form and POST/GET form having Chinese or advanced French characters to server.
  • At server decode the data using UTF-8 scheme.
  • Now repeat same test by swapping client and server encoding.

You will see that none of times you were able to see the expected characters at server. But if you will use same encoding scheme then you will see expected character. So, browsers do implements accept-charset and its effect kicks-in.

Having correct decoding and encoding set at server side to read request and write response back to client

There are hell lot of ways talked about that you can do to achieve this (sometime some configuration may be required based on specific scenario but below solves 95% cases and holds good for your case as well). For example:

  1. Use character encoding filter for setting encoding on request and response.
  2. Use setCharacterEncoding on request and response
  3. Configure web or application server for correct character encoding using -Dfile.encoding=utf8 etc. Read more here
  4. Etc.

My favorite is first one and will solve your problem as well - "Character Encoding Filter", because of below reasons:

  • All you encoding handling logic is at one place.
  • You have all the power through configuration, change at one place and everyone if happy.
  • You need not to worry that some other code may be reading my request stream or flushing out the response stream before I could set the character encoding.

1. Character encoding filter

You can do following to implement your own character encoding filter. If you are using some framework like Springs etc. then you need not to write you own class but just do the configuration in web.xml

Core logic in below is very similar to what Spring does, apart from a lot of dependency, bean aware thing they do.

web.xml (configuration)

<filter>
    <filter-name>EncodingFilter</filter-name>
    <filter-class>
        com.sks.hagrawal.EncodingFilter
    </filter-class>
    <init-param>
        <param-name>encoding</param-name>
        <param-value>UTF-8</param-value>
    </init-param>
    <init-param>
        <param-name>forceEncoding</param-name>
        <param-value>true</param-value>
    </init-param>
</filter>

<filter-mapping>
    <filter-name>EncodingFilter</filter-name>
    <url-pattern>/*</url-pattern>
</filter-mapping>

EncodingFilter (character encoding implementation class)

public class EncodingFilter implements Filter {
    private String encoding = "UTF-8";
    private boolean forceEncoding = false;

    public void doFilter(ServletRequest request, ServletResponse response, FilterChain filterChain) throws IOException, ServletException {
        request.setCharacterEncoding(encoding);
        if(forceEncoding){ //If force encoding is set then it means that set response stream encoding as well ...
            response.setCharacterEncoding(encoding);
        }
        filterChain.doFilter(request, response);
    }

    public void init(FilterConfig filterConfig) throws ServletException {
        String encodingParam = filterConfig.getInitParameter("encoding");
        String forceEncoding = filterConfig.getInitParameter("forceEncoding");
        if (encodingParam != null) {
            encoding = encodingParam;
        }
        if (forceEncoding != null) {
            this.forceEncoding = Boolean.valueOf(forceEncoding);
        }
    }

    @Override
    public void destroy() {
        // TODO Auto-generated method stub

    }
}

2. ServletRequest.setCharacterEncoding()

This is essentially same code done in character encoding filter but instead of doing in filter, you are doing it in your servlet or controller class.

Idea is again to use request.setCharacterEncoding("UTF-8"); to set the encoding of http request stream before you start reading the http request stream.

Try below code, and you will see that if you are not using some sort of filter to set the encoding on request object then first log will be NULL while second log will be "UTF-8".

System.out.println("CharacterEncoding = " + request.getCharacterEncoding());
request.setCharacterEncoding("UTF-8");
System.out.println("CharacterEncoding = " + request.getCharacterEncoding());

Below is important excerpt from setCharacterEncoding Java docs. Another thing to note is you should provide a valid encoding scheme else you will get UnsupportedEncodingException

Overrides the name of the character encoding used in the body of this request. This method must be called prior to reading request parameters or reading input using getReader(). Otherwise, it has no effect.

Wherever needed I have tried best to provide you official links or StackOverflow accepted bounty answers, so that you can build trust.

Tags:

Html

Forms

Java

Utf 8

Jsp

Recent Posts