What does HTTP/2 mean for a reverse proxy server?

Yes, it is exactly as you say. Conversion from HTTP/2 to HTTP/1.1 must happen in one direction, and from HTTP/1.1 to HTTP/2 must happen in the other case.

In practice this means that although HTTP/2 the protocol doesn't need a traditional text-based parser, a comprehensive HTTP/2 server needs an HTTP/1.1 parser, not only to work with clients which are HTTP/1.1 only (among them crawlers) but also for talking to inner applications.

By use, one of the most important application protocols is FastCGI. FastCGI also requires parsing of HTTP/1.1 responses from the application and conversion to HTTP/2 responses to the client.

As dsign discussed, your understanding is correct.

However I thought it was worth pointing out there are still massive advantages to HTTP/2 at your edge connection (i.e. Your reverse proxy) as the issues HTTP/2 solves (primarily latency) are less of an issue over the typically shorter, typically high bandwidth hop from the reverse proxy to the content server.

For example if you have a 100ms delay to the reverse proxy at the edge and only 1ms delay between the reverse proxy and the content server, then the fact the content server is speaking HTTP/1.1 to the proxy server probably won't have much impact on performance due to the super fast 1ms latency. So the end client (speaking HTTP/2 to the reverse proxy) still sees a massive performance boast over HTTP/1.1.