When to use JMS and when to use REST?
Always use REST. It is the most modern, advanced and scalable integration approach available today. Load balancing a REST based service is achieved simply with hardware or software HTTP load balancer and can be considered just as free as load balancing in JMS.
MOM (Message Oriented Middleware) doesn't scale easily (but may scale big enough for your needs). REST works at web scale.
MOM does not have economies of scale. For data retrieval requests, each time a particular piece of data is requested, another message must be sent to the server and responded to by the server. In a REST based system, requests for the same data can be serviced by a HTTP cache. This means that as the volume of requests increase over time, a MOM based system will see the server load increase at the same rate as the requests. A REST based system will see the the server load increase at a slower rate than the requests.
MOM will tempt you with fire-and-forget messages with guaranteed delivery, only to bite you with the chain of custody problem.
MOM is terrible for synchronous request-reply as it will fail slowly (i.e. wait for timeout) when the server is down. When a request is going to fail, you want it to fail fast. A HTTP request to a REST based service will fail immediately (on the TCP connect) if the server is down.
MOM is useful for asynchronous request-reply messaging, but then you'll be left with the problem of where to store the state in-between the request and the reply (Hint: Your options are File or Regular Database, the Message or a NoSQL Database). Often the extra implementation effort is not worth the perceived advantages of asynchronicity. Also REST based services do support asynchronous requests if you really need it. 202 Accepted is your friend in this situation.
Finally, the use of caching allows REST based systems to implement pull-based integrations, which are far easier to support. For instance, just say we want to move data from system A to system B. The MOM approach would be to send messages from A to B. A REST based approach would be to create a data feed service in A (like an RSS feed) that B polls for new data (the same way your RSS reader polls for new articles). When B fails, in the MOM example, the support team will need to monitor the message queues to make sure they don't overflow, while someone else get's B back up. In the REST example, the support team just has to worry about getting B back up. There isn't much of a difference when A fails. In the MOM example B doesn't know and doesn't care. In the REST example B does know that A is down, but it still doesn't care because obviously there is no new data from A when it's down. Initially the polling that pull-based integration requires seams very inefficient, however HTTP caching makes this a non-issue.
In other words, instead of investing in a JMS server, invest in a good caching HTTP load balancer.
You can't compare these two technologies.
REST is a service/pattern to give you an organized way to access a stateless resources.
MOM Systems/JMS is a pattern designed aroubd sharing messages between systems. Its about data in, data out in a reliable fashion.
You can't really compare JMS to REST because they solve different problems.
But if your question is more along the lines of do I need a REST interface for my JMS queues? Its all situation, I have seen people use REST to shield thin clients from the logic nessessary to queue messages in JMS. E.g. if you have an android client that wants to talk JMS, its a lot harder to do that naitvely versus pushing messages to a "rest" interface which can then translates and push to a JMS.