Can JavaScript be used to capture the user's screen?

JavaScript has full access to the document object model, so at least in theory, it could capture what's on its own web page (but not anything outside the browser window) and there's a library to do that: http://html2canvas.hertzen.com/ (I haven't tried it.)

The same-origin policy prevents JavaScript from accessing the DOM of another site. Since JavaScript cannot access the DOM of another site, it cannot leak material from the other site. So, if your question boils down to whether a script running in one tab, or even an iframe, can capture the banking password from elsewhere in the browser, then no, provided same-origin is properly implemented in the browser itself.

Same origin applies to domain from which the page was served, not from which the script was served. So, my page at hxxp://bbrown.spsu.edu/ (it wasn't interesting, and now it's dead because I've retired) can load a script from google-analytics.com, as it does, and that script has access to the DOM of the page from which it was loaded; it can also send stuff back to Google through a bit of sleight-of-hand. The point is, it can do that only because I trusted Google Analytics enough to load their script in my page; the code that loads the page is in markup I wrote. If you load my page into your browser, that script from google-analytics.com can see only the DOM of my page in your browser, and not anything else you may have open in your browser.

Besides capturing the screen with Javascript, a common thing is tracking a users mouse movements/actions on a web page. This previous question on StackOverflow shows how to capture the position of the mouse in Javascript and jQuery. With this information, people can take those mouse positions and map them with a screenshot of the web page (created using another method). Some common tools like Lucky Orange and Crazy Egg create heat maps of web pages so webmasters can see what people are doing on the website.

As this question relates to security, there is some security concern over being able to see mouse movements on a web page. Hackers may use this information for or as a part of phishing attacks. But I don't think it is a real concern.

getUserMedia(); can capture the entire desktop.

This functionality is experimental, so you will need to direct the user to enable it.