Is WebGL a security concern?
Yes, WebGL is indeed a potential security risk, though the magnitude of the risk is hard to assess and open to debate. There are some tricky issues here. The browsers have put in place some defenses against the security risks, but there seems to be some debate about whether those defenses will prove adequate in the long run.
One major risk is that WebGL involves running code directly on the video card, and exposing APIs that provide direct access to video card APIs. The browser does attempt to sandbox this code (to a certain extent), and browsers do enforce a number of security restrictions designed to prevent malicious behavior. However, many of these APIs and their implementations were not originally designed to be provided to untrusted entities (they were only usable by native applications, which are fully trusted), so there are concerns about whether exposing them to arbitrary web sites might enable web sites to attack your system.
There was one high-visibility white paper (see also the sequel) which looked at the security of the WebGL implementation in browsers at the time, and found a number of vulnerabilities. They found some memory safety issues in several WebGL APIs, and also found some attacks that would allow one web site to read pixel data of other web sites (which could enable a breach of confidentiality). See also this third study, which demonstrated the existence of these vulnerabilities on a number of browsers and web cards (at the time).
Browsers have responded to this with a variety of defenses: they have blacklisted video cards with known security problems; they have tried to fix the known memory safety problems; and they have restricted use of WebGL per the same-origin policy, to prevent a malicious web site from using WebGL to spy on users' use of other web sites.
There is some ongoing debate over whether these defenses will prove adequate in the long term. Microsoft has taken the position that WebGL is too great a security risk and the existing defenses are not robust enough. On the other hand, Mozilla takes the position that the defenses they have put in place will be adequate, and that WebGL provides important value to the web. Ars Technica has an excellent round-up of the issue; and here is another press report.
P.S. I completely disagree with your statement about it being particularly a problem for open source web browsers. That's a myth. See Open Source vs Closed Source Systems, which already covers these arguments. (See also Chrome vs Explorer - how to explain in plain words that open-source is better? for additional thoughtful discussion on this topic.)
Some key points:
- WebGL is not just exposing OpenGL to JavaScript. All the entry points have been restricted to remove possibilities of out-of-bounds memory accesses, so that it is always possible for the browser to check for out-of-bounds accesses (and doing so is covered by conformance tests).
- WebGL does allow to run almost arbitrary shaders on the GPU. However, keep in mind that shaders are not arbitrary general-purpose code. They can only access very specific memory in a way that browsers check for out-of-bounds accesses. Shaders are validated and translated by a shader compiler embedded in the browser before being passed to the GPU driver.
- There was only exactly one security flaw in a WebGL spec, ever: the WebGL spec originally allowed usage of cross-origin images as WebGL textures, and it was demonstrated that a timing attack could successfully read these. This was corrected in mid-2011 and the current version of the WebGL spec, 1.0.1, is secure.
- More on WebGL security can be found here: http://www.khronos.org/webgl/security/
It seems that security would especially be a problem with open source web browsers.
You sir are very wrong. It has proven many times that opensource programs are often a lot more secure than closed source ones, because there are a lot more eyes that check the code.
Also all of the browsers run these things in a sandbox. Breaking out of the sandbox will be difficult, but it will be as much of a problem in closed source as in open-source browsers.