Do websockets allow for p2p (browser to browser) communication?

No. Browsers can only initiate WebSockets connections, not receive them. The W3C browser API spec only defines how to start an outbound connection.

You can make an application that would both initiate and accept WebSockets connections, but browsers do not do this.

You might look at Pusher App which you could use to build a WebSockets application with multiple clients. The service provides up to 20 simultaneous WebSockets clients for free (you can pay for higher scaling).

Update:

The WebRTC protocol and API is making rapid progress and allows a Data Channel to be established between two peers (you still may need a STUN/TURN server for the initial NAT traversal and setup).

I was reading about websocket and peer 2 peer and found PeerJS.

I still haven't made anything though, but by the examples it looks promising.

In theory it is possible with WebRTC DataChannel:

RTCDataChannel is a WebRTC API for high performance, low latency, peer-to-peer communication of arbritary data. The API is simple—similar to WebSocket—but communication occurs directly between browsers, so RTCDataChannel can be much faster than WebSocket even if a relay (TURN) server is required (when 'hole punching' to cope with firewalls and NATs fails).

"In theory" because it isn't supported by a stable browser yet and you still need a relay server (TURN) if one of the browsers is behind a symmetric NAT. Nevertheless, it is a really promising feature.

Update: Chrome 26 and Firefox 22 support RTCDataChannel by default and Firefox 19-21 if you enable WebRTC by setting media.peerconnection.enabled to true (about:config).