Decode a websocket frame

If packet_length is 126, the following 2 bytes give the length of data to be read.
If packet_length is 127, the following 8 bytes give the length of data to be read.
The mask is contained in the following 4 bytes (after the length).
The message to be decoded follows this.

The data framing section of the spec has a useful illustration of this.

If you re-order your code to something like

• Read packet_length
• Check for packet_length of 126 or 127. Reassign packet_length to value of following 2/4 bytes if required.
• Read mask (the 4 bytes after packet_length, including any additional 2 or 8 bytes read for the step above).
• Decode message (everything after the mask).

then things should work.

The sticking point is at > 125 bytes payload.

The format is pretty simple, lets say you send ten a's in JavaScript:

ws.send("a".repeat(10))

Then the server will receive:

bytes[16]=818a8258a610e339c771e339c771e339

• byte 0: The 0x81 is just an indicator that a message received
• byte 1: the 0x8a is the length, substract 0x80 from it, 0x0A == 10
• byte 2, 3, 4, 5: the 4 byte xor key to decrypt the payload
• the rest: payload

But now lets say you send 126 a's in JavaScript:

ws.send("a".repeat(126))

Then the server will receive:

bytes[134]=81fe007ee415f1e5857490848574908485749084857490848574908485749084857490848574908485749084857490848574908485749084857490848574908485749084857490848574908485749084857490848574908485749084857490848574908485749084857490848574908485749084857490848574908485749084857490848574

If the length of the payload is > 125, the byte 1 will have the value 0xfe, the format changes then to:

• byte 0: The 0x81 is just an indicator that a message received
• byte 1: will be 0xfe
• byte 2, 3: the length of the payload as a uint16 number
• byte 4, 5, 6, 7: the 4 byte xor key to decrypt the payload
• the rest: payload

Example code in C#:

List<byte[]> decodeWebsocketFrame(Byte[] bytes)
{
    List<Byte[]> ret = new List<Byte[]>();
    int offset = 0;
    while (offset + 6 < bytes.Length)
    {
        // format: 0==ascii/binary 1=length-0x80, byte 2,3,4,5=key, 6+len=message, repeat with offset for next...
        int len = bytes[offset + 1] - 0x80;

        if (len <= 125)
        {

            //String data = Encoding.UTF8.GetString(bytes);
            //Debug.Log("len=" + len + "bytes[" + bytes.Length + "]=" + ByteArrayToString(bytes) + " data[" + data.Length + "]=" + data);
            Debug.Log("len=" + len + " offset=" + offset);
            Byte[] key = new Byte[] { bytes[offset + 2], bytes[offset + 3], bytes[offset + 4], bytes[offset + 5] };
            Byte[] decoded = new Byte[len];
            for (int i = 0; i < len; i++)
            {
                int realPos = offset + 6 + i;
                decoded[i] = (Byte)(bytes[realPos] ^ key[i % 4]);
            }
            offset += 6 + len;
            ret.Add(decoded);
        } else
        {
            int a = bytes[offset + 2];
            int b = bytes[offset + 3];
            len = (a << 8) + b;
            //Debug.Log("Length of ws: " + len);

            Byte[] key = new Byte[] { bytes[offset + 4], bytes[offset + 5], bytes[offset + 6], bytes[offset + 7] };
            Byte[] decoded = new Byte[len];
            for (int i = 0; i < len; i++)
            {
                int realPos = offset + 8 + i;
                decoded[i] = (Byte)(bytes[realPos] ^ key[i % 4]);
            }

            offset += 8 + len;
            ret.Add(decoded);
        }
    }
    return ret;
}