End-to-end Encrypted Group Chat Considerations

Overall

I would recommend that you use hybrid encryption.

It's more performant than using "only asymmetric encryption" (public-private-key). Some problems could even occur when you encrypt very long messages with asymmetric encryption.

The server should never know the symmetric keys. It's main task is to guarantee the transmission of the messages.

2-party-E2EE

The first "message" between two parties is the exchange of a symmetric key via asymmetric encryption. This is to increase the performance of all succeeding messages (symmetric encryption and decryption is faster than asymmetric encryption).

Mulit-party E2EE

This basically works the same way as the 2-party-end-to-end-encryption. The simplest method would be this:

1. The admin of the group chat creates a (random) symmetric key.
2. The admin sends this symmetric key to each recipient of the group via asymmetric encryption.
3. All further messages are then encrypted & decrypted with this symmetric key.
4. If a new party joins the group then this symmetric key can be sent to this party as well, if you don't care that these members then also can decrypt previous messages. If you don't want them to decrypt older messages, then the 1. step is repeated.
5. If a party leaves the group then the 1. step is repeated.

---

Edit: I should probably also add that there are definitely more details to it, but this shows the basic functionalily.

There is no need to reinvent the wheel or design your own cryptographic protocol that will likely be flawed due to lack of experience.

Message Layer Security (MLS) is a trend in the cryptology academy due to the complexity of implementing end-to-end encryption for large scale chat applications (like WhatsApp/Snapchat/Telegram).

There are multiple proposals that were accepted by the academy and some were already implemented in real world applications like WhatsApp and Signal. WhatsApp uses a Noise Protocol and Signal uses is own protocol also called Signal.

Per the recent development progress made by Nadim Kobeissi (and team) the Noise Protocol Framework currently is the best option for any real world application that wants to implement a secure end-to-end protocol that is based on DH key exchange and is not TLS.

I strongly recommend you to watch the following Real World Crypto talks on the Noise Protocol Framework and Noise Explorer, I'm sure this will immensely enlighten you:

The Noise protocol framework| | Trevor Perrin | RWC 2018

Noise Explorer: Fully Automated Modeling and Verification for Arbitrary Noise Protocols