Is it possible to host telegram on my own server?
You could implement a full working Telegram-API, then have hosted clients on your server via this API.
Your users would login on your web, then you sign them in via the hosted clients on your servers.
You are basically performing a proxy service to these users , and you can even integrate other value added features for you users this way.
According to the official telegram FAQ the current answer is no:
Q: Can I run Telegram using my own server?
Our architecture does not support federation yet. Telegram is a unified cloud service, so creating forks where two users might end up on two different Telegram clouds is unacceptable. To enable you to run your own Telegram server while retaining both speed and security is a task in itself. At the moment, we are undecided on whether or not Telegram should go in this direction.
So as long as the server itself is not open-source the entire Telegram eco-system cannot be considered open-source, even though there is an open API and official open-source clients.
There seem to be some unofficial telegram servers, but it's not clear how compatible they are with existing clients.
Some possible telegram alternatives
Matrix is allegedly providing "an open network for secure, decentralized communication" and has both open-source clients (Element being the 'official' one) and an open-source server that can be self-hosted. BUT while it looks good on the surface, there are indications that companies behind it have undisclosed intimate links with governmental actors (similar to Signal).
XMPP/Jabber has been around for a longer time, is an open protocol with multiple server and client implementations, and might be the least tainted by third-party interests. XMPP was the underlying protocol behind the original Google Talk messenger before it was rebranded to Google Hangouts and switched to a proprietary protocol.
Teamspeak a collaborative platform for teams, intended originally for gamers, free client and server.
Mumble a voice oriented solution which allows self-hosted servers.