Can I use Firebase for sending SMS to users without using Twilio?

You can send SMS Verification using Firebase and you cannot send ordinary SMS through Firebase. You can send using Twilio instead.

Even for SMS verification, you have to wait until they release the API docs for SMS verification through Android. https://firebase.google.com/docs/auth/android/phone-auth

There is no built-in API for sending SMS through Firebase. You will have to use a third-party service (such as you're currently doing with Twilio) to send SMS messages.