FirebaseUI - Validate displayName before user creation
As it's said in the documentation:
Currently, FirebaseUI does not offer customization out of the box. However, the HTML around the widget is not affected by it so you can display everything you want around the widget container.
So you have no way to add a pre-hook, validation, etc. However, you can try to intercept the requests using xhr-intercept or fetch-intercept and do anything you want when there is a request for creating a user. You can cancel the request if the displayName is not unique and show an error via JS.
As firebaseUI cannot be customized, the possible solutions can be.
Backend solution - Subscribe to firebase hooks that notifies any data change. where you can check for unique displayName.
Frontend Solution - Design a custom Auth UI screen integrated with firebase UI plugins. Here you will have complete control over the data before sending it to firebase.