Search for email containg attachment of specific file type

It's a little backwards, but you can do this with

xlsx hasattachment:true

I just tried this and it works well:

ext:xlsx

Try this - it's just slightly different than what you tried and it works for me. Via the documentation.

attachments:.xlsx