Non-binary, more modern alternative to "Dear Sir/Madam"

I would suggest something along the lines of:

Dear Admissions Committee,

or

Dear Department of (STEM),

etc. You can usually find what a particular department/program/etc is called from their website. Avoid addressing the "Department of Chemical Engineering" as the "Engineering Department"; it would be most polite to use their own phrasing because it shows you've paid some attention and aren't sending out a copy-pasted letter to everyone.

For another example, I just randomly looked at a particular department, the "Department of Chemical Engineering and Biotechnology" at the University of Cambridge; for their "MBE" program they have a "contact" page that says to "please send an email to the MBE Office": in that case I'd simply address the email:

Dear MBE Office,

For any of the above, you can also omit the "Dear".

In a comment, @DanRomik suggests modifying the salutations to refer to particular groups of people (like "committee" does) rather than abstract departments: examples include: “Dear MBE program administrators”, “Dear MBE office team”. I think that's perfectly fine, too, if you find that more logical.

One that is traditional is

To Whom it may concern:

That has been around for decades, at least, and is generally used when the recipient is unknown or may be more than one person.

In my opinion, actually, anything with "Dear..." is a bit trite if you don't actually know the person.

A useful option, although I guess some people would consider it a step less formal than 'Dear Sir/Madam', is

Good morning,

or

Good afternoon,

as appropriate to the time you're sending the email.