What does "in-house hash function" mean?

From the explanation of in-house in the Cambridge Directory: "Something that is done in-house is done within an organization or business by its employees rather than by other people".

Here it means developing your own hash algorithm instead of using a public one. Usually that means that it is developed by only a few people with only limited expertise in the problem area and without any public input. Thus it is very likely that the self-developed one gets eventually broken once more experts in cryptography take a look at it.

See also Why shouldn't we roll our own? and How valuable is secrecy of an algorithm?.

In the context of cryptography "in-house" is a synonym for "questionable origin and unverified strength".

It specifically means that they developed their own hashing function (or in other cases encryption, key-exchange scheme, etc.).

This, in cryptography, is a Bad Idea with capital letters. While developing your own library of common functions or your own webservice framework or whatever can have a perfectly good use-case, cryptography is one of the fields where a tiny mistake can make the whole thing incredibly fragile in a way that you will never find out. If you build your own webserver ("our high-performance in-house webserver...") and there's a problem, you have a good chance of finding out sooner rather than later because it crashes, or sends the wrong files, or performs badly. But if your crypto algorithm has a problem that destroys its cryptographic strength, you have to be very lucky that someone who breaks it actually tells you. The people who try to break it are almost certain to be attackers, because very few cryptographers waste their time on some in-house crypto hack. They know to stick with public algorithms where it actually matters if they find something, to more than one company.