Extent of "unscientific" or wrong papers in research mathematics

I would say this is a common, but not fatal, issue. I think most mathematicians are aware of several papers in their field with serious errors/gaps without retractions/errata, and over time at least the wrong papers which are important get discovered. (e.g., see this MO post for temporary counterexamples---there aren't too many that went unnoticed for too long). There are also some grey areas where most people aren't sure if they can trust certain papers or not, and in certain areas/topics it's more of an issue than others.

The culture in mathematics is that retractions are typically for academic dishonesty, and incorrect papers either get corrected (or at least have their flaws exposed) in other papers or with errata.

What to do:

• for an individual paper, check its review on MathSciNet, and check papers that reference that paper: if many people use it freely, it's probably okay, but if someone later found an issue they will often mention it in another paper. if the paper has a corrigendum, it will be linked in MathSciNet.

• if you start reading the paper and for this or other reasons become suspicious, talk to experts---usually if a paper is important and there is a serious issue, experts become aware of the situation relatively quickly

• check if the author has errata/comments on their webpage

• some authors are more reliable than others, and they get a reputation for it; this is something you typically learn over time from your own research and talking to experts.

• don't panic: if you happen to use a result that was wrong and don't realize it (or publish a paper yourself with a major error), that's okay. it happens from time to time and you can't check everything. you might discover the error later or someone might point it out. in any case, you are often able to then write another paper correcting the situation.

• don't contribute to the problem: put errata for your own papers on your website and if there is a major error not corrected in a later paper, publish a corrigendum

• (optional) keep track of known issues in a personal notebook along with their resolution. i do this because there are quite a number of papers in my area with errors/gaps and i have a bad memory. i contemplated making this public and letting other people contribute to this list, but i haven't so far mainly because i'm not sure i want to put in the effort to update it and make sure it is fair to all parties involved. however i have shared this list with some people individually.

Journals don't retract scientific papers because they are now known to be wrong. If this were the case, they would have to retract a zillion papers from the past because, well, science evolves: new findings disprove old ones, old mistakes get corrected, and new ones are introduced.

And this happens in mathematics too.

This answer on MathOverflow makes a reference to the paper Errors and Corrections in Mathematics Literature which analyses the number of published corrections in mathematics literature, divided by field. The fraction of errata is less than 1%, but indeed this doesn't count all the undiscovered mistakes and those discovered which will never receive an errata.

Just how widespread is the issue?

This is not the be an exact answer to the question but a mention of some related famous anectodes and a mention of a research field dealing with the issue. This write-up by Voevodsky gives several examples of wrong proofs being belief to be true for many years. He eventually "pioneers" a research field called univalent foundations which aims to create a computer asisted proof checker (or create proofs that can be checked by computers).