How harmful is it to submit side-results that are not new?

Your impulse to try to find out whether your "mathematical side results" are new and/or publishable is a great one, but I think you need to go about it in a different way. If you are not a mathematician and working without the guidance / mentorship of an experienced mathematician, trying to get your results published in math journals will be somewhere between difficult and quixotic.

Have you tried making use of the site mathoverflow.net? Asking whether a result exists in the literature should be on-topic for that site providing the result is viewed to be a sufficiently contentful piece of mathematics. (Yes, there is a certain amount of elitism / snootiness here, which most people seem to agree is necessary in order to keep a research level math Q&A site afloat). Anyway, you can certainly try: what is great about MO (and in fact most SE sites) is that it is very likely that whatever happens will happen with almost magical quickness: with high probability, if within a few hours no one comes along to answer your question, then the answer is not "well known" by the mathematical community.

Here are some specific issues with the float-it-to-the-journals plan that you propose. (It is not an exhaustive list: probably that is not necessary.)

What if I just submit a side-result to a medium-class journal, e.g. American Mathematical Monthly?

The American Mathematical Monthly is not a "medium-class journal". It occupies a specific niche, and within that (small) niche it is one of the top journals. For instance, I have submitted four papers to the AMM. Two of them were rejected. One of these rejected papers was easy to publish in a more "mainstream" journal; the other was not. The other two papers got accepted...eventually. But I turned in a larger number of revisions on these two papers (I believe they were 3 and 7 pages long) than on most of my other papers. Most of these revisions were made after the mathematics was agreed upon to be correct and of the sort they wanted to publish.

After my first four submissions, many years of close reading and half a dozen referee reports, I think I have a pretty good idea of what the AMM wants to publish (not necessarily the same good idea as everyone on the editorial board). I now sometimes advise other people as to whether their manuscripts ought to be submitted there. Even so I have just written an article that I had at first thought I wanted to submit to the AMM, but now I am leaning towards submitting elsewhere, and I am not sure. It's complicated!

If the results are new, they will be published and this may help other people.

No, just because the results are new does not mean they will be published by any reputable mathematical journal. I have a few manuscripts that I tried to publish at several different places but did not succeed. Eventually I stopped. I agree that in theory they could be published somewhere, but in practice this can be a lot of trouble.

(I agree that the results may help other people! Consider making them available in some other form, e.g. in a manuscript on your webpage and on the arxiv. You can put things on the arxiv that you do not have plans to publish, so long as you are reasonable about it.)

If the results are not new, the paper will just be rejected. This is not a big deal for me since these are only side-results.

Hard no. Results get republished all the time. The refereeing process for math journals is that the editors seek opinions from one or two people. If those one or two people have not seen the results before -- maybe they made a reasonable attempt to search the literature, or maybe they didn't; both occur frequently -- then the paper will likely not be rejected for that reason. There are some notorious examples in the mathematical community of papers that were published containing content this is not only "not new" but is well known material from undergraduate courses, or is actually at a lower level than that. This is actually a pretty bad way to be sure that the results are not new.

My only concern is that the editor and reviewers will be angry at me for having wasted their time on results that are not new. This may be harmful to my reputation.

I think that is a legitimate but rather minor concern. The mathematical community is vast, and although we talk about people in some ways, gossiping about poor submissions (that never appeared) is very rare (in my experience, obviously). If you are willing to "take no for an answer" and spread out your submissions among multiple journals, I don't think this will be very harmful to your reputation. Yes, you might be wasting people's time and that is worth thinking about, although the less appropriate the paper is, the quicker and easier it should be for a qualified party to see that. I think the main consideration here is wasting your own time. Are you aware that if you submit a paper to a math journal you should expect to wait at least six months for a referee report, and that delays of a year or more are not uncommon? That's way too long to wait to find out whether some lemmas should be put in your game theory paper or not, isn't it?

In summary: I urge you not to use the journal refereeing process the way you suggest, but more for your own good than for the sake of your reputation. I hope you know that even if you don't have any mathematical connections, you could still make some. Universities have multiple departments for a reason: have you tried to make contact with anyone in the math department of your university? If your colleagues there do not have the specific expertise to help you, they are still your colleagues so some of them should be willing to take a little time to direct you to the people who do. I encourage you to take advantage of this.

As a CS guy, I would never send something to a Math journal, without a good mathematical collaborator. Sending half-baked results, without prior solid literature search and sufficient background knowledge is not a good way to promote your career.

I also think you make one crucial mistake:

If the results are not new, the paper will just be rejected. This is not a big deal for me since these are only side-results. My only concern is that the editor and reviewers will be angry at me for having wasted their time on results that are not new.

This is not your major problem. The main problem is, if those results slip through the cracks of peer-review and get published and (i) they are wrong (ii) they are not new (iii) this thing happens more than once. Then you will get a reputation for sending half-baked ideas, bad papers or try to present other people's results for novel and yours.

TL, DR; Only send the best papers you can write and only in areas you have sufficient knowledge to judge their merits. Not every paper has to be seminal, but they must all be correct and with a proper literature search.

One possible strategy is to submit to the arXiv first and then look for verification that it is new and potentially think about publishing. This gets your results out of the door and out into the open where they can be helpful to others, with a lot less hassle than it would take to publish through a journal. If you go through this route, you have something concrete to talk about in, say, MathOverflow, or with any mathematicians you care to approach, and you are protected regarding priority.

If you do this, I would recommend you to be very clear in the abstract and comments, and in the title too if possible, about the exact nature of the piece: say that it is something you found but that you are unsure of its relation to the literature, and that you're looking for comments regarding it.

This is a feasible route as long as you can submit to the arXiv; this is generally fairly inclusive but it is not universal, and if you are outside of a university then you might struggle finding endorsers for your submission. Nevertheless, it might be worth a shot, and if you say you're looking for an endorsement it may (or may not) help in getting a mathematician in the know to look at your manuscript and help you figure out if it is novel or not. If you cannot submit to the arXiv, there is also viXra, but honestly speaking I would think twice about submitting there.