Will declining a research award be problematic for me, or other people?

Don't give in to Imposter Syndrome!

Both your reasons are fundamentally not sound. Somebody nominated you for an award. The awards committee thinks you are deserving. You should not refuse the award simply because you think others may be more deserving. It's the task of the award committee to establish this, and their decision was you.

Rejecting the award will probably not "hurt" your colleagues, but it will probably hurt you. First, you won't have the award, and when you finally realize that your work was maybe in fact good enough to warrant decoration it will be too late to change your mind. Second, you will presumably not get nominated by the same colleagues again for something because they will assume you are not interested in awards. Third, people will look at this as strange behavior, and it is certainly possible that you get flak from your administration or senior professors for refusing what essentially amounts to free marketing for your department. Maybe you are tenured and senior enough that you don't care, but, truth be told, very few people are actually distinguished enough that they really don't need to care at all about what their university thinks about them.

Minor point: I would also have the option of giving a talk, I'd rather not, but I do have some work to talk about. Wondering if declining the offer to give a talk is a problem too.

I think it's less of a problem to decline a talk than the award itself, but even that is prone to raise some eyebrows, mainly because giving talks is such an inherent part of being an academic that people will wonder why you do not wish to partake in it.

Will declining a research award be problematic for myself or other people?

It's problematic in the sense that it would very likely be a mistake, and undermine the goal that the award is trying to achieve, to the detriment of yourself (mainly) and to a lesser extent of the scientific community you belong to. In a system that is supposed to function as a meritocracy, each time a talented person or their work does not get recognition that they deserve, everybody is hurt a little bit.

As for why it is likely to be a mistake, the main point to keep in mind is that it is a very rare individual who has the capacity to assess him/herself as objectively and accurately as he/she is assessed by others -- almost everyone suffers a bias in one direction or another in how accurately they perceive their level of talent and achievements. In many people this manifests itself in the direction of thinking they are smarter than they really are and that their achievements are more significant than they really are - this is the Dunning-Kruger effect, tied to narcissism and egotism. In other people one sees the opposite effect where the person thinks they and their achievements are less worthy of praise than they really are, a phenomenon known (at least in certain contexts) as impostor syndrome, and sometimes associated with issues of depression and low self-esteem. Both of these effects are well-known examples of cognitive biases.

Now, we don't know whether you suffer from any of these biases. But the general principle is that people are untrustworthy judges of their own worth. For this reason, when we finish papers and grant proposals we send it off for other people to critique, and accept their judgment of how good the work is. The same principle goes for awards; people who are more senior and established than you have considered possible candidates for the award and decided that you are the most worthy among them. You don't have either the information that they had of who they were comparing you to, or the ability of the committee members to look at you and the other candidates objectively (or at least more objectively) without being influenced by the very common biases I described above. The logical conclusion is that their judgment that you should receive the award is much more likely to be accurate than yours.

Anyway, good luck no matter what you end up deciding. Although I have an opinion on what would be the better choice, I do think you have a full right to decline the award, and disapprove of the "get over yourself" sentiment expressed in this answer (which I downvoted) linked to from Stella Biderman's answer (which was excellent and I upvoted, along with xLeitix's also excellent answer).

In a comment, you say

The problem is I worry that accepting might shine a light on my research not being quite there just yet and people would look at my bibliometrics and think poorly of me, if I hadn't accepted then I'd just be yet another researcher, if that makes sense. I'm not using the bibliometrics as a yard stick for research quality, but rather perceived research quality. The actual research quality comparison I am basing on papers published.

With all due respect, this is a terrible reason to decline the award. I strongly agree with all the reasons that xLeitix list for why it is a bad idea for your career. I further think it's rude to decline the award.

You don't get to decide who gets the award. I mean that quite seriously; you don't sit on the committee that hands out the award. Your opinion on who is or is not deserving of the award is entirely irrelevant to everything. Turing down the award is rude because it says, both to your colleagues and to the committee, "I know better than you do who deserves this award." You wouldn't dream of writing them a letter telling them that they chose the wrong person if it was anyone else who was honored, right? So don't do it here. In this example it's all the worse because you're doing so in a blatantly biased capacity as you have a close relationship with the awardee.

Also, awards exist so that communities and organizations can praise their members, hold up examples to junior members, and give the community a chance to highlight important work. See the comments to the effect here. When people turn down awards the community very frequently feels hurt or slighted.