What is the threshold for a software paper to be publishable?

There is no bright line threshold for such things and the answer should be the same as for any scientific paper, hinging on questions of novelty and extension of knowledge. If it doesn't have that, then it probably isn't a good candidate, though the standards of different journals vary widely.

But you seem to describe a small-if-any advance with little novelty and you also seem to have made up your mind. We can't help you with the judgement and you just need to take a risk and call it. Others may disagree with you, but that is always the case in reviewing.

I don't really understand your point 5 and might disagree with your emphasis on it. Why is work done within a group less valuable that work that crosses institutional lines? Lots of papers are done by one or a few people within a research group. However, I might agree more if you mean all of their cited papers come from those same people. That isn't necessarily a red flag, but might be.

I'm going to accept Buffy's answer, because even if it is not the "here's how to make it easy on you" answer I'd hoped for, it seems to be the generally-agreed correct answer.

That said, after a lot of thought (and considering feedback from here) I've come up with a list that pretty much satisfies me. I guess that everything after the first two points is more to check "is there novelty/notability here that you're overlooking because your gut reaction is that this isn't enough work to count as a paper?"

• Does this implement any non-trivial science? If the tool implements significant data generation or data analysis logic, then it should be considered. "Significant" is still a subjective term, but just reading an output format would not be significant.
• Is the software tool distributed independently? One important purpose of software papers is to make the community aware of plugins that otherwise wouldn't be noticed (lost in some dark corner of the internet). These may be relatively small (but useful) bits of code that would be overlooked without a journal to serve as an announcement forum. This criterion is probably only applicable to edge cases.
• Is the tool re-usable in many contexts? Even a very simple tool can be a valuable contribution if it is designed to facilitate interoperability. For example, a simple "universal translator" for domain-specific formats might be very small, but might be worth publishing.
• Is the approach novel? Even for journals that do not have a novelty requirement, the existence of a novel approach can be a consideration in whether a paper should be accepted.
• Does implementation of the tool require significant domain expertise? Packaging significant domain expertise in a way that makes it available to non-experts is a potential purpose of a scientific software publication.

For the fifth point, I'm imagining something like a GUI that prevents users from selecting incompatible options, where the incompatibilities may require domain expertise. I know in my field we have script-based data generation programs where users can accidentally select incompatible options and get nonsense as a result.

I'm not positive that this covers all cases, but it covers everything I can think of. And I feel better having used a rubric that I could apply to any software package.

Posting with the hope of community feedback, and so the next person in a similar situation can see what I did!