Retry failed jobs in github actions

I would say that you need to manage the retry logic on code level. Meaning, implement/integrate such mechanic to handle and execute again only the failed tests. I'm afraid simply

want to retry the test step (or the whole job) once if the tests fails.

will execute all your tests, may even overwrite outputs, like reports and logs from the first run.

In my experience, I have used a wrapper (shell) script. Here is how it could be achieved:

#!/usr/bin/env bash
{ # try
    # your npm test command here, that saves as output -  failed tests
} || { # catch
    # retry failed tests
      if [ -f ./rerun-failed-file ]; then
        echo "============= [WARN] Rerun file found! =============="
        echo "============= [WARN] Rerunning FAILED tests mode. =============="
        # run npm test command that picks up the failed tests & aggregate test artifacts
      fi
} 

For your particular use case, just do:

npm test || npm test

Action Retry seems to work well in my testing

https://github.com/nick-invision/retry/

I was able to use it with multi-part commands as long as they were single line (for example do-this && do-that)