Should I throw an error or return a rejected promise inside an async function?

A throw within a Promise/Promise chain will automatically cause that Promise/Promise Chain to be rejected.

const myFunc = input => {
  return new Promise((resolve, reject) => {
    if (badInput) {
      throw new Error('failed')
    }

    return resolve(sns.createTopic(input).promise())
  })
}

return myFunc('some input')
  .then(result => { 
    // handle result
  })
  .catch(err => { console.log(err) }) // will log 'failed'

However, your myCustomFunction isn't wrapped in a Promise, it is using throw before the Promise is returned by sns.createTopic().promise(). To create and return a Promise already in a rejected state you would use Promise.reject(new Error('failed')) instead of throw.

I disagree with whomever told you

never throw an error inside these kinds of promise-based functions

In my opinion, you should throw a TypeError() synchronously when it indicates a programmer error rather than an operational error.

To quote Joyent | Error Handling:

Operational errors represent run-time problems experienced by correctly-written programs. These are not bugs in the program. In fact, these are usually problems with something else [...]

Programmer errors are bugs in the program. These are things that can always be avoided by changing the code. They can never be handled properly (since by definition the code in question is broken).

Your colleague seemingly fails to differentiate between these types of errors, and the code you have written is almost as it should be, with the exception of using a generic Error() instead of the semantically correct TypeError().

Why should you care about the difference?

You started out by saying that you're writing a wrapper for the AWS SDK. So, from the point of view of developers using your library, do you think they'd prefer to debug a program that throws immediately where they're doing something wrong, or would they prefer to debug a program that fails silently, attempting to resolve their misuse of your API without informing them of incorrect code?

If you think the first option sounds easier to deal with, you'd be absolutely right. Your program should always be as transparent as possible when telling a programmer what they've done wrong. Attempting to resolve misuse is what results in buggy APIs with undefined, undocumented, and just plain bizarre behavior.

What were they trying to recommend I do instead?

To give an extremely basic example (and possibly unfair comparison, since I don't have any context as to what constitutes badInput), your colleague appears to be informing you that you should do this:

try {
  if (badInput) {
    throw new Error('failed')
  }

  ...
} catch (error) {
  // expected error, proceed as normal
  // ...except not really, you have a bug
}

instead of this:

process.on('uncaughtException', function (error) {
  // deal with programmer errors here and exit gracefully
})

if (badInput) {
  throw new Error('failed')
}

try {
  ...
} catch (error) {
  // deal with operational errors here and continue as normal
}

Some real-world examples in the Node.js runtime environment that differentiate these errors, even in asynchronous functions can be found in the chart here:

Example func | Kind of func | Example error  | Kind of error | How to   | Caller uses
             |              |                |               | deliver  |
==========================================================================================
fs.stat      | asynchronous | file not found | operational   | callback | handle callback
             |              |                |               |          | error
-------------+--------------+----------------+---------------+----------+-----------------
fs.stat      | asynchronous | null for       | programmer    | throw    | none (crash)
             |              | filename       |               |          |

Conclusion

I'll leave it to you to decide whether your particular issue is due to a programmer error or an operational error, but in general, the advice that was given to you is not sound advice, and encourages buggy programs that attempt to proceed as if there was nothing wrong.

TL;DR

Any function that is expected to return a Promise under operational conditions should throw synchronously when the error is due to a bug, and should reject asynchronously when an exogenous error occurs within a correctly-written program.

This reflects the official recommendation of Joyent:

The best way to recover from programmer errors is to crash immediately.

They are correct.

The call to myCustomFunction assumes that a promise is returned at all times (.then and .catch deal with resolved and rejected promises, respectively). When you throw an error, the function doesn't return a promise.

You could use this to catch the error:

try {
  myModule.myCustomFunction(someInput).then(result => {
    // carry on
  })
  .catch(err => {
    // do something with the error
  })
} catch(err) {
  ...
}

But as you can see, this results in two error handlers: try/catch for the synchronously thrown error, and .catch for any rejected promises that sns.createTopic(someParams) may return.

That's why it's better to use Promise.reject():

module.exports.myCustomFunction = input => {

    if (badInput) {
        return Promise.reject('failed');
    }

    return sns.createTopic(someParams).promise()
}

Then, the .catch will catch both types of errors/rejections.

NB: for newer versions of Node.js (v7.6 and up, I believe), the following will also work:

module.exports.myCustomFunction = async input => {

    if (badInput) {
        throw new Error('failed');
    }

    return sns.createTopic(someParams).promise()
}

The key here is the async keyword. By using this keyword, the function results are wrapped by a promise automatically (similar to what @peteb's answer is showing).