What is the alternation operator in Lua patterns?

First, note that Lua patterns are not regular expressions; they are their own simpler matching language (with different benefits and drawbacks).

Per the specification I've linked to above, and per this answer, there is no alternation operator in a Lua pattern. To get this functionality you'll need to use a more powerful Lua construct (like LPEG or a Lua Regex Library).