Russian Caesar cipher

JavaScript (ES6),  148 ... 110  108 bytes

Saved 12 bytes thanks to @Grimy!

This function applies some maths to the code points.

s=>s.replace(/./g,s=>String.fromCharCode(...(n=s.charCodeAt()-304)%80-1?[(n^16*(n>0))+304]:[1060+n%48,776]))

Try it online!

How?

For each character in the input string, we define \$n\$ as its code point minus \$304\$.

 characters    | code points  | n
---------------+--------------+---------------
 А to Я        | 1040 to 1071 | 736 to 767
 а to я        | 1072 to 1103 | 768 to 799
 Ё             | 1025         | 721
 ё             | 1105         | 801
 ASCII         | 0 to 126     | -304 to -178

Then we apply the following logic:

// neither 'Ё' nor 'ё'?
n % 80 - 1 ?
  // output a single code point
  [
    // invert the case if this is a Cyrillic character
    (n ^ 16 * (n > 0))
    // and restore the original offset
    + 304
  ]
:
  // output two code points
  [
    // the first one is either 1061 (Х) or 1093 (х)
    1060 + n % 48,
    // the 2nd one is the combining diaeresis
    776
  ]

05AB1E, 16 40 39 33 bytes

63ÝD16^‚Ž4K+ç`‡•2.w2γ•3äçDl‚vy`«:

Would be just the first 15 bytes without the edge case of mapping Ёё to Х̈х̈.

-7 bytes thanks to @Grimy.

Try it online.

Explanation:

63Ý              # Push a list in the range [0,63]
   D             # Duplicate it
    16^          # Bitwise-XOR each value with 16: [16..31, 0..15, 48..63, 33..47]
        ‚        # Pair it with the initial [0,63] list we duplicated
         Ž4K     # Push compressed integer 1040
            +    # Add it to each integer in the inner lists
             `   # Push both lists separated to the stack again
              ‡  # Transliterate the characters in the first list to the second list
                 # in the (implicit) input-string
•2.w2γ•          # Push compressed integer 10251061776
       3ä        # Spit it into three parts: [1025,1061,776]
         ç       # Convert each to a character: ["Ё","Х","̈"]
Dl               # Create a lowercase copy: ["ё","х","̈"]
  ‚              # Pair them together: [["Ё","Х","̈"],["ё","х","̈"]]
   v             # Loop over both these lists `y`:
    y`           #  Push the characters of the current list separated to the stack
      «          #  Append the top two together: Х̈/х̈
       :         #  And replace the Ё/ё with Х̈/х̈ in the (modified) input-string
                 # (after which the result is output implicitly)

See this 05AB1E tip of mine (section How to compress large integers?) to understand why Ž4K is 1040 and •2.w2γ• is 10251061776.


Retina 0.8.2, 41 bytes

T`А-Па-пя-рЯ-Р`Ro
Ё
Х̈
ё
х̈

Try it online! Just a transliteration and a fixup of the two special cases. The characters to be transliterated are listed in mirror order so that Ro can be used to specify the reverse order for the transliteration.