Return neighbors index in an 3x3 grid

MATL, 17 16 bytes

9:qWIe1Y6Z+i)BPf

The array is 1-based, that is, contains numbers from 1 to 9.

Try it online! Or verify all test cases.

Explanation

Consider input 2 as an example.

9:q  % Push [0 1 2 ... 8]
     % STACK: [0 1 2 ... 8]
W    % Rise to 2, element-wise
     % STACK: [1 2 4 ... 256]
Ie   % Reshape as 3-row matrix (column-major order)
     % STACK: [1   8  64;
               2  16 128;
               4  32 256]
1Y6  % Push [0 1 0; 1 0 1; 0 1 0]
     % STACK: [1   8  64;
               2  16 128;
               4  32 256],
              [0   1   0;
               1   0   1;
               0   1   0]
Z+   % Convolution, maintaining size
     % STACK: [10  81 136;
               21 170 336;
               34 276 160]
i    % Take input, n
     % STACK: [10  81 136;
               21 170 336;
               34 276 160],
               2
 )   % Get n-th entry (1-based; column-major order)
     % STACK: 21
B    % Convert to binary
     % STACK: [1 0 1 0 1]
P    % Flip
     % STACK: [1 0 1 0 1]
f    % Find: gives indices of nonzeros. Implicitly display
     % STACK: [1 3 5]

Mathematica, 32 bytes

GridGraph@{3,3}~AdjacencyList~#&

Uses a graph instead of an array. GridGraph@{3,3} constructs a 3x3 grid-shaped graph, shown below, which Mathematica helpfully labels with the numbers 1–9 for the vertices by default. Then ~AdjacencyList~#& tells you the neighbours of a vertex.

Mathematica, 40 bytes

{24,135,26,157,2468,359,48,579,68}[[#]]&

1-indexed. Just looks up the answer. Can someone do better in Mathematica?