New Order #6: Easter Egg

Jelly,  16 14 11 10 9  8 bytes

-1 thanks to Lynn (mod-2; logical NOT; add to self: Ḃ¬+ -> bitwise OR with 1:|1)

|1×r)ẎQi

A monadic Link accepting an integer, n, which yields an integer, a(n).

Try it online! (very inefficient since it goes out to layer \$\lceil\frac n2\rceil\$)

An 11-byte version, ½‘|1×rƲ€ẎQi, completes all but the largest test case in under 30s - Try it at TIO - this limits the layers used to \$\lceil\frac{\lfloor\sqrt n\rfloor+1}2\rceil\$.

How?

The permutation is to take the natural numbers in reversed slices of lengths [1,5,3,11,5,17,7,23,9,29,11,35,13,...] - the odd positive integers interspersed with the positive integers congruent to five modulo six, i.e [1, 2*3-1, 3, 4*3-1, 5, 6*3-1, 7, 8*3-1, 9, ...].

This is the same as concatenating and then deduplicating reversed ranges [1..x] of where x is the cumulative sums of these slice lengths (i.e. the maximum of each slice) - [1,6,9,20,25,42,49,72,81,110,121,156,169,...], which is the odd integers squared interspersed with even numbers multiplied by themselves incremented, i.e. [1*1, 2*3, 3*3, 4*5, 5*5, 6*7, 7*7,...].

Since the differences are all greater than 1 we can save a byte (the reversal) by building ranges [x..k] directly, where k is the 1-indexed index of the slice.

Due to this structure the permutation is a self-conjugate permutation, i.e. we know that \$P(n) = v \iff P(v) = n\$, so rather than finding the value at (1-indexed) index n (|1×r)ẎQị@) we can actually get the (1-indexed) index of the first occurrence of n (|1×r)ẎQi).

|1×r)ẎQi - Link: integer, n       e.g. 10
    )    - for each k in [1..n]:  vs = [ 1, 2, 3, 4, 5, 6, 7, 8, 9,10]
|1       -   bitwise-OR (k) with 1     [ 1, 3, 3, 5, 5, 7, 7, 9, 9,11]
  ×      -   multiply (by k)           [ 1, 6, 9,20,25,42,49,72,81,110]
   r     -   inclusive range (to k)    [[1],[6..2],[9..3],[20..4],...,[110..10]]
     Ẏ   - tighten                     [1,6,5,4,3,2,9,8,7,6,5,4,3,20,...,4,......,110,...,10]
      Q  - de-duplicate                [1,6,5,4,3,2,9,8,7,20,...,10,......,110,...82]
       i - first index with value (n)  20

JavaScript (ES7),  46 45  41 bytes

0-indexed.

n=>((x=n**.5+1&~1)*2-(n<x*x+x)*4+3)*x+1-n

Try it online!

How?

This is based on the 1-indexed formula used in the example programs of A090861.

\$x_n\$ is the layer index of the spiral, starting with \$0\$ for the center square:

$$x_n=\left\lfloor\frac{\sqrt{n-1}+1}{2}\right\rfloor$$

Try it online!

\$k_n\$ is set to \$6\$ for the bottom part of each layer (including the center square), and to \$-2\$ everywhere else:

$$k_n=\begin{cases} -2&\text{if }n\le 4{x_n}^2+2x_n\\ 6&\text{otherwise} \end{cases}$$

Try it online!

Then \$a_n\$ is given by:

$$a_n=8{x_n}^2+k_nx_n+2-n$$

Try it online!

Which can be translated into:

n=>8*(x=(n-1)**.5+1>>1)*x+(n<=4*x*x+2*x?-2:6)*x+2-n

Making it 0-indexed saves 5 bytes right away:

n=>8*(x=n**.5+1>>1)*x+(n<4*x*x+2*x?-2:6)*x+1-n

The formula can be further simplified by using:

$${x'}_n=2\times\left\lfloor\frac{\sqrt{n}+1}{2}\right\rfloor$$

which can be expressed as:

x=n**.5+1&~1

leading to:

n=>2*(x=n**.5+1&~1)*x+(n<x*x+x?-1:3)*x+1-n

and finally:

n=>((x=n**.5+1&~1)*2-(n<x*x+x)*4+3)*x+1-n

Wolfram Language (Mathematica), 60 bytes

8(s=⌊(⌊Sqrt[#-1]⌋+1)/2⌋)^2-#+2+If[#<=4s^2+2s,-2,6]s&

Try it online!