Trigonometric diophantine equation $8\sin^2\left(\frac{(k+1)\pi}{n}\right)=n\sin\left(\frac{2\pi}{n}\right)$

(4,0) and (12,3) are the only "primitive and non-trivial" solutions $(n,k)$ with $n\leq 10^5$.

Considering the symmetries that are present both in your geometry question and in the resulting equation, let a "primitive and non-trivial" solution be a pair $(n,k)$ with $n,k\in \mathbb{Z}$, $n\geq 3$, and $0\leq k \leq n/2 - 1$, satisfying$$8\sin^2\left(\frac{(k+1)\pi}{n}\right)=n\sin\left(\frac{2\pi}{n}\right).$$ The fact that $k$ can be restricted to a finite interval for each $n$ opens the problem up for a direct numerical search. Here's my implementation in PARI/GP:

N_MIN=3;
N_MAX=100000;
N_BLOCKSIZE=100;
TOLERANCE=1e-20;

\\ We express the problem in terms of m,n, where m=k+1.
\\ We only call the (slow) sin function once for each computation of the LHS.
lhs(m,n)={sqrt_lhs=sin(m*Pi/n); sqrt_lhs*sqrt_lhs;}
rhs(n)=sin(2*Pi/n)*n/8;

\\ Check m,n for |LHS-RHS| < TOLERANCE
for(n=N_MIN,N_MAX,cur_rhs=rhs(n); if(n%N_BLOCKSIZE==0,printf("# Done checking all n<%u\n", n)); for(m=1,n/2,{
    cur_lhs=lhs(m,n);
    if(abs(cur_lhs-cur_rhs)<TOLERANCE, printf("%u\t%u\t%e\n", n, m-1, cur_lhs-cur_rhs))
    }));
quit

---

Note: If I'm not mistaken, this is guaranteed to find all solutions $(m,n)$ with $n<n_\max = 10^5$, even though floating point math is used (assuming there are no bugs in PARI/GP or my computer). The point is that PARI/GP claims a default precision of 38 decimal digits, while the tolerance for root-finding in the above code is equivalence to 20 decimal digits. If you calculate the error propagation assuming 38-digit precision for the sin functions, the maximum absolute error after the few arithmetic operations per $n,m$ pair is still on the order of $10^{-38}$. The code can in principle give false positives if the RHS and LHS aren't equal but happen to be within the tolerance of one another, but it can't give false negatives. (In fact for these parameters there were no false positives, either.)

---

Update: Here is an implementation in C using 128-bit floating point arithmetic:

#include <stdio.h>
#include <quadmath.h>

// To compile: gcc -O3 -std=c99 -lquadmath trig.c -o trig

#define M_2PIq 6.2831853071795864769252867665590058q

#define N_MIN 3
#define N_MAX 2000
#define N_BLOCKSIZE 100
#define TOLERANCE 1e-20q
#define DIGITS 35

int main(){

  // We express the problem in terms of m,n, where m=k+1.
  for(unsigned n=N_MIN; n<=N_MAX; n++){

    // Print out a status update each N_BLOCKSIZE iterations.
    if(n%N_BLOCKSIZE==0) printf("# Done checking all n<%u\n", n);

    // Compute the RHS only once per n
    __float128 cur_rhs = sinq(M_2PIq/n)*n/8.0q;
    for(unsigned m=1; 2*m<=n; m++){

      // We only call the (slow) sin function once for each computation of the LHS.
      __float128 cur_lhs=sinq(m*M_PIq/n);
      cur_lhs*=cur_lhs;

      // Print out (n,k) if a solution is found
      if( fabsq(cur_lhs-cur_rhs) < TOLERANCE ){
        char str_difference[DIGITS];
        quadmath_snprintf(str_difference, DIGITS, "%Qe", fabsq(cur_lhs-cur_rhs));
        printf("%u\t%u\t%s\n", n, m-1, str_difference); 
      }
    }
  }

}

Small remark: It is clear that if pair $(n,k)$ with $0\le k<\left\lfloor\frac{n}{2}\right\rfloor$ satisfies the equality then pairs $(n, k+nt)$ and $(n, n-2-k+nt)$ also satisfy it (where $t\in\mathbb{Z^+}$).

As @Peter Košinár noted this equality is equivalent to $$4\cos\left(\frac{2\pi(k+1)}{n}\right)+n\sin\left(\frac{2\pi}{n}\right)=4$$ so for known $n$ the value of $k$ can be found as $$k=\frac{n}{2\pi}\arccos\left(1−\frac{n}{4}\sin \frac{2\pi}{n}\right)-1$$ If $\frac{n}{2\pi}\arccos\left(1−\frac{n}{4}\sin \frac{2\pi}{n}\right)$ is an integer then an appropriate pair is found.

The simple program written in C++ (which uses the type long double for calculations) gives the following pairs $(n,k)$ for $n\le 10^9$:

         n         k
         4         0
        12         3
 104758793  36318061
 211160884  73205826
 211708650  73395727
 317015209 109903690
 423417300 146791455
 423965066 146981356
 634030418 219807381
 742623573 257454750
 846286834 293393010
 846834600 293582911
 847382366 293772812
 847930132 293962713
 848477898 294152614
 952688925 330280775
 953236691 330470676
 953784457 330660577
 954332223 330850478
 954879989 331040379

Is it a result of the errors of numerical calculations or not? I don't know.