Simplifying $\prod_{k=3}^{n-1}\cos\left(\frac{\pi}{k}\right)$

As said in comments and answer, a closed form is more than likely improbable.

However, we could make some approximations. Let $a_k=\cos \left(\frac{\pi }{k}\right)$ and consider $$P_n=\prod_{k=3}^{n-1} a_k =\frac{1}{2\sqrt 2}\prod_{k=5}^{n-1} a_k$$ Taking logarithms $$ \log(P_n)=\log\left(\frac{1}{2\sqrt 2}\right)+\sum_{k=5}^{n-1} \log(a_k )$$ Now, using composition of Taylor series $$\log(a_k)=-\frac{\pi ^2}{2 k^2}-\frac{\pi ^4}{12 k^4}-\frac{\pi ^6}{45 k^6}-\frac{17 \pi ^8}{2520 k^8}-\frac{31 \pi ^{10}}{14175 k^{10}}-\frac{691 \pi ^{12}}{935550 k^{12}}+O\left(\frac{1}{k^{14}}\right)$$ Just for your curiosity $$\log(a_k)=-\sum_{p=1}^\infty\frac{2^{2 p-3} \pi ^{2 p} (E_{2 p-1}(1)-E_{2 p-1}(0))}{p\, (2 p-1)!\,k^{2p}}=\sum_{p=1}^\infty \frac {c_p}{k^{2p}}$$ where appear Euler polynomials but I am afraid that we cannot do much for the summation over $k$ beside using the fact that $$\sum_{k=7}^{n-1}k^{-2p}=H_{n-1}^{(2 p)}-1-2^{-2 p}-3^{-2 p}-4^{-2 p}-5^{-2 p}-6^{-2 p}$$

Using the expansion as written above, we should arrive to some ugly expression such as $$\sum_{k=5}^{n-1} \log(a_k )=K-\frac{\pi ^2}{2} H_{n-1}^{(2)}-\frac{\pi ^4 }{12} H_{n-1}^{(4)}-\frac{\pi ^6}{45} H_{n-1}^{(6)}-\frac{17 \pi ^8}{2520} H_{n-1}^{(8)}-\frac{31 \pi ^{10} }{14175} H_{n-1}^{(10)}-\frac{691 \pi ^{12}}{935550} H_{n-1}^{(12)}$$ where appear genralized harmonic numbers and where $$K=\frac{205 \pi ^2}{288}+\frac{22369 \pi ^4}{248832}+\frac{607493 \pi ^6}{26873856}+\frac{7339467953 \pi ^8}{1083553873920}+\frac{76853883151 \pi ^{10}}{35107145515008}+\frac{6162541526621059 \pi ^{12}}{8341457774365900800}$$

Trying for a few values of $n$, the decimal values are $$\left( \begin{array}{ccc} n & \text{approximation} & \text{exact} \\ 3 & 0.99918368 & 1.00000000 \\ 4 & 0.49999441 & 0.50000000 \\ 5 & 0.35355339 & 0.35355339 \\ 6 & 0.28603083 & 0.28603070 \\ 7 & 0.24770997 & 0.24770985 \\ 8 & 0.22317897 & 0.22317887 \\ 9 & 0.20619049 & 0.20619039 \\ 10 & 0.19375568 & 0.19375558 \\ 11 & 0.18427260 & 0.18427251 \\ 12 & 0.17680827 & 0.17680818 \\ 13 & 0.17078367 & 0.17078359 \\ 14 & 0.16582101 & 0.16582093 \\ 15 & 0.16166353 & 0.16166345 \\ 16 & 0.15813079 & 0.15813072 \\ 17 & 0.15509235 & 0.15509228 \\ 18 & 0.15245161 & 0.15245154 \\ 19 & 0.15013553 & 0.15013546 \\ 20 & 0.14808788 & 0.14808780 \end{array} \right)$$

Still using the above expansion and the asymptotics of harmonic numbers, converting to decimals, we should have for large $n$ $$P_n \sim \frac{1}{2 \sqrt{2}} \exp \left(c+\frac{\pi ^2}{2 n} +\frac{\pi ^2}{4 n^2} \right)\qquad \text{with} \qquad c \approx -1.12361$$

Applied to $n=20$, this last formula would give $0.148018$.

Edit

For the infinite product

$$\prod_{k=3}^{\infty}\cos\left(\frac{\pi}{k}\right)=0.1149420448532962007010401576568126847536004314847$$ while the given appoximation would lead to $0.11494164$.

Update

Remembering that we know the exact values up to $k=6$, using as an approximation $$\cos(x) \simeq\frac{\pi ^2-4x^2}{\pi ^2+x^2}\qquad (-\frac \pi 2 \leq x\leq\frac \pi 2)\implies \cos\left(\frac{\pi}{k}\right)=\frac{k^2-4}{k^2+1}$$ we arrive to $$P_n=\prod_{k=3}^{n-1}\cos\left(\frac{\pi}{k}\right)=\frac{\sqrt{6}+\sqrt{30}}{32} \prod_{k=7}^{n-1}\cos\left(\frac{\pi}{k}\right)=\frac{40885 \left(\sqrt{6}+\sqrt{30}\right) \pi \text{csch}(\pi )}{774144} \frac{ \Gamma (n-2)\, \Gamma (n+2)}{ \Gamma (n-i) \,\Gamma (n+i)}$$ giving $$P_\infty=\frac{40885 \left(\sqrt{6}+\sqrt{30}\right) \pi \text{csch}(\pi )}{774144}\approx 0.113881$$

Using instead the $[2,2]$ Padé approximant $$\cos\left(\frac{\pi}{k}\right)=\frac{12 k^2-5 \pi ^2}{12 k^2+\pi ^2}$$ we should get $$P_n=\frac{K_1}{K_2} \frac{\Gamma \left(n-\frac{1}{2} \sqrt{\frac{5}{3}} \pi \right) \Gamma \left(n+\frac{1}{2} \sqrt{\frac{5}{3}} \pi \right)}{\Gamma \left(n-\frac{i \pi }{2 \sqrt{3}}\right) \Gamma \left(n+\frac{i \pi }{2 \sqrt{3}}\right)}$$ where $$K_1=\sqrt{5} \left(\sqrt{6}+\sqrt{30}\right) \left(12+\pi ^2\right) \left(48+\pi ^2\right) \left(108+\pi ^2\right) \left(192+\pi ^2\right) \left(300+\pi ^2\right) \left(432+\pi ^2\right)$$ $$ \sin \left(\frac{1}{2} \sqrt{\frac{5}{3}} \pi ^2\right) \text{csch}\left(\frac{\pi ^2}{2 \sqrt{3}}\right)$$ $$K_2=800 \left(\pi ^2-60\right) \left(5 \pi ^2-432\right) \left(5 \pi ^2-192\right) \left(5 \pi ^2-108\right) \left(5 \pi ^2-48\right) \left(5 \pi ^2-12\right)$$ giving $$P_\infty=\frac{K_1}{K_2}\approx 0.114938$$

It is normal that the Padé approximant leads to better results sincd $$\int_0^{\frac \pi 7} \left(\cos (x)-\frac{\pi ^2-4 x^2}{\pi ^2+x^2}\right)^2\,dx\approx 1.028 \times 10^{-7}$$ while $$\int_0^{\frac \pi 7}\left(\cos (x)-\frac{12-5 x^2}{12+x^2}\right)^2\,dx\approx 9.574 \times 10^{-12}$$

Well, you could write $$ \prod_{k=3}^n \cos(\pi/k) = 2^{2-n} \sum_{signs} \cos\left(\pm \frac{\pi}{3} \pm \frac{\pi}{4} \pm \ldots \pm \frac{\pi}{n}\right) $$ where the sum is over all $2^{n-2}$ possible choices of the $\pm$ signs.
If $n$ is moderately large, those $\pm \pi/3 \pm \ldots \pm \pi/n$ will be rather nasty rational multiples of $\pi$. So it's not exactly a "simplification".

For example, if $n=7$ I get $$ 16^{-1} \left(\cos \left( {\frac {11\,\pi}{420}} \right) +\cos \left( {\frac {13\, \pi}{140}} \right) +\cos \left( {\frac {27\,\pi}{140}} \right) +\cos \left( {\frac {109\,\pi}{420}} \right) \\+\cos \left( {\frac {43\,\pi}{ 140}} \right) +\cos \left( {\frac {179\,\pi}{420}} \right) +\cos \left( {\frac {59\,\pi}{420}} \right) +\cos \left( {\frac {83\,\pi}{ 140}} \right) \\+\cos \left( {\frac {57\,\pi}{140}} \right) +\cos \left( {\frac {199\,\pi}{420}} \right) +\cos \left( {\frac {97\,\pi}{ 140}} \right) +\cos \left( {\frac {319\,\pi}{420}} \right)\\ +\cos \left( {\frac {31\,\pi}{420}} \right) +\cos \left( {\frac {113\,\pi}{ 140}} \right) +\cos \left( {\frac {151\,\pi}{420}} \right) +\cos \left( {\frac {153\,\pi}{140}} \right) \right)$$