Why are values of Eisenstein $E_2^*$ algebraic integers?
For $|q|<1$ we consider the null Jacobi theta functions $$ \theta_2(q):=\sum^{\infty}_{n=-\infty}q^{(n+1/2)^2}\textrm{, } \theta_3(q):=\sum^{\infty}_{n=-\infty}q^{n^2}\textrm{, } \theta_4(q):=\sum^{\infty}_{n=-\infty}(-1)^nq^{n^2}. $$ For $q=e^{-\pi \sqrt{r}}$, $r>0$ the elliptic singular modulus $k=k_r$ is given by $$ k_r=\left(\frac{\theta_2(q)}{\theta_3(q)}\right)^2. $$ The complete elliptic integrals of the first and second kind are: $$ K(x)=\frac{\pi}{2}{}_2F_1\left(\frac{1}{2},\frac{1}{2};1;x^2\right) $$ and $$ E(x)=\frac{\pi}{2}{}_2F_1\left(-\frac{1}{2},\frac{1}{2};1;x^2\right). $$ Then
Theorem.
Let $$ f(-q):=\prod^{\infty}_{n=1}(1-q^n)\textrm{, }|q|<1,\tag 1 $$ then if $q=e^{-\pi\sqrt{r}}$, $r>0$, we have $$ P(q^2)=1-24\sum^{\infty}_{n=1}\frac{nq^{2n}}{1-q^{2n}}=\frac{3}{\pi\sqrt{r}}+\left(1+k^2_r-\frac{3\alpha(r)}{\sqrt{r}}\right)\frac{4}{\pi^2}K^2.\tag 2 $$
Proof.
Let $q=e^{-\pi\sqrt{r}}$, $r>0$. Differentiating with respect to $r$ the relation $$ \log\left(f(-q^2)\right)=\sum^{\infty}_{n=1}\log\left(1-q^{2n}\right)\textrm{, }|q|<1, $$ and using (see [W,W] Chapter 21, Miscellaneous examples 10, pg. 488): $$ f(-q^2)^6=\prod^{\infty}_{n=1}\left(1-q^{2n}\right)^6=\frac{2kk'K(k)^3}{\pi^3q^{1/2}}, $$ we get $$ \frac{1}{6}\frac{d}{dr}\log\left(\frac{2k_rk'_rK^3}{\pi^3q^{1/2}}\right)=-2\sum^{\infty}_{n=1}\frac{nq^{2n-1}}{1-q^{2n}}\frac{dq}{dr}. $$ After some calculations we arrive to (reminder: $P(q^2)=1-24P^{*}(q^2)$): $$ \frac{1}{24}\left(\frac{\pi}{\sqrt{r}}+12\frac{1}{K}\frac{dK}{dk_r}\frac{dk_r}{dr}+\frac{4}{k_r}\frac{dk_r}{dr}+\frac{4}{k'_r}\frac{dk'_r}{dr}\right)=2q^{-1}P^{*}(q^2)\frac{q\pi}{2\sqrt{r}} $$ Using the known relations (see [Ber3] Chapter 17 Entry 9 pg. 120 and [Ber2] Chapter 11 Entry 30 pg. 87-88): $$ \frac{dk_r}{dr}=\frac{-k_r(k'_r)^2K^2}{\pi\sqrt{r}}, $$ $$ \frac{dk_r'}{dr}=\frac{k_r^2k_r'K^2}{\pi\sqrt{r}} $$ and ([Borw,Borw] Chapter 1, Section 1.3, pg. 7-11): $$ \frac{dK}{dk_r}=\frac{E}{k_r(k_r')^2}-\frac{K}{k_r}, $$ $$ \alpha(r)=\frac{\pi}{4K^2}-\sqrt{r}\left(\frac{E}{K}-1\right), $$ we arrive to $$ P^{*}(q^2)=-\frac{1}{24}+\frac{K^2}{6\pi^2}+\frac{K^2k_r^2}{6\pi^2}-\frac{\alpha(r)K^2}{2\pi^2\sqrt{r}}+\frac{1}{8\pi\sqrt{r}}. $$ From this along with $P(q^2)=1-24P^{*}(q^2)$, we get the result. qed
Remark that $\alpha(r)$, $r>0$ is the elliptic alpha function in [Borw,Borw] book. It is proven in [Borw,Borw] that $\alpha(r)$ is algebraic number whenever $r$ is positive rational.
Proposition.
Let $r>0$ and $q=e^{-\pi\sqrt{r}}$, $K=K(k_r)$, then $$ 1-24\sum^{\infty}_{n=1}\frac{nq^n}{1-q^n} =\frac{6}{\pi\sqrt{r}}+\left(1+k^2_r-\frac{6 \alpha(r)}{\sqrt{r}}\right)\frac{4K^2}{\pi^2}= $$ $$ =\frac{6}{\pi\sqrt{r}}+s_1(r)\theta_3(q)^4,\tag 3 $$ where $$ s_1(r):=1-\frac{6\alpha(r)}{\sqrt{r}}+k_r^2.\tag 4 $$
Proof.
From (1) setting $r\rightarrow r/4$ and using $$ k_{r/4}=\frac{2\sqrt{k_r}}{1+k_r}\textrm{, }M_2(r)=\frac{1+k'_r}{2} $$ and (see [Boew,Borw]) $$ \alpha(4r)=(1+k_{4r})^2\alpha(r)-2\sqrt{r}k_{4r},\tag a $$ we get the result. qed
The relations you are looking for are (3) and (4).
Notes. The upper half plane formulation can constructed if in place of $k_r$ we define for $q=e^{2\pi i \tau}$, $Im(\tau)>0$, the singular modulus: $$ m(q):=\left(\frac{\theta_2(q)}{\theta_3(q)}\right)^2\textrm{, }q=e^{2\pi i \tau}\textrm{, }Im(\tau)>0\textrm{, }-1/2\leq Re(\tau)<1/2 $$ and $m'(q)=\sqrt{1-m(q)^2}$, then $$ \theta_2(q)^2=\frac{2m(q)K(m(q))}{\pi}\textrm{, }\theta_3(q)^2=\frac{2K(m(q))}{\pi}\textrm{, }\theta_4(q)^2=\frac{2m'(q)K(m(q))}{\pi}. $$ Then if also we have $$ i\frac{K\left(m'(q)\right)}{K(m(q))}=2z $$ see [Arm,Eb],[Borw,Borw]$\ldots etc$. You have to find the complex analog of elliptic alpha $\alpha(r)$.
References
[Arm,Eb]: J.V. Armitage, W.F. Eberlein. 'Elliptic Functions'. Cambridge University Press. (2006)
[Bag]: N.D. Bagis. 'On certain theta functions and modular forms in Ramanujan theories'. arXiv:1511.03716v2 [math.GM] 6 Dec 2017.
[Ber2]: Bruce. C. Berndt. 'Ramanujan`s Notebooks Part II'. Springer-Verlag, New York. 1989.
[Ber3]: B.C. Berndt. 'Ramanujan`s Notebooks Part III'. Springer Verlag, New York (1991).
[Borw,Borw]: J.M. Borwein and P.B. Borwein. 'Pi and the AGM: A Study in Analytic Number Theory and Computational Complexity', Wiley, New York, 1987.
[W,W]: E.T. Whittaker and G.N. Watson. 'A course on Modern Analysis'. Cambridge U.P. 1927.
Continuing...
From relations (3) and (4) we get $q=e^{-\pi\sqrt{r}}$, $r>0$ $$ \left(E_2(q)-\frac{6}{\pi\sqrt{r}}\right)\frac{1}{\theta_3(q)^4}=1-\frac{6\alpha(r)}{\sqrt{r}}+k_r^2\tag 5 $$ If $r$ is positive rational, then $\alpha(r)$ and $k_r$ are algebraic numbers.
Moreover for any integer $N$ we can get easily the evaluation (in each case separate) of $k_r$ usng
$$
\left(\frac{256}{r_0^{16}}-r_0^8\right)^3=j\left(\frac{1+\sqrt{-N}}{2}\right),
$$
where $r=N-$integer and
$$
j\left(z\right)=\left(\left(\frac{\eta(z/2)}{\eta(z)}\right)^{16}+16\left(\frac{\eta(z)}{\eta(z/2)}\right)^8\right)^3
$$
is the known $j-$invariant. Then
$$
k_N^2=\frac{1}{2}-\sqrt{\frac{1}{4}-\frac{16}{r_0^{24}}}
$$
An interesting article is [Broad] which reduces these calculations very considerably.
Another way is to evaluate it using $$ j_r=j(i\sqrt{r})=\frac{256(k_r^2+(k'_r)^4)^3}{(k_rk'_r)^4}\textrm{, }k'_r=\sqrt{1-k_r^2}.\tag b $$ The definition and evaluations of $\alpha(r)$ function are given in [Borw,Borw] and you must read the related theory (book).
The elliptic alpha function $\alpha(r)$ also has been introduced for evaluation of billions of digits of $\pi$. The Borwein bothers give in their book the underlined theory of $\pi-$formulas and algorithms.
Note also that [W,W] ($q=e^{-\pi\sqrt{r}}$, $r>0$): $$ \prod^{\infty}_{n=1}(1-q^n)=2^{1/3}\pi^{-1/2}q^{-1/24}(k_r)^{1/12}(k'_r)^{1/3}K(k_r)^{1/2}\tag 6 $$ and $$ \theta_3(q)=\sqrt{\frac{2K(k_r)}{\pi}}\tag 7 $$ and $k_r$ is algebraic number when $r-$positive raional.
Note also that the above two formulas have analytic continuations in upper half plane $Im(z)>0$.
References
[Broad]: David Broadhurst. 'Solutions by radicals at singular values $k_N$ from new class invariants for $N\equiv 3\textrm{mod}8$'. arXiv:0807.2976v3 [math-ph] 31 Jul 2008.
...Continuing
$$ A_r=\sqrt{-r}\left(E_2(q)-\frac{6}{\pi\sqrt{r}}\right)\frac{1}{\eta_1(q)^4}=\sqrt{-r}\frac{1+k_r^2-6\frac{\alpha(r)}{\sqrt{r}}}{2^{-2/3}(k_r)^{1/3}(k'_r)^{4/3}}\textrm{, }q=e^{-\pi\sqrt{r}}\textrm{, }r>0,\tag 8 $$ where $$ \eta_1(q):=q^{1/24}\prod^{\infty}_{n=1}(1-q^n)\textrm{, }|q|<1. $$ Your case is $q=e^{2\pi i z_N}$, where $z_N=\frac{N+i\sqrt{N}}{2}$, $N$ positive integer. Hence if $N$ is even, then $q=e^{2\pi i \left(N+i\sqrt{N}\right)/2}=e^{-\pi\sqrt{N}}$, $N$ positive even integer. Hence I can evaluate all your $N-$even cases.
1) If $N=r=4$, then $$ k_4=3-2\sqrt{2}, $$ $$ k'_4=2\sqrt{-4+3\sqrt{2}}, $$ $$ \alpha(4)=6-4\sqrt{2}. $$ Hence $$ A_4=\ldots=0 $$ 2) If $N=8=r$, then $$ k_8=-2 \sqrt{2 \left(7+5 \sqrt{2}\right)}+4 \sqrt{2}+5, $$ $$ k'_8=2 \sqrt{\sqrt{2 \left(799+565 \sqrt{2}\right)}-4 \left(7+5 \sqrt{2}\right)} $$ $$ \alpha(8)=-8 \sqrt{41+29 \sqrt{2}}+26 \sqrt{2}+36 $$ Hence $$ A_8=4i $$ 3) For $N=r=12$ $$ k_{12}=15-10 \sqrt{2}+8 \sqrt{3}-6 \sqrt{6} $$ $$ k'_{12}=2 \sqrt{3 \sqrt{2 \left(4801-1960 \sqrt{6}\right)}+85 \sqrt{6}-208} $$ $$ \alpha(12)=2 \left(132-94 \sqrt{2}+77 \sqrt{3}-54 \sqrt{6}\right) $$ Hence $$ A_{12}=6i4^{1/3} $$ For $N=r=16$, we have $$ k_{16}=-4 \sqrt{140+99 \sqrt{2}}+24 \sqrt{2}+33, $$ $$ k'_{16}=2 \sqrt{6 \sqrt{69708+49291 \sqrt{2}}-8 \left(140+99 \sqrt{2}\right)} $$ $$ \alpha(16)=4 \left(-4 \sqrt{15900+11243 \sqrt{2}}+252 \sqrt{2}+357\right) $$ Hence $$ A_{16}=12i\sqrt{2} $$ For $N=r=28$, we have $$ k_{28}=255-180 \sqrt{2}+96 \sqrt{7}-68 \sqrt{14} $$ $$ k'_{28}=2 \sqrt{30 \sqrt{9321998-6591648 \sqrt{2}}+45798 \sqrt{2}-64768} $$ $$ \alpha(28)=82464-58312 \sqrt{2}+31170 \sqrt{7}-22040 \sqrt{14} $$ Hence $$ A_{28}=54 i $$
Where for all evaluations I have used [Borw,Borw] pg.172-173 and relation (a).
For odd $N$ we have $q=-e^{-\pi\sqrt{N}}$, which is very hard.
...CONTINUED
If $k_r=m\left(e^{-\pi\sqrt{r}}\right)$, with $m(q)$ is as below: $$ m(q):=\left(\frac{\vartheta_2(q)}{\vartheta_3(q)}\right)^2, $$ changing $q\rightarrow-q$, we have $$ m(-q)=\left(\frac{i^{1/2}q^{1/4}\sum_{\scriptsize n\in\textbf{Z}}q^{n^2+n}}{\sum_{\scriptsize n\in\textbf{Z}}(-1)^nq^{n^2}}\right)^2=i\left(\frac{\vartheta_2(q)}{\vartheta_4(q)}\right)^2=i\frac{m(q)}{m'(q)}. $$ Hence $$ m(-q)=i\frac{\cdot m(q)}{m'(q)}\textrm{, }m'(q)=\sqrt{1-m(q)^2}. $$ and from $$ \frac{1}{\sqrt{1-x}}K\left(\frac{x}{x-1}\right)=K(x) $$ we get $$ K^{*}=K(m(-q))=m'(q)K(m(q))=m'(q)K. $$ Hence $$ \eta_1(-q)=\frac{e^{i\pi/24}2^{1/3}}{\sqrt{\pi}}\left(m(q)m'(q)\right)^{1/12}K(m(q))^{1/2} $$ Hence if $N=r=$odd, we have $q=-e^{-\pi\sqrt{r}}$ and $$ A_N:=\sqrt{-N}\left(E_2(z_N)-\frac{3}{\pi Im(z_N)}\right)\frac{1}{\eta(z_N)^4}= \sqrt{-r}\left(P(-q)-\frac{6}{\pi\sqrt{r}}\right)\frac{1}{\eta_1(-q)^4}. $$ But as Paramanand Singh noted, we have $$ P(-q)=\left(\frac{2K}{\pi}\right)^2\left(\frac{6E}{K}+4m(q)^2-5\right). $$ Also $$ E=E(m(q))=\frac{\pi}{4\sqrt{r}K}+\left(1-\frac{\alpha(r)}{\sqrt{r}}\right)K. $$ Hence for $N=r=$odd $$ A_N=A_r=\ldots=\frac{e^{i\pi/3}4^{1/3}\left(-6\alpha(r)+\sqrt{r}(1+4m(q)^2)\right)}{\left(m(q)m'(q)\right)^{1/3}}\textrm{, }q=e^{-\pi\sqrt{r}}\tag 9 $$
Hence if $k_r$, $\alpha(r)$, for $r=3,7,11,19,27,43,67,163$ are known, then $A_r=A_N$ are known and hence answer to the evaluations of the case $z_N=\frac{N+\sqrt{-N}}{2}$, with $N-$odd. For all odd cases, since $h(-N)=1$ you can evaluate the $j-$invariant $j(i\sqrt{N})$ which (see [Broad]) have minimal polynomials of degree at most 3. Then using (b) you can find $k_r$.
For the elliptic alpha function, you have to search. With a quick view I find $r=3,7,27$ only (see [Borw,Borw] pg.173)
This is too long to fit in a comment.
Ramanujan established in his monumental paper Modular Equations and Approximations to $\pi$ that the desired expression is an algebraic number if $\tau_n=\sqrt{-n} $ where $n$ is a positive rational number.
Let $$P(q) =1-24\sum_{j=1}^{\infty}\frac{jq^{j} }{1-q^{j}}\tag{1}$$ and then Ramanujan proved that $$P(e^{-2\pi\sqrt{n}}) =\left( \frac{2K}{\pi}\right)^2A_n+\frac{3}{\pi\sqrt{n}}\tag{2}$$ where $A_n$ is an algebraic number dependent on $n$ provided that $n$ is a positive rational number. Here $K=K(k) $ is the complete elliptic integral of first kind with modulus $k$ and $k$ corresponds to nome $q$ so that $$k=\frac{\vartheta_{2}^{2}(q)}{\vartheta_{3}^{2}(q)},q=e^{-\pi\sqrt{n}}\tag{3}$$ and $k$ is an algebraic number if $n$ is a positive rational number. Ramanujan's proof is presented in one of my blog posts.
If $q=\exp(\pi i\tau) $ then we have $P(q^2) =E_2(\tau)$. Further we have $$\eta(\tau) =q^{1/12}\prod_{j=1}^{\infty}(1-q^{2j}),q=e^{\pi i\tau}\tag{4}$$ It is well known that the eta function can be expressed in terms of $k, K$ as $$\eta(\tau)=2^{-1/3}\sqrt{\frac{2K}{\pi}}(kk')^{1/6}\tag{5}$$ and therefore the equation $(2)$ can be written as $$A_n=\dfrac{E_{2}(\tau_n)-\dfrac{3}{\pi\Im{\tau_n}}}{\eta^4(\tau_n)}$$ which is an algebraic number.
Note that if $\tau_n=\dfrac{n+\sqrt{-n} }{2}$ then we have $E_2(\tau_n)=P(q^2)$ if $n$ is even and $q=e^{\pi i\tau_n} $ and if $n$ is odd then $E_2(\tau_n)=P(-q^2)$. Using Ramanujan's technique one can prove that $$P(e^{-\pi\sqrt{n}})=\left(\frac{2K}{\pi}\right) ^2B_n+\frac{6}{\pi\sqrt{n}}\tag{6}$$ (just replace $n$ in $(2)$ by $n/4$ and $B_n=A_{n/4}$) and $$P(-e^{-\pi\sqrt{n}}) =\left(\frac{2K}{\pi}\right)^2C_n+\frac{6}{\pi\sqrt{n}}\tag{7}$$ where $B_n, C_n$ are algebraic numbers and $n$ is a positive rational number and thus the expression mentioned in the question is an algebraic number. Proving that it is an algebraic integer is unfortunately not possible via Ramanujan methods.
I would like to outline a conceptual idea which does not answer this question directly.
The contents of Section 14, Chapter VIII of Weil's book Elliptic Functions according to Eisenstein and Kronecker implies that $$E_2(\tau)-\frac{3}{\pi\, Im(\tau)}=\frac{3}{\pi^2}K_2(0,0,2),$$
where $K_2(0,0,s)$ is defined by $$K(0,0,s):=\sum_{w\in W}{^*}\frac{\bar{w}^2}{|w|^{2s}}$$ with $Re(s)>2$, $W$ is the lattice spanned by $1,\tau$ and $K(0,0,2)$ is defined by analytic continuation of $K(0,0,s)$.
Let $1,\tau$ be the basis of algebraic integers of some imaginary quadratic field of class number $1$. Then $K(0,0,s)$ is an integral multiple of Hecke $L$-function whose character is of $(2,0)$-type. We would like to remark that to evaluate $K(0,0,2)$ is to evaluate a critical value of a Hecke L-function. The observation attributed to Deligne et al. asserts that the critical values of these L-functions are some algebraic periods times an algebraic number, and the claim in the OP might be solved by refiner studies of these Hecke L-functions. See p. 12 of M. Watkins' note for an example.
P.S. This idea can be extended to other Ramanujan series arising from quadratic fields of class number 2.