Show that this process is not a martingale

Here's an approach that comes from Li, Xue-Mei, Strict local martingales: examples, Stat. Probab. Lett. 129, 65-68 (2017). ZBL1386.60159, https://arxiv.org/abs/1609.00935. Indeed, she mentions this very example after Corollary 4 (bottom of page 4 in the arXiv version).

Lemma. Suppose $X_t$ is a continuous martingale, and let $\langle X \rangle_t$ be its quadratic variation. Then for every $0 < \alpha < 1$ and every $t$, we have $E[\langle X \rangle_t^{\alpha/2}] < \infty$.

Proof. Let $M_t = \sup_{0 \le s \le t} |X_s|$. By Doob's maximal inequality, for any $x > 0$ we have $$P(M_t \ge x) \le \frac{1}{x} E|X_t|.$$ So $$\begin{align*} E[M_t^\alpha] &= \int_0^\infty P(M_t^\alpha \ge x)\,dx \\&\le 1 + \int_1^\infty P(M_t^\alpha \ge x)\,dx \\& \le 1 + E|X_t| \cdot \int_1^\infty x^{-1/\alpha}\,dx \\&< \infty.\end{align*}$$ Then by the Burkholder-Davis-Gundy inequality we have$$E[\langle X \rangle_t^{\alpha/2}] \le C_\alpha E[M_t^\alpha] < \infty$$ as desired. $\Box$

Now for the process at hand, we have $\langle X \rangle_t = \int_0^t e^{2 B_s^2}\,ds$. By Hölder's inequality or Jensen's we have $\langle X \rangle_t^\alpha \ge t^{\alpha-1} \int_0^t e^{2 \alpha B_s^2}\,ds$, so by Fubini/Tonelli $E[ \langle X \rangle_t^\alpha] \ge t^{\alpha - 1} \int_0^t E[e^{2 \alpha B_s^2}]\,ds$. The integrand is infinite for all $s > 1/4$, so by the lemma $X_t$ cannot be a martingale for any $t > 1/4$.

---

I feel like perhaps there should be a more direct way to relate $E[\langle X\rangle_t^{\alpha/2}]$ to $E[X_t]$, perhaps by some clever application of Itô's formula and Hölder's inequality, but I don't quite see how.

Let $B_t:=B(t)$. By the Itô formula $$f(B_1)-f(B_0)=\int_0^1 f'(B_t)\,dB_t+\frac12\,\int_0^1 f''(B_t)\,dt$$ with $f(b):=\int_0^b e^{a^2}da$ (with $\int_0^b:=-\int_b^0$ for $b<0$), we have \begin{equation*} X_1=\int_0^1 f'(B_t)\,dB_t =f(B_1)-\frac12\,\int_0^1 f''(B_t)\,dt =f(B_1)-\int_0^1 B_t e^{B_t^2}\,dt. \tag{1} \end{equation*} From here, it is not hard to see that $EX_1$ does not exist.

Indeed, consider the event \begin{equation*} A:=\{B_u\in[b,b+1/b],B_1-B_u<-2/b,M_u>-b\}, \end{equation*} where $b\to\infty$, \begin{equation*} u:=1-1/b^2, \end{equation*} \begin{equation*} M_u:=\min_{t\in[u,1]}(B_t-B_u). \end{equation*} Note that on the event $A$ we have $B_u\ge b$, $B_1<b-1/b$ and $B_t>0$ for all $t\in[u,1]$. Therefore and because (by the l'Hospital rule) $f(b)\sim e^{b^2}/(2b)$, it follows from (1) that on $A$ \begin{align*} X_1&\le\frac{e^{(b-1/b)^2}}{(2+o(1))b}-\int_0^u B_t e^{B_t^2}\,dt \\ &=\frac{e^{b^2-2}}{(2+o(1))b}-\int_0^u B_t e^{B_t^2}\,dt \\ &\le\frac{e^{b^2-2}}{(2+o(1))b}-\int_0^u (\tfrac tu\,b+B_t^u) \exp\{(\tfrac tu\,b+B_t^u)^2\}\,dt, \end{align*} where $B_t^u:=B_t-\frac tu\,B_u$; for the latter, displayed inequality, we recall that $B_u\ge b$ on $A$ and use the fact that $se^{s^2}$ is increasing in real $s$.

The Brownian bridge $(B_t^u)_{t\in[0,u]}$ is a zero-mean (Gaussian) process independent of $(B_u,B_1-B_u,M_u)$; so, $(B_t^u)_{t\in[0,u]}$ is independent of the event $A$. Introduce also the event $$C_x:=\{\max_{t\in[0,u]}|B_t^u|\le x\}$$ for real $x>0$, which allows us to use the Fubini theorem to get \begin{align*} EX_1\,1_{A\cap C_x}\le\Big(&\frac{e^{b^2-2}}{(2+o(1))b}P(C_x) \\ &-\int_0^u E(\tfrac tu\,b+B_t^u) \exp\{(\tfrac tu\,b+B_t^u)^2\}1_{C_x}\,dt\Big)\,P(A). \tag{2} \end{align*} Because $g_a(b):=(a+b) e^{(a+b)^2}+(a-b)e^{(a-b)^2}$ is convex and even in real $b$ for each real $a\ge0$, we have $g_a(b)\ge g_a(0)=2ae^{a^2}$. Therefore and because the distribution of the Brownian bridge $(B_t^u)_{t\in[0,u]}$ is symmetric,
\begin{align*} E(a+B_t^u) \exp\{(a+B_t^u)^2\}1_{C_x} =\tfrac12\,Eg_a(B_t^u)\,1_{C_x} \ge ae^{a^2}\,P(C_x) \end{align*} for $a\ge0$. So, by (2), \begin{align*} EX_1\,1_{A\cap C_x}&\le\Big(\frac{e^{b^2-2}}{(2+o(1))b}-\int_0^u \tfrac tu\,b \exp\{(\tfrac tu\,b)^2\}\,dt\Big)\,P(C_x)P(A) \\ &=\Big(\frac{e^{b^2-2}}{(2+o(1))b}-\frac u{2b} \, (e^{b^2}-1)\Big)\,P(C_x)P(A) \\ &=-e^{b^2(1+o(1))}\,P(C_x)P(A). \end{align*}

On the other hand, letting now $x\to\infty$, we have $P(C_x)\to1$ and, still with $b\to\infty$, \begin{align*} P(A)&=P(B_u\in[b,b+1/b])P(B_1-B_u<-2/b,M_u>-b) \\ &\ge P(B_u\in[b,b+1/b])[P(B_1-B_u<-2/b)-P(M_u\le-b)] \\ &=e^{-b^2/(2+o(1))}[P(B_1<-2)-o(1)]=e^{-b^2/(2+o(1))}. \end{align*} Thus, \begin{align*} EX_1\,1_{A\cap C_x}&\le-e^{b^2(1+o(1))}\,(1-o(1))\,e^{-b^2/(2+o(1))}=-e^{b^2/(2+o(1))}\to-\infty, \end{align*} which shows that indeed $EX_1$ does not exist.