If you have two envelopes, and ...
The assumption that is unrealistic is that there is a $\frac12$ chance that the other envelope contains twice the money. Realistically, there is an underlying distribution of values and that distribution dictates the probability that a given amount is the smaller.
1. Analysis of the Two Envelope Paradox
Let the value of a pair of envelopes (POE) be the smaller of the values of the envelopes. Let the pdf of the value of the POE be $f(a)$. That is, the probability that the value of a POE is between $a$ and $a+\mathrm{d}a$ is $f(a)\,\mathrm{d}a$.
The expected value of a randomly chosen envelope is $$ \begin{align} E &=\frac12\int f(a)\,a\,\mathrm{d}a+\frac12\int f(a)\,2a\,\mathrm{d}a\\ &=\frac32\int f(a)\,a\,\mathrm{d}a\tag{1} \end{align} $$ We will assume that this exists
2. Conditional Probabilities
The probability that the value of the POE was between $\frac a2$ and $\frac a2+\frac{\mathrm{d}a}2$ and that we chose the larger is $\frac12f\left(\frac a2\right)\frac{\mathrm{d}a}2$. The probability that the value of the POE was between $a$ and $a+\mathrm{d}a$ and that we chose the smaller is $\frac12f(a)\,\mathrm{d}a$. Thus, the probability that the value of the envelope we chose is between $a$ and $a+\mathrm{d}a$ is $\frac14\left(f\left(\frac a2\right) + 2 f(a)\right)\mathrm{d}a$. Therefore, we define $$ P(a)=\frac14\left[f\left(\frac a2\right) + 2 f(a)\right]\tag{2} $$ Furthermore, given that the value of the envelope we chose was between $a$ and $a+\mathrm{d}a$, the probability that we chose the envelope with the larger value is $$ L(a)=\frac{f\left(\frac a2\right)}{f\left(\frac a2\right)+2f(a)}\tag{3} $$ and the probability that we chose the envelope with the smaller value is $$ S(a)=\frac{2f(a)}{f\left(\frac a2\right)+2f(a)}\tag{4} $$ This is where the unrealistic assumption falls apart. Without knowledge of $f$, we cannot know the conditional probabilities $L$ and $S$; they are usually not $\frac12$ and $\frac12$.
3. Strategies
Always Switch
Suppose we switch all the time. Then our expected value is
$$
\begin{align}
&\int\left[L(a)\frac a2+S(a)2a\right]P(a)\,\mathrm{d}a\\
&=\frac14\int\left[f\left(\frac a2\right)\frac a2+2f(a)\,2a\right]\,\mathrm{d}a\\
&=\frac32\int f(a)\,a\,\mathrm{d}a\\[8pt]
&=E\tag{5}
\end{align}
$$
Always Stay
Suppose we stay all the time. Then our expected value is
$$
\begin{align}
&\int\left[\vphantom{\int}L(a)\,a+S(a)\,a\right]P(a)\,\mathrm{d}a\\
&=\frac14\int\left[f\left(\frac a2\right)\,a+2f(a)\,a\right]\,\mathrm{d}a\\
&=\frac32\int f(a)\,a\,\mathrm{d}a\\[8pt]
&=E\tag{6}
\end{align}
$$
Therefore, the expected value is $E$ whether we switch all the time or stay. This is comforting since intuition says that switching should not help.
Better Strategy
However, there is a strategy that does give us a better expected value. Choose any function $k:[0,\infty)\to[0,1]$ such that $k(2a)\gt k(a)$; a monotonic increasing function for example. If an envelope has value $a$, keep it with probability $k(a)$ and switch otherwise. Then the expected value is
$$
\begin{align}
&\int L(a)\left[k(a)a+(1-k(a))\frac a2\right]P(a)\,\mathrm{d}a\\
&+\int S(a)\left[\vphantom{\int}k(a)a+(1-k(a))2a\right]P(a)\,\mathrm{d}a\\[3pt]
&=\frac14\int f\left(\frac a2\right)\left[k(a)a+(1-k(a))\frac a2\right]\,\mathrm{d}a\\
&+\frac14\int2f(a)\left[\vphantom{\int}k(a)a+(1-k(a))2a\right]\,\mathrm{d}a\\[3pt]
&=\frac32\int f(a)\,a\,\mathrm{d}a
+\frac12\int f(a)\left[\vphantom{\int}k(2a)-k(a)\right]a\,\mathrm{d}a\\[3pt]
&=E+\frac12\int f(a)\left[\vphantom{\int}k(2a)-k(a)\right]a\,\mathrm{d}a\tag{7}
\end{align}
$$
which, if $k(2a)\gt k(a)$, is better than $E$. If $k(a)$ is constant, as it is in the previous strategies, the expected value is $E$.
My Opinion: You aren't using the notion of random variables correctly. You can't have that $E[Y]=X$ if these are random variables. $E[Y]$ is a fixed number, not a random variable. So your reasoning is incorrect. And, I would think, it really does come down to how you define what $X$ and $Y$ are. You can't be vague about that and expect to apply mathematical techniques to an ill-defined quantity with meaningful results.