Is the boundary of a manifold topologically unique?

I don't think this is true, by a classical `swindle' that I think is due to Stallings. You can find variations on this argument eg in Milnor's paper, Two complexes which are homeomorphic but combinatorially distinct (Annals 1961). He refers to some notes of Stallings.

Here is how I recall the argument, as applied to your question. I might be off by a sign, depending on the dimension.

Consider a manifold $M$ for which there is an h-cobordism $Y$ from $M$ to itself with non-zero Whitehead torsion; then your $X$ will be two copies of $M$, or (taking orientations into account) $X = -M \cup M$. Then obviously $\partial Y = X$. On the other hand, you could take $Z = M \times I$.

Claim: $Y^0 \cong Z^0 = M \times R$.

Proof: Consider an h-cobordism $Y'$ from $M$ to itself whose torsion is the negative of that of $(Y,M,M)$. Then $$Y \cup_M Y' \cong M \times I \cong Y' \cup_M Y.\quad (*)$$ Now consider $$ \cdots (Y \cup Y') \cup (Y \cup Y') \cup Y \cup (Y'\cup Y) \cup (Y'\cup Y) \cdots $$ By (*), this is $(-\infty,0] \times M \cup Y \cup [0,\infty)\times M = Y^0$.

On the other hand, you can put the $Y$ in the middle onto one side, and redistribute the parentheses to see that this is an infinite union of copies of $Y \cup Y' = M \times I$, and hence is $Z^0$.

Since you say in a comment that you might be satisfied with a homotopy equivalence, let me sketch a proof that the homotopy type of the boundary depends only on the interior.

Let $Y$ be the interior of $X$ and let $DY$ be the space of all proper maps $[0,1)\to Y$, suitably topologized. Then $DY$ must be homotopy equivalent to $\partial X$.

To see this, first observe that $DY$ is homotopy equivalent to the space $D'Y$ of germs of such maps, where two maps $[0,1)\to Y$ have the same germ if they agree on $[a,1)$ for some $0\le a<1$.

Then observe that $D'Y$ depends only a neighborhood of $\partial X$, and that by using a collar neighborhood $C\cong I\times\partial X$ we find $D'Y$ to be homeomorphic to the space of all germs of proper maps $[0,1)\to [0,1)\times\partial X $.

The latter space of germs is homotopy equivalent to the space of all proper maps $[0,1)\to [0,1)\times\partial X$.

Finally, this last space is the product of

(1) the space of all proper maps $[0,1)\to [0,1)$ and

(2) the space of all continuous maps $[0,1)\to \partial X$.

(1) is contractible. (2) is homotopy equivalent to $\partial X$.