Definitions of a proper embedding

I'd be interested to know where you've seen the second definition. As Thomas said, the two definitions are definitely not equivalent. However, there is a certain close relationship between them, which might explain what somebody had in mind when introducing the second definition.

Lemma. Suppose $M$ and $N$ are topological manifolds with boundary, and $f\colon M\to N$ is a continuous map. If $f$ restricts to a proper map from $\operatorname{Int} M$ to $\operatorname{Int} N$ (in the sense that preimages of compact sets are compact), then $f^{-1}(\partial N)=\partial M$.

Proof: Assume $f$ restricts to a proper map from $\operatorname{Int} M$ to $\operatorname{Int} N$, and let $g\colon \operatorname{Int} M\to \operatorname{Int} N$ denote the restricted map. To show that $f^{-1}(\partial N)=\partial M$, suppose first that $x\in f^{-1}(\partial N)$, meaning that $f(x)\in \partial N$. If $x$ were in $\operatorname{Int} M$, then our hypothesis would imply $f(x)\in \operatorname{Int} N$, which is disjoint from $\partial N$; so we must have $x\in \partial M$.

Conversely, suppose $x\in \partial M$, and assume for contradiction that $x\notin f^{-1}(\partial N)$. This means $f(x)\in \operatorname{Int} M$. Let $B$ be a precompact coordinate ball centered at $f(x)$ and completely contained in $\operatorname{Int} N$. There is a sequence of points $x_i \in \operatorname{Int} M$ such that $x_i \to x$ in $M$. Since $f$ is continuous, $f(x_i)\to f(x)$, and therefore by discarding finitely many terms in the sequence we may assume that $f(x_i)\in B\subseteq \overline B$ for all $i$. This means that all of the $x_i$'s lie in the set $f^{-1}(\overline B)\cap \operatorname{Int} M = g^{-1}(\overline B) $, which our hypothesis guarantees is compact. Therefore the point $x = \lim_{i\to\infty} x_i$ must also lie in this compact set, which is a contradiction because $x\notin\operatorname{Int} M$. $\square$

Note however, that the converse is not true: If we take $M=(0,1]$, $N=[-1,1]$, and $f\colon M\to N$ to be the inclusion map, then $f^{-1}(\partial N) = \partial M$ but the restriction of $f$ to $(0,1)$ is not a proper map from $(0,1)$ to $(-1,1)$.

Edit: Mike Miller pointed out that a typical context in which the second definition is used is when everything in sight is compact. In that case, definition (2) is equivalent to the restriction of $f$ to the interior being proper. In fact, only compactness of $M$ matters.

Lemma. Suppose $M$ and $N$ topological manifolds with boundary, and $f\colon M\to N$ is a continuous map. If $M$ is compact and $f^{-1}(\partial N) = \partial M$, then $f$ restricts to a proper map from $\operatorname{Int} M$ to $\operatorname{Int} N$.

Proof: Suppose $M$ is compact and $f^{-1}(\partial N) = \partial M$. First we need to show that $f(\operatorname{Int} M)\subseteq \operatorname{Int} N$. This follows from \begin{align*} x\in \operatorname{Int} M & \implies x\notin \partial M\\ & \implies x\notin f^{-1}(\partial N)\\ & \implies f(x)\notin \partial N\\ & \implies f(x)\in \operatorname{Int} N. \end{align*}

Let $g$ denote the restricted map from $\operatorname{Int} M$ to $\operatorname{Int} N$. Suppose $K\subseteq \operatorname{Int} N$ is compact. Since every continuous map from a compact space to a Hausdorff space is proper, $f^{-1}(K)$ is a compact subset of $M$. If we can show that $f^{-1}(K)\subseteq \operatorname{Int} M$, it then follows that $f^{-1}(K) = g^{-1}(K)$ and thus $g$ is proper. We have \begin{align*} x\in f^{-1}(K) & \implies x\in f^{-1}(\operatorname{Int} N)\\ & \implies x\notin f^{-1}(\partial N)\\ & \implies x\notin \partial M\\ & \implies x\in \operatorname{Int} M. \end{align*} This completes the proof. $\square$

I never came across the second definition, but it's quite obvious that the two definitions don't coincide (consider one of the obvious coverings of $\mathbb{R}^+\times \mathbb{R}\rightarrow \mathbb{R}^+\times S^1 $, e.g $(x, t) \mapsto (x, e^{it})$, where $ \mathbb{R}^+ := \{x\in\mathbb R| x \ge 0\}$ to see that the second does not imply the first)

It is not unusual in Mathematics that one term is used in two different meanings -- a typical example is the use of the term bounded map in metric spaces vs the term in the context of, say, linear maps between Banach spaces. This has then usually historic reasons.

As a supplement to Thomas's answer, to see that the first also doesn't imply the second, consider a constant map from $\overline{B}^2$ to $\overline{B}^2$ that doesn't map into the boundary. This is proper in the first sense, but not in the second.