Definition of continuity at a point
The real question is, why is the $0<d_X(x,p)$ condition in the definition of the limit?
Basically, it doesn't hurt to allow $x=p$ in the continuity example, because (1) we know that $p\in E$, and (2), when $x=p$, $d_Y(f(x),f(p))=0<\epsilon$, so there is no reason to leave it out.
Essentially, in the continuity case, the case $x=p$ is trivially true.
On the other hand, when $p\in E$ in the limit definition, we don't want the limit to depend on $f(p)$, but only on the values $F(x)$ when $x\neq p$.
For example, when $f(x)=0$ for $x\neq p$ and $f(p)=1$, we want $\lim_{x\to p} f(x) = 0$, but that would not be true if we didn't have the condition $0<d_X(x,p)$ - without that condition, the limit is undefined.
Isolated points:
We consider isolated points to be points of continuity because we want in general that if:
$f:E\to Y$ is continuous at $p$ and $p\in E'\subset E$ then $f_{|E'}:E'\to Y$ to also be continuous at $p$.
However, note that the continuity definition could have said $0<D_X(x,p)$. That doesn't affect the continuity at $p$ one bit, so if you wanted to define all isolated points as points of discontinuity, you'd want some other definition completely.
It's fine to let $x=p$ in the definition of continuity, since then $f(x)=f(p)$, and so $$d_Y(f(x),f(p))=0<\epsilon.$$ It just isn't very interesting. When simply discussing limits of functions, though, we may want to examine behavior near but not at a point (such as when taking derivatives), so we require $x\neq p$ in such cases.
Note: It's possible that $p$ is isolated in $X$, so that we can't talk about the limit as $x$ approaches $p$ at all. A function $f:X\to Y$ is continuous at $p\in X$ if and only if either (i) $p$ is isolated in $X$ or (ii) $p$ is a limit point of $X$ and $f(p)=\lim\limits_{x\to p}f(x)$.
Edit: Even with your proposed alteration to the definition, a function is necessarily continuous at isolated points of its domain. Suppose that $f:E\to Y$ with $E\subseteq X$ and $p$ an isolated point of $E$. In other words, there is some $\delta>0$ such that for any $x\in E$, we have $d_X(x,p)<\delta$ if and only if $x=p.$ In particular, there is no $x\in E$ such that $0<d_X(x,p)<\delta$.
Now, take any $\epsilon>0$. Then for all points $x$ in the (empty) set $$\{x\in E: 0<d_X(x,p)<\delta\},$$ we have that $$d_Y\bigl(f(x),f(p)\bigr)<\epsilon,$$ vacuously. Since $\epsilon>0$ was arbitrary, then $f$ is continuous at $p$, under your proposed alteration to the definition. In fact, your altered version is mathematically equivalent to Rudin's version.
This sort of vacuous truth is bothersome to many, and permitting $x=p$ as in Rudin's definition of continuity allows us to avoid such things. As I said, allowing $x=p$ isn't a problem, but it isn't very interesting, either. The primary reason that we have to have $0<...$ in the limit definition, but not necessarily in the continuity definition, is so that we can examine limits of functions with (for example) removable discontinuities at a point, or limits of functions as we approach points at which they are undefined.