What's so special about a prime ideal?

The easiest way to answer this, I think, is with an example. Let $R=\mathbb{Z}$, and let's consider the ideal $I=(6)$. This is the set of all integers that are multiples of $6$. You can see that it's an ideal because if you take any multiple of $6$ and multiply it by any other integer, the result is still a multiple of $6$. So $I$ is closed under multiplication by any element of the ring.

But $I=(6)$ is not a prime ideal. You can see this because $2 \notin (6)$ and $3 \notin (6)$, but $2 \cdot 3 \in (6)$.

On the other hand, the ideals $(3)$ is prime: If you multiply two numbers together and the result is a multiple of $3$, then at least one of the two numbers you began with must also be a multiple of $3$.

If you ponder this example, you will also understand why the word "prime" is used for this property.

If the question is "What's so special about it?", which I might take to mean "Why does the concept matter?", I might mention that the quotient ring of of a commutative ring with unit by a prime ideal is an integral domain.

However, later it looks as if maybe what you meant is that you're trying to understand what the definition says.

The set of all multiples of $10$ is an ideal in $\mathbb Z$. That means that if $a$ is a multiple of $10$ and $b\in\mathbb Z$, then $ab$ is a multiple of $10$.

But that ideal is not a prime ideal: $2$ and $5$ are not in that ideal but $2\times5$ is. The definition tells us that if it were a prime ideal, then if $2\times 5$ is in the ideal, then either $2$ or $5$ is in the ideal. That means that if $2\times5$ is a multiple of $10$ then either $2$ or $5$ is a multiple of $10$. But that is not true, so this ideal is not prime.

It seems to me you're confused about quantifiers.

$a,b$ are always elements of $R$, though the prime ideal definition doesn't specify that one has to be in $J$

The definition of "ideal" does not say anything to the effect that either $a$ or $b$ "has to be" in $J$. It says that if one of them is in $J$, then their product has to be in $J$. The definition of "prime ideal" also includes that, in that it says it's an ideal.

One statement says that if one of two things is in $J$, then so is their product.

The other says that if their product is in $J$, then so is one of them.

There is a big difference between "If P then Q" and "If Q then P".

Now consider the set of all multiples of $11$. That is also an ideal, since if just one of $a,b$ is a multiple of $11$, then so is $ab$. But this time you cannot find two numbers $a,b$ that are not multiples of $11$ but for which $ab$ is a multiple of $11$. With $10$ we were able to do that just by factoring $10$ as $2\times5$, and we could do that because $10$ is not prime. The set of all multiples of a prime number is a prime ideal in $\mathbb Z$; the set of all multiples of a composite number (like $10$) is not. It's easy to see why the latter kind is not a prime ideal, just as we did above. The other statement, that if a number is prime, then it divides a product $ab$ only if it divides either $a$ or $b$, is a bit more work to prove, and is called Euclid's lemma.