What's the difference between isomorphism and homeomorphism?

Homomorphism - an algebraical term for a function preserving some algebraic operations. For a group homomorphism $\phi$ we have $\phi(ab)=\phi(a)\phi(b)$ and $\phi(1)=1$, for a ring homomorphism we have additionally $\phi(a+b) = \phi(a) + \phi(b)$ and for a vector-space homomorphism also $\phi(r\cdot a)=r\cdot\phi(a)$, where $r$ is a scalar and $a$ is a vector.

Isomorphism (in a narrow/algebraic sense) - a homomorphism which is 1-1 and onto. In other words: a homomorphism which has an inverse.

However, homEomorphism is a topological term - it is a continuous function, having a continuous inverse.

In the category theory one defines a notion of a morphism (specific for each category) and then an isomorphism is defined as a morphism having an inverse, which is also a morphism.

With such an approach, morphisms in the category of groups are group homomorphisms and isomorphisms in this category are just group isomorphisms. Similarly for rings, vector spaces etc.

In the category of topological spaces, morphisms are continuous functions, and isomorphisms are homeomorphisms.

Extra remark: A fundamental difference between algebra and topology is that in algebra any morphism (homomorphism) which is 1-1 and onto is an isomorphism - i.e., its inverse is a morphism. In topology it is not true: there are continuous and bijective functions whose inverses are not continuous. That's (one of the reasons) why we like compact Hausdorff topological spaces: for them inverses are always continuous, just like in algebra inverses of homomorphisms are homomorphisms.

Isomorphisms (in general) are "structure-preserving" bijections, whether they be preserving group, ring, field, topological, (partial) order, or some other sort of structure. Homeomorphisms, specifically, are topology-preserving isomorphisms.