Category with colimits but no limits
Sure, take for example the category $$A \rightarrow B \leftarrow C$$
Other than the identities, there is exactly one arrow from $A$ to $B$ and one arrow from $C$ to $B$.
It has all colimits (almost all, that is; there's no initial object) -- the colimit of every diagram that contains two different objects is $B$, and of a diagram that contains only one object it is that object itself.
But it doesn't have all limits; for example there is no product of $A$ and $C$.
Of course, the opposite category of a category with all colimits and not all limits will be a category with all limits and not all colimits ...
I object to the premise of your question, namely that the existence of a category with colimits but not limits has anything to do with the intuition you discuss. Note that in an arbitrary category, morphisms need not have any relation to functions. Moreover, given any category $C$, you can form the opposite category $C^{op}$ which reverses the directions of all the arrows. So, domains and codomains are really completely symmetric if you're just talking about arbitrary categories. For instance, if $C$ has all colimits but not all limits, then $C^{op}$ has all limits but not all colimits.
The intuition that Ravi Vakil is talking about is not applicable to arbitrary categories. Instead, it's applicable to the sort of "concrete" categories we tend to encounter in practice (but not their opposite categories), where morphisms are closely related to functions and limits and filtered colimits are constructed similarly to how they are in the category of sets.
As for the way you interpreted the intuition, I'm not entirely sure what you mean but it seems not entirely unreasonable. The way I might phrase it is that a map $X_n\to X$ gives you actual elements of $X$ for each element of $X_0$. So, if $X$ is the colimit of the sequence, it has specific elements which come from the elements of each $X_n$ (and it can be deduced from the universal property that every element of $X$ comes from some $X_n$, at least in the category of sets). On the other hand, a map $X\to X_n$ doesn't tell you any specific elements of $X$. So, you can't describe what an element of $X$ can look like using just a single $X_n$; you really need the whole sequence before you can even name a single element of $X$.