Fundamental group as topological group
Andrew's answer is right, but I'll just throw in a few comments since "topological" homotopy invariants are of great interest to me. Here Paul Fabel has shown that $\pi_{1}^{top}$ on the Hawaiian earring is not a topological group. It turns out that multiplication can fail to be continuous even for some reasonably nice spaces (like locally simply connected planar continua). This and a nice connection with free topological groups appears in a manuscript I just posted on arXiv (perhaps shameless self-promotion...but I'll post the link when it becomes available).
This quotient map mistake has appeared many places in the literature, even in an appendix by Peter May from the 70's who described the topological fundamental groupoid (giving each hom-set the quotient topology of fixed endpoint Moore path spaces). The same false assertion that products of quotient maps are again quotient maps was also used to "show" the higher topological homotopy groups are topological groups.
With this quotient topology, the topological fundamental group(oid) is discrete on spaces that have universal covers (are path connected, locally path connected, and semi-locally simple connected). When it is non-discrete, this topology is often difficult to deal with. After all...there are going to be many homotopic loops that we identify in the quotient but which "look nothing alike" in the space.
While we don't always have a topological group, we're not completely out of luck. $\pi_{1}^{top}(X)$ is always a quasitopological group in the following sense:
definition: A quasitopological group is a group $G$ with topology such that inversion is continuous and multiplication $G\times G\rightarrow G$ is continuous in each variable.
A basic theory of these objects can be found in "Topological Groups and Related Structures" some of which you can get on google books.
Edit: The topological fundamental group and free topological groups is the new paper I mentioned. It is a bit denser than the one from Andrew's answer but if anyone is interested enough to read it, I'd greatly appreciate any comments, suggestions, or corrections.
Update: A bit of a digital paper chase led me, via David Robert's thesis (note that in the latest version, it is Chapter 5, section 2 that is most relevant), to this paper on the arXiv. The last sentence of the abstract is:
These hoop earring spaces provide a simple class of counterexamples to the claim that $\pi_{1}^{top}$ is a functor to the category of topological groups.
I recommend reading this article.
(Added later: In case it's not clear, the author of that paper is Jeremy Brazas who added an answer afterwards, so if you vote for my answer, you should definitely vote for his!)
Original Answer: These were my initial thoughts before I found the references above. These were what made me sufficiently intrigued to do the paper chase and find the above-mentioned thesis and article.
The proof given in the second paper (by Biss) that is mentioned in the question is short enough that I think it reasonable to copy it out here. I shan't copy out the obvious diagram so need to establish some notation first:
- $m \colon \pi_1^{Top}(X,x) \times \pi_1^{Top}(X,x) \to \pi_1^{Top}(X,x)$ is the multiplication map in question
- $p \colon \operatorname{Hom}((S^1,1),(X,x)) \to \pi_1^{Top}(X,x)$ is the quotient map
- $\overline{m} \colon \operatorname{Hom}((S^1,1),(X,x)) \times \operatorname{Hom}((S^1,1),(X,x)) \to \operatorname{Hom}((S^1,1),(X,x))$ is the "upstairs" multiplication map. (It's a tilde in the original, but that isn't displaying correctly for me so I daren't use it.)
The proof then proceeds:
To show that $m$ is continuous, it suffices to show that $\overline{m}$ is continuous, for then if $U \subset \pi_1^{Top}(X,x)$ is open, $(p \times p)^{-1} m^{-1}(U) = \overline{m}^{-1}p^{-1}(U)$ is open, but by the definition of a quotient map, $(p \times p)^{-1} m^{-1}(U)$ is open if and only if $m^{-1}(U)$ is.
There then follows a proof that $\overline{m}$ is continuous, a fact that I trust does not need proving.
Comments on your comments:
We don't need the quotient map to be open since we are only ever dealing with preimage sets. It is certainly not always true that if $q \colon X \to Y$ is a quotient that $q(U)$ is open in $Y$ for every open $U$ in $X$. But it is true by definition that $q^{-1}(U)$ is open in $X$ if and only if $U$ is open in $Y$. This is because the topology on $Y$ is precisely that to make this true. So since we are only dealing with sets of the form $(p \times p)^{-1}(A)$ then the assertion is valid assuming that $p \times p$ is a quotient map.
Here, I find myself worried. A quick back-of-envelope check seems to show that one can't simply assume that the product of quotients is again a quotient in Top (a counterexample eludes me as I don't have Counterexamples in Topology to hand and I'm too used to dealing with "nice" spaces). It may be the case that for Hom-spaces then there's some magic that can be done (though such is not mentioned in the paper); but again the best that I can do on the back of an envelope is observe that (modulo some basepoint mess) by construction $\operatorname{Hom}((S^1,1),(X,x)) \times \operatorname{Hom}((S^1,1),(X,x))$ quotients to $\pi_1^{Top}((X,x) \times (X,x))$. But to proceed, one would need to know that $\pi_1^{Top}$ was a product-preserving functor. This is morally the same as saying that it is representable - which looks good since we have an obvious representing object $S^1$! However, this can't be made into a proper argument since although we have a representing object, we don't have an enriched Hom-functor $hTop \times hTop \to Top$ which to evaluate at $S^1$.
So I would look for a counterexample to the product of quotients being a quotient, and see where that leads you. Either you'll find a proper counterexample to the proposition in question, or you'll see why in this special case, such a counterexample could not occur.
(Of course, I may well be missing something obvious!)
Let $X$ be a topological space and let $a$ be a point of $X$. There are three interesting topologies on $\pi_1(X,a)$: the quotient topology of the topology of compact convergence, and two other one which you'll let me call admissible and adequate.
In all three topologies, the classes of open normal subgroups coincide; moreover they are simultaneously Hausdorff (or not).
1) As observed, the topology of compact convergence is not a group topology, although the multiplication is separately continuous.
2) Say a subgroup $H$ of $\pi_1(X,a)$ is admissible if, any $b\in X$ has a neighborhood $V$ such that forany path class $\gamma\in\pi(X,a,b)$ (fundamental groupoid) and any loop $c$ in $V$ based at $b$, $\gamma c \gamma^{-1}$ belongs to $H$.
There is a unique group topology on $\pi_1(X,a)$ such that the normal admissible subgroups form a basis of neighborhoods of the identity. For this topology, the open subgroups are the admissible subgroups.
If $X$ is locally arcwise connected, then a subgroup is admissible if and only if it is the stabilizer of a point above $a$ in a covering of $X$.
The admissible topology is coarser than the topology of compact convergence, and may be strictly coarser.
3) Let us say that a subgroup $H$ of $\pi_1(X,a)$ is adequate if, for any $b\in X$ and any path class $\gamma\in\pi(X,a,b)$, there exists a neighborhood $V$ of $b$ such that for any loop $c$ in $V$ based at $b$ $\gamma c \gamma^{-1}$ belongs to $H$. (Note the switch in the order of quantifiers.)
There exists a unique group topology on $\pi_1(X,a)$ for which adequate subgroups form a basis of open subgroups.
This topology is finer (and possibly strictly finer) than the topology of compact convergence. However if $X$ is locally arwise connected, both topologies have the same open subgroups.