Does the Baker-Campbell-Hausdorff formula hold for vector fields on a (compact) manifold?

To answer your first question, the composition of two time-1 flows won't necessarily be another time-1 flow.

One way to see this is to note that when a time-1 flow $\phi_X$ has a periodic point $P$ (period > 1), then $P$ can't be hyperbolic since it lies on a closed orbit of the flow for $X$. (The eigenvector of $D\phi_X$ tangent to this orbit has corresponding eigenvalue 1.)

Now, take a flow on $S^2$ whose time-1 map rotates the sphere, switching the north and south poles. Take a second flow for which both poles are hyperbolic attracting fixed points. Composing the two time-1 maps gives you a new diffeomorphism with hyperbolic points of period 2.

The answer is no in the symplectic case. I believe there are $X$ and $Y$ such that $\phi_X\circ\phi_Y$ is not generated by any time-independent Hamiltonian flow but I can't produce an example. However I can prove that $\phi_X\circ\phi_Y$ is not generated by a flow defined by a local formula.

Consider the cylinder $S^1\times [0,1]$ and close up its ends to make a sphere. The first field $Y$ rotates the thing along the $S^1$ factor, say, with angular velocity $\sqrt 2$. The second field $X$ is supported is a small round disc $D$ inside the cylinder, and it rotates this disc around its center (with angular velocity decaying to 0 at the boundary).

Let $f=\phi_X\circ\phi_Y$ and suppose it is a time 1 map of a time-independent Hamiltonian flow. Since the sphere is simply connected, there is a globally defined Hamiltonian $H$, and the orbits of the flow are its level curves. Suppose also that the flow field is given by a local formula in terms of $X$ and $Y$. Outside $D$, we have $X=0$, so the formula yields $Y$. This implies that the orbits contain large pieces of the original circles (namely part of the circles outside $D$). Consider such an interval $s$ of the circle through the center of $D$. The image $f(s)$ must be within the same orbit. But $f(s)$ intersects $s$ transversally at some point outside $U$. Moving along $Y$ shows that the orbit contains an open region, a contradiction.

The BCH formula holds in a finite-dimensional Lie algebra for $X,Y$ in some open neighborhood of $0$. Therefore, the coefficients of BCH are bounded by $a^n$ for some $0<a<\infty$. So suppose that $X,Y$ are vector fields for which degree-$n$ bracket monomials (things like $[X,[X,Y]]$, with $n$ many $X$s and $Y$s total) don't grow too fast in your topology — not faster than $b^n$ for some $0 < b < \infty$. Then for $s < b/a$, ${\rm BCH}(sX,sY)$ converges, in your topology, if your topology is complete.

Now, I'm not much of one for infinite-dimensional topological vector spaces, so I can't say which topologies are great to use.