Formal power series coefficient multiplication

$Y$ is known as the Hadamard product of $A$ and $B$, and there is no simple way to find it from $A$ and $B$ in general. (If you don't believe me, set $A = B = e^x$.) The closest thing I know to a formula is (with all the analytic caveats that makes everything converge)

$$Y(r) = \frac{1}{2\pi } \int_0^{2\pi} A(e^{i \theta}) B(re^{-i \theta}) d \theta$$

which follows by Parseval's theorem. If $A$ and $B$ are sufficiently simple (for example if they are rational functions) then it is possible to evaluate this integral, but in general there's not much you can do; $Y$ can be much more complicated than $A$ or $B$. (Another idea is to write the above as a contour integral, and if the integrand ends up being meromorphic you can try to use the residue theorem.)

Computing Hadamard products is a special case of computing the diagonal of a two-variable generating function, a problem which I describe with a few examples here using the residue theorem.

(Of course for very special $A$ it is possible to say more, e.g. when $a_k$ is a polynomial in $k$.)

It is known as the Hadamard product $\rm\; f \star g\;$. There is no closed form for general hadamard products, but some classes of functions are known to be closed under such products, e.g. rational power series. However, algebraic series are not generally closed under Hadamard products, the standard example being $\rm\ f = g = (1-4x)^{-1/2} \;=\; \sum \binom{2n}{n}\ x^n.\ $ However, $\rm\: f\:$ rational, $\rm\: g\:$ algebraic $\rm\;\Rightarrow\; f\star g\;$ algebraic. Also D-finite power series are closed under Hadamard products, i.e. power series satisfying a linear differential equation with polynomial coefficients or, equivalently, series whose coefficients satisfy a linear recursive equation with polynomial coefficients.

Hadamard products can also be defined in terms of diagonals of multivariate series (and vice versa), e.g. see this paper, where you'll find some interesting connections with finite automata which, e.g. help one to easily observe that algebraic series over $\rm\mathbb{F}_p$ are closed under Hadamard product (which fails in characteristic 0)

For more on Hadamard Products or Hadamard Multiplication Theorem look for papers by Louis R. Bragg in American Mathematical Monthly, Jan. 1999, pp 36-42 or SIAM J. Math. Anal., Vol. 17, 1986, pp 220-230 for starters.