Are there some original papers or books related to applications of algebraic topology and algebraic geometry in complex dynamic systems

The first application of algebraic geometry to dynamical systems that comes to my mind is the following preprint of Gromov -- very old one : ON THE ENTROPY OF HOLOMORPHIC MAPS

https://www.ihes.fr/~gromov/wp-content/uploads/2018/08/1024.pdf

A recent, excellent survey of dynamics on algebraic surfaces is :

http://perso.univ-rennes1.fr/serge.cantat/Articles/dyn-aut.pdf

The emerging field of arithmetic dynamics has seen a lot of attention lately, and it is very intimately connected to algebraic geometry (and, to a lesser degree, algebraic topology as well). I think you would enjoy reading Silverman's ``The Arithmetic of Dynamical Systems''.

Another good place to start is to track the output of Marion Mrozek and Konstantin Mischaikow, and their various co-authors. There is a whole group at the University in Krakow centered on Mrozek doing algebraic topology applications to dynamical systems.