Algorithm with chapter number

Since you are using both the algorithm2e and algorithm packages, and both of them implement some same commands/environments, you had to use the algo2e option for algorithm2e to prevent a clash and this overrides the settings for the counter that you set as options for algorith2e, since the environment that will be used is the one from algorithm. You have three possible solutions:

1. If you don't need the algorithm package, load only algorithm2e and not algorithm and load the former without the algo2e option:

   \documentclass[11pt]{book}
   \usepackage[ruled,vlined,linesnumbered,resetcount,algochapter]{algorithm2e}
   
   \begin{document}
   
   \chapter{chapter1}
   \begin{algorithm}[H]
   \centering
   test
   \caption{algo}
   \label{algo1}
   \end{algorithm}
   
   \end{document}
   

   

2. If you really need both algorithm2e and algorithm in your document, and you want to use the algorithm environment (from the algorithm package) add the following lines to your preamble, after loading the packages:

   \makeatletter 
   \renewcommand\thealgorithm{\thechapter.\arabic{algorithm}} 
   \@addtoreset{algorithm}{chapter} 
   \makeatother
   

   A complete example:

   \documentclass[11pt]{book}
   \usepackage[ruled,vlined,linesnumbered,algo2e,resetcount,algochapter]{algorithm2e}
   \usepackage{algorithm}
   
   \makeatletter 
   \renewcommand\thealgorithm{\thechapter.\arabic{algorithm}} 
   \@addtoreset{algorithm}{chapter} 
   \makeatother
   
   \begin{document}
   
   \chapter{chapter1}
   \begin{algorithm}[H]
   \centering
   test
   \caption{algo}
   \label{algo1}
   \end{algorithm}
   
   \end{document}
   

   

3. If you really need both algorithm2e and algorithm in your document, and you want to use the algorithm environment (from the algorithm2e package), use the algorithm2e environment (the algo2e option renames algorithm to algorithm2e):

   \documentclass[11pt]{book}
   \usepackage[ruled,vlined,linesnumbered,algo2e,resetcount,algochapter]{algorithm2e}
   \usepackage{algorithm}
   
   \begin{document}
   
   \chapter{chapter1}
   \begin{algorithm2e}[H]
   \centering
   test
   \caption{algo}
   \label{algo1}
   \end{algorithm2e}
   
   \end{document}
   

By the way (not related to the question), you are loading packages more than once; this should be avoided.