Is it widely followed that only some parts of academic books are taught to undergraduates?

Not covering the entirety of a textbook is true in absolutely ever course I've ever taken, and is true of every institution I'm familiar with. However, I would not say that it is true that any given subset of sections of any given book are the ones that every professor covers; in fact, at least in the US, each professor has a wide latitude in choosing what parts of the book they want to cover, and to what extent their lectures and/or tests cover the same material as the book at all. Some professors, in some classes, prefer only to cover whats more-or-less written in the book, and some intentionally lecture on topics that are not in the book and rely on reading the book to cover other topics. Some professors even have a book only as optional reading and you don't need to read any of it.

All of the above is true at the undergraduate level, and even more true - in my experience - at the graduate level, as professors deviate even further from any available textbook.

As to some books being designed for both undergraduate and graduate levels, this varies by book. Some books are almost never used at the graduate level, and some are almost never used at the undergraduate level, and some are used in both but to different extents.

But to answer your core question: yes, it is very common not to cover the whole book, and no you shouldn't worry about it. Its always good to lightly skim through the material that isn't required, so you get an idea of what you are skipping, but generally most professors make it a point to select out the material they believe is most important and relevant and skip what they don't deem necessary. Few professors ever follow the order of the book, either, and prefer to select their own ordering - and rarely do two professors agree on what that order would best be.

And don't worry - if you end up doing anything challenging, you will always feel tremendously under-prepared and under-trained, no matter how many textbooks you've read. That just comes with the territory :)

Yes, it is common to teach a course using only parts of an associated textbook.

In particular, read the "Preface" (or "Foreword", or "To the Instructor") part of your textbook(s); it usually specifies one or more suggested course sequences, which sections are optional, which sections more strongly depend on other sections, etc.

If you're really concerned about being under-prepared, then there's nothing stopping you from just reading the other sections on your own. This itself will be a skill broadly expected in graduate school, so it's not bad to practice itself at this time.

Most textbooks simply have far too much content for a one-semester undergraduate course. So I would say, yes, this is the norm.

In some cases, instructors might cover only a rather small section of a textbook. Introduction to Algorithms (CLRS) and Compilers: Principles, Techniques, and Tools ("Dragon book") are two good examples: while they can and are often used as undergraduate texts, the books themselves are more like handbooks and can be a valuable reference even for senior researchers. It is certainly impossible to cover much of the book in an undergraduate course in this case.