Why is a circle not a convex set?

They mean just the edge of the circle, without the interior. That is not a convex set.

The confusion arises from the notation. A circle, in the second part of the sentence, is defined as just the points on the boundary, excluding the interior. From your picture you can see that the line connecting A and B has A and B on the boundary, but the other points in the segment lie in the interior of the circle. Therefore, if we define a circle as the set of points on the boundary, it is not convex.

On the other hand, if a circle is defined as the set of points in the boundary plus those on the interior, it is a convex set.

The sentence in your book is misleading since it did not define what a circle is, and it seems to be using both interchangeably, adding more confusion.

The circle is given by the black curved line. Note that the white interior is not part of the circle.

Now consider the intersection point of the line AB and the line CD. This intersection point clearly lies on the line AB between the points A and B. The points A and B are on the circle, so if the circle were convex, then the intersection point would need to be on the circle as well. But it obviously is not; it in inside the circle. Therefore the circle is not convex.

On the other side, the inside of the circle is convex, as if you chose any two points inside the circle, then the full line segment connecting them is also inside the circle.