My revised paper was accepted surprisingly quickly; should I worry?

Unless you have some reason to believe your revised paper has some serious additional problems, I don't think you should worry. It may not be best practices for referees and editors to operate this way (just looking to see whether the changes requested in the first report have been adequately completed). Ideally, the referee should check over carefully that none of the changes have introduced new problems, but that isn't always done.

The referee and editor may have felt that the revisions were not really that big a deal. You may have been overthinking things when you made the changes; they might have intended some considerably less sweeping editing. Or the referee may have seriously underestimated how much additional work and explication would be required to fix the problems the referee pointed out. The referee might even have told the editors that another round of refereeing would not actually be necessary; if you made the changes, then the paper would be ready for publication. (For several of the journals I have refereed for, this is one of the options in the recommendation menu of the reviewing software; however, I also know, as an author, that this part of the review is not shared with the authors.)

It's perfectly possible to review a revised paper in a short time (assuming it even makes it back to the reviewer). I've seen more than one reviewer turn around a short (few pages) paper with minor revisions in a couple of hours; I've done same-day myself. If the reviewer has a copy marked up with the changes, then the changes themselves need a proper review against the initial report, but the rest of the paper only needs eneough attention to check that the whole thing makes sense taking into account the revisions.

If you turned the revisions round fast, the paper could still be fresh in the reviewer's mind, or they may have taken good notes or have a good memory. If the paper is noteworthy to the reviewer (first they've reviewed or first in a long time; particularly interesting) they're likely to remember it quite well.

This probably means that the editor checked that you had addressed the comments, and did not feel the need to send it back to reviewers (which is often the time-consuming part). It's still a fast turnaround, so think yourself lucky!