Is there a height limit on bookshelves enhancing an enchantment table?

Yes. The height limit is two.

Only bookshelves the same height or one block higher than the enchantment table are counted when setting the maximum level the table can enchant. Additionally, only bookshelves exactly two blocks away from the table are counted, not bookshelves directly adjacent, nor bookshelves further away.

There is a popular version which was necessary until the changes in version 1.3: A two-high 5x5 square of bookshelves with a single one block wide door way, separated from the table in the center by a 3x3 square path of absolutely nothing except air. It's important to note anything placed between a bookshelf and the enchanting table will prevent the bookshelf from contributing to the table's maximum level.

There is an upper limit on the enchantment table's level. Beyond 15 bookshelves there is no additional effect to adding more, so there is no need to build this entire structure since version 1.3. With just 15 bookshelves in any of the valid positions you can achieve the maximum enchanting level which is 30.

Yup

You can find the wiki article here but a quick quote from it:

In order to have an effect, a bookshelf must be placed exactly 2 blocks, laterally, of the enchantment table and be on the same level or one block height above the table, and the space between the bookshelf

They have a nice little set of diagrams on the wiki that help explain the layout a bit better but it pretty much means the bookshelves must rest at the same level or one block above where the enchantment block is placed.