What PDF plugin does Magento use to generate invoices?

Magento does not use any plugin to generate the pdfs. It uses Zend_Pdf.
You can take a look at all the classes in here and see how it's done.
Unfortunately Zend_Pdf is not something magical. It requires a lot of work and some knowledge of geometry.
It is like drawing something on a canvas. You ca use methods as drawRectangle or drawText to actually place your elements on the page.
You can find a more detailed documentation about Zend_Pdf here

By default magento using ZEND_PDF to customization on pdf invoice,order and shipment.

You have to refer link for more customization of your task,

You can start from here,

Zend Pdf Customization Basics Link

Thanks.

By default magento uses Zend_Pdf here is sample fo creating your own pdf

simple zend pdf file not generate in magento

If you want to use third party library then i recommend use the tcpdf library it is easy to use.