Can I use a PS4 controller on a PS3?

Yes, it is possible to use a DualShock 4 on a PS3. You can plug it in directly to the USB port by default, but if your PS3 has the correct system firmware, it can be used wirelessly as well.

There are a few features and games that aren't 100% compatible, the instructions for connecting the controller to the console and the compatibiliy list is in this post:

1. First, plug the micro usb cable into your PS3.
2. Next, plug your Dualshock 4 into the micro usb cable.
3. Turn on your PS3, and then hit the button on your Dualshock 4. Note: make sure your Dualshock 4 is assigned as Controller 1
4. You can now navigate the XMB, load games, and use it as a controller for games listed below!
5. (additional step if a PS3 controller is required) Lastly, turn on your PS3 controller, and make sure it's assigned as Controller 2. In certain games' menus, you will need to use on this controller before you can use your Dualshock 4.

For wireless:

6. Go to Accessory Settings, then Manage Bluetooth Devices
7. Start scanning for device then unplug your DS4
8. While the PS3 is scanning, hold down the PS and Share buttons on the DS4 simultaneously until the lightbar on the front of the DS4 is flashing rapidly, indicating it has synced
9. Plug your DS4 back in, and select Wireless Controller
10. Once your PS3 is done registering the device, you may unplug your DS4 and use it wirelessly!