What is torrent encryption and does it make my traffic anonymous?
Think of it like an underground fight club. Encrypting the traffic means nobody on the outside can see you enter or leave, but once you're inside, everybody there knows who you are and can monitor your participation.
This feature is really only useful if you have an ISP that blocks torrent traffic. Encrypting it means it doesn't appear to be torrent traffic, it's just an encrypted stream, but once you get past the ISP and connect to the swarm everybody else participating knows exactly who you are and what you're doing.
The "encryption" hides the content of the torrent data traffic from a casual observer and makes it harder to determine that the traffic is in fact torrent data traffic. It was designed to make it harder for ISPs to snoop on torrent traffic (and either block it, throttle it or send nastygrams). It can also be useful for evading the effects of buggy firmware in network devices*.
The cryptography used is relatively weak. The DH key exchange is only 768 bit with a fixed prime which is almost certainly crackable by a well-funded attacker. The actual encryption is rc4 which is known to have weaknesses though I don't know how relevant they are to this particular application. MITM attacks are possible if the attacker knows the "info hash" of the torrent in question.
Also it only protects data connections. It DOES NOT hide the fact you are present in the swarm from someone scanning the tracker or dht. It DOES NOT stop the copyright enforcers from connecting to your client and downloading a copy of the file from you to demonstrate that you are illegally offering it for distribution.
* I have encountered cases where the same peice of a file was repeatly failling hash checks, turning on encryption fixed it. I suspect a buggy and/or overzealous NAT implementation was responsible.
what does this feature encrypt?
It encrypts the entire communication stream with other BT peers.
Name of the file, it's contents, size?
All of the above. Note that it is completely and utterly useless to encrypt these things as they are already PUBLIC.
Could it prevent DMCA notices?
It can prevent a residential ISP from casually snooping the traffic. In some countries this doesn't matter, as residential ISPs have no business snooping your traffic. In other countries, rightsholders have programs to cooperate with ISPs to detect piracy and serve out nastygrams.
If not, what does it actually do?
It is mainly security theatre. It will still be obvious to your ISP that you are using BT. It will still be easy for rightsholders to find out which IPs are sharing their content. So, the encryption doesn't solve any real problem. Someone added it to their BT client to draw dumb users who don't understand the security aspects at all. Then everyone else had to add it to their BT client too, even though it is mostly useless.