Can a different DNS server be used for subdomains?

The other answer made the assumption that:

You want to resolve *.dev.example.com ---> *.example.com

But that's not how your question seems to read. It sounds to me like you want to delegate control over the dev.example.com domain to your own machine. If that's the case, then the delegation is easy:

• On whatever machine is your nameserver for example.com, you'd add an NS record that says your dev machine is the nameserver for dev.example.com
• You'd also add a glue record, an A record giving the IP of your dev machine.

In your case, if this is Bind, you'd have something like this on the nameserver for example.com:

$ORIGIN dev.example.com.
@             IN      NS     dev.example.com.
dev           IN      A      1.2.3.4 

And then on your dev machine you'd set up Bind to be authoritative for dev.example.com (You're on your own for this part, but it's straightforward.)

Yes, without a problem, this is one of the basic functions of DNS.

On the DNS server for example.com, you define the IP for dev.example.com, and that the NS for dev.example.com is dev.example.com, and on dev.example.com you define all the sub-names of dev.example.com.