How can I defeat the Caretaker?

No panic, this monster is beatable as all the others. Keep in mind these things:

  1. Use Quen all the time, when using it Caretaker will not suck your life and heal himself.
  2. Always move and dodge from his attacks, Caretaker has a pattern so just learn that, when he charges to hit the ground, dash from him, then to him and start attacking because he will be stuck, but not too long, when you see blue flash, get back from him.
  3. In the middle of the fight he summons spirits. Kill them as fast as you can because they will heal him.
  4. use Igni and Grapeshot bombs just in between dodging.
  5. If you have oil - he is a Relic so use Relic oil on him
  6. He is susceptible to Northern Wind (the bomb that freezes opponents). Use this especially when he has low health to prevent him from healing.

Make sure you have applied Relict Oil on your silver-sword. Also, equipping Ekimmara (Damage dealt to foes regenerates Vitality) or Ekhidna Decoctions (Performing actions that consume Stamina regenerates Vitality) can tremendously help you in this endeavor.
Next, use the Quen sign on Geralt as frequently as possible. This debilitates the Caretaker's life-steals when he hits you. If possible, learn the Fast Attack, 'Whirl' to deal more damage to the Caretaker in his moment of weakness.
Wait, weakness? Yes. This boss has one weakness, or opening so to speak, to deal extra damage.
When the Caretaker begins charging his petrification attack, (as seen when he charges his shovel into the ground and a circle of blue light emits around him) use your crossbow to stop him midway, thus causing him to momentarily be stunned. Use this opportunity to attack him with 'Whirl'. Normal attacks are fine too.
When he starts to summon his spirits, don't fret. Focus your offence on the spirits themselves. Attacking the Caretaker would only lead you being flung further away and gives him ample time to regenerate. Precise, the spirits disappears after one or two blows. One last tip of advice is to practice patience.
The more you rush, the more mistakes you would likely make. Go in steady. Keep your wits about you. Don't rush when attacking and don't get greedy too. Hope this helps.
P.S: It took me half an hour to learn and counter his techniques. If I can do it, so can you.
Godspeed.