Drunk man with a set of keys.

The key thing here is this: let $T$ be the number of tries it takes him to open the door. Let $D$ be the event that the man is drunk. Then $$ P(D\mid T=3)=\frac{P(T=3, D)}{P(T=3)}. $$ Now, the event that it takes three tries to open the door can be decomposed as $$ P(T=3)=P(T=3\mid D)\cdot P(D)+P(T=3\mid \neg D)\cdot P(\neg D). $$ By assumption, $P(D)=P(\neg D)=\frac{1}{2}$. So, we just need to compute the probability of requiring three attempts when drunk and when sober.

When he's sober, it takes three tries precisely when he chooses a wrong key, followed by a different wrong key, followed by the right key; the probability of doing this is $$ P(T=3\mid \neg D)=\frac{4}{5}\cdot\frac{3}{4}\cdot\frac{1}{3}=\frac{1}{5}. $$

When he's drunk, it is $$ P(T=3\mid D)=\frac{4}{5}\cdot\frac{4}{5}\cdot\frac{1}{5}=\frac{16}{125}. $$

So, all told, $$ P(T=3)=\frac{16}{125}\cdot\frac{1}{2}+\frac{1}{5}\cdot\frac{1}{2}=\frac{41}{250}. $$ Finally, $$ P(T=3, D)=P(T=3\mid D)\cdot P(D)=\frac{16}{125}\cdot\frac{1}{2}=\frac{16}{250} $$ (intentionally left unsimplified). So, we get $$ P(D\mid T=3)=\frac{\frac{16}{250}}{\frac{41}{250}}=\frac{16}{41}. $$

Let's first compute the probability that he wins on the third try in each of the two cases:

Sober: The key has to be one of the (ordered) five, with equal probability for each, so $p_{sober}=p_s=\frac 15$.

Drunk: Success on any trial has probability $\frac 15$. To win on the third means he fails twice then succeeds, so $p_{drunk}=p_d=\frac 45\times \frac 45 \times \frac 15 = \frac {16}{125}$

Since our prior was $\frac 12$ the new estimate for the probability is $$\frac {.5\times p_d}{.5p_d+.5p_s}=\frac {16}{41}=.\overline {39024}$$

I tried focusing instead on the number of times he tries a key and fails. So if he gets it on the 3rd try, he misses $2x$. The probability of doing this, given that he's drunk, is $(4/5) * (4/5) = 16/25$. On the other hand, the probability of him missing twice in a row given that he's sober is $(4/5) * (3/4) = 3/5$. Applying Baye's rule, I get

$$Pr(\text{drunk}\mid \text{missed twice}) = (16/25)/((16/25 + (3/4)(4/5)) = 0.51$$

Given that he misses $3x$, I get

$$Pr(\text{drunk}\mid \text{missed }3x) = ((4/5)^3)/((4/5)^3 + (2/3)(3/4)(4/5)) = 0.62$$

$$Pr(\text{drunk}\mid \text{missed }4x) = ((4/5)^4)/((4/5)^4 + (1/2)(2/3)(3/4)(4/5)) = 67.2$$

$$Pr(\text{drunk}\mid \text{missed } 5x) = ((4/5)^5)/((4/5)^5 + 0) = 1$$

The result has the desirable property that the probability starts at $0.5$ and gets higher the more we observe he starts missing the lock. I'm thinking the success on the $x$ attempt should not enter the calculation. I justify this because, we're given the observation that he finally opens the door, so that's not part of our probability calculation. What's really uncertain is the number of times he has to try before he opens it.