Need help finding flaw in the voltage regulator circuit

Your output transistor is a high-voltage part and has exceptionally low hFE, as low as 5 at 5A.

If the hFE is actually, say, 10, at 5A out, the poor BC547 is being called upon to deliver 0.5A with around 13V across it. That's 7.5W in a TO-92.

Reminds me of a problem I had recently (actually it was metalworking rather than electronics, but the principle is the same) that caused a couple of my devices to stop unlocking from the fingerprint sensor for a month or so.

Overview

Re-drawn in more readable format:

^{simulate this circuit – Schematic created using CircuitLab}

The whole thing now basically makes sense to me. The way you drew it is more like what I see in a magazine, somewhere.

Also, I notice that you are using an MJE13007. Did you source that power BJT from a failed compact CFL? It's the kind of BJT that may be needed in such devices (and other mains-powered switcher-derived light bulbs.)

I am preferring the D44H11, these days, in TO-220. They have relatively much higher guaranteed \$\beta\$. In your application, I don't think there's any question which of the two is better:

Comparison:

Note that the D44H11 doesn't handle the same high voltages that the MJE13007 can support. But your application doesn't have to handle mains voltages. So the MJE13007 HV capability is wasted on your application. Note also that the D44H11 has a minimum guarantee for \$\beta\$ that is an order of magnitude larger. (It's Early Effect is terrible, but your application doesn't care about that fact.) Also, the D44H11 is at least as capable, if not more so, in handling high collector currents. And in fact, is rated for \$10\:\text{A}\$, continuous, which the MJE13007 is not. (Not that you should be normally running either of these devices right at, or exceeding, their Maximum Ratings.)

Analysis

Before we go anywhere, let's work out a rough estimate for the output voltage.

Assuming about \$V_{D_{_1}}=700\:\text{mV}\$ and another \$V_{\text{BE}_{_3}}=700\:\text{mV}\$, we can find that the output voltage should be around \$\left(V_{D_{_1}}+V_{\text{BE}_{_3}}\right)\cdot\left(1+\frac{R_4}{R_5}\right)\approx 4.5\:\text{V}\$. This suggests about \$\frac{15\:\text{V}\cdot\sqrt{2}-4.5\:\text{V}-1.4\:\text{V}}{1\:\text{k}\Omega+1\:\text{k}\Omega}\approx 7\:\text{mA}\$ for the current in \$R_3\$.

(Also, given the worst case \$\beta=200\$ for the BC547B, this might require as much as \$8.5\:\text{mA}\$ of base current for the BC547B. And we've already calculated that we don't have that much. So yet another possible problem.)

Next, let's look at the following figure from the MJE13007 datasheet:

Here, we can easily see that we need about \$1.3\:\text{A}\$ of base current for a collector current of \$8\:\text{A}\$. Let's assume that works all the way up to a collector current of \$10\:\text{A}\$ (beyond the maximum specifications.) This would imply that we need support for a base current that could be as high as \$1.7\:\text{A}\$.

But the BC547B is specified as having an absolute maximum collector current of \$500\:\text{mA}\$. So we are already well beyond the BC547B maximum capability. So this may already be a problem here.

This is why I'm suggesting that you consider the D44H11, instead. In this case, the BC547B would at best only need to support about \$170\:\text{mA}\$ of base current. And that is well within the specifications for the BC547B.

---

I see that an answer has been selected already, so at this point I'll hold short of the analysis. It was touched on by Spehro and there's no need to flog this horse, further.

The first capacitor --- 10,000 uF --- is too small.

That only supports 1 amp drain at 1 volt ripple at 120 Hertz.