Mix two non-opaque colors with "hue" blend mode

The Hue alpha you have at your second image does not represent the alpha color composition formula, but it rather reflects the Porter Duff alpha composition Source Over as defined here 9.1.4. Source Over and it uses the following formula:

If you want to achieve that kind of blending, which is not proper Hue blending, you can use the following formula in javascript:

PDso = { // Ported Duff Source Over
    r: ((S.r * S.a) + (B.r * B.a) * (1 - S.a)) / aR,
    g: ((S.g * S.a) + (B.g * B.a) * (1 - S.a)) / aR,
    b: ((S.b * S.a) + (B.b * B.a) * (1 - S.a)) / aR,
};

// where
// S : the source rgba
// B : the backdrop rgba
// aR : the union alpha (as + ab * (1 - as))

Hue Blending Mode with Alpha Channel

Here is a screenshot of the exact hue blend source over backdrop using the alpha color composition formula that I have created in Photoshop:

The middle square with the green highlighted letters is the correct blend representation. Here is the CSS Hue mix blend with the source color inside the backdrop color, using the new CSS mix-blend-mode (run the code snippet):

.blends div {
    width:140px;
    height:140px;
}

.source {
    mix-blend-mode: hue;
}

.backdrop.alpha {
    background-color: rgba(141, 214, 214, .6);
    isolation: isolate;
}

.source.alpha {
    background-color: rgba(255, 213, 0, .6);
}

<div id="main">
 
 <div class="blends alpha">
  <div class="backdrop alpha">
   <div class="source alpha"></div>
  </div>
 </div>

</div>

If you use a color picker, you'll get almost the same values (211, 214, 140 <> 210, 214, 140). That can be due to slightly different algorithms, or different rounding methods, but it doesn't really matter. The fact is, that this is the correct result when blending alpha colors with hue blend mode.

So, now we need the formula to have the proper color values for the alpha color composition applied to our hue blend mode. I have searched a little bit and I found everything inside the Adobe Document management - Portable document format - Part 1: PDF 1.7. We can find the color composition formula at the page 328 after the Blend Modes:

11.3.6 Interpretation of Alpha

The colour compositing formula

This is the formula I managed to get the proper and closer to Photoshop match for the Hue Blending Mode with alpha channel. I wrote it like this in javascript:

function Union(ab, as) {
    return as + ab * (1 - as);
}

function colourCompositingFormula(as, ab, ar, Cs, Cb, Bbs) {
    return (1 - (as / ar)) * Cb + (as / ar) * Math.floor((1 - ab) * Cs + ab * Bbs);
}

var aR = Union(B.a, S.a); // αr = Union(αb, αs) // Adobe PDF Format Part 1 - page 331

var Ca = {
    // Adobe PDF Format Part 1 - page 328
    r: colourCompositingFormula(S.a, B.a, aR, S.r, B.r, C.r),
    g: colourCompositingFormula(S.a, B.a, aR, S.g, B.g, C.g),
    b: colourCompositingFormula(S.a, B.a, aR, S.b, B.b, C.b)
};

// where
// C : the hue blend mode result rgb
// S : the source rgba
// B : the backdrop rgba
// aR : the union alpha (as + ab * (1 - as))
// Ca : the final result

body {
  padding:0;
  margin:0;
}

iframe {
  width: 100%;
  height: 200px;
  border:0;
  padding:0;
  margin:0;
}

<iframe src="https://zikro.gr/dbg/html/blending-modes/"></iframe>

My test example can be found here. At the 2.5 With Alpha (Hue Blending Algorithm Computed), you cay see the final hue blend mode result with alpha. It has slightly different values than Photoshop result but I got the exact same result (202, 205, 118) in Fireworks, hue blending the source and backdrop colors:

All applications have their own slightly different algorithms and maybe the formula I have used is rather old and maybe there is a newest version.

Starting from here

Hue blending creates a color with the hue of the source color and the saturation and luminosity of the backdrop color.

I can come up with some formulas, but they might be rubbish, although they completely reproduce the original numbers posted:

h: hSource + deltaH * (1 - aSrouce) * aBackdrop * 0.41666666 = 50; 63

s: sBackdrop * 0.9 + deltaS * (1 - aBackdrop) * aSource * 0.20833333 = 45; 47.5

l: lBackdrop * 0.957142857 + deltaL * (1 - aBackdrop) * aSource * 0.77 = 67; 63.3

a: 1 - (1 - aSource)^2 matches always