Defining scalable "white curly brackets" {| and |} (⦃ and ⦄)

With scalerel, you can definitely take care of the \left \right problem. And I think I took care of the tine & scriptsize problem, too. What I did was to save a good copy of the normalsize bbraces in boxes, so that \lxbrace and \rxbrace would place the boxes containing good copies of the bbraces. I then used scalerel to scale these boxes to the desired sizes. So, essentially \lxbrace and \rxbrace replaced your \llbrace and \rrbrace.

The key here is that I am scaling the \normalsize version of the bbraces, whereas what I think happened to your original glyph is that tiny versions of a LaTeX brace are not merely scaled versions of the normalsize brace, but modified (by LaTeX) for legibility. But inyour case, you don't want modifications to your glyph's components as they are shrunk.

Lastly, scalerel is able to emulate the \left \right thingy to scale your brace up to the proper size, whatever that might be. I did use a width-limited scale, to avoid the scaled brace looking too clunky.

\documentclass{article}
\usepackage{amsmath}
\usepackage{scalerel}
\usepackage[only,llbracket,rrbracket,llparenthesis,rrparenthesis]{stmaryrd}
\parskip 1ex
\parindent 0ex
\usepackage{accsupp} % for ensuring the right Unicode codepoint upon pasting

\newcommand*{\llbrace}{%
  \BeginAccSupp{method=hex,unicode,ActualText=2983}%
    \textnormal{\usefont{OMS}{lmr}{m}{n}\char102}%
    \mathchoice{\mkern-4.05mu}{\mkern-4.05mu}{\mkern-4.3mu}{\mkern-4.8mu}%
    \textnormal{\usefont{OMS}{lmr}{m}{n}\char106}%
  \EndAccSupp{}%
}
\newcommand*{\rrbrace}{%
  \BeginAccSupp{method=hex,unicode,ActualText=2984}%
    \textnormal{\usefont{OMS}{lmr}{m}{n}\char106}%
    \mathchoice{\mkern-4.05mu}{\mkern-4.05mu}{\mkern-4.3mu}{\mkern-4.8mu}%
    \textnormal{\usefont{OMS}{lmr}{m}{n}\char103}%
  \EndAccSupp{}%
}

\begin{document}
\newsavebox{\lXbrace}
\savebox{\lXbrace}{$\llbrace$}
\newsavebox{\rXbrace}
\savebox{\rXbrace}{$\rrbrace$}
\def\lxbrace{\scalerel*{\usebox{\lXbrace}}{\llbrace}}
\def\rxbrace{\scalerel*{\usebox{\rXbrace}}{\rrbrace}}

\noindent
\verb|\tiny|: {\tiny\(\llbrace abc \rrbrace\)}
{\tiny\(\lxbrace abc \rxbrace\)} 
 $<-$scale of original brace

  \verb,ORIGINAL:,%
\indent \(\llbrace \left\llbracket\left\{\left[(a) \cdot
bc^{2^3}\right]^4\right\}^5\right\rrbracket \rrbrace\)
\\\verb,FIXED:   ,%
\def\core{ \left\llbracket\left\{\left[(a) \cdot
bc^{2^3}\right]^4\right\}^5\right\rrbracket}%
\indent \(\scalerel[1.5ex]{\lxbrace}{\core}\scalerel*[1.5ex]{\rxbrace}{\core}\)
bbraces scaled, but limited to 1.5ex width

\end{document}

I edited the original image (via zoom) to show how scalerel can take the bbraces and vertically scale them to fit the mathematical object they enclose. The top equation is without scalerel, the bottom with it.

I have added also a zoom on the \tiny size objects, so that one can see the effect of scaling the normalsize glyph rather than letting LaTeX scale both the strut and the brace and then sticking them together. The left version is letting LaTeX scale the glyphs that compose the character, while the right image is letting scalerel package scale the \normalsize character to \tiny proportions.

---

Based on request to be able to provide horizontal compression, to help make glyphs better match comparable counterparts in the literature, I upgraded the scalerel package today (to v1.3), adding the following composite commands

\newcommand\scaleleftright[4][99in]{%
  \if.#2\def\leftarg{\bl@nk}\else\def\leftarg{#2}\fi%
  \if.#4\def\rightarg{\bl@nk}\else\def\rightarg{#4}\fi%
  \scalerel[#1]{\leftarg}{#3}\scalerel*[#1]{\rightarg}{#3}%
}
\newcommand\stretchleftright[4][10000]{%
  \if.#2\def\leftarg{\bl@nk}\else\def\leftarg{#2}\fi%
  \if.#4\def\rightarg{\bl@nk}\else\def\rightarg{#4}\fi%
  \stretchrel[#1]{\leftarg}{#3}\stretchrel*[#1]{\rightarg}{#3}%
}

\newcommand\hstretch[2]{\stretchrel*{\scalebox{#1}{$#2$}}{#2}}

\newcommand\vstretch[2]{\stretchrel*{#2}{\scalebox{#1}{$#2$}}}

The ...leftright commands provide automated scaling for symbols in the manner of mathmode's \left and \right commands. But the \hstretch and \vstretch commands allow for the glyph stretching that was desired by the user. Using his prior example, I further refined the definition of his white-curly-braces symbol as

\def\lxbrace{%
   \hstretch{0.6}{\scalerel*{\usebox{\lXbrace}}{\llbrace}}}
\def\rxbrace{%
   \hstretch{0.6}{\scalerel*{\usebox{\rXbrace}}{\rrbrace}}}

which provided a 40% horizontal compression of his composite-glyph character. Thus, to compare (now in normal, not tiny size), my original correction to this latest correction: