biblatex: sentence case and nested quotation marks

As a workaround, you may use csquote's \MakeAutoQuote macro to define active characters which may be used instead of \mkbibquote. (In my example, I use « and ».)

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[american]{babel}
\usepackage{csquotes}
\MakeAutoQuote{«}{»}
\usepackage[style=authoryear]{biblatex}
\usepackage{filecontents}

\begin{filecontents}{test.bib}
@article{ref,
  sorttitle = {Handle Quotation Marks within Article Titles},
  author = {Lastname, Firstname},
  title = {Handle «Quotation Marks» within Article Titles},
  journal = {Journalname},
  volume = {10},
  date = {1},
  pages = {5--10}}
\end{filecontents}

\bibliography{test.bib}

\DeclareFieldFormat{sentencecase}{\MakeSentenceCase*{#1}}
\renewbibmacro{title}{%
    \ifthenelse{\iffieldundef{title}\AND\iffieldundef{subtitle}}{}
        {\ifthenelse{\ifentrytype{article}\OR\ifentrytype{inbook}%
            \OR\ifentrytype{incollection}\OR\ifentrytype{inproceedings}}
            {\printtext[title]{%
                \printfield[sentencecase]{title}%
                \setunit{\subtitlepunct}%
                \printfield[sentencecase]{subtitle}}%
                \newunit}%
            {\printtext[title]{%
                \printfield[titlecase]{title}%
                \setunit{\subtitlepunct}%
                \printfield[titlecase]{subtitle}}%
                \newunit}}%
    \printfield{titleaddon}}

\begin{document}
    \textcite{ref}
    \printbibliography
\end{document}

Just for the records, it also works if you nest the quotation macro in a further pair of braces:

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage[american]{babel}
\usepackage{csquotes}
\usepackage[style=authoryear]{biblatex}
\usepackage{filecontents}

\begin{filecontents}{test.bib}
@article{ref,
  sorttitle = {Handle Quotation Marks within Article Titles},
  author = {Lastname, Firstname},
  title = {Handle {\mkbibquote{Quotation Marks}} within Article Titles},
  journal = {Journalname},
  volume = {10},
  date = {2018},
  pages = {5--10}}
\end{filecontents}

\bibliography{test.bib}

\DeclareFieldFormat{sentencecase}{\MakeSentenceCase*{#1}}
\renewbibmacro{title}{%
    \ifthenelse{\iffieldundef{title}\AND\iffieldundef{subtitle}}{}
        {\ifthenelse{\ifentrytype{article}\OR\ifentrytype{inbook}%
            \OR\ifentrytype{incollection}\OR\ifentrytype{inproceedings}}
            {\printtext[title]{%
                \printfield[sentencecase]{title}%
                \setunit{\subtitlepunct}%
                \printfield[sentencecase]{subtitle}}%
                \newunit}%
            {\printtext[title]{%
                \printfield[titlecase]{title}%
                \setunit{\subtitlepunct}%
                \printfield[titlecase]{subtitle}}%
                \newunit}}%
    \printfield{titleaddon}}

\begin{document}
    \textcite{ref}
    \printbibliography
\end{document}

Note that this does not always work (beginning of title gives all-caps, forcing of upcased constituents, such as names, in such quotes is not possible).

For these specific cases, if you cannot use active characters as proposed by @lockstep (for instance since you need to run biber with --output-safechars), you can also use csquotes text macros (\textooquote, \textcoquote, \textoiquote and \textciquote).