pdfx: CMYK color profile in PDF/A

You can force the output intent with :

\RequirePackage{filecontents}
 \begin{filecontents*}{\jobname.xmpdata}
 \setCMYKcolorprofile{coated_FOGRA39L_argl.icc}
                     {Coated FOGRA39}
                     {FOGRA39 (ISO Coated v2 300\% (ECI))}
                     {http://www.argyllcms.com/}
        \Title{Title}
        \Author{Author\sep}
        \Language{de-DE}
        \Keywords{keyword1\sep keyword2\sep keyword3}
        \Publisher{Publisher}
      \end{filecontents*}
\documentclass[]{scrbook}

\usepackage[a-3b]{pdfx}

% -- snipp --

\immediate\pdfobj stream attr{/N 4} file{coated_FOGRA39L_argl.icc}
\pdfcatalog{%
/OutputIntents [ <<
/Type /OutputIntent
/S/GTS_PDFA1
/DestOutputProfile \the\pdflastobj\space 0 R
/OutputConditionIdentifier (Coated FOGRA39)
/Info(FOGRA39L)
>> ]
}

% -- snapp -- 

\begin{document}
Test
\end{document}

But preflight will complain, that the output intent isn't RGB

… preflight will complain, that the output intent isn't RGB

If it does not pass at least one validator it's probably not a solution. In this answer about CMYK PDF/A I showed how to produce a valid PDF/A with and without hyperlinks. The solution is to avoid pdfx because, as you already found out, it has numerous bugs.

if I actually include CMYK images, it does not pass as proper PDF/A since the color profiles of the PDF and the included image differ

You can include both RGB and CMYK images and graphics in a PDF/A. But they must be colour managed. Imagine a CMYK document with an embedded image that comes with three colour values per pixel: That must throw an error.

• The state-of-the-art solution is to embed graphics that are already PDF/A because this forces both colour management and accessible metadata.

• You can include RGB .jpg images with an embedded .icc profile but the metadata will not be PDF/A compliant. Some validators will complain, Acrobat by default will not (but you can enable embedded documents metadata checking in preflight settings).

• What does not work is embedding .png because those don't support neither embedded image profiles nor CMYK.

• You could embed CMYK .jpg images without a profile but that does not make much sense. Embed PDF vector graphics instead or PDF raster graphics. Those support lossless compression.