How to extract aac audio from an mp4 file to m4a
I finally found the solution myself in using mp4 or m4a as output format and adding -vn to prevent also copying the video.
ffmpeg -i input.mp4 -vn -c:a copy output.m4a
(Source: https://superuser.com/a/706622/180675)
Seems simple now :]
Fast and lossless stripping
Another option:
MP4Box -add infile.mp4#audio outfile.m4a
MP4Box is part of the gpac package.