ext4 listing of files very slow in one specific directory that contained lots of files before

Directories only ever grow in size, not shrink. Try moving all those files out into a a temporary directory (like log2) then rmdir the old directory and rename the temp one as the new permanent one.