Accuracy problem of a DDS at very low frequencies

It's not an accuracy thing- it's resolution.

The front of the data sheet specifies tuning resolution of 0.0291Hz with a 125MHz clock.

\$0.0291 \approx \dfrac{125\times 10^{6}}{2^{32}}\$ Hz (since the phase accumulator is 32 bits)

So that's about 30% of your desired output frequency. That comes from the result of adding the LSB of the tuning word to the phase accumulator at 125MHz- for a given clock frequency, it's inherent to the chip and the number of bits they chose for the phase accumulator and tuning word.

You can try reducing the clock frequency- the minimum is 1MHz so you should be able to improve the resolution by more than two orders of magnitude, to around +/-0.23% at 0.1Hz.

\$0.23\times 10^{-3} \approx \dfrac{1\times 10^{6}}{2^{32}}\$ Hz resolution with a 1MHz clock

Unfortunately, other things are going to have to change for optimal performance (especially the output filter- which is typically a 7th order elliptical LC filter on these modules).

If you never need to go above, say, 1Hz, you can simply add an RC filter with a cutoff of, say, 100Hz to the existing output and it will be acceptable for many purposes.

What you're looking for on the datasheet is the frequency tuning resolution. For this chip it is 0.0291 Hz for a 125 MHz reference clock input. Your frequency will be rounded to a multiple of this number. This number is based on the frequency of the clock input of the chip.

For example, 0.1 Hz will be rounded to 0.0873 Hz (0.0291*3). The period for 0.0873 Hz is 11.5 seconds which is what you're seeing.

A lower input clock frequency will give higher accuracy at lower frequencies. So if you wanted better accuracy at lower frequencies then lower the clock frequency.