Should I use useEffect() for dom manipulation or just do it directly?

Its almost the same, see my related answer, useEffect in-depth.

The difference is a notable "gotcha", useEffect callback executed after the render phase.

const App = () => {
  useEffect(() => {
    console.log("executed after render phase");
  });

  console.log("executed at render phase");

  return <></>;
};

Will result:

executed at render phase
executed after render phase

You better keep DOM manipulations and other side effects inside useEffect.

useEffect(() => {
    document.title = `You clicked ${count} times`;
}, [count]); // set your dependencies

Reason is: This approach is compliant with React Strict Mode and upcoming Concurrent Mode.

What Concurrent mode is:

[...] React may invoke render phase lifecycles more than once before committing, or it may invoke them without committing at all (because of an error or a higher priority interruption).

In your case, the document.title assignment might be repeated multiple times or React might even decide to abort the whole commit. In general, this can lead to inconsistencies:

Because the above methods might be called more than once, it’s important that they do not contain side-effects. Ignoring this rule can lead to a variety of problems, including memory leaks and invalid application state. (docs)