How can I initialize a class instance in a stateless function component in React?

I'd go for useMemo solution, but here's how you can do it with useRef + useEffect to clear up some confusion:

const StatelessFunction = (props) => {
   const helper = useRef(); // ref is equivalent of instance prop

   useEffect(() => {
     helper.current = new HelperClass();
     helper.current.doSomething();
   }, []); // this is equivalent of componentDidMount
}

just use useRef, no need to combine it with useEffect.

const instance = useRef();
if (!instance.current) {
  instance.current = new HelperClass();
}

see React Docs for details.

Looking back at this and some of the comments I can see that useMemo can not be trusted to not run the HelperClass constructor again and useRef will only set the helper after the first render because it's initial value can't be a function. Probably useState is the easiest and most reliable way to do this:

const [helper] = useState(()=>new HelperClass());

You can use useMemo to create an instance of HelperClass and useEffect to call it. Giving them both empty array of dependencies means they will only be called "on mount". I put on mount in quotes because memo will be called only on first render and effect will be called after first render cycle is finished.

const StatelessFunction = props => {
  const helper = useMemo(() => new HelperClass(), []);
  useEffect(() => {
    helper.doSomething();
  }, [helper]);
  return (<JSX />);
};

If the only thing you'll ever do is just call doSomething and never use the helper instance again you can just do it with useEffect:

useEffect(() => {
  new HelperClass().doSomething();
}, []);

If you do plan to use the helper instance at some later time then you could use the previous example with useMemo or useRef:

const helper = useRef();
useEffect(() => {
  helper.current = new HelperClass();
  //only called once after first render
  helper.current.doSomething();
}, []);
//you still have the helper instance as helper.current