What is the purpose of a custom React reconciler within a DOM host?

There are some interesting advantages when it comes to using a custom reconciler in React. As you see in the README.md file of react-pixi-fiber, it's totally possible to use ReactDOM to render the pixi elements instead of using the custom render from react-pixi-fiber.

Why create a custom renderer/reconciler then?

In this specific case the reason is that ReactDOM doesn't really deal with canvas elements. As you said though, that could have been achieved by a combination of custom hooks/components. If you read the why section of react-three-fiber you will see that by using their custom reconciler you can achieve two things, compared to custom components:

1. Everything from threejs will work here, because the support is built into the reconciler.
2. The performance is the same as when using threejs directly, because the reconciler has more control over what is rendered and when.

You can take a look here where there's an in depth explanation of the difference between render and reconcile and how the reconciler has fine grained access to: components lifecycles, decides the diffing and how elements are added/removed from the view (be in DOM, canvas, iOS etc).

In order to understand this we need to know how React’s old reconciliation algorithm worked.

You only have one main thread in javascript that does your UI updation, state change, network computation & responding to user’s action.

At any point in time React has 2 tree’s (VDOM tree) in memory, the current tree which is already rendered and the updated tree with all the latest changes. The Stack reconciler finds the difference between 2 tree’s synchronously in a single pass This prevents the main thread from doing potential urgent work until the recursive process has finished. If user happens to type in a text input during this phase, your app becomes unresponsive as main thread is busy. A similar kind of thing happens when an animation needs to happen and your main thread is busy, that’s when you experience Jank.

Please take a look this article.