How do I solve the quintic $n^5-m^4n+\frac{P}{2m}=0$ for $n$?

Solving a quintic is probably not the best idea for your problem.

Your numbers $m,n$ and $f$ are integers and so you should be using techniques for finding integer solutions.

For example, $2mn(m^2-n^2)(m^2+n^2)=f$ means that $m$, $n$, ... are all factors of the integer $f$.

(By the way, you seem to have $m>n$ at first and then $n>m$ later in your post.)

For your work, you may find it helpful to know of the following list of differences of coprime 4th powers https://oeis.org/A147858

I doubt that solving a quintic is the way to go. Consider the congruent number problem, which asks if an integer can be the area of a right triangle with rational sides. In your problem, the area will always be a congruent number. If you read the article, you will notice first that the congruent number problem reduces to solving a cubic, and second that it is unsolved.

Of course, your problem is different. You are requiring integral sides, not rational sides, and you are starting with the product of the sides, not the area. Still, it seems to me that the two problems are somehow related, and the congruent number problem is unsolved, even though it involves a cubic rather than a quintic.

I don't think you are paying sufficient attention to the fact that you require a solution in integers. I think the "number-theory" tag is more appropriate than any of the tags you have applied.

Here are some thoughts of the top of my head. Suppose we are given the easiest case, $ABC=60$. Now $60=2^2\cdot3\cdot5$, and we may assume that $A<B<C.$ We can test the various factors of $60$ as possible values for $C$. For example, can $C=4$? Then $B\leq3$ so $A\leq2$ and $ABC\leq2\cdot3\cdot4<60$, contradiction. In general, we need $$(C-2)(C-1)C^2\geq F$$

Once we have chosen a possible value for $C$, we have the equations $$\begin{align} A^2+B^2 &= C^2\tag1\\ AB &= \frac{F}{C}\tag2 \end{align}$$ so that $$(A+B)^2=C^2+\frac{2F}{C}$$ If the right-hand side is a perfect square, we can solve for $A+B$ and eliminate $B$ from $(2)$.

So, if $F$ is small enough to factor, there's an easy way to test $F$, but I think it will be hard to find a formula that says, "An $F$ of one of these forms is acceptable, and an $F$ of any other form is not."

I know that this is really more of a comment than an answer, but it's too long for a comment box.