EstimatdDistribution returns evaluated expression, not function name/
After a bit of spelunking, you could try the following to define a new distribution:
betaPrimeDistributionEval[a_,b_] := ProbabilityDistribution[x^(a-1) (1+x)^(-a-b)/Beta[a,b],{x,0,\[Infinity]},Assumptions->a>0&&b>0]
betaPrimeDistribution /: DistributionDomain[_betaPrimeDistribution] = Interval[{0, Infinity}];
betaPrimeDistribution /: DistributionParameterQ[_betaPrimeDistribution] = True;
betaPrimeDistribution /: DistributionParameterAssumptions[betaPrimeDistribution[a_,b_]] := a>0 && b>0
betaPrimeDistribution /: PDF[betaPrimeDistribution[a_, b_], c___] := PDF[betaPrimeDistributionEval[a, b], c]
betaPrimeDistribution /: RandomVariate[betaPrimeDistribution[a_, b_], c___] := RandomVariate[betaPrimeDistributionEval[a, b], c]
Then I think the following does what you want:
data = RandomVariate[betaPrimeDistribution[1, 2], 100];
EstimatedDistribution[data, betaPrimeDistribution[a, b]]
betaPrimeDistribution[0.899712, 1.704]
What do you mean? The beta prime distribution is right here. Even if it wasn't, you could easily define it with TransformedDistribution
as
betaPrimeDistribution[α_, β_] :=
TransformedDistribution[
\[FormalX]/(1 - \[FormalX]),
\[FormalX] \[Distributed] BetaDistribution[α, β]
]