Replace custom functions, leave built in functions untouched?

The best method I am aware of to handle this kind of problem is to filter by context.⁽¹⁾

SetAttributes[user, HoldFirst]
user[s_Symbol] := Context@Unevaluated@s =!= "System`";

a[x, y]/(b[x, y] c[x, y]) /. f_?user[x1_, y1_] :> f[2 x1, 3 y1]

a[2 x, 3 y]/(b[2 x, 3 y] c[2 x, 3 y])

One could include other contexts in the exclusion besides System, or use the inverse and test only for user symbols existing in the "Global`" context. Without additional examples my example is as specific as I can make it.

---

Regarding the unusual evaluation of the ? operator (PatternTest) please see:

• Why doesn't PatternTest work with Composition?

You can impose conditions on the patterns to restrict their matching:

a[x, y]/(b[x, y] c[x, y]) /. 
 f_?(MemberQ[{a, b, c}, #] &)[x1_, y1_] :> f[2 x1, 3 y1]

or the alternative, equivalent:

a[x, y]/(b[x, y] c[x, y]) /. 
 f_[x1_, y1_] :> f[2 x1, 3 y1] /; MemberQ[{a, b, c}, f]

Both expressions return your desired result:

Because built-in functions are Protected, the following also works.

a[x, y]/(b[x, y] c[x, y]) /. 
    f_[x1_, y1_] :> f[2 x1, 3 y1] /; ! MemberQ[Attributes[f], Protected]

(* a[2 x, 3 y]/(b[2 x, 3 y] c[2 x, 3 y]) *)