How to prevent arguments UpValues from being used in selective functions

You might consider putting the "type" label inside an Association itself. This will complicate key addressing but simplify other handling.

asc = <|"atomData" ->
    <|"Atom Names" -> {"N", "C", "O", "C", "H", "H"}, 
     "Atom Nr" -> {56, 23, 117, 81, 211, 5}, 
     "Resname" -> {ALA, ALA, TYR, LEU, GLY, GLY}, 
     "Bias Value" -> {1, 5, 1, 5, 1, 1}, 
     "getRandomAtom" :> RandomChoice[{56, 23, 117, 81, 211, 5}], 
     "atomExists" :> (MemberQ[{"N", "C", "O", "C", "H", "H"}, #] &)|>|>;

aDtest[a_Association /; Keys[a] === {"atomData"}] := True;

f1[a_?aDtest, args__] := a[[1]]["getRandomAtom"] + Total[a[[1]]["Atom Nr"]]

f1[asc, 2]
549

Using a[[1]] each time is only one way to approach this; others include:

f1[aa_?aDtest, args__] := 
  With[{a = aa[[1]]}, a["getRandomAtom"] + Total[a["Atom Nr"]]]

Or:

f1[a_?aDtest, args__] := f1core[a[[1]], args]

f1core[a_, args__] := a["getRandomAtom"] + Total[a["Atom Nr"]]

For Mathematica 10.4 or later

The method above can be improved for recent versions of Mathematica as follows:

ClearAll[f1]

p1 = <|"atomData" -> a_|>;

f1[p1, args__] := a["getRandomAtom"] + Total[a["Atom Nr"]]

f1[asc, 2]
549

Reference:

  • MatchQ-ing Associations (MMA 10)

You should definitely go the route of using an inert wrapper for the association, atomData[Association[...]].

You can make a general subvalue like what you have,

atomData[a_Association][key_] := a[key];

and you can make more specific definitions like

atomData[a_Association]["getRandomAtom"] := 
       RandomChoice[a["Atom Names"]

and

atomData[a_Association]["atomExists", s_] := MemberQ[a["Atom Names"], s]

so that you don't need to waste space in your atomData for computed properties. This definition however,

atomData /: h_[atomData[a_Association],args___] := h[a,args]

is, as you say, pure rubbish suboptimal. This upvalue is so general that it renders your object useless for any other purpose. Just try {atomData[<|a->b|>],2}.

I use UpValues so seldom, that I tend to see it as code smell, an indication you should rethink things (except Nothing, I use Nothing in a lot of cases). In the above, the definition should be on h and not on atomData.

In your workarounds, you try to implement a list of special heads that shouldn't trigger an upvalue on atomData. Instead, it should be in the definition of the other functions to access the data in atomData:

f1[a:atomData[assoc_Association], args_] := f[ assoc, args]
f1[a_Association, args_] := ....

Write a definition like the above for every function you want to just work with the raw data directly. But the general behavior is for atomData head to not disappear whenever it's in some other expression.

A couple of other points:

You should use a more specific pattern than _Association. Even _?AssociationQ is better, since it rules out things like Association[1]. But in your case you could define a pattern:

$atomDataPattern = KeyValuePattern[ 
     {"Atom Names" -> {__String}, "Atom Nr" -> {__Integer} }
]

and so then you can use this in function definitions:

f[atomData[ad:$atomDataPattern], args] := f[ad,args]

You can then make a nice formatting for atomData using the functions described in this post.

If you want to program in Mathematica but keep to OOP methods you are used to, you might benefit from the package described in this post.