Custom atomic expressions - modern tutorial

I have a new package for speeding up the construction of things like these which I discuss at the end

Atomic vs Valid

These are two related but distinct concepts. The former indicates that we can't access subparts of the expression. The second indicates that the expression has already fed through its entire constructor process and we don't need to update it anymore.

I use both of these in my InterfaceObjects package so let me describe what I do there. In these objects I use a Head as both an object type and a constructor. The way I circumvent an infinite eval loop is by checking for one of these flags (the registered type declares which one it uses). Here's a simple example:

construct =
  (* 
     I'm injecting the SetNoEntry so I can explicitly show that it's my
     validation function. The HoldFirst prevents infinite recursion.
  *)
  With[{validator = System`Private`SetNoEntry},
   Function[Null, validator[Unevaluated[#]], HoldFirst]
   ];
unconstructedQ =
  With[{validator = System`Private`EntryQ}, 
   Function[Null, validator[Unevaluated[#]], HoldFirst]
   ];
preprocessData = Identity;
myObj[data_]?unconstructedQ :=
 With[{d = preprocessData[data]},
  construct[myObj[d]]
  ]

Here preprocessData can be arbitrarily complex and we can put in logic to not construct an object if the data is invalid. This means that we can then write a quick validQ function that just checks that the object has been constructed, rather than having to dig into and validate the data over and over. This is conceptually how Association works. One quick thing to note, is that for v11+ there SetNoEntry and NoEntryQ also have corresponding HoldSetNoEntry and HoldNoEntryQ functions that obviate the need for the HoldFirst.

We could easily have done this by substituting System`Private`ValidQ for System`Private`EntryQ and System`Private`SetValid for System`Private`SetNoEntry, though.

And here's where the difference is important. In the example given, we can't access into a myObj directly:

o = myObj[{1, 2, 3}];
o[[1]]

Part::partd: Part specification myObj[{1,2,3}][[1]] is longer than depth of object.

myObj[{1, 2, 3}][[1]]

But if I'd done the same with Valid and friends I could:

construct =
  With[{validator = System`Private`SetValid},
   Function[Null, validator[Unevaluated[#]], HoldFirst]
   ];
unconstructedQ =
  With[{validator = System`Private`ValidQ}, Function[Null, validator[Unevaluated[#]], HoldFirst]
   ];
preprocessData = Identity;
myObj2[data_]?unconstructedQ :=
 With[{d = preprocessData[data]},
  construct[myObj2[d]]
  ]

o = myObj2[{1, 2, 3}];
o[[1]]

{1, 2, 3}

Why I Only Use NoEntry

As I see it, the point of objects is to hide complexity from the user. The user doesn't need to know that you're storing your stack as a linked list or that you have 16 different fields in your data structure. In fact, it's cleaner if the user doesn't know. You need a well designed API that provides all the necessary methods your user might want or need and which works fast. Beyond that pushing all the implementation details out of reach is good practice.

To my eyes, making an object atomic helps achieve that abstraction. It makes it more difficult for a user (and myself as the developer!) to go in an tinker and get in the bad habit of manipulating the direct data rather than going through the API. Long-term this will make the code harder to maintain and cause more breakages when I go through and optimize later. I think of this by analogy to public and private fields in a C++ or python class, although the correspondence is not direct. For a discussion of why those are useful, see here.

Immutable vs Mutable

The question of whether to use a reference to some data or the data itself as the first argument to your object is I think dependent on the type of environment you're working in as well as personal preference, to some degree. Consider this:

editObj[o : myObj[stateSym_], ...] := (
  editState[stateSym, ...];
  o
  )

editObj[myObj[data_], ...] := (
  quickConstructor@editData[data, ...]
  (* 
    where quick constructor will build a new myObj object in the fastest way 
     possible w.r.t type-checking, setting of NoEntry, etc.
  *)
  )

These are the two idiomatic ways to edit object data. In the former we edit the object state and return the original object directly. In the latter we edit the object data and have a quick constructor for when we know the data is valid. Both of these will get the job done and which you prefer is really up to you.

On the other hand, there are cases where mutable vs. immutable really does matter. As an example, say you want to synchronize state across many parts of a program without having to use some kind of global variable as a synchronizer. This is exactly where mutability comes in handy. I could write a syncState object like:

syncState~SetAttributes~HoldFirst
syncState[] :=
  Module[{state}, construct@syncState[state]];

And then all my functions would take a state argument like:

handler1[s_syncState, ...] := ...;
handler2[s_syncState, ...] := ...;

This way they could directly call into the syncState API and ensure synchronization across the entire program in a modular fashion.

On the other hand, this mutability means it's harder to serialize the state. What I mean by that is you now have references to a given symbol floating about, like: myObj[state$132424]. If you write this to file you now need to destruct state$132424 so that it's in a serializable form (e.g. Association). If you would like to serialize multiple parts of an app, but were relying on the mutability of state$132424 this can add a whole new layer of complexity, as now you'll have to serialize the fact that state$132424 had that data attached to it rather than the data itself. This can be done with, e.g. DumpSave, but it is non-trivial to make entirely robust.

As an aside, in my anecdotal experience it tends to be a bit slower to mutate things than simply write then in terms of basic immutable primitives which really work quite efficiently in general.

In general, I tend to prefer to use immutable data structures whenever possible, and only introduce the mutability when I need it or when it will seriously help performance.

Mutation Handler

One thing to mention here is the MutationHandler family of functions. These make it possible for even immutable expressions to operate mutably when bound to a Symbol or other mutable expression. I won't get into that here since that's been treated in detail here but it's definitely worth checking out. Writing a good set of mutation handlers will make writing code much more natural when it's warranted.

ExpressionStore

One place where using mutable versions of an expression is helpful is with regards to ExpressionStore. As noted there, ExpressionStore can cache computed properties by explicit object identity. So you could create something like:

$cache = Language`NewExpressionStore["<ObjectCache>"];

And then you can write a function that only calculates a value if it's not in the cache, like:

calcCached[obj_, prop_, meth_, args___] :=
 Replace[$cache@"get"[obj, prop],
  {
   Null :>
    With[{val = obj@meth[args]},
     $cache@"put"[obj, prop, Hold[val]];
      (* using Hold just so we know the Head it must have *)
     val
     ],
   Hold[v_] :> v
   }
  ]

In the mutable setup, we can modify our objects without worry, but in the immutable setup, every modification will create new object (though quickly and cheaply) which will lose its attachment to its cached properties. In this kind of case it definitely does make sense to use a mutable ref.

Typesetting

When it comes to typesetting, I try to be consistent with what people are used to. I generally avoid writing fancy typeset forms, and instead call into the mechanism that all of WRI's objects use, which is ArrangeSummaryBox.

As far as passing excessive data to the FE goes, this actually handles it! If you have a huge object, it doesn't pass the entire thing to the FE but instead returns it back with one of those little "store in notebook?" attached cells. You can prune this down even further by setting "Interpretable"->False I believe, which is also probably a good setting in a mutable object setting, as you can't ensure the object will retain its validity from session to session.

One thing I always make sure to do, though, is check if I have a valid object before typesetting it. What I mean is that I always check my NoEntry bit like:

myObjQ = Function[Null, System`Private`NoEntryQ[Unevaluated[#]], HoldFirst];
Format[m_myObj?myObjQ, StandardForm] :=
 RawBoxes@
  BoxForm`ArrangeSummaryBox[
   myObj,
   ...
   ]

Sometimes myObjQ will be a little bit more sophisticated, but usually it's pretty much just that.

As I see it, going beyond the standard in terms of typesetting won't really buy you much, as you should be more focused on writing a good API for working with your objects flexibly and efficiently.

Methods

This is a place where my preferred style of operation is probably not best for the average Mathematica development project. As I see it, there are three ways to get methods into an object, but they all require one thing: you've got to write lots of little API functions. What I mean by that is if I have myObj as my base type and I want to do four different types of manipulations on it I write:

myObjManip1[myObj[data_], ...] := (* work with data *);
myObjManip2[myObj[data_], ...] := (* work with data *);
myObjManip3[myObj[data_], ...] := (* work with data *);
myObjManip4[myObj[data_], ...] := (* work with data *);

Note that in methods you can also make use of the fact that we now have NoEntry or Valid set to handle object validation up front. This means you could rewrite this as:

myObjManip1[myObj[data_]?validQ, ...] := (* work with data *);

where validQ simply checks that bit or whatever other quick tests you would like to have.

At this point, I can go one of a three ways:

Expose ALL the Functions!!!

If I have a large set of API functions, it might make sense just to expose them to users directly. One the one hand, this gives them really targeted control over the manipulation of my object. On the other, they now need to find and learn about tens of new functions in an average case.

Expose stuff as UpValues

A slick way around this is to write the manipulation functions in a developer context (e.g. "MyApp`Package`") and then expose the API as a bunch of UpValues. This has the benefit of putting things in a context that people are more familiar with and not flooding the global namespace. The issue with this is that we need to find a good set of top-level functions we can shoehorn things into and if the shoehorning isn't done well it can be confusing that before. On top of that, it takes more work to discover all the available UpValues.

Expose stuff as SubValues

This is my favorite way by far. In this setup, we again put all the functions into the developer context, but now we expose all the methods as "SubValues" keyed by their string values. This means something like:

myObj[data_]["Manip1", ...] := myObjManip1[myObj[data], ...];
myObj[data_]["Manip2", ...] := myObjManip2[myObj[data], ...];

or in my preferred syntax (it looks the most "normal" to a python/Java/C++ programmer):

myObj[data_]@"Manip1"[...] := myObjManip1[myObj[data], ...];

The issue with this would appear to be that discovery is hard, but that's why you always need something like:

myObj[data_]@"Methods" = {...};

And if you have properties you need a rule for that too.

When I work with the objects I make in my InterfaceObjects package I'd say 80% of the time this is how I prefer to expose things to users and 20% of the time I like to use UpValues for cases where it's really clear that the system function should support your type.

There, too, I wrote up better logic to automatically curate and set up all the "Methods" and "Properties" lists and whatnot.

A few concrete examples:

  • I made a RubiksCube object that implements all these thing I talk about.

  • My InterfaceObjects package implements all this except that it only ever uses immutable data structures.

  • I also worked with many of these ideas in a context that only ever used Symbol as its data ref so it'd be mutable always. That lives in my SymbolObjects package (which is on GitHub too).

  • I'm working on a DataStructures package that uses the formatting and NoEntry ideas but takes an expose-all-the-functions approach to its API.

See also: this discussion

Simple Constructor

I wrote up a simple constructor for these data types for my DataStructures package. The package itself will be on the paclet server in a few days, otherwise feel free to load the constructor directly like:

BeginPackage["DataStructures`Developer`"];
Get["https://github.com/b3m2a1/DataStructures/raw/master/Packages/Developer/Register.m"];
EndPackage[];

Then you can use it like:

RegisterDataStructure[MyObj, MyObj[_Association]]

Then maybe add a convenience constructor:

MyObj[] := MyObj[<||>]

It's reasonably fast to make one of these:

MyObj[] // RepeatedTiming

{0.0000109, MyObj[<||>]}

But if you know you have valid data you can speed this up a lot by using a dedicated fast constructor:

`MyObj`New[<||>] // RepeatedTiming

{2.8*10^-6, MyObj[<||>]}

This fast constructor can be specified in the RegisterDataStructure call like:

RegisterDataStructure[MyObj, MyObj[_Association], "FastConstructor" -> MyObjNew]

MyObjNew[<||>] // RepeatedTiming

{2.7*10^-6, MyObj[<||>]}

By default it's tagged as "Atomic":

MyObj[][[1]]

Part::partd: Part specification MyObj[<||>][[1]] is longer than depth of object.

MyObj[<||>][[1]]

But you can turn that off and use ValidQ instead by calling RegisterDataStructure with "Atomic"->False.

There are a number of other levers you can play with, here. If you'd like a to supply a custom data prep or data validation function you can do so with the "DataPrepper" and "DataValidator" options.

The "Validator" option allows you to pass a custom symbol to bind as the function that checks if a data structure is valid. By default it'll be something like `MyObj`ValidQ but probably a more commonly desired choice would be MyObjQ to imitate built-in functions.

The "FormattingRules" and "Formatted" options let you specify how you want BoxForm`ArrangeSummaryBox to work with your structure. If "Formatted" is False it doesn't format at all. Otherwise, the "Icon" supplied in the "FormattingRules" specifies the little icon you want for your structure. The "DisplayedFields" should be an Association of keys mapping to functions to extract the displayed value for that field. The "HiddenFields" will be the fields that are hidden until the little + button is clicked on the summary box.

More settings and options (particularly for mutation handling and things) could potentially come in the future, but since these data structures are intended to be as efficient as possible while remaining convenient, I don't think many more will come.


I recently had to do something just like this, but chose not to make my object AtomQ. I'm not a fan of making things AtomQ unless they need to be*.

I do however think it is useful to be able to mark a data structure as validated so that you no don't need to go through a possibly expensive validation every time you want to use your data structure. For this, I use a combination of Unevaluated with System`Private`ValidQ and System`Private`SetValid.

ds:MyObject[args___] /; !validObjectQ[Unevaluated @ds] := Module[
    {canonical = Catch[canonicalizeMyObject @ args, $tag]},
    canonical /; validObjectQ[canonical]
];

validObjectQ[ds:MyObject[Association[___]]] := 
    System`Private`ValidQ[Unevaluated @ ds];

validObjectQ[___] := False;

createValidObject[args___] := 
    System`Private`SetValid[Unevaluated[ MyObject[ args]]];

canonicalizeMyObject[a_ ? AssociationQ] := Module[
    {validAssociation},
    (* put expensive validation/canonicalization code here *)
    validAssociation = KeyExistsQ[a, "specialKey"];
    If[validAssociation, createValidObject @ a, $Failed]
];

In the above code, you see that whenever you create an instance of MyObject it will trigger the single definition. Then canonicalizeMyObject is called and will attempt to return a validated MyObject. After this, canonicalizeMyObject will no longer be called on this instance of MyObject.

obj1 = MyObject[bob];
obj2 = MyObject[<|"A" -> 3|>];
obj3 = MyObject[<|"A" -> 3, "specialKey" -> 2|>];

validObjectQ /@ {obj1, obj2, obj3}
(* {False, False, True} *)

You can run TracePrint on that last command with a second argument of _canonicalizeMyObject to verify that it isn't called.

A few points about this ValidQ flag (all that I have gleamed from spelunking and playing around, I'm not aware of any documentation):

  • It is passed on when copying an expression, so if obj3 is ValidQ, then obj4 = obj3 infers this flag on obj4 without calling the canonicalizeMyObject code.
  • It goes away if you modify the object. So if you do AppendTo[obj3, 4] then obj3 is re-validated.
  • It is saved when serializing to an MX file.
  • It is undocumented, so user beware.

*If anyone had asked me, I would have said not to overload Part for SpaseArray, but I think I'm in the minority on this point, also no one asks me about such important things.

Tags:

Function Construction

Guidelines

Expression Manipulation

Recent Posts