Returning multiple results from a LibraryLink function

I think number 2 is the way to go. This was one of the reasons why LibraryLink was written. The TetGenLink interface is an example where multiple LibrayLink functions query a single tetgen c++ instance and this works very efficiently.

Unfortunately, there is no LibrayFree function that you could connect to.

I see a third way, which is a combination of your two choices, but its usefulness depends on the problem you are solving. I used this once to get the best from both worlds. What I needed was a similar thing to ComponentMeasurements in 3D. Therefore, I needed a component labeling algorithm (similar to MorphologicalComponents) which takes a volume, binarizes it and assigns each separate component a unique ID. For this I needed only read access to the volume and therefore, I wanted to use it inside the library without copying it.

For this part I wrote a function using your second point which stored the result, the labeled volume, statically inside the library instance.

For each component in the labeled volume I wanted to have certain measures. My result was therefore a list which consisted of a set of several measurements for each component. This was something like

{"ID"->1, "Volume"->123, "BoundingBox"->{{..},{..},{..}}}

for each component, so highly inhomogeneous. To retrieve this result I wrote another function using MathLink where it didn't matter that I couldn't use the memory advantages of LibraryLink because I didn't have to transfer the large volume data.

I personally like how you can build very complex expressions with MathLink and for me it would have been very inconvenient if I had to write a LibraryLink function to retrieve every single property of my components.

void MLPutComponentProperties(MLINK mlp, const Component3d<mint> &c) {
    MLPutFunction(mlp, "List", c.number_of_properties);
    MLPutFunction(mlp, "Rule", 2); MLPutString(mlp, "Id"); MLPutInteger(mlp, c.label);
    MLPutFunction(mlp, "Rule", 2); MLPutString(mlp, "Volume"); MLPutInteger(mlp, c.volume);
    MLPutFunction(mlp, "Rule", 2); MLPutString(mlp, "BoundingBox");
        MLPutFunction(mlp, "List", 3);
        MLPutFunction(mlp, "List", 2);
          MLPutInteger(mlp, c.bounding_box.x1+1);
          MLPutInteger(mlp, c.bounding_box.x2+1);
        MLPutFunction(mlp, "List", 2);
          MLPutInteger(mlp, c.bounding_box.y1+1);
          MLPutInteger(mlp, c.bounding_box.y2+1);
        MLPutFunction(mlp, "List", 2);
          MLPutInteger(mlp, c.bounding_box.z1+1);
          MLPutInteger(mlp, c.bounding_box.z2+1);
}

void MLPutComponentList(MLINK mlp, const Component3dList components) {
    MLPutFunction(mlp, "List", components->size());
    Component3dListIterator it;
    for (it = components->begin(); it != components->end(); ++it) {
        MLPutComponentProperties(mlp, **it);
    }
}

Summary: I vouch for option 2 as LibraryLink is great with memory (sharing as opposed to copying). Furthermore I did a little investigation into the possible types of tensors with a negative/conservative result. The main point of posting the analysis to prevent that other people do the same.

Memory

Addressing: "Did anyone compare LibraryLink/MathLink data transfer performance so I can more easily assess what the performance hit would be"?

I think sending data over a MathLink connection always causes a copy of the data. In LibraryLink this is not necessary.

Firstly, you can load a function with the instruction that a tensor argument has to be shared between Mathematica and LibraryLink. This prevents copying information when giving the information from the kernel to the C program. Secondly, I think any tensor "returned" from a LibraryLink function using MArgument_setMTensor most likely does not have to be copied either. The main evidence for this is that the docs say

"Arguments passed to and from a library function can share data, saving on memory consumption and the time to copy large amounts of data."

in LibraryLink/tutorial/Introduction#163333181. This doc page also lists other differences between LibraryLink and MathLink. In particular the overhead of calling a LibraryLink function is much lower.

Example

Please see my answer here for an example of returning multiple outputs. It involves calling a separate function to access a global variable. I hope to learn more about this so that I can improve this code. At the moment it crashes for large inputs.

Analysis

From the LibraryLink user guide we have: "You can exchange not only C-like data types such as integers, reals, packed arrays, and strings, but also arbitrary Mathematica expressions".

Of course, from LibraryLink we can call MathLink, so I guess in this sense that statement is true (I am now quite confident that MathLink is necessary for this). However, I was hoping we could maybe make ragged arrays.

I investigated a bit, and found in WolframLibrary.h

#define MType_Integer 2
#define MType_Real 3
#define MType_Complex 4

My hope was now that there would be another type corresponding to 1. The following code can be used to investigate this further

DLLEXPORT int raggedArray_T_I(WolframLibraryData libData, mint Argc, MArgument *Args, MArgument Res) {

    MTensor T_arg = MArgument_getMTensor(Args[0]);
    int type = libData->MTensor_getType(T_arg);
    MArgument_setInteger(Res, type);
    return 0;
}

It doesn't even really matter what the code above is. The point was to see if we could call a library function with something other than the usual "packable" arrays as arguments. I loaded the function above as follows

rAFu = LibraryFunctionLoad[libraryName, "raggedArray_T_I", {{_, _}}, 
  Integer]

Of course for the usual arrays the output was expected. One interesting thing is that any combinations of empty lists apparently has type 3 as a tensor, corresponding to an array of reals.

Unfortunately, I was unable to find any other input that qualified as a tensor. For example raFu[{{1},1}] gives an error that {{1},1} is not a tensor (of rank -1 of Removed[$$Failure] oddly).

Far fetched

My suspicion is now that maybe the valid types are 2,3 and 4 instead of 1, 2 and 3 is because of the following reason. Possibly expressions contain an indicator (maybe in the sense that they are structs have the indicator as an attribute, I read somewhere that expressions "are" structs (and unions)) containing the information what kind of expression the expression is. Then 1 could stand for "normal expression" and 2, 3 and 4 for the PackedArrays. Other types of expressions could be Integer, Real or Complex. That is what I did when I programmed my own little prototype of Mathematica anyway. I would be glad if somebody could shine more light on this.