Difference between param[out] and return in doxygen?

Out parameters are different from return values. Take this example in C:

/**
 * \param[in]  val      Value calculations are based off.
 * \param[out] variable Function output is written to this variable.
 *
 * \return Nothing
 */
void modify_value(int val, int *variable)
{
    val *= 5;
    int working = val % 44;
    *variable = working;
}

The function returns nothing, but the value to which variable points is changed, hence we call it an output parameter. It represents an 'output' of the function in that we expect it to be modified somehow by the function. val, on the other hand, is an 'input' parameter because it is not modified (and, indeed, cannot be modified from the perspective of the function's caller, since it is passed as a value).

Here's a slightly more useful and realistic example:

typedef struct data {
    int i;
    int j;
    ...
} data;

/**
 * \param[in]  val Initialising parameter for data.
 * \param[out] dat Data pointer where the new object should be stored.
 *
 * \return True if the object was created, false if not
 *         (i.e., we're out of memory)
 */
bool create_data(int val, data **dat)
{
    data *newdata;
    newdata = (data*)malloc(sizeof(data));
    if(newdata == NULL)
    {
        *dat = NULL;
        return false;
    }
    newdata->i = val;
    *dat = newdata;
    return true;
}

In this case, we construct some complex object inside the function. We return a simple status flag that lets the user know the object creation was successful. But we pass out the newly-created object using an out parameter.

(Although, of course, this function could easily just return a pointer. Some functions are more complex!)

As a simpler answer, [out] parameters are only for results returned via parameters not the return value. It is quite reasonable to have a function which has a return value and also has optional return data, eg: one I'm just writing has the signature:

  /**
  Determine UTF type of a file. 
  Unless a UTF8 file has a BOM, it is regarded as unknown.

  @param [in] path Path to file suitable for ifstream
  @param [out] bomWasFound optional return flag to indicate a BOM was found, really only useful for UTF8
  @return an enum indicating type, default utf_unknown
  */
  UtfType CheckFileType(const std::string& path, bool* bomWasFound=0);