what does Out store?

You should look at Out's down-values. Here is what I get.

You can see the Out is essentially an indexed variable with assignments.

Update

Out[ ] is equivalent to

Out[Length[DownValues[Out]]]

Update 2

OK, lets look at the code exactly as you wrote it.

In this case, Out[3] has its MatrixForm wrapper stripped off, so it is the same as Out[2]. This is explained in this Documentation Center article in the section The Main Loop. So what you see is entirely as expected. My example where the assignment to mfA was followed by ; is more interesting. In this case the special rule for ; is invoked and that rule apparently does not strip off wrappers.

The value stored in Out is, as a rule, what is displayed in the output. The only exception from that are the various *Forms, MatrixForm being an example of that. When one of such "wrappers" (as the help calls it) is detected in the top level of the output, it is removed from the expression prior to assigning to Out. The front end makes it clear that what's displayed is not the value of Out:

In[1]:= {{1,2},{0,1}} // MatrixForm

Out[1]//MatrixForm= 1   2
         ^^^^^^^^^^    
                     0   1

In[2]:= Out[1]

Out[2]= {{1, 2}, {0, 1}}

The reason for this is that it's convenient for further processing of the output. The MatrixForm itself would be inoperable:

In[3]:= ({{1,2},{0,1}} // MatrixForm) * 2

Out[3]= 2 1   2

           0   1

One would need to manually remove the outer layer every time of using Out[] as the price for wanting to have the output nicely displayed. So Mathematica does that for the user as a convenience.

In your example you assigned MatrixForm[mA] to mfA so that's what mfA is. (This is often exactly the undesirable behaviour.) As a result of the assignment, MatrixForm[mA] is printed. But now the rule gets applied, the top-level MatrixForm is shaved, and only mA stored in Out for that input line.

---

Update: the list of functions which behave as wrappers is $OutputForms, as found here:

? $OutputForms

$OutputForms is a list of the formatting functions that get stripped off when wrapped around the output.

You can unprotect it, add or remove some (or all of them) and cause MatrixForm to be stored in the Out or StringForm not to:

In[1]:= Unprotect[$OutputForms];                                                

In[2]:= $OutputForms = {StringForm};                                            

In[3]:= {{1,2},{0,1}} // MatrixForm                                             

Out[3]= 1   2

         0   1

In[4]:= % + 2                                                                   

Out[4]= 2 + 1   2

             0   1

In[5]:= "abc" // StringForm                                                     

Out[5]//StringForm= abc
         ^^^^^^^^^^

In[6]:= Head[%]                                                                 

Out[6]= String