VBA: Why would the Not operator stop working?

It's because of the internal Long value coming from this property, as explained by Cindy Meister. We should always use CInt to avoid this.

Sub testChangeBoolean2()
  Dim X As Boolean                     ' again, default is False
  X = CInt(Selection.Cells(1).FitText) ' [Fixed] does set X to True
  X = Not X                            ' X is False!
End Sub

I believe the explanation has to do with how older programming languages (WordBasic and early VBA) stored the integer values of True and False. In those days, True = -1 and False = 0.

Newer programming languages still use 0 for False, but 1 for True.

The majority of Word's Boolean type properties continue to use -1 for True (Font.Bold, for example), which has been cause for confusion and frustration for programmers working with the Interop in newer languages. So, at some point, some developers at Microsoft decided to use the new way and assigned the integer value of 1 to True for some new functionality. Such as FitText.

Considering the following code sample, where X is of type Boolean and y of type Integer:

• If FitText is True, the integer value is 1
• If reversing the values, using Not shows that the Boolean remains "True" because its integer value is not 0, it's -2
• Setting the integer value directly to True gives -1

This is confusing, indeed, but does explain why Not is not giving the expected result.

Sub testChangeBoolean()
  Dim X As Boolean                  ' again, default is False
  Dim Y As Integer
  X = Selection.Cells(1).FitText    ' does set X to True
  Y = Selection.Cells(1).FitText
  Debug.Print X, Y                  ' result: True    1
  X = Not X                         ' X is still True!
  Y = Not Y
  Debug.Print X, Y                  ' result: True   -2
  X = False
  Y = True
  Debug.Print X, Y                  ' result: False  -1
End Sub

To add on to Cindy's excellent answer, I want to point out that while VBA normally has safeguards to coerce the values when assigning to a Boolean data type, this can be circumvented. Basically, if you write a random value to a memory address that's not yours, then you should expected undefined behavior.

To help demonstrate this, we'll (ab)use LSet which essentially allow us to copy the value without actually assigning.

Private Type t1
  b As Boolean
End Type

Private Type t2
  i As Integer
End Type

Private Sub Demo()
  Dim i1 As t2
  Dim b1 As t1
  Dim b As Boolean

  i1.i = 1

  LSet b1 = i1

  b = b1.b

  Debug.Print b, b1.b, i1.i
  Debug.Print CInt(b), CInt(b1.b), i1.i

End Sub

Note the line b = b1.b is basically equivalent to what we did in the OP code

X = Selection.Cells(1).FitText

That is, assigning a Boolean to another Boolean. However, because I wrote to the b1.b using LSet, bypassing VBA runtime checks, it doesn't get coerced. When reading the Boolean, VBA does implicitly coerce it into either True or False, which seems misleading but is correct because any falsy results is one that equals 0 (aka False), and any truthy results is one that doesn't. Note that the negative for truthy means that both 1 and -1 are truthy.

Had I assigned the 1 to a Boolean variable directly, VBA would have had coerced it into -1/True and thus there'd be no problem. But evidently with FitText or LSet, we are basically writing to the memory address in an uncontrolled fashion, so that VBA start to behave strangely with this particular variable since it expects the Boolean variable to already had its contents coerced but wasn't.