What do parentheses in binding paths mean?

The () parentheses refer to Attached Properties.

Binding to an Attached Property

Quoting the MSDN library (I'm quoting MSDN here because I couldn't have written it down better):

This syntax is generally used for one of the following cases:

• The path is specified in XAML that is in a style or template that does not have a specified TargetType. A qualified usage is generally not valid for cases other than this, because in non-style, non-template cases, the property exists on an instance, not a type.
• The property is an attached property.
• You are binding to a static property.

For use as storyboard target, the property specified as propertyName must be a DependencyProperty.

(Validation.Errors) references the attached property Errors in the Validation class. Since the binding has a RelativeSource = Self, it's gonna look for the value of that attached property with respect to the TextBox itself.