when to use {x:Type …}?

There is no difference in effect; in both cases the TargetType property will be set to typeof(Border)

The first version {x:Type Border} was needed in the first version of WPF because the compiler did not use the TypeConverter class to convert the string into a Type object and you needed to specify the TypeExtension class to do that for you.

The second version was introduced, if I remember correctly, with Silverlight and quickly found its way to the WPF compiler.

EDIT

My assumption on the TypeConverter class was wrong; this is implemented by the FrameworkElementFactory:

From the documentation:

Type Properties That Support Typename-as-String

WPF supports techniques that enable specifying the value of some properties of type Type without requiring an x:Type markup extension usage. Instead, you can specify the value as a string that names the type. Examples of this are ControlTemplate.TargetType and Style.TargetType. Support for this behavior is not provided through either type converters or markup extensions. Instead, this is a deferral behavior implemented through FrameworkElementFactory.

Silverlight supports a similar convention. In fact, Silverlight does not currently support {x:Type} in its XAML language support, and does not accept {x:Type} usages outside of a few circumstances that are intended to support WPF-Silverlight XAML migration. Therefore, the typename-as-string behavior is built-in to all Silverlight native property evaluation where a Type is the value.

Although in the given example it makes no difference but actually there is difference between x:Type and TypeName-as-String.

I have recently encountered a situation which shows that x:Type is different from TypeName-as-String when it comes to custom types. From my experience -

x:Type considers the strong name or the version of the assembly (in which type resides) but not TypeName-as-String.

I have explained about my scenario and other details in my blog here -

Importance of specifying AncestorType with x:Type in RelativeSourceBinding

Apart from this, there is also difference in how WPF infers the type. For x:Type TypeExtension is used, whereas for TypeName-as-String FrameworkElementFactory is used (as Erno mentioned).

Setting this property (TargetType) to Border without assigning the style with an x:Key allows the style to be applied to all Border elements. But when you sets the x:Key to {x:Type Border}, This means that if you give the Style an x:Key value of anything other than {x:Type Border}, the Style would not be applied to all Border elements automatically. Instead, you need to apply the style to the Border elements explicitly.