Binding a Button's visibility to a bool value in ViewModel

Assuming AdvancedFormat is a bool, you need to declare and use a BooleanToVisibilityConverter:

<!-- In your resources section of the XAML -->
<BooleanToVisibilityConverter x:Key="BoolToVis" />

<!-- In your Button declaration -->
<Button
 Height="50" Width="50"
 Style="{StaticResource MyButtonStyle}"
 Command="{Binding SmallDisp}" CommandParameter="{Binding}" 
Cursor="Hand" Visibility="{Binding Path=AdvancedFormat, Converter={StaticResource BoolToVis}}"/>

Note the added Converter={StaticResource BoolToVis}.

This is a very common pattern when working with MVVM. In theory you could do the conversion yourself on the ViewModel property (i.e. just make the property itself of type Visibility) though I would prefer not to do that, since now you are messing with the separation of concerns. An item's visbility should really be up to the View.

There's a third way that doesn't require a converter or a change to your view model: use a style:

<Style TargetType="Button">
   <Setter Property="Visibility" Value="Collapsed"/>
   <Style.Triggers>
      <DataTrigger Binding="{Binding IsVisible}" Value="True">
         <Setter Property="Visibility" Value="Visible"/>
      </DataTrigger>
   </Style.Triggers>
</Style>

I tend to prefer this technique because I use it in a lot of cases where what I'm binding to is not boolean - e.g. displaying an element only if its DataContext is not null, or implementing multi-state displays where different layouts appear based on the setting of an enum in the view model.

2 way conversion in c# from boolean to visibility

using System;
using System.Windows;
using System.Windows.Data;

namespace FaceTheWall.converters
{
    class BooleanToVisibilityConverter : IValueConverter
    {
        public object Convert(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
        {
            if (value is Boolean && (bool)value)
            {
                return Visibility.Visible;
            }
            return Visibility.Collapsed;
        }

        public object ConvertBack(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
        {
            if (value is Visibility && (Visibility)value == Visibility.Visible)
            {
                return true;
            }
            return false;
        }
    }
}