WPF: System.ArgumentException => {"'{0}' is not a Visual or Visual3D."}

That indeed seems to be a bug. I ran your repro project and checked out the call stack when the exception is thrown. It happens in DataGridCell.RemoveBindingExpressions during a call to VisualTreeHelper.IsAncestorOf. The latter method throws an exception when it is passed an object that is not Visual or Visual3D. But DataGridCell is passing it whatever element is the target of the binding. In your case that happens to be a Run which does not derive from Visual.

I was thinking you might be able to work around it by using an IValueConverter to create the FlowDocument and binding RichTextBox.Document so that the binding is being applied to the RichTextBox. But since Document isn't a dependency property, it can't be a target of binding.

So instead what you might want to do is create a UserControl that hosts the RichTextBox control:

<DataGridTemplateColumn.CellTemplate>
    <DataTemplate>
        <Local:HomeworkControl Text="{Binding Homework}" />
    </DataTemplate>
</DataGridTemplateColumn.CellTemplate>

Then in that user control you would take care of building the RichTextBox, document, run, etc. Unfortunately I think this is just a limitation (aka bug) in the DataGrid control.

Interestingly this happened to me as well. What Josh said got me thinking. It seems like once you select the cell and select it again it tries to load the CellEditingTemplate which is not specified in my case and yours and it throws the Visual/Visual3d exception.

I got it fixed by specifying IsReadOnly="True" on my DataGridTemplateColumn. I don't use the CellEditingTemplate anyway because I am doing bulk inserts with TextBoxes/DatePicker/Checkboxes etc. loaded in the cell templates.

I had the same problem with a Datagrid with a custom column with a Hyperlink with embedded run, with the binding set on the Run's Text property. When the run Text binding was not explicitly set to be BindingMode.OneWay I got this error. Setting it explicitly solved the problem. Note I got the exception when editing ANY columns in the datagrid not just this one.