How to zero (or replace) the diagonal of a square matrix?

As requested, posting my comment as an answer:

UpperTriangularize[arg, 1] + LowerTriangularize[arg, -1]

seems to meet all the criteria, quite quick (surprisingly so to me).

The following works for a numeric matrix, should be OK for symbolic ones

exmat = {{0, 5, 2, 3, 1, 0}, {4, 3, 2, 5, 1, 3}, {4, 1, 3, 5, 3, 2}, 
       {4, 4, 1, 1, 1, 5}, {3, 4, 4, 5, 3, 3}, {5, 1, 4, 5, 2, 0}}; 

MatrixForm[ReplacePart[exmat, {i_, i_} -> 0]]



(*   0   5   2   3   1   0

    4   0   2   5   1   3

    4   1   0   5   3   2

    4   4   1   0   1   5

    3   4   4   5   0   3

    5   1   4   5   2   0 *)

Method one:LinearAlgebra`SetMatrixDiagonal

mat = RandomReal[10, {3, 3}];
LinearAlgebra`SetMatrixDiagonal[mat,Array[0 &, Length[mat]]] // MatrixForm

Developer`PackedArrayQ[mat]

True

Method two:LinearAlgebra`AddVectorToMatrixDiagonal

mat = RandomReal[10, {3, 3}];
LinearAlgebra`AddVectorToMatrixDiagonal[mat, -Diagonal[mat]] // MatrixForm

Developer`PackedArrayQ[mat]

True

ps:Note this two method will change the original mat.

Compare with ciao's answer here

arg = RandomReal[10, {10^4, 10^4}];
LinearAlgebra`SetMatrixDiagonal[arg,Array[0 &, Length[arg]]]; // AbsoluteTiming

{0.406152, Null}

arg = RandomReal[10, {10^4, 10^4}];
AbsoluteTiming[LinearAlgebra`AddVectorToMatrixDiagonal[arg, -Diagonal[arg]];]

{0.356579, Null}

arg = RandomReal[10, {10^4, 10^4}];
AbsoluteTiming[UpperTriangularize[arg, 1] + LowerTriangularize[arg, -1];]

{1.41651, Null}