Select layer by attribute using SQL statement LEN() in ArcMap?

To calculate the length in characters of a string expression use CHAR_LENGTH(string_exp)

Field name delimiters for file geodatabase is double quotes (not square brackets).

So this should work:

CHAR_LENGTH(“ColumnName“) > 5

Here is a SQL reference for query expressions used in ArcGIS.