Is there any way to validate a filename?

So, this question is a bit old but in case you're looking: With XE and newer you can use the TPath class in System.IOUtils.

Result := TPath.HasValidFileNameChars(AFileName, UseWildcards); 

As far as I know, there is not an RTL or Windows API function to validate a filename, so you must write your own function following the Windows File Naming Conventions:

The following fundamental rules enable applications to create and process valid names for files and directories, regardless of the file system:

  • Use a period to separate the base file name from the extension in the name of a directory or file.
  • Use a backslash () to separate the components of a path. The backslash divides the file name from the path to it, and one directory name from another directory name in a path. You cannot use a backslash in the name for the actual file or directory because it is a reserved character that separates the names into components.
  • Use a backslash as required as part of volume names, for example, the "C:\" in "C:\path\file" or the "\server\share" in "\server\share\path\file" for Universal Naming Convention (UNC) names. For more information about UNC names, see the Maximum Path Length Limitation section.
  • Do not assume case sensitivity. For example, consider the names OSCAR, Oscar, and oscar to be the same, even though some file systems (such as a POSIX-compliant file system) may consider them as different. Note that NTFS supports POSIX semantics for case sensitivity but this is not the default behavior. For more information, see CreateFile.
  • Volume designators (drive letters) are similarly case-insensitive. For example, "D:\" and "d:\" refer to the same volume.
  • Use any character in the current code page for a name, including Unicode characters and characters in the extended character set (128–255), except for the following:

    • The following reserved characters:
      • < (less than)
      • > (greater than)
      • : (colon)
      • " (double quote)
      • / (forward slash)
      • \ (backslash)
      • | (vertical bar or pipe)
      • ? (question mark)
      • * (asterisk)
    • Integer value zero, sometimes referred to as the ASCII NUL character.
    • Characters whose integer representations are in the range from 1 through 31, except for alternate streams where these characters are allowed. For more information about file streams, see File Streams.
    • Any other character that the target file system does not allow.
  • Use a period as a directory component in a path to represent the current directory, for example ".\temp.txt". For more information, see Paths.
  • Use two consecutive periods (..) as a directory component in a path to represent the parent of the current directory, for example "..\temp.txt". For more information, see Paths.
  • Do not use the following reserved device names for the name of a file:

    CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9. Also avoid these names followed immediately by an extension; for example, NUL.txt is not recommended. For more information, see Namespaces.

  • Do not end a file or directory name with a space or a period. Although the underlying file system may support such names, the Windows shell and user interface does not. However, it is acceptable to specify a period as the first character of a name. For example, ".temp".

you can check this C++ article Validating file names for an complete example function to validate the windows file names.