How to validate file type of HttpPostedFileBase attribute in Asp.Net MVC 4?

Because your attribute inherits from and existing attribute, it would need to be registered in global.asax (refer this answer for an example), however do not do that in your case. Your validation code does not work, and a file type attribute should not be inheriting from RequiredAttribute - it needs to inherit from ValidationAttribute and if you want client side validation, then it also needs to implement IClientValidatable. An attribute to validate file types would be (note this code if for a property which is IEnumerable<HttpPostedFileBase> and validates each file in the collection)

[AttributeUsage(AttributeTargets.Property, AllowMultiple = false, Inherited = true)]
public class FileTypeAttribute : ValidationAttribute, IClientValidatable
{
    private const string _DefaultErrorMessage = "Only the following file types are allowed: {0}";
    private IEnumerable<string> _ValidTypes { get; set; }

    public FileTypeAttribute(string validTypes)
    {
        _ValidTypes = validTypes.Split(',').Select(s => s.Trim().ToLower());
        ErrorMessage = string.Format(_DefaultErrorMessage, string.Join(" or ", _ValidTypes));
    }

    protected override ValidationResult IsValid(object value, ValidationContext validationContext)
    {
        IEnumerable<HttpPostedFileBase> files = value as IEnumerable<HttpPostedFileBase>;
        if (files != null)
        {
            foreach(HttpPostedFileBase file in files)
            {
                if (file != null && !_ValidTypes.Any(e => file.FileName.EndsWith(e)))
                {
                    return new ValidationResult(ErrorMessageString);
                }
            }
        }
        return ValidationResult.Success;
    }

    public IEnumerable<ModelClientValidationRule> GetClientValidationRules(ModelMetadata metadata, ControllerContext context)
    {
        var rule = new ModelClientValidationRule
        {
            ValidationType = "filetype",
            ErrorMessage = ErrorMessageString
        };
        rule.ValidationParameters.Add("validtypes", string.Join(",", _ValidTypes));
        yield return rule;
    }
}

It would be applied to a property as

[FileType("JPG,JPEG,PNG")]
public IEnumerable<HttpPostedFileBase> Attachments { get; set; }

and in the view

@Html.TextBoxFor(m => m.Attachments, new { type = "file", multiple = "multiple" })
@Html.ValidationMessageFor(m => m.Attachments)

The following scripts are then required for client side validation (in conjunction with jquery.validate.js and jquery.validate.unobtrusive.js

$.validator.unobtrusive.adapters.add('filetype', ['validtypes'], function (options) {
    options.rules['filetype'] = { validtypes: options.params.validtypes.split(',') };
    options.messages['filetype'] = options.message;
});

$.validator.addMethod("filetype", function (value, element, param) {
    for (var i = 0; i < element.files.length; i++) {
        var extension = getFileExtension(element.files[i].name);
        if ($.inArray(extension, param.validtypes) === -1) {
            return false;
        }
    }
    return true;
});

function getFileExtension(fileName) {
    if (/[.]/.exec(fileName)) {
        return /[^.]+$/.exec(fileName)[0].toLowerCase();
    }
    return null;
}

Note that you code is attempting to also validate the maximum size of the file which needs to be a separate validation attribute. For an example of a validation attribute that validates the maximum allowable size, refer this article.

In addition, I recommend The Complete Guide To Validation In ASP.NET MVC 3 - Part 2 as a good guide to creating custom validation attributes

Be aware that EndsWith is case-sensitive by default. So I would change this:

if (file != null && !_ValidTypes.Any(e => file.FileName.EndsWith(e)))

to this

if (file != null && !_ValidTypes.Any(e => file.FileName.EndsWith(e, StringComparison.OrdinalIgnoreCase)))