using custom deleter with unique_ptr

Should be

unique_ptr<FILE, int(*)(FILE*)>(fopen("file.txt", "rt"), &fclose);

since http://en.cppreference.com/w/cpp/memory/unique_ptr

or, since you use C++11, you can use decltype

std::unique_ptr<FILE, decltype(&fclose)>

The above answer while its intent is OK and in practice compiles and works is wrong, because it is not specified that you are allowed to take the address of a standard library function. A C++ library implementation is allowed to provide different overloads or more parameters (with default arguments). Only calling the library function is sanctioned by the standard. Therefore, you need to wrap the call to fclose in your own function implementation or lambda, such as

unique_ptr<FILE, int(*)(FILE*)>(fopen("file.txt", "rt"),
   [](FILE *fp)->int{ if(fp) return ::fclose(fp); return EOF;});

or wait for unique_resourceof https://wg21.link/p0052 to become standardized, but even there you need to use the lambda or a deleter function (object), see the more recent versions of p0052.

Note that in a real program you may want to check and act upon the return value of fclose, which could be awkward from within a destructor: you don't get to return a value and throwing exceptions from destructors is a bad idea. Similar considerations may or may not apply for other types of pointer.

With that caveat out of the way, an alternative approach would be to specify the deleter as a functor:

struct file_deleter {
    void operator()(std::FILE* fp) { std::fclose(fp); }
};

using unique_file = std::unique_ptr<std::FILE, file_deleter>;

The type alias allows you to simply write:

unique_file f{ std::fopen("file.txt", "rt") };

This is more ergonomic than having to pass an additional pointer or a lambda every time you create a pointer. The use of a functor type also means that the unique_ptr does not have to carry around a separate pointer for the deleter, which allows for space savings relative to the other approaches. To see this, I use the following code:

int main()
{
    std::unique_ptr<FILE, decltype(&fclose)> f1{ nullptr, &fclose };
    std::unique_ptr<std::FILE, void(*)(std::FILE*)> f2{
        nullptr, [](std::FILE* p) { std::fclose(p); } };
    unique_file f3{ nullptr };
    std::FILE* f4{ nullptr };

    std::cout << "sizeof(f1) = " << sizeof(f1) << '\n';
    std::cout << "sizeof(f2) = " << sizeof(f2) << '\n';
    std::cout << "sizeof(f3) = " << sizeof(f3) << '\n';
    std::cout << "sizeof(f4) = " << sizeof(f4) << '\n';
}

Using MSVC building for an x64 target, I get the following output:

sizeof(f1) = 16
sizeof(f2) = 16
sizeof(f3) = 8
sizeof(f4) = 8

In this specific implementation, for the case using the functor the unique_ptr is the same size as a raw pointer, which is not possible for the other approaches.