How AV software makes sure quarantined files never get executed?

There are a couple ways an antivirus program might go about preventing the execution of a quarantined file:

• Hold it open for exclusive access. It can simply use the normal CreateFile API and pass zero for dwShareMode, preventing all other kinds of access to the file. If an executable can't be read, it can't be executed. Since antivirus programs are usually one of the first things to start, they can reopen all such files when the system reboots.
• Use Software Restriction Policies. When the antivirus program is installed, it could create an SRP that blocks the execution of programs in the quarantine directory.
• Hook a function that's critical for the creation of a process. Process starting always involves NtCreateSection, so the antivirus program's kernel-mode driver could hook that and determine whether the file in question is an executable and check whether PAGE_EXECUTE is requested.

Of these, only the first is useful for non-programs (e.g. documents that may contain exploits). To keep an eye on all file loads, the antivirus program could hook NtCreateFile, since no files can be read without invoking that. Alternatively, to watch for the marking of pages as executable, it could hook NtProtectVirtualMemory. Either way, the kernel-mode driver could look around a bit at what's going on in userspace before allowing the operation to proceed.

The easiest solution, however, would be to encrypt the quarantined files with a key that's not stored on disk. With such a system, the user would have to type a password to decrypt and un-quarantine the files. This could be accomplished fairly easily with some asymmetric-key cryptography.

The quarantine AV features mentioned in the reply before make sure that the file cannot be executed or accessed while under quarantine. So no, as long as the antivirus is working, there is no danger for the virus to escape quarantine. Practically, some store the file in a different non-runnable format and some just rename the file and deny read/write.execution access to them.

A simple example: you get a standard file with ASCII .exe content, convert that content to binary and re-write it as zeros and 1s into a text file. Nothing can go wrong from there. There's nothing to escape since the file is now binary content saved as text.

-Edit-

So the elements you asked about are Deny R/W/X access and encode the file into anything else but the original format.

Suppose you have an important file(say, some presentation) that is flagged by your antivirus as "malicious". You do not wish to delete it(it's important after all, even if it is malicious!), but you don't want your machine to get infected as well.

So, what option do you have? Your antivirus product can quarantine it. It keeps the files in an isolated environment on your disk, so that malicious files do not pose a threat to your machine.

There are different approaches as to how these files can be handled: 1) moving the file to a different location 2) renaming the file, changing the format of the file 3) changing the permissions of the file, marking it as hidden 4) encrypting or encoding the file, so that even accessing the file would render the malicious code useless

Now what if you want the file back, even after you understand that it might wreak havoc, or might be just a false-positive? Antivirus products give you the option to move the quarantined file from quarantine to the original location, in the original format, so it is usable again for you(at your own risk, of course!).

If you strongly suspect that your file is harmless, you can try scanning it with other antivirus(say, on Virustotal). You can also ask your antivirus product vendor to recheck the file for any malicious behavior. Once you're satisfied that it is harmless, you can safely go ahead and restore the file from quarantine.

And yes, you can rely on quarantine features of any reputed antivirus product.