How can I evaluate untrusted Mathematica code in a sandbox?

You should consider using the sandbox functionality. You can create a subkernel and put it in sandbox mode this way:

link = LinkLaunch[First[$CommandLine]<> " -wstp -noicon"];
LinkWrite[link, Unevaluated@EvaluatePacket[Developer`StartProtectedMode[]]];

You can then interact with this subkernel using the standard LinkWrite and LinkRead functions. If you don't mind your master kernel being sandboxed, you can even just evaluate Developer`StartProtectedMode[] there, but it disables a lot of functionality (mostly import/export and file system manipulation).

Note that sandbox mode also will only allow you to load .m/.wl files from very specific directories. You can set this in the call itself as well:

Developer`StartProtectedMode[{"Read" -> {$myPath}, "Write" -> {$myPath}, "Execute" -> {$myPath}}]

where $myPath is the path to where you store the code you wish to interact with.

I have been solving exactly the same problem about 2 years ago (http://community.wolfram.com/groups/-/m/t/125587?p_p_auth=aZGMz5bs). Students are uploading piece of Mathematica (Wolfram Language) code which is run by a testing script (in Mathematica) and the results are compared with a reference solution.

To prevent the students to run potentially dangerous code I am using this initialization part (maybe I have missed some functions):

ClearAttributes[{Run,RunThrough,CopyFile,DeleteFile,RenameFile,CreateDirectory,CopyDirectory,DeleteDirectory,RenameDirectory,OpenRead,OpenWrite,Read,Write,BinaryRead,BinaryWrite},Protected]
Run[___]:=$Failed;
RunThrough[___]:=$Failed;
CopyFile[___]:=$Failed;
DeleteFile[___]:=$Failed;
RenameFile[___]:=$Failed;
CreateDirectory[___]:=$Failed;
CopyDirectory[___]:=$Failed;
DeleteDirectory[___]:=$Failed;
RenameDirectory[___]:=$Failed;
OpenRead[___]:=$Failed;
OpenWrite[___]:=$Failed;
Read[___]:=$Failed;
Write[___]:=$Failed;
BinaryRead[___]:=$Failed;
BinaryWrite[___]:=$Failed;
SetAttributes[{Run,RunThrough,CopyFile,DeleteFile,RenameFile,CreateDirectory,CopyDirectory,DeleteDirectory,RenameDirectory,OpenRead,OpenWrite,Read,Write,BinaryRead,BinaryWrite},Protected]

To detect if the testing script finished correctly I am using predefined exit code (different than the default 0). Example:

Exit[123 (* some secret code *)]

This code is tested from the "top-level" bash script which runs the Mathematica. If the student uses Quit command in his solution, it will skip my Exit with exit code and the whole test will fail.

An example how to run the testing script and test the results (exit codes):

timeout -s 9 10 math -noprompt -run < ./tester.wl >./results.txt
RET=$?

# use the same "secret code"
if [ $RET -eq 123 ]
then
# OK
fi

if [ $RET -eq 137 ]
then
# TIMEOUT
fi

I am using the timeout command, because the TimeConstrained function in Mathematica worked unreliably.

i am doing the same thing as you did，and i am using MSP module which is a security solution for webMathmatica。please refer my topic How to adapt MSPToExpression function in $PrePrint?

sandbox seems another good solution for security，mathematica online is using it。 can i ask whether you solved this by using sandbox？