What did the system call "tuxcall" do?

tuxcall is the place-holder for the tux system call which was used by user-space tools to communicate with the TUX kernel module, which implemented the TUX web server. This was a web server running entirely in the kernel; it was maintained by Ingo Molnar until improvements in other parts of Linux, notably thread support with NPTL, brought user-space web server performance up to the level attained by TUX.

You can still find the TUX 3 patches for Linux 2.6.18 among Ingo’s patches, including the implementation of sys_tux (the system call in question). The user-space portion, which includes the documentation, can be found on the Wayback Machine (thanks hvd!).

I've found something from non-standard-syscalls:

tuxcall() - This call comes from a TUX module and is sent to the kernel. The call asks the kernel to perform some task for the module. A TUX module is basically a server application/daemon in the form of a Linux module. Imagine an Apache server being a kernel module; that is essentially how TUX works.