Is it possible to do a traceroute in the browser?

Why don't you just sign the applet? Isn't the problem actually more you don't know how to sign the applet? If so, then start here: jarsigner. Here is a more clear tutorial.

There is actually no simpler/better solution than actually running some piece of code and/or commands at the client machine. The traceroute really have to originate at the client machine.

Javascript and Actionscript cannot do this due to security restrictions. They lives in the webpage context only. Silverlight might be able to do, but don't pin me on that. I don't do NET stuff.

You can't do this at all from a browser. Javascript can at best open a connection back to its originating server for AJAX requests, but can only do so via HTTP. Flash can talk to arbitrary hosts, but only if they're listed in a crossdomain.xml file on the originating server, and again only via TCP. UDP support in Flash is apparently pending.

Traceroute and ping are both ICMP-based protocols and cannot be created/controlled from Flash or Javascript. They also both require 'raw' access to build custom packets, and this definitely cannot be done browser-side. This is why 'ping' is an 'SUID' program on Unix systems, as raw packet access requires root privileges.

At best you can do a server-side implementation and have the output sent to the browser. And even then, you most likely could not do it from an in-server process on a Unix box, as the web server is unlikely to be running as root. You'd have to execute the system ping and/or traceroute and redirect the output back to the browser.