Send data over telnet without pressing enter

There actually is a way for the server to request this: It's called telnet option negotiation. Typically telnet will default to configure the local tty in "raw" mode when you use port 23 and "cooked" (or "line") mode on other ports. Line mode is where you have minimalistic local editing and the data is sent when you hit return.

Once you disable linemode you can separately configure things like local echo.

EDIT: I think a reasonable sequence would be:

255, 253, 34,  /* IAC DO LINEMODE */
255, 250, 34, 1, 0, 255, 240 /* IAC SB LINEMODE MODE 0 IAC SE */
255, 251, 1    /* IAC WILL ECHO */

That enables TELOPT_LINEMODE (34) and then sets the linemode LM_MODE to 0 (which I think is the right way to tell the client not do to any local editing). Finally it says WILL ECHO indicating the server will echo (so the client will not).

The client (if it supports telnet negotiation) will reply with sequences like IAC blah blah or "quoted" sequences like IAC SB ... IAC SE which you can detect and filter out of your input stream.

You should be able to do this with telnet option negotiation. The protocol defaults to half-duplex mode, and at a minimum for an interactive session, the server should negotiate the suppress go ahead option and echo option.

At the bare minimum you could just spit out ff fb 01 ff fb 03 (will echo, will suppress-go-ahead) at the begining of the session, then reply to any ff fd 01 (do echo) with ff fb 01 (will echo) and reply to any ff fd 03 (do suppress-go-ahead) with ff fb 03 (will suppress-go-ahead).

Edit to add that the linemode negotiation mentioned by Ben Jackson is a better answer. Suppress go-ahead won't be enough for most clients connecting on ports other than 23.

However I think the other problem you're running into is that Java is sending Unicode characters. For example, when you say (char)0xff, Java assumes you're referring to UTF-16 character U+00ff which is ÿ. It's probably sending it over the socket using UTF-8 encoding, so the telnet client sees two bytes: c3 bf which it passes on and displays as ÿ.

What you can do is explicitly tell Java to use ISO-8859-1 encoding. For example, you may have been doing something like this before:

out = new PrintStream(connection.getOutputStream());
out.print((char)0xff);  // sends 0xc3 0xbf
out.print((char)0xfb);  // sends 0xc3 0xbb
out.print((char)0x01);  // sends 0x01
out.flush();

Instead, you can use the OutputStreamWriter to specify the encoding you want:

out = new OutputStreamWriter(connection.getOutputStream(), "ISO-8859-1");
out.write((char)0xff);  // sends 0xff
out.write((char)0xfb);  // sends 0xfb
out.write((char)0x01);  // sends 0x01
out.flush();