Unknown process on my ubuntu machine communicating over multiple ports with UDP proto

So can anyone give me a heads up of what has happened

You got hacked. But it can just be anything (SQL injection, remote file inclusion, Shellshock...) and it is impossible to tell without having a close look at the system. And even then it might be impossible to tell because the attacker might have removed any traces how the system was attacked.

Btw I use the iptables-persistent package...

A packet filter firewall like iptables is not able to protect you against application level attacks like SQL injection or similar.

One is an executable called facebook, while the other is called gameover.so (a dynamically linked library?).

Names of files do not matter much, code matters.

...and what I can do to protect against this?

Find out how you got hacked. Fix the problems. Rebuild the server. Don't just restore the server from backup because then you will be probably hacked shortly again with the same attack.

If you are unable to do it yourself get professional help.

Edited:

You seem to be victim of a SQL injection. There is a script (PHP probably) receiving data from web and querying database without enough sanitizing input.

Before you kill 63911, have a look at ps ho lstart 63911 !!

This will let you search in your web server logs what's happened just before process 63911 was run!

First post

1. What's /tmp/facebook ??

   A) you could strace -p 63911 to have an idea of what this do...

   B) you could tcpdump -ani eth0 udp port 51802 to randomly follow connections.

   C) you could use less /tmp/facebook, or if it's a binary file, strings /tmp/facebook | less (on /tmp/gameover.so too, of course).

2. What about /sbin/init?

   Build another host with the same version of the same distribution and compare size and date (with ls -l) and sha1sum /sbin/init.

3. And again: keep your system up to date! And have backups!!

Note: If your system is strongly (and efficiently) infected, you have to shut it down, before doing checksums, string or even ls. But as netstat seems to work without hiding anything, that seems not to be the case...

This answer addresses, in particular:

what I can do to protect against these attacks in the future

as identifying the actual original infection vector is very difficult, and if there were multiple vulnerabilities - which is likely - insufficient in and of itself. Start with general hardening of the entire set of systems, and add in logging (IDS/IPS) that both can catch infections as they happen, and can block them while they're being attempted.

To really improve, block both directions and start running an IDS/IPS; don't forget to look at those logs, which will be able to help identify the NEXT infection vector.

I recommend you do the following:

• Unplug your firewall from the Internet in addition to your machines:

  • Buy a new one

  • Or give it a hard reset, and use a non-compromised machine on another network to download any firmware updates for it. If there aren't any because it's old... see "Buy a new one"

    • If they were able to use a computer inside your LAN, they were able to try to, or succeed at, logging into your firewall.

    • Also check to see if your firewall has one of many vulnerabilities; if so, see "Buy a new one"

    • Buy a serious hardware router/firewall, like running pfSense free software on an old machine with 2 NICs, or their own devices, or a fitlet tiny fanless PC, or whatever.

      • Then install the Snort or Suricata package - those are IDS/IPS packages which have a chance of noticing attacks in progress and optionally blocking IPs for a period of time - in your case, I'd say block for a few days and watch your logs very carefully for them to try and get back in.

      • And buy a subscription to the Snort VRT rules; consider the ET Pro rules as well, though they're more expensive.

      • Block EVERYTHING; you'll have massive blocking at first; whitelist on a per-IP*rule basis, and watch them keep trying to get back in.

    • Or at least a Ubiquiti Edgerouter Lite - much cheaper, but very limited in the GUI (all the really advanced stuff is in the command line, and it's definitely not for running an IDS/IPS).

    • And set it up to block EVERYTHING, both inbound and outbound. Open up individual outbound ports on individual IPs or small subnets/blocks as required (for OS updates, for example).

      • This WILL be very annoying - make use of aliases in pfSense, if you go that route

      • This will also seriously restrict the ability of anything inside the firewall to get out to command and control servers.

  • If possible, put the web server and the database server on entirely different VLAN's or firewall ports, so they can only talk to each other via the database connection, one-way, with encryption enabled.

• Make CERTAIN all your software's fully up to date, and kept there

• Seek out hardening guides for every piece of software in the mix.

  • And follow them.

• Change all your passwords on the firewall, servers, database, routers, etc.

  • Use something like KeePass to generate long random ones for each service.

  • Don't forget to go to Database Settings, Security, hit the "1 second delay" link or button, and then increase it - taking 3 or 4 or 7 seconds to open KeePass is a small price to pay for making attackers have to work incredibly much harder than the defaults.

• Keep the machines off while you:

  • Buy a new hard drive

  • Plug in power but NOT networking (and especially not Wifi; physically remove/turn off all wifi)

  • Install a fresh OS from scratch

  • Set up your software and OS/app level security, and patch it.

  • If the drives in your compromised machines are solid state, take them apart and crack the chips, or apply a torch to them until they melt.

    • Or perform another physical destruction method that protects whatever important data you have left on there, and prevents the drive from being used again.

  • If the drives in your compromised machines have spinning platters, purchase a sledgehammer, a box of gallon Ziplock bags, and a small kitchen towel.

    • Wrap the compromised hard drives in the small kitchen towel

    • Put the wrapped HD in at least 6 Ziplock bags, each sealed

    • Smash the drive with the sledgehammer until glass shards are leaking through the towel, OR it's got significant dents in it (in the case you have metal platters).

    • Restore your valuable data from existing backups

• Or, much riskier if you don't replaced the compromised HD, wipe it with DBAN or similar and start over

  • If you have any Windows machines on the same network, do several offline virus scans.

    • Use several because no one product covers everything, but by using several different products, you reduce the uncovered space significantly. I'd recommend:

    • at least one of AVG and Avira (or both)

    • at least one of Dr. Web and Kapersky (preferably both), to get some Russian involvement.

    • Comodo Rescue Disk (it advertises rootkit scanning, too)

    • Pick another couple of your favorites.

    • PCSupport.about.com has a pretty good top 15 list as of Feb 2016

    • If you're truly worried, do this on every machine on the network.

      • It won't really take much more time - you can put AVG in one, Avira in another, Kapersky in a third, Dr.Web in a fourth, and then simply move them to the machine on the right when they're done in round-robin fashion.

    • They're almost certainly going to find tracking cookies - that's normal, and nothing to be concerned about, though I'd always delete them.

After all of this, once you're fully online again

• Make sure your OS is patched, and stays patched.

• Read your Snort/Suricata logs regularly, and tune the settings.

• pfSense and the Edgerouter Lite can both set up VPNs of a variety of types. Use a certificate based one, regardless, for any inbound access other than to your webserver

• Make sure your (new) firewall is patched and up to date.

• Make sure all other software is patched and up to date.

• Forever.