What is the best home wireless network encryption algorithm to use?
From a security perspective, I think you are asking the wrong question. WPA2 is the basic answer. But it's entirely incomplete! A more complete answer will view WPA2 as one component of your wireless network defence. Of course there's strong encryption methods using certificates/vpn etc but these are too difficult for most people to set up and are usually reserved for businesses. So let's assume WPA-2 is the 'best' answer to the basic question. However... as you'll see, there's many weaker points that attackers go for, that ultimately reveal your WPA2 password, so I've included them in the points below.
I'm assuming many people will land on this page and see answers saying 'yeah just use a good password and WPA2 encryption', which is bad advice. Your WPA2 network is still completely vulnerable, as you will see:
the main thing you can do, is be the hardest person to hack around you. That's the biggest deterrent. If I'm going to hack you, but you're taking too long or are too expensive to crack, I'll try the next person. This will require some playing around in your router settings.
I'll assume you would never use WEP. 10 minutes on youtube and your mom can crack it.
Switch off WPS. this is EXTREMELY vulnerable to brute force attacks and can be hacked in seconds, even if you are using WPA2 with a ridiculously complex password. Tools like reaver and revdk3 or bully make light work of these. You're only a little bit more protected if your router supports rate-limiting, which slows down, but doesn't prevent brute force attacks against your routers pin. Better to be safe and just switch WPS off and be 100% safe against these attacks.
turn off remote access, DMZ, UPNP, unecessary port forwarding
turn on, any inbuilt intrusion detection systems, MAC address filtering (tedious to set up if visitors to your house want access to your wifi (you will have to add your friends device to the router's MAC white-list to enable access) This can be hacked by faking a MAC address easily, and getting your MAC is also easy with an airodump-ng scan, but nevertheless, this will slow down attackers, requires them to be near a client device (mobile phone, or laptop in the whitelist) It will be pretty effective against some remote attacks.
have a very long, non-human, complex password. If you have ever tried to decrypt a password you'll know that it gets exponentially harder to crack a password the more complex, less predictable and longer it is. If your password even remotely resembles a word, or something that could probably be a set of words (see: markov chains) you are done. Also don't bother adding numbers to the end of passwords, then a symbol... these are easily hacked with a dictionary attack with rules that modify the dictionary to flesh it out to cover more passwords. This will take each word or words in the dictionary, and add popular syntax and structures, such as passwords that look like this 'capital letter, lowercase letters, some numbers then a symbol. Cat111$, Cat222# or whatever the cracker wants. These dictionaries are huge, some can be investigated on crackstation or just have a look at Moxie Marlinspikes' cloudcrackr.com. The goal here is to be 'computationally expensive'. If you cost too much to crack using ultra high speed cloud based cracking computers then you're safe against almost anyone. So ideally you want to use the maximum 64 characters for your password, and have it look like the most messed up annoying symbol infused piece of incoherent upper-lower-case dribble you've ever seen. You'll probably be safe after 14 characters though, there's quite a bit of entropy here, but it's far easier to add characters than it is to decrypt.
change your routers default password and SSID. nobody does this, but everyone should. It's literally the dumbest thing. Also, don't get lazy. and don't keep the router's model number in the SSID, that's just asking for trouble.
update your router's firmware. Also, if your router is old. throw it out and buy a newer one, because it's likely your router is on some website like routerpwn.com/ and you've already lost the battle. Old routers are full of bugs, can be easily denial-of-serviced, don't usually have firewalls or intrusion detection systems and don't usually have brute-force WPS rate limiting among other things. just get a new one.
learn about evil-twin hacks. The easiest way to protect against this is to stop your device from auto-connecting. However, this might still snag you. Become familiar with software like wiphishing and airbase-ng, these apps clone your router, then Denial of service your router making your device connect to the attackers cloned router, allowing them to intercept your traffic. They'll usually try to phish the WPA2 password from you here. You're safer from these attacks if you actually know what your router's web console looks like, because the default phishing pages that come with these types of apps are usually pretty old looking, however a sophisticated attacker can create a good landing page. Put simply, if your 'router' ever wants you to type in a password don't type it! You'll only ever be asked when you are creating the password, when you specifically log in to the 192.168.0.1 or 10.1.1.1 user interface, then you are being phished and it's game over. To prevent this attack you could also artificially reduce the range of your router. pull out the antenna's and create a little faraday cage around it, leaving a small area that points to your most ideal wifi position. Alternatively, just use a cable to your laptop or computer until the attacker gives up.
handshake attacks are pretty popular, this is where the attacker sends a deauthorisation packet to anyone connected to your router using your password, then when that device (say an iPhone) tries to reconnect, it captures the '4 way handshake' which let's the device and router authenticate using your WPA2 password. This is what hackers use to crack offline using the password attacks in point 6. However if you have used a strong password (as described in point 6) then you've mitigated this attack already.
So i've focussed on router based defence, but there's actually even easier ways to be attacked. If the attacker knows who you are, you're screwed. With a tiny bit of social engineering, they can find your facebook your email or some other way to contact you and insert some malicious snippet of code that's invisible and hijack your entire computer, which therefore lets them simply check the wifi settings in your computer and obtain the ultra strong password you've spent so long making. One popular method is to send you an email that's junk, and keep sending it until you click unsubscribe, as you usually would for junk mail, except this link is exactly the worst thing to do. You've broken the cardinal law of email. Don't click links in emails. If you have to click one, at least check where it goes first.
If someone has access to any of your devices, or plugs/gets your to plug a device into your laptop, you're gone. things like usb sticks 'usb rubber ducky' can compromise your computer and reveal your WPA2 password to a relatively novice hacker.
if you use a wireless keyboard, and you live near an attacking neighbour, they can use things like keysweeper to compromise your wifi, and a lot more. This could be creatively used with an evil twin attack to increase the likelihood you type your password (it listens to wireless keyboard signals). The way to prevent this attack is to not use a wireless microsoft keyboard.
There's plenty of other ways, and you'll never prevent them all,
but usually if your router is locked down, has a nice password, has WPS off, WPA2 on, a strong (new) router with a password, no remote-web access, unnecessary ports are closed, MAC filtering is used and intrusion detection in the router is switched on you will usually prevent even pretty dedicated attackers. They'll have to try harder methods and will probably just give up.
In a nutshell, WPA2 is currently the most secure wireless security scheme.
Personal and Enterprise
It supports two main modes of authentication, known as WPA2-Personal and WPA2-Enterprise. The former utilises a pre-shared key (PSK) and is generally considered to be most suitable for home networks, whereas the latter is 802.1x which requires an authentication server.
WPS
A third mode of authentication, Wi-Fi Protected Setup (WPS), is known to be vulnerable and should be disabled on all wireless networks. When this mode of authentication is enabled (and it often is by default) the associated PIN can typically be enumerated in a matter of hours.
Pre-Shared Keys
PSK authentication, the type used in home networks, is vulnerable to offline brute-force attacks. If an attacker can capture a WPA/WPA2 handshake, they can use brute-force and dictionary attacks (like you might do with a hash), essentially going through large numbers of possible values until a match is found. Fortunately, generating WPA handshakes is fairly slow which makes this harder for an attacker, but once the handshake has been captured they don't have to stay in the vicinity, so could potentially go away for months to crack it offline (if they're very determined)!
Potential countermeasures against these PSK attacks include:
A sufficiently strong key that is long, complex, and not based on a dictionary word or common phrase (ideally random), such that it would take an extremely long time to crack.
A key that is changed at regular intervals, such that it is unlikely that an attacker would be able to crack it before it changes.
Do not use the default SSID. Changing the "name" of the wireless network will prevent rainbow tables from being useful. Rainbow tables have been compiled for many common SSIDs and these can significantly decrease the time it takes to crack the PSK.
WPA2-Enterprise at Home
If you're really security conscious then it is entirely possible to setup WPA2-Enterprise in a home environment, although you'll need to configure a RADIUS server and use a router that supports it - so it's a much more complex process. Example
The above recommendations are only related to reducing the likelihood of WPA2 being cracked specifically. In any wireless network, a range of other considerations need to be made such as changing the router's configuration username and password and whether device lists should be monitored or MAC address filtering used.
Short answer is: use WPA2. WPA would be somewhat tolerable, but WPA2 should really be preferred. Do not use WEP, which is not really better than nothing (arguably, WEP is worse than nothing, because it gives to users the impression that security is happening, whereas it is not).
More importantly, be sure to use a strong password (meaning: very random) and try to avoid "common SSID" like (say) "homewifi" (some people have compiled big tables of precomputed password hashes for some common SSID values; you can still defeat attackers in that situation by using an even more random password, but using an uncommon SSID improves your chances). Note that normal users type the WiFi password only once; afterwards, the password is stored within the entrails of their computer or mobile device; thus, there is no real problem with having a long, fat, random, unmemorizable password for your WiFi network.
Hidden SSID don't improve security (though some people are convinced that they do). With a non-hidden SSID, users never have to type it even once, so a randomly chosen SSID can be used with no ill effect; a randomly chosen SSID is likely to be "uncommon" in the sense expressed above.
(In all of the above, by "random", I mean "generate with coins/dice/computer, not with your human, meaty brain, the latter being totally incapable of producing randomness of non-pathetic quality.)
I am not aware of any ongoing plan for making a newer, improved WPA3. WPA2 is already quite strong, within the limitations of the WiFi design -- in particular, WPA2 is about protecting the network from outsiders, but does not mean that regularly connected users cannot spy on each other. If you want to go further, you would have to add another layer, e.g. enforce IPsec usage between user machines and the gateway.