Can a hacker, that knows my IP address, remotely access accounts I have left logged in on my computer?
No, they would have to have access to your browser cookies in order to abuse them to log into a site you left logged in. Merely knowing your public IP address would not allow them to log into any website. If you are asking this question though, I would not be so sure that there are "immense barriers" between them and your personal computer. A good hacker can do a lot more than you may think.
In theory, a vulnerability in your router could be exploited which typically requires knowing your IP address, but there are dozens of ways to get your IP address anyway. Not to mention, the IPv4 space is small enough that a decent server can scan every single possible IP address in under a day (only 232, or 4,294,967,296 in total, including a large number of reserved or invalid ones). It is more likely that an attacker would exploit a vulnerability in, say, your browser than your router through your IP address. That is not to say that vulnerabilities in routers are uncommon, but the risk of an infection or compromise through some vulnerable or out of date program is far greater.
IP addresses are not very dangerous when a malicious user knows them. This is somewhat of a myth caused by script kiddies (especially of the video gaming variety) who ominously proclaim that they have your IP address and you better watch out, often with the implication that knowledge of an IP address amounts to full access to a network. The worst common scenario is that a malicious user mounts a denial of service attack on your router, causing your network connection to slow down or break. This can be irritating, but is not particularly dangerous.
There are two real situations where your IP address is sensitive information:
If you are dealing with a bitter player for an online video game who you just beat (because that headshot totally didn't hit him) or a spiteful troll on IRC, they may mount a DoS attack against your network in vengeance. In this case, you may want to call your ISP. They may be able to change your IP address or protect you from the attack in order to restore your connection to the network. Even if that does not work, these types of attacks quickly subside. You should probably just avoid associating with the type of person who falls into this category.
If your adversary is a law enforcement agency or any other legally-privileged entity whose goal it is to tie your IP address to your real-life identity, you should be using an anonymity network such as Tor (for web browsing) or a VPN (for P2P). This is the case when your adversary is able to subpoena your ISP to obtain your subscription details. In the past, it was easy to social engineer ISPs to get this information (folks on IRC used to do this to get someone's real address), but nowadays it tends to take a legally-binding court order, in which case your ISP will barf up all the personal information it has on you without giving it a second thought.
If neither of those cases applies to you, you have nothing to worry about.
There are few malicious things people can do if they know your public IP address, and the main two are router hacking and DDoS.
Router hacking involves someone scanning your router, and depending on the make, model and version, find vulnerabilities on it. If any is found, a hacker can change your router settings, usually changing the DNS servers to point to some server he controls, therefore being able to MitM any non-encrypted connection that depends on DNS. Configuring your router to not accept any connection on the WAN side helps a lot avoiding this kind of attack.
DDoS involves sending a lot of traffic to your IP address. Depending on your connection speed, a couple of zombie computers can slow down your connection or even knock you offline very fast.
So, why so many sites asks me to logoff? It's because of something called CSRF - Cross Site Request Forgery.
Let's say you are logged in at your favorite browser-based game (say, www.game.com), and you receive a mail claiming someone will attack you, and with a link telling you to read and see the screenshot. You click there, but the site have a hidden picture which source is www.game.com/donate-all-resources.php?to=hacker-nick. If www.game.com does not have CSRF protection, your browser will load the donate-all URL and, well, donate all your resources.
And about the immense physical and cryptological barriers: they are way smaller than you think: hack your router, change the DNS to another one, MitM every HTTP connection, and if you try to download anything over HTTP, poison the download. Done.
I don't know your level of knowledge in this area, but I'd like to answer your question with a layman-level explanation of what an IP address is.
An IP address is very much like your postal address. If you want to correspond with someone through the postal system, you need their postal address and they need yours so you can send each other letters. Similarly, to exchange data or send messages online, your computer and the other computer or server must each know the IP address of the other.
When you type in a web address, your browser first corresponds with an IP address it already knows, which is the IP address of a DNS server. Google's DNS server has IP address '8.8.8.8', for example. It asks this server for the IP address of the server that hosts the website you are looking for. The DNS server gives it if it exists, and your computer then corresponds with that IP address to get the website data.
So there are many servers on out there that have known your IP address, just like any business you've sent letters to or received services from likely know your home address.
Your IP address is not a secret, or a password. It is the address used to do all legitimate internet transactions and services with you.
A person who wants to hurt or harass you can't cause you much trouble if they don't know where you live. If they don't have your home address they can't even find you to do anything. So getting your street address is the very first minimal prerequisite to messing with you or your home, and thus many people who are the victims of stalking or harassment try to conceal their home address, and would be distressed to learn that their stalker or angry ex has learned it.
Similarly a hacker can't do anything without knowing which computer to hack. If they don't know your IP address they can't even find you to begin their work. But it's really just the minimal first step. While not knowing the address prevents them from working, knowing the address doesn't really get them far. Like an angry ex who wants to steal things from your house, knowing where to go is just the first step. Once they find out where to go, they don't really have super secret info or some crazy advantage. Once they get there, they have no more power than any random person who walks by your house. They are still left with all the work of breaking in.
A hacker who knows your IP has not gained any special privileges or powers. A hacker who brags about knowing your IP is like a burglar bragging that they know your postal address. In fact, it's an even more ridiculous brag on the hacker's part, because a determined burglar can often break into a house, and even if they fail their attempts usually damage the house. But hacking is different. When a better physical home security is invented, only some homes get it, because the upgrade costs money to install. When new computer security is invented, most everyone gets it, because the software provider pushes the security patch to everyone's system automatically for free. So if your computer has the latest security updates, then it has the same security system that world-class experts have built to resist world-class hackers. The only disadvantage you have is if you have configured your system in a way that grants strangers access, like accidentally leaving your garage door unlocked (leaving your router set to use the manufacturer-default password) or if you didn't change your locks when you broke up with your ex (someone knows your iCloud password).
If you've got the latest security updates (you do update your computer right?) and an antivirus program, and you haven't left anything vulnerable (you don't give your passwords away or leave them as the default right?) then your hacker friend still faces 99.9% of the challenge of hacking after he learns your IP address. He's found your castle, but now has to attack it.
The one thing an amateur can do to a properly-configured system is called a DDOS attack. And it is basically abusing the legitimate internet protocols to overwhelm your system with spam messages. It's like someone sending you 10,000 junk letters every day, making it impossible for you to sort through and find your real mail.
So your IP address is like the street address of your home. And the hacker is like an angsty punk you met downtown shouting "I know where you live!". Sure buddy, plenty of people know where I live, that's why doors have locks.