Can hackers detect my operating system?

Yes, this is very possible.

In fact, support for this is built into the most popular port scanning tool in use, nmap. This feature, OS detection, can be activated by passing the -O flag to the nmap tool. What nmap does is send different TCP and UDP packets to the target and examine the response. By comparing the response to a database of existing operating systems fingerprints, nmap can determine with reasonable accuracy what operating system is running on the system.

My first two cents:

What is the actual problem here? Attackers knowing you are running a vulnerable operating system or you not upgrading it?

Now in regards to your question. There are a number of utilities which can be used to fingerprint an operating system. There are several JavaScript libraries which allow you to get information about a person's browser and operating system.

Browsers are not the only thing you can try to fingerprint. Other services like SMB also can be used (as they will advertise certain versions) to attack a machine. Basically any service you are running could be used to try and guess your operating system. There are hardening guides which will assist you in making your print as low as possible (have a quick google).

Also note that if you are a regular consumer, chances are most of those services will not be accessible from the internet as you will be behind a NAT router or even a firewall.

The short answer is that yes attackers can usually detect your operating system in standard configurations.

Usually this would come while your browsing, and the browser will provide that information to systems that you contact in the User-Agent String (N.B. with some browsers and plugins this can be changed to anything you like).

Also as @terrychia notes, attackers who can send traffic directly to your system (e.g. if they're on the same network as you) can likely work out your OS version based on responses to requests they make to your computer.

To address the point at the end of your questions, 2 things. One I wouldn't rely on A-V alone to protect a system which is unsupported (e.g. Windows XP after April), secondly specifically for MSE, AFAIK there will be no more updates for MSE on XP after April (i.e. support for MSE on XP ends when XP goes out of support)