Apostrophes and double quotes don't show up until I type the next letter

The reason is because you are using US-international keyboard.

Here is how to change this:

1. In the Windows run box (Windows+R) type control intl.cpl or control international.
2. Click the "Keyboards and Languages" tab
3. Click "Change Keyboards..."
4. AT THIS POINT MAKE SURE YOU ARE USING "English (United Kingdom) - US" as the default input language, meaning you set the keyboard to US, not US-international

Edit: since people are complaining that this answer does only answer the "Why" question, and does not provide a solution to the problem (which wasn't the question asked by the OP), please find the solution to the problem in the next answer https://superuser.com/a/343483

This is because you have a keyboard where ' ~ " are set to be a dead-key.

The behaviour of a dead key is ... well... to be dead! So no reaction when you press it.

A dead key is gonna be waiting for the next key to be pressed, to alter what that key is gonna print. For example ' + a might produce á if the quote key is configured as a dead key.

Usually, when a keystroke has no meaning after a dead key, it defaults to printing the character of the dead key, followed by the new character that has been typed.

So you can follow advice of other posters on how to change your keyboard (look at which KB you have configured). You can also get the Microsoft Keyboard Layout Creator to map a new keyboard and see how you can program dead keys.

Update: with the keyboard editor, you can also load the current keyboard and modify the dead key setting

I checked my settings and they are set to English/US but the problem persisted. Then after closing the dialogue an icon appeared on the task bar allowing me to change from EN/International keyboard to EN/US Keyboard - Problem fixed!