Is there a way to upgrade api versions of all apex classes without manually updating each of them?
WARNING
First of all, I would like to say, do this only if you are certain you have sufficient unit tests to catch anything that might break. Also, make certain you back up your code. I cannot stress this strongly enough.
You can do this pretty easily with any IDE, Salesforce DX, workbench, etc. The general process is either two or three steps.
First, download the files you want to update the API version for. Next, use a tool to "Find and Replace in Files". All major IDEs have this feature. Replace <apiVersion>.+</apiVersion> (a regular expression match) to the desired version. Third, if your IDE doesn't automatically deploy the changes, deploy the changes back to the server.
Force.com IDE (Search > File)
Edit: 2020 Edition
You can use VS Code to do the same thing. Click on Edit > Replace in files, and use the Regular Expression option (far right button next to Search), and you can do this with VS Code. You need to perform a SFDX deploy afterwards.
You can also do this with sed/awk and other *NIX (Unix, Linux) shell tools, too. Check the online documentation for your preferred tool.
Firstly, do heed sfdxfox's warning. It is very important to backup your metadata before embarking on such a large scale update of the api versions.
Make sure you run all the test cases before and after to look for things breaking. They might break in strange and unexpected ways.
If you can, try this out in a scratch org first so it is easy to experiment and roll back if required.
René Winkelmeyer has created a SFDX plugin that can automate this process for you. It will either target the highest version in the target org or you can specify a target version.
sfdx muenzpraeger:source:api:set -a 41.0
Reading content of package directories
45 files have been set to API version 41.0.