Should I update trigger API version?

Apex language evolves every 4 months, with every Salesforce release.

I can definitely say there would have been many bugs in v20 which have been fixed over time.

Now there can be code, which relies on bug or side-effect of the bug to get things, done. Now when you increase API version the side-effect would go and in some case can break your logic.

eg: Parameterized Typing and Interfaces

Your code might be using Parameterized Typing and Interfaces, which was removed in v25(W13).

 public virtual interface Pair<T, U> {
      T getFirst();
      U getSecond();
      void setFirst(T val);
      void setSecond(U val);
      Pair<U, T> swap();
 }

Now increasing API verson will make classes using Parameterized Typing and Interfaces absolute.

That being said, its always a good idea to have the highest available Api Version as it has latest performance and security patches. Try it in sandbox for few weeks and then move to prod.

Whenever Salesforce releases a new API version, any classes/triggers written in older API version are still supported. That's one of the reasons to support backward compatibility to have the API versions.

To aid backwards-compatibility, classes and triggers are stored with the version settings for a specific Salesforce API version.

So to answer your question:

Should I update them to the newer version (44 or 45) using the UI?

Depends. If there's no significant reason to upgrade all your classes/triggers to latest version, you can just leave them as is. Salesforce ensures that classes/triggers written in older API version still work. Usually it's recommend to sync up with the latest API version. Any upgrade should be carefully planned though.

Possible downside of keeping it as 20?

The only issue you can encounter is if you are using ConnectAPi where methods specific to an API version requires that particular API version.

The classes and methods of the ConnectApi namespace are supported only in the API versions specified in the documentation

You'll want to see my answer on what I believe you should do, but also this answer should be relevant.

API versions determine how your code behaves. At v20, it acts like Salesforce is on v20, while at v45, it behaves like v45. This is mostly compatible, but some functions, such as JSON.serialize, have different behavior depending on the API version. In other words, leaving it at v20 likely won't break anything for at least a few years, but upgrading it now might break stuff now.

Before you decide, you'll want to make a sandbox, run all tests, and make sure nothing breaks. Also, make sure you manually test the triggers to verify that nothing breaks (not all developers write good unit tests). Also, if possible, have a developer review your code just to make sure. Some non-obvious bugs could theoretically appear and be hidden by poorly written code. After you're satisfied, you should upgrade if possible; this will give you a lot more time before you have to do any more revisions.