Should I ask my co-author before I upload an article (that he already told me is okay to publish) to arXiv?

Yes, you should check with any co-authors before publicly posting a preprint of work that is not yet published.

They may prefer that you not post it at this time (e.g. if they have a particular timeline in mind for when they want to publicly share the work, given other related things they are working on). Even if they have no reason to object, you should check with them as a courtesy.

Submitting to ArXiv is a form of publication. You always need the consent of all coauthors to publish anything, anywhere.

About "virtually no downsides": the following is a real story. A mathematician proves his best theorem ever. Writes the paper, sends it to a very good journal, and uploads to the arxiv. The very good journal takes a year to referee the paper, and rejects it (happens all the time, at least in my area). During that year, a group of other mathematicians read the paper in the arxiv, worked on it, proved a generalization, and got it published in a journal. So, a year later, the original paper is virtually unpublishable, and our friend has to scrap it.

As you can imagine, this person never again uploaded anything to the arxiv.