Can I write a referee report too fast?

Fast is great! Just be warned that it means editors will like you and send you more requests so you’ll have to learn to say no. If you also say “no” quickly and suggest alternatives, then you’ll still leave a good impression with the editors.

There's no drawback (from the editor's/author's point of view) to you submitting a quick review, but you should mention to the editor that you've already seen the paper before. It can be suspicious as a case of the author reviewing his/her own paper:

The reviews themselves were not remarkable: mostly favourable, with some suggestions about how to improve the papers. What was unusual was how quickly they were completed — often within 24 hours. The turnaround was a little too fast, and Claudiu Supuran, the journal's editor-in-chief, started to become suspicious.

See the source for more.

Statistical outliers always attract attention.

a) Your speed will be judged against the content and professionalism of your report: Were you speedy or just in a hurry to finish with the damn thing?

b) Even if your speed is not useful to the Editor (because, maybe, there is a second referee, so what will matter for the publication process is the lowest speed, not the fastest), still, if a) above is positively concluded, then your speed will be appreciated and remembered.

c) Always "let sink" anything you write. For how much it depends, but my experience says that if I have a week, I give it a week, so that I am distanced not only from the content but also from syntax and overall structure.