Is there a way to convert a .rofl file to .mp4 or .avi?

The *.rofl files contain only binary game data and no media frames.
For example a very simplified layout of the file could look like:

id   | champ. name | action | parameters
0000 | annie       | move   | 155 | 188
0001 | fizz        | attack | annie

and so on.

With this data the client is able to "reproduce" the game you just played. If they recorded movie data instead there would be certain disadvantages:

• File size would probably be bigger
• Movie is bound to a fixed resolution
• Can't move the camera around in the spectator view

If you want to record that game play in a common movie format you can use the built-in recorder.

Just click the red Record Button on the new Annotated Timeline to start recording, (watch your sick play), then click again to stop.
We’re calling these video clips “Highlights” and they’re in the .webm format. Your Highlight will be saved on your computer in the following location: C:\Users\Username\Documents\LeagueofLegends\Highlights

No, there is no way to directly convert .rofl files to any other video format. This is because they are not video files themselves - they just contain "spectator data" so that the game client knows what actions happened at what times, which allows it to replay those actions in sequence for your viewing pleasure.

If you want to make an actual video recording, you'll need some kind of screen-recording software. There's a whole bunch of options - OBS, Plays.TV, and Fraps are probably the most commonly used ones, but there's tons more too. These will record to actual video files (.avi or .mp4, usually) that you can then convert or use as you please.

As mentioned in another answer, you can use the built-in recorder as well, if you just want to save some clips. The result of using the built-in recorder is .webm file, which is a lightweight, web-friendly video file. After testing it, I can't see any hard limit on the amount of time that a single highlight can take (I was able to take 20 minutes of video without any problems). However, the quality of that recording isn't the greatest. I don't have a tool to take exact measurements, but I would estimate the resolution is about 1280x720. There are a lot of compression artifacts, though, so it doesn't look very crisp or high definition.

"replay" files for games are generally just instructions to the game engine, not actual video.

So if you want to convert them to an actual video you will most likely need to replay them in the game while recording the output of the game with a screen-recording soloution.