Why do smartphones limit the amount of fingerprints that can be recorded?

No, theres no specific security reason to limit the number of fingerprints stored.

However, given that these fingerprint modules normally is a HSM, like a smart card, storing fingerprints, the number of, and the size of the secrets such a fingerprint module can store, is limited.

Note that a fingerprint template can also become quite a large, because storing a fingerprint template is not like storing a password hash in a password database. Everytime a fingerprint is read, this will give a completely different result than the previous read, because you are not putting the finger exactly as you did when you registred you, or "enrolled" yourself.

Now, the fingerprint reader needs to make a difficult calculation, on the input fingerprint image, and the template image, to come out with a conclusion if the fingerprint just read, is enough similiar to the fingerprint currently registred, to succeed authentication. This similiarity, can be expressed as a input to the authentication function, where you can select how "sensitive" the authentication should be (too sensitive = it will reject your finger too often, too weak = it may allow a unauthorized person to log in as you)

This is also the reason the fingerprint reader asks you to put your finger multiple times to enroll, because the reader uses these to calculate a optimal "sensitivity" value for your finger, that is stored inside the template.

Think like this: You take out your camera and take a photo of your house. This photo you put in your portfolio and store as "this is my house". After a couple of days, you see a house, and want to know if its your house. You take a photo of the house. As you might understand, you can't just bitwise compare these 2 images, or compare its SHA256 hash, as the images will be slightly different angles, rotation and light, even if they are a picture of the same house. If you did, "authentication" would always fail.

You need to make a complex mathematical calculation on the picture, to find out if the pictures are of the same object or not.

This means the "template" of a fingerprint reader, can get quite a large, depending on the algoritm used in the fingerprint reader.

There ARE fingerprint readers, that can encrypt a template using its internal key, and then export the template as a AEAD (Authenticated Encryption with Associated Data) that is then stored in standard unprotected memory (that it exist gigabytes of). But think of the security consequences of this. This would mean, that you could "steal" this AEAD with a enrolled fingerprint image, and later, lets say when you have sold that phone on Craigslist and the new owner has enrolled his fingerprint, just use the stolen AEAD along with your finger, to unlock the phone.

Thats why the phones must store the fingerprint templates inside secure memory, and thats why the number of fingerprint templates are limited.