Restful API - handling large amounts of data

There are different ways in general by which one can improve the API performance including for large API sizes. Each of these topics can be explored in depth.

• Reduce Size Pagination
• Organizing Using Hypermedia
• Exactly What a User Need With Schema Filtering
• Defining Specific Responses Using The Prefer Header
• Using Caching To Make Response
• More Efficient More Efficiency Through Compression
• Breaking Things Down With Chunked Responses
• Switch To Providing More Streaming Responses
• Moving Forward With HTTP/2

Source: https://apievangelist.com/2018/04/20/delivering-large-api-responses-as-efficiently-as-possible/

You can change your API to include additional parameters to limit the scope of data returned by your application.

For instance, you could add limit and offset parameters to fetch just a little part. This is how pagination can be done in accordance with REST. A request like this would result in fetching 10 resources from the messages collection, from 21st to 30th. This way you can ask for a specific portion of a huge data set:

myapi.co.uk/messages?limit=10&offset=20 

Another way to decrease the payload would be to only ask for certain parts of your resources' representation. Here's how facebook does it:

/joe.smith/friends?fields=id,name,picture

Remember that while using either of these methods, you have to provide a way for the client to discover each of the resources. You can't assume they'll just look at the parameters and start changing them in search of data. That would be a violation of the REST paradigm. Provide them with the necessary hyperlinks to avoid it.

I strongly recommend viewing this presentation on RESTful API design by apigee (the screencast is called "Teach a Dog to REST"). Good practices and neat ideas to approach everyday problems are discussed there.

EDIT: The video has been updated a number of times since I posted this answer, you can check out the 3rd edition from January 2013