HTTP status code to return by a REST API when image size exceeds limit

You can use 420 or even 422, but I would avoid that until you have really good reason to have separate code for it. Usually is better to keep number of different status codes rather small. Check top 10 on that list: http://www.restapitutorial.com/httpstatuscodes.html

You should avoid using more than 10 codes, because your API will become too complex.

So my answer is: use 400 with proper error message returned to the client like: "Image too large, you can upload files up to XX MB"

This seems like it would be an ideal candidate for 413 Payload Too Large. From Section 6.5.11 of RFC 7231:

The 413 (Payload Too Large) status code indicates that the server is refusing to process a request because the request payload is larger than the server is willing or able to process.