Regenerate catalog cache images issues

You should try using the image resize command to pre-generate all necessary resizes.

php bin/magento catalog:image:resize

This command gets all the images sizes that have been defined in the theme XML and pregenerates the images in their correct folders.

You can also check the command documentation for more information http://devdocs.magento.com/guides/v2.1/frontend-dev-guide/themes/theme-images.html

Answer on November 20, 2019:

Regenerate image cache by command is not a feasible solution for all because it will take a lot of time for some website which has a lot of products. Also, I faced some issues like If we generate a cache image from CLI, it will work. When we flushed images from admin or delete the cached image manually at that time it won't generate a cache image again on page load so I need to run the regenerate command again and again. As per my point of view, the best solution is to generate image cache on page load.

Default Flow

Default Magento flow is whenever it load image(media), it will always pass through the request to pub/get.php and check whether the image exists or not. If it not exist, it will generate a new cached image. If it exists, it will return that path. So by default image should generate on page load.

We can check this pass-through logic in the below files

pub/media/.htaccess for apache server

RewriteRule .* ../get.php [L]
.............................
.............................

nginx.conf.sample for nginx server

location /media/ {
    try_files $uri $uri/ /get.php$is_args$args;
    .......................................
    .......................................

How to check this logic is working or not?

Put echo "test";exit; in the starting of pub/get.php and load any cached media URL, it should print test. Otherwise something wrong in your server configuration.

For me, whenever I deleted the catalog cache directory (rm -rf pub/media/catalog/product/cache/*) after that when we load the page it will not generate a new cached image and it goes to 404 pages not found and also it never reach get.php. I then noticed that many of the folders were having incorrect permissions different from 755 for folders and 644 for files. After I set the right permission, it works fine.

I hope it gives some idea.