How to execute more than one background process in laravel?

Finally, I found a solution. It is very very easy. So, Here is how it works.

First of all, I divide records into the number of chunks (For example, 5 chunks. It will divide 3 million items into 5 chunks having 600k items each)

Then, I can push a command to queues that I create instantaneously for the chunks and execute queue worker for that queue only once using --once option. To make it simple to understand, Here is the code that I am using.

$chunk_id = 0;
foreach($chunks as $chunk){
    // Adding chunk to queue
    Artisan::queue('process:items',[
        'items' => $chunk,
    ])->onQueue('processChunk'.$chunk_id);

    // Executing queue worker only once
    exec('php artisan queue:work --queue=processChunk'.$chunk_id.' --once > storage/logs/process.log &');

    $chunk_id++;
}

With exec command, we are executing queue worker for the specific queue created for the specific chunk. Also, we've added & at the end of the command which forces the command to execute in the background at OS level.

This is how it can be done. I tested it and it is working smoothly! Anything else to improve or are there any drawbacks of using this method?

Just to add something from my personal experience.

First install and configure supervisor for your OS accordingly, following are the confs for linux basaed OS e.g. Ubuntu

Supervisor confs: (/etc/supervisor/conf.d)

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work database --sleep=3 --tries=3
autostart=true
autorestart=true
user=username
numprocs=25
redirect_stderr=true
stderr_events_enabled=true
stderr_logfile=/var/www/app/storage/logs/worker.error.log
stdout_logfile=/var/www/app/storage/logs/worker.log

Then create jobs according to your needs and dispatch.

Supervisor will process jobs simultaneously, in this particular case 25 jobs will be processing at a time.