GPS taking long time to fix out in the open
The first thing that comes to my mind is that a first fix starts with unknown position and with unknown time. The device has to find as much satellites and then calculate the position.
All the subsequent fixes are easier to find, because you already have an approximate positon and you already have the correct time, then a GPS tracker would spend less time to lock in, by doing a Cross-correlation of known sattelite sequence and received signal. But in case the initial conditons are not met (no approximate position or no correct time) it has to do the first fix, every startup.
Make sure you have a battery backed RTC time and position, and a kind of mechanism to upload the initial conditions to the GPS receiver at startup.
EDIT:
Looking at datasheet, the SoC has a BBRAM - Battery Backed RAM, but there s no mention about the battery neither it has any pad to connect an external battery. Probably it has an super capacitor or small LiPo battery embedded in the module. It could be that this battery is empty, since it is new device. Try to power the device for couple of hours or the entire day, because that battery would charge very slowly.
Once it will be charged try to see if the fix comes within specific time.
EDIT 2: I think I have found the root of the problem read this:
17.2. POWER SUPPLY It is recommended to keep the power supply on all the time in order to maintain RTC block active and keep satellite data in RAM for fastest possible TTFF. When VCC is removed settings are reset to factory default and the receiver performs Cold Start on next power up.
Question
How come my GPS module takes so long time to fix, after TTFF (Time To First Fix)?
Answer
Update - I now realise my answer below will apply to many similar GPS signal acquisition problems but does not answer the OP's question in this specific case.
The OP asks why first fix is reasonably long, but later fixes are disappointingly not that much shorter.
@Marko Buršič already gave the correct answer, with the reason that no battery backup is the root cause of the problem.
End of update
Note - As explained above, my answer below is not the correct answer, but can provide some background knowledge for newbies to understand the situation better.
Now let me share my GPS testing and fixing experience. My experience is about 100 hours learning and testing, with 5 of so different GPS/GNSS modules.
Yes, sometimes it takes 3 to 5 minutes to get the first fix TTFF (Time To First Fix). But almost always, afterwards it takes much shorter to do subsequent fixes, in average 30 seconds or so (shortest 15 seconds, longest 2 minutes).
I always use my GPS module at home, near two windows facing two directions. I found that I need to extend the antenna at least 15cm from window, but no need farther, even my open sky is not that open (see Appendices for some photos). I found the optimum distance is just some two meters from window.
I surprisingly found that the "openness" of an open sky does not matter that much. Often I just hang my GPS module outside the wall of my flat, some 20 stories high, with a one meter long connecting wire to the active antenna.
I also surprisingly found that power supply noise does not mater that much. Using a Lipo battery power bank, hoping to get little noise, and best performance, does not help at all.
I once used GPSD with ATK1218-BD (BeiDou), but the learning curve very steep and not friendly to newbies. One the other hand, for NEO-6m/7M/8M, I found uBlox uCenter "IDE" very good. Tip - SparkFun has newbie GPS tutorials.
I have also written basic Rpi4B Thonny python 3.7.3 programs to extract GPS data, based on uBlox uCenter. I found everything straight forward and easy (Of course the GPS NMEA1083 protocol is already text based and easy to play with python.)
For more GPS setup and testing details, you might like to read my answers to similar GPS questions in the reference list below.
References
(1) Rpi Neo-6M / Neo-8M GPS Module Setup / Fix / Update Problems
(2) How can Rpi listen to a GPS module?
(3) How can Rpi connect a GPS module?
(4) The Basics of GPS - SparkFun
(5) How GPS Works (YouTube 12 min watch) - SparkFun 2017jul31
Appendices
Appendix A - Photos of my open sky at home
Appendix C - YouTubel showing a GPS module blinking once per second after fixing
Video showing GPS module blinking once per second after fixing
Notes
(1) The little backup battery that the OP must feel jealous is at the left side the blinking blue LED.
(2) The tiny connector connecting the black wire to the active anetnna is tricky to connect, must read SparkFun for a newbbie tutorial on how to connect.
(3) Even if you have learned how to connect, be warned that the number of connections allowed is only 5. That means after 5 times plugging in, the contacts will become loose and very easy to fall out, so you see I am using cable ties to fix the cable in secure position to "release strain" caused by careless guys like me pulling or tripping long antenna cables.
And if you find fixing intermittently fails, it is probably the poor contact connector due to you poor craftsmanship, so don't curse half of the Universe for its clear blue sky, or the ridiculously packed match box size flats blocking the sky waves. And yes, I also live in a match box, or what we locals called a "coffin" room, and I am playing GPS toys inside a coffin, and I am loving it :)
Appendix D - Crowded Buildings
/ to continue, ...