STM32F4 and HAL
Let me tell you that a lot of us share the same disappointment you have with HAL libraries , they are indeed poorly and vaguely documented and still new contains many bugs.
So to answer your question why did ST decided on HAL is simple:
They want to create a hardware abstraction layer which in plain English means they want software development and code to be independent of the micro-controller , so if you write a code today for stm32f4 and you need after a couple of years to migrate to stm32f7 it will be easy and code will be highly modular.
This also allows more developers like software programmer to work with the micro-controller without really understanding or going into the deep details of how the hardware is achieving a task. Companies like ST and TI (starting this road now) are trying to make embedded development similar to PC code development where you use high level drivers to develop code FAST. The clumsiness and lack of optimization in their drivers and libraries is compensated by the high performance of the ARM devices.
I think STM32cubeMX is a great tool if you are using HAL libraries , because the most time consuming job is initializing peripherals and now you can do it in a very short time , with visual interface that can be easily changed without affecting the user code ( if you write your code in the appropriate place ) You can use Stm32cubeMx and then review the code and try to understand how and why they are using each function , in this way you are trying to solve an exercise and having solution manual nearby for correction which is great IMO.
ARM core is quiet complex so the old methods we used on 8-bit micro controller like handling registers directly (writing C in assembly fashion) is not feasible , it is time consuming and makes the code hard to maintain due to complex architecture (check the clock setting for example)
Creating your own libraries is quite simple. Their register spec documentation is pretty good, most if not all of the peripherals are easy to setup. I find it much more painful to use their libraries. but perhaps that is just me. This is true for st, nxp, ti, atmel to name a few (not so much for intel and microchip).
Why do they change libraries, could be any number of reasons, some new boss took over, some division got shut down another took over. Marketing wanted a new image for the product. As ElectronS mentioned, could be an attempt to abstract away from the hardware more to attract users not willing or able to do bare metal. I would go further on that and say they are probably trying to compete with the Arduino phenomenon. Which mbed and everyone else has always tried to do and failed (even before Arduino).
In any case the further away from the hardware it gets the more bloated and slower, so the more you have to spend per unit for rom, ram and mhz. Just so you could spend the same amount of time programming? Just doing it differently?
You say you come from the PIC world, now they did do an okay job with tools, their chip docs were dreadful though, some of the worst. they compensated with libraries and sandboxes.
At the end of the day, try the various options, try the competing products to see how their tools compare. A lot of that you can do for free just to see if it makes sense and you can compile stuff. Maybe even use an instruction set simulator. Find the one that matches for you.
Note, the without canned libraries option is ALWAYS available to you. You are not limited as to what toolchain you can use, what host operating system, what ide, editor, etc. They might stick it to you on programming the parts, if their options are extremely limited in that respect move on to some other chip or vendor if you can.
To sell a chip product like this they have to provide a development environment whether it is all theirs or free stuff they glued together. And they tend to put a library of some kind together. It only has to look just good enough and the blink the led example work just well enough to get either your management or your hardware team to design in their product, then when your board product is tossed over the wall to software, is when the pain does or doesnt arrive. If it almost works but not quite is a big win for the chip vendor as you will now pay for tech support for that last little bit. So it is in their best interest to be almost there but not quite.
The chip vendors only have to look good enough to get the design win. They have to keep improving (? changing) the product to attract new and old customers. So they will have do overs, how far apart and how many prior libraries the continue to support, varies. So just about any library you get used to will go away eventually. So learn to adapt (or dont use their stuff and go your own, which you can support indefinitely). Granted, ideally, you only need to develop the application once per product, make your firmware perfect (good luck if using third party libraries), and you wont need to go back and find a computer that will load their toolchain if you can find a copy of it, and remember how to use that old library. Remember not only should you save your source code, but you should save all of their tools and docs.
Their libraries are only supported on usually one toolchain, under one maybe two IDEs and sometimes only on Windows, and certain versions. Again you have none of those limitations, most definitely not for ARM, if you do your own thing. You can always read any/all of their libraries to see how they do things. But that is often very scary, they dont use their A team developers for libraries, I have extracted a few lines of code to ask interview candidates what is wrong with this code.
to save time and effort both on the silicon side and the software side they very often recycle the same ip, so once you see how the peripheral works on one of their chips it often works the same way on many other of their chips. Yes the clock systems can be tricky with or without their libraries. High chance of bricking the chip, that is where most of my chip/board bricking has happened. Helps to understand how their chips work for example the AVR's, most if not all, can be reprogrammed while the chip is in reset, so any bad code that messes up the pins needed to re-program, or hangs logic needed to reprogrammed, doesnt matter, you can reprogram those chips. Some of these vendors (st is one) has an internal bootloader which you can select using a strap (BOOT0 for example in the st world), you could easily put your own strap in while developing and should whether or not you use a library.
One size fits all fits no-one well. Particularly true for software. So any attempt to abstract away the hardware, just makes it slow and bloated. Might as well get a bigger chip and run linux on it, if that is what you are truly after. A lot of this is as a result of the developers though, not wanting to get their hands dirty, so we have basically asked for this, and they are trying to supply it.
Again, dont lock yourself into st or any one vendor (unless it is too late and management and or the hardware team has stuck it to you, note the stm32 products are nice and easy to use). Shop around. TI is putting a lot of eggs in the cortex-m4 basket. You can do the mbed thing on a number of these arm products as well as the vendor supported solutions.
One thing you can always rely on, is that they will change libraries from time to time and eventually stop supporting the one you got used to.
I know this is going to be long and opinionated but as we've just (successfully) released our new product using the HAL, I think it is worth being considered. Also, I don't work for ST, I have hated every bit of the HAL, almost restarted the project with StdPeriph, I have felt the pain - but now I understand why.
First, a bit of background. We develop ultra low power telemetry systems, and our product is powered by an STM32L1. When we started working on the firmware, we had the usual choices for (bare metal) ST devices: do everything by hand, use the StdPeriph libraries, or go with the HAL. The ST guys convinced us to go with the HAL - so we did. It was painful, we had to work around bugs in the software (the I2C part drove us crazy for quite some time) and I still dislike the overall architecture. But it works.
When I switched from desktop to embedded, a bit more than a year ago, I was stunned by something strange I couldn't name or even grasp. Over time I was able to understand what was - or rather, what is - going on: the embedded world is in transition. Silicon gets cheaper everyday and MCUs are more powerful and versatile. More and more devices, regardless of their size, power needs, rely on generic MCUs. More and more companies join the game, bringing in a horde of new developers with various backgrounds. The "mean" culture drifts away from the traditional "EE guy with programming wizardry skills" to "SW guy with vague hardware knowledge".
Whether this is good or bad is irrelevant. It just happens. Actually, it also happened to the software world, more than once. The web boom in 2000 attracted PHP/MySQL newbies - go tell them there are registers in the CPU, they will answer "I'm using Linux, so there is no Registry in my OS". Earlier, multi-user OSes running in protected mode allowed lazy developers to never set up an ISR in their entire career and be fine. Even earlier, keyboards and screen made card punchers and printer manufacturers go crazy.
And yes, the current trends make me personally sad, as I see ignorant developers in awe with the latest shiny techs, while being totally unable to link them with history. When I see a younger me coding a game in Javascript with WebGL in 2015, I want to shout "There is nothing new! I did the same with C++ and the 3Dfx SDK back in 1995!". What the story doesn't tell is that his game runs on my mobile phone, whereas mine needed a gamer PC (and an installer, and I couldn't push updates by the web). The truth is, he could develop is game in one month, where I did the same in six, or twelve.
Obviously, neither ST or TI or Intel or whoever manufactures chips wants to miss the turn. And they are right. The HAL is ST's response, and is actually quite sound, not only on the business or marketing side, but on the engineering side as well. The reason why it is sound lies in the name:
Hardware Abstraction Layer
If anything else, this is what you should remember. The HAL is an effort to move away from the hardware. It is good engineering because it allows us to decouple the functionality from the details. Layering is what allows to develop complex programs - one abstraction on top of another, down to the hardware. Abstraction is actually the most powerful tool we have to manage complexity. I very much doubt anyone on this planet would be able to program a web browser in assembly for a specific CPU.
The cultural shift is indeed hard to digest, but as I have to reckon one does not need to have read Knuth's Art of Computer Programming to develop web applications, the EE world must admit there are newcomers that can (and will!) develop embedded code without having read the Fucking Holy Reference Manual.
The good news is new players doesn't mean less work for older players - quite the opposite, IMHO. Who are they going to call when things "don't work?". If you have RTFM (unlike them), and if you know what each bit of this obscure configuration register do, you have an advantage.
Between your readings and experiments, just go with the HAL. New MCU? No problem. New MCU line? No problem as well (I coded a test on a STM32F4 Nucleo in just a day with CubeMX, and then simply ported it to our device ... it just felt right). Mocking for unit tests? No problem. The list goes on and on, because abstraction is good.
Of course, the HAL itself is not 100% OK. It's documentation is awful (but you have RTFHRM, didn't you?), there are bugs, ST just dumped a beta version at us (seems pretty standard these days) and their public support is a joke. But not releasing the HAL would have been even worse.