Apple - What deficiencies in Classic Mac OS led Apple to replace it?
Did Not Multitask
It had limited, cooperative multi-tasking, but this was spliced on top of a fundamentally single-tasking OS. That meant if a (say, background) task was greedy or had a problem, the whole system ran badly or locked up.
There was a point when Microsoft was "eating Apple's lunch" because multitasking worked so much better on Windows - and Apple lost a lot of market share.
No crash protection
It did not have any way to contain the failure of one particular app. So when one app "crashed", the system crashed. Being able to run multiple applications concurrently, to say nothing of the array of often dozens of desk accessories, extensions and whatnot, that created a huge vulnerability.
It was common for a Mac to come into the shop with so many desk accessories installed that you had to scroll to see them all. I said "They did to this Mac what a pimp does to a Cadillac." No wonder it was in the shop.
No security
Also related, there was no hardware-based memory protection, so one app could scribble another app's memory from a simple bug.
This also meant an application, DA etc. could snoop on the contents and activities in system memory. What's the point of having password logins when your free screensaver DA can simply scan memory looking for the passwords?
Without an ability to keep applications contained, this also meant filesystem security was a lost cause. Among other things, there was no way to keep applications from damaging or hacking the OS.
Viruses started on the Mac, you know. This sort of thing is why. They also virtually ended when OS X came out.
No shell
Okay, that only matters to a few people, but there was absolutely no robust back-end from which to keyboard/script system tasks. No way to, say, "rsync" a directory to a backup; you had to rely on packaged products. Of course there were developer tools, but that was not readily available to end users, you had to sign up and if I recall, pay some fees.
Upgrading OS 8/9 to proper multitasking was a hard problem - a veritable Gordian knot. It needed an "Alexander the Great" solution, and that's where Steve came in.
From the Wikipedia article on Classic Mac OS:
Mac OS is characterized by its monolithic system. From its original release through System 4, it ran only one application at a time. Even so, it was noted for its ease of use. Mac OS gained cooperative multitasking with System 5, which ran on the Macintosh SE and Macintosh II. It was criticized for its very limited memory management, lack of protected memory, no access controls, and susceptibility to conflicts among extensions that provide additional functionality such as networking or support for a particular device.