Are there instances where you must attack enemies?

While a no-kills run is possible, the FIGHT command is still required to complete the game.

Unless you clearly demonstrate you have no intent to show mercy, there are exactly two encounters (both at predetermined points) in the game where the FIGHT command must be chosen even if you do not want to kill anyone. Both of those occur at the very end of the game on the first playthrough. Every other encounter in the game can be peacefully resolved without ever using the FIGHT command.

With regards to the fight you mentioned:

If you remember that one of the froggits mentioned that you may have to use the spare command even if a monster's name tag isn't yellow to resolve a battle without killing anyone, the solution becomes a bit easier to fathom. SPAREing a monster means telling them you do not wish to fight. Toriel wants to make sure you are strong enough to survive on the other side of the door before allowing you to pass through it. If you have enough determination, though, she'll concede and stop attacking you (and the flee option will be removed from the MERCY menu when this happens).

Additionally, since it was also asked, I'll go ahead and answer the question raised in the spoiler block at the end:

When using "ingame" methods to handle your save data (like restarting from a save point after ALT-F4'ing out of the game), the game will have already saved some information about what you've done. This includes whether you had killed or spared Toriel (and how many times for each), what cutscenes you've seen (and how many times you've seen them), and so on, and this is kept separate from the actual "savegame" file. With one "major spoiler" exception, though, none of these will actually block you off from any ending. It is, however, possible to fully reset this by deleting the save folder (located at AppData\Local\UNDERTALE) and disabling steam cloud saves (if applicable, since it will "helpfully" restore the files you just deleted). Doing this will give you a fresh start, as if you had just played Undertale for the first time.

Keeping as spoiler free as possible, yes. There are instances where you must attack to progress through the game. This isn't the same as killing however, and Toriel is not one of these instances.

The way to save Toriel:

Go to the Mercy menu and keep sparing her (about 20 times). Eventually she will relent.

As far as I know there's no way to make the game forget you failed to save her the first time.

There are two instances where you must attack to proceed with the game.

The first instance in the game where you must attack to progress:

Towards the end of game when you fight Asgore Dreemurr in the neutral story route, he starts the battle by destroying the 'MERCY' option. You still have the 'ACT' option, but while this helps lower Asgore's stats, you have to attack him until his health gets low. Once you've weakened him enough, you'll do a critical hit that will put him a single hit away from dying, and the fight effectively ends here.

The one other time you must fight:

Is the fight against Flowey. You don't choose the 'FIGHT' option in the menu as you do in a standard fight, but the option appears on the field towards the end of the fight where you can use it by selecting it with your heart over the icon.