Would these magnets be enough to damage a portable HDD?

Considering the drive itself contains a magnet that I would expect to be far stronger than these are, I would doubt there to be any issues from them coming into contact. It's largely a myth that hard drives are easily wiped by a magnet (VHS, tapes and floppy media are far more likely to). In fact, a test was done here involving huge neodymium magnets, far more powerful than most magnets you will have, which caused zero data loss at all (more of a chance of mechanically damaging the drive due to the power of them if they'd been closer to the read heads).

I wouldn't state it's impossible for some form of damage to occur, but it's very unlikely. Don't worry too much about it, but keep them separate if you can.

If you want to damage a harddrive with a magnet, the drive needs to be running and busy writing/reading stuff. And even then you need a damn powerful magnet.

I once disabled a harddrive with a 10x10x15 big magnet, and I believe it's more damaged because the internal mechanics got twisted/damaged during reading writing. I suspect a head crash

All in all you need a pretty big magnet.

Want a magnet that is guaranteed to destroy a hard drive — or for that matter, a CD/DVD drive? You need something like this 2-inch N52 neodymium disc magnet that has a pull force of approximately 450 lbs. Get your hand clamped between this and a piece of magnetic metal, and you could lose your hand. This won’t just erase the hard drive, it will wreak it, along with the CD/DVD drive and any fans.

http://www.forbes.com/sites/adriankingsleyhughes/2012/11/28/can-a-magnet-destroy-a-pc/#3c30672856a2