Why use a Smartcard for (Two Factor) Auth instead of another medium?

A smart card works by keeping a secret hidden and answering a challenge that proves it has the secret. It, theoretically, should never reveal that secret to anyone and it should be unrecoverable. There are some technical ways you might be able to get around it, but most of them are destructive to the card. This means you know if your smartcard has been compromised.

An encrypted USB drive or memory card on the other hand can simply be copied. There is no mechanism protecting it from being cloned by an attacker. There are some USB sticks that do provide hardware protection to prevent unauthorized access and these would make a more viable option, but it would be a toss up as to whether even those were as well protected as a good smartcard.

It's (theoretically) harder to duplicate a Smart Card. You can duplicate a USB drive easily.

If I steal both, you are equally in trouble, but if I steal the USB, duplicate it, then replace it without you knowing, then you are in trouble and you don't know it.

Disk encryption requires the host to keep or derive a master key which is kept somewhere in memory so that's your biggest issue. I've implemented aspects of the SafeNet ProtectFile product and other smart-card stacks so I'm intimately aware of the challenges.

Don't for a minute think you have any real security. There are "digital forensic devices" that snapshot and analyse system memory just by plugging a USB stick into a device port, they exploit USB hardware "bugs". Banks have been known to remove or Aryldyte USB ports on workstations because of these known problems. Its almost trivial to acquire the disk encryption key out of memory and then decrypt a hard disk with the tools that are available. This is one aspect of how law enforcement is able to gather evidence and bring criminals to justice.

For your particular use case, nothing is more secure than what you can hold in your head. So its a question of whether you think your brain is more likely to lose information than some piece of technology.

I would use a long meaningless phrase and change it periodically based on your level of paranoia. Pronounceable pass phrases are easier to remember and providing you use a long enough one you have better security than any device (including smartcards) can provide in this instance. I include Availability as a dimension of security and for personal use I prefer the Availability characteristics of my person more than flash which could be acquired if I am incapacitated. Enterprise's may prefer an alternate model that doesn't rely on wet-ware to be alive and functioning to maintain availability.

Smart-card messaging protocols cannot prevent a MITM attack between the card chip and the host computer without a pre-agreed secret (this is distinct from the card PIN and is actually used to protect message flow between card and host/computer) and there are many surreptitious ways to eavesdrop USB. Setting up this secret on all cards to be used on all machines/workstations represents a royal configuration nightmare in enterprise deployments, so it is rarely ever done and fixed well known keys built into the smartcard stack are used. Even if you did go to the trouble, you now have a secret installed in all over the place, so its hardly secret anymore. For this use case a smartcard is just hand-waving, and really no better than typing something on a keyboard, but it can fail more easily or be "lost"/"stolen".

So given a smartcard cannot give you more than keyboard level security in this particular use case, I would use either use a long pass-phrase or consider a read-only flash stick containing a large chunk of random data with a passphrase/password and make sure you keep a secure offline backup of that cheap Chinese flash!