An alternative to traditional passwords: is there some merit to this idea?
That sounds great to me.
The entire point of generating passwords is to pick one that is extremely hard for an attacker to guess, and easy for the authorized users to remember and type in. Ideally a password generating process should generate such extremely hard-to-guess passwords even if the attacker knows exactly which password generating process you used ( Kerckhoffs's principle ). If the "perfect knowledge" attacker finds it hard to guess the password, then other attackers will find it even harder.
I'm going to assume that you use a /dev/random that is not broken (it's connected to a hardware random number generator or a cryptographically secure pseudorandom number generator).
Many people seem to feel that a so-called "10-random-character password" formed from a randomly selected uppercase letter, randomly selected lowercase letters, followed by a randomly-chosen digit, a total of 10 characters long, is more than adequate for many purposes. That gives ln2( 26^9 * 10 ) = 45.6 bits of password strength.
If you completely randomly pick words from any list of 7776 words ( 6^5 words ), such as the the standard Diceware list or the easier-to-type-on-a-phone Dialdice list, and the attacker knows exactly what wordlist you use, and your user chooses the first 4 words you select, in the order you select them, the user gets ln2( (6^5)^4 ) = 51.7 bits of password strength.
increasing randomness?
If the user hits the refresh key a random number of times, and then rolls the dice to randomly selects 4 words from among the ones you present, and then rolls the dice to shuffle those words into a random order to generate the 4-word Diceware passphrase, then (again assuming /dev/random is not broken) I can a priori calculate that it has no effect -- the user still has 51.7 bits of password strength.
If the user thinks that's not enough password strength, then I think your system should allow the user to add additional letters, symbols, words, etc.
effects of non-random selection
Your user is unlikely to make a completely random selection, but that's OK. In the worst case -- your user puts things in alphabetical order or chooses some other order, and the attacker somehow knows exactly what order that is, you've lost -ln2( 4! ) = -4.6 bits of password strength from a 4-word Dialdice passphrase.
If we assume the user always rejects certain words, and keeps refreshing until he can make a passphrase without those words, and those words make up about 1/6 of the dictionary, and somehow the attacker knows which words those are, then for a 4-word Dialdice passphrase we've lost about -1 total bit of password strength. (The worst case is a loss of N bits of password strength for every 2^N times the user pushes the refresh button -- but that doesn't seem to be realistic to me).
The combination of these two effects results in your users having 46.1 bits of password strength with a 4-word Dialdice passphrase -- stronger than the 10-random-character password, and much easier to memorize.
If you later think that's not enough password strength, it's pretty straightforward to increase the minimum number of words in the passphrase, or switch to a wordlist with more words in it, or both. (You could use the approach in "Migrate old md5 passwords to bcrypt passwords" to force everyone with old, low-strength passwords to upgrade to new, higher-strength passwords).
That sounds great to me.
The idea of the xkcd comic is to generate a passphrase based on 4 words from a small dictionary of about 2000 to 3000 extremely common words.
If you let users pick words from a refreshable list, you significantly reduce entropy. The same is true for letting them change the order. Have a look at XKCD #936: Short complex password, or long dictionary passphrase?
Therefore you need to provide a much larger dictionary with less common words to compensate.
On the practical side: Users don't like typing in long passwords, especially on devices without a proper keyboard.