Password lists are perhaps the most widely used source of seed information to attempt to reverse engineer passwords by both security professionals and malicious actors alike. The proliferation of these lists are perhaps due to nature of password cracking, and how humanization tends to yield more successful results, more on this in a bit.
Security professionals, when describing the methodology used in this activity, draw a distinction between a ‘brute force’ and ‘dictionary’ attack. A secure password meets the following two conditions: 1. It costs more in resources to recover the information, than the information is worth, and 2. By the time the information is inevitably recovered, it no longer has worth. As real world examples, you would not spend two million dollars to break into a safe that contains one million dollars, and to learn the position of an enemy spy the week after this spy has gone elsewhere is no longer useful information.
A brute-force attack uses truly randomly selected characters in sequence to attempt to guess the correct password, and is only limited by time, and attack speed, where speed is limited to compute hardware. In theory, this model has a success rate of 100% given enough time, though time is also the limiting factor in this approach. The complexity of a password influences the theoretical amount of guesses one must make to guess correctly, this is another way to describe a key-space. A key-space is the total surface area of complexity, for instance, a password that uses only lower-case alphabetic characters would have a key-space of 26. Adding capitalization into the mix increases the key-space to 52, as one would need to account for the possibility of all capital letters as well in a reverse engineering attempt (this repeats for special characters, and numbers). Indeed, this is the reason for password complexity requirements. To calculate the amount of time required to reverse engineer a password, one would multiply the guess rate by the key-space (combinations of which add possibilities exponentially). Divide this number by two, and you have the roughly expected rate (the middle of the bell curve, success could come earlier or later). The sum of this amount of time is the time in which success is guaranteed. Notably, powerful hardware and flaws in older implementations of Windows password security has allowed this time space to be reduced to roughly 8 hours. I digress..
A dictionary attack uses real words to try and increase the chances of success, based on the logic that humans tend to use inspiration from a spoken and written language when coming up with a password. After all, a password is ‘something you know’, and rather than opt for a higher cognitive load in memorization of something not meaningful, the tendency is to use something meaningful to more easily remember. The dictionary attack attempts to take advantage of this by using a ‘wordlist’ as input, and first attempting to guess these words. Theoretically, given the same key-space and guess rate, the guarantee of success is the same, however this technique has been so successful it’s become a standard in the security professionals arsenal.
It may be clear at this point that attempts to increase the success rate of a password reverse engineering effort tend to try and take advantage of the predictability of humans being humans. While security strategists have attempted to remedy this by mandating so-called ‘complex’ passwords, this does not often adequately negate the tendency of humans to satisfy this requirement in the most simplistic way possible, potentially with capital letters in front, @’s as As, $’s as S characters, !’s as I, & as G, numbers on the end, and special characters at the very end as an afterthought (If I’m describing your password, I hope this brings a smile). While this creates a theoretically huge number of possibilities, often the reality is that the real password is not really complex at all, but masked in just enough character exposition to satisfy key-space requirements and not all that different from a simple alphanumeric implementation.
Some of the most advanced research into this area involve statistical analysis and calculating positional probability of certain letters, numbers, and character positions (this is a great blog that details some of this work), often resulting in a significantly higher success rate.
So, how does one win the war on password security? The short answer appears to be avoiding all humanity in selecting a password, which is the only way to unequivocally sidestep all attempts at optimizing the password recovery process (implied as a negative in this case). Additionally, a separate password for every service you use prevents lateral escalation if your password was ever to be compromised. A real example of the consequences of password reuse include the Uber ‘hacks’, determined to be perfectly legitimate use of accounts, just by an unauthorized party who acquired the account key elsewhere.
‘The best password is the one you don’t even know’ is a relatively new mantra adopted by purveyors of password managers, which are tools offered as a solution to using a different strong password per service, that do the work of remembering for you. The astute reader may point out that using a solution like this (such as Lastpass) that puts your most sensitive information on the Internet is in itself also ludicrous. While in theory true, it seems to be a choice between letting a company with their reputation on the line take care of security, taking this on yourself in its entirety, or some happy alternative to be determined. No perfect solution exists, but while passwords continue to be the most widely used means for authentication, securing them will continue to be a ongoing tug of war.