TOFU... It's used for SSH iirc, though I could be wrong.
No, you're right. That that single, assumed to be legitimate, and first introduced key, is trusted and used for all subsequent encounters. Any later unvalidated change in key would indicate suspect brokenness. Authentication of said former key, via any particular mechanism, is a secondary bonus. For instance, you may first check mail to a given fingerprint gets you to the mail/context you expect. Then a web search of that fingerprint may yield independent bloggers affirming their similar expierience, then some reasonable trust of that key is established. Though it is encouraged that such lone keys be signed by some web of trust that you can then reach. This new environment of weak CA's will, in hope, yield a stronger more articulated sense of what we all are signing for each other.