There are a few things to watch out for. First, it's really easy for subtle (or not so subtle) biases to be present in a noise source. These can be due to component drift, external noise (i.e., power supply coupling), etc. You want a design that isn't sensitive to such things, if possible. Second -- and it's partly a corollary to the first -- the designs I've seen for real RNGs have always included a scrambler step, to mix up the bits, account for biases, etc. The first such scrambler was, I think, described by von Neuman himself. I have the citation in my office; I'll try to post it tomorrow.