You wrote:
My question is how do I know it is encrypted?
Calculate an entropy measure of some sort. Entropy is a measure of disributional skew. Maximum entropy means minimum skew.
For human-readable text of any sort, the monogram entropy, i.e. the entropy of individual characters, will _always_ be detectably less than maximal. Encrypted text will always be near maximal. The two are easy to distinguish. ASCII-armored encrypted text will always be right at 6 bits per byte.
For speed of implementation, you don't need even to look at much text. You can get a statistically significant measure quite quickly from the first couple of kilobytes.
And since you're only really worried about detecting non-randomness, you don't even need to calculate the exact entropy but rather an approximation of it. This approximation can be done with entirely fixed point arithmetic, if you're a bit clever about it.
A practical system would cut out a notch at 6/8 for ASCII armor, which would make approximation techniques a bit tricky. More practical is just to detect ASCII armor with a regular expression recognizer and de-armor it before the entropy check.
Eric
Won't work! You can always embed an encrypted message in what 'looks' like plaintext. A trivial example: Encrypt a message with a caesar cypher, then build a story where the first char of each word maps to each subsequent char from the encrypted text. At the cost of expanding the size of the message by a factor of 5 to 10 you've hidden the encrypted message in what looks like a letter to your mother (or a news story in the NY Times, etc.) This is old technique. Dale H.