Cracking a program

Let's say I have this encryption program and I don't know what's the algorithm used in it. Is there a technique I can analyse and break it? Are there any books or sites which teaches us how to do it? Any help is much appreciated Thank you. YeN ChOoN