In this post we will discuss about various common problem you face when we are trying to scrap a site for some valuable information and ways to approach such problems. Then at the end we will discuss few ideas on how to design a bot and we will discuss on how to interface with a web app.