Many Internet business are built on a foundation of valuable and useful information. Collecting useful data, whether for an investment firm, community newspaper or airline, can be time-consuming and expensive. To avoid the bother and expense, some businesses resort to simply extracting useful data from a competitor’s website. The pilfering of data without authorization is referred to as “web scrapping” or “web harvesting.”
In its most basic form one company can manually copy the valuable commercial data from a competitor. This is an expensive and impractical process that can be performed much more efficiently by software bots. These bots, or software robots, take actions which replace human behavior. Using these bots, a business can target and collect specific sets of data available on the public website from a competitor’s business. The bots can repeatedly enter a website and harvest thousands of items of data per minute as soon as the data is posted. The targeted data may include valuable information such as real estate listings, product price listings and product reviews. The airline industry, with its constantly changing information on flight schedules, flight delays and price changes, is a prime target for web scrapping.
