Brief History of Web Scraping

Web scraping is becoming a more widely known term. Most associate it with web data extraction, the most efficient and the simplest way of copying large chunks of information online; however, did you know that web scraping was born for a completely different purpose and it took almost two decades for it to transform into web scraping we are familiar with now?

Here is the timeline:

The birth of the World Wide Web

The origins of very basic web scraping can be dated back to 1989 when a British scientist Tim Berners-Lee created the World Wide Web. Originally the idea was to have a platform where information could be automatically shared between scientists in universities and institutes all around the world. However, with the World Wide Web came three very important features that are the key elements for every web scraping tool nowadays:

• the URLs which we now use to designate a scraper to a specific website, 
• embedded hyperlinks that allow us to navigate through the designated website, 
• and web pages that contained various types of data - text, images, audios, videos, etc.

First web browser

Continuing his work, two years later, Tim Berners-Lee created the very first web browser, an http:// web page, all run on a server from his NeXT computer, giving a way for people to access and interact with the World Wide Web.

The Wanderer

Time-wise not much apart, in 1993, the first concept of crawling was born. The Wanderer, more precisely - the World Wide Web Wanderer developed by Matthew Gray at the Massachusetts Institute of Technology was a first of its kind, Perl-based web crawler whose sole purpose was to measure out the size of the web. The same year, the Wanderer was used to generate an index called the Wandex. Even though the author does not claim it, the Wanderer with Wandex had the potential to become the first general-purpose World Wide Web search engine.

JumpStation

However, the same year, 1993, the technology that laid grounds for big names such as Google, Bing, Yahoo, and other search tools on the web today - JumpStation was born and became the actual very first crawler-based web search engine. With it, millions of web pages indexed - the internet turned into an open-source platform of data in various forms.

BeautifulSoup

A bit more than a decade later, in 2004, came BeautifulSoup - HTML parser, a library of commonly used algorithms written in Python programming language. BeautifulSoup helped to grasp the sense of site structure and parse the contents within the HTML containers; therefore, saving hours of work for programmers. And since the internet had become this immense source of information that anyone with a computer and internet connection had access to, as well as it being easily searchable, people had started to take advantage of this by extracting the information available to them. For some time websites did not prohibit the ability to download the content of their sites; however, slowly that changed, and for the amount of data that was getting downloaded - simply manually copy-pasting was not an option; therefore, other ways of obtaining the information was bound to be developed.

Rise of visual web scrapers

Soon after, web scraping as we know it was born. The visual web scraping software Web Integration Platform version 6.0 which was launched by Stefan Andresen, allowed users to highlight the necessary information of a web page and structure that data into a usable excel file, or database which provided an opportunity for non-programmers to join and easily extract data from the web.

Nowadays, as technologies and industries progress, companies are looking to gain an advantage over their competition. And, due to the fact, that the amount of information available on the internet is growing exponentially, Web scraping is becoming one of the most prominent and widely-used methods of acquiring data at scale across various industries and business spheres

Future of web scraping

Web scraping has grown immensely in recent years, and almost guaranteed to continue upward growth. Currently, the commercial web scraping scene is mostly for gaining a competitive advantage by collecting leads, scraping competitors, price monitoring, etc. However, as technology develops, such as Artificial Intelligence, and data becomes even more accessible and crucial to different aspects of life, web scraping will advance with it and produce various new and remarkable applications that we are only looking forward to experimenting with.