Blog

In the world of web scraping and data collection, efficiency and precision are paramount. That's why we're excited about the new feature that is being added to Web Scraper functionality: Website State Setup. This feature you to execute conditional actions based on specific criteria, ensuring you collect the most relevant data.

New Update: Web Scraper 1.79.3

Web scraping enthusiasts and data aficionados, rejoice! The latest release of Web Scraper, version 1.79.3, brings several enhancements and fixes.

Exciting Updates Alert!

We're thrilled to announce some powerful features and enhancements in our latest Web Scraper update!

Check out what's new:

We're thrilled to share the latest enhancements in our extension to make your experience even smoother. Check out the new features and improvements in our latest release:

Hey there! We've got some major news to share with you that will up your scraping game. We're thrilled to introduce the latest and greatest version of Web Scraper. Take your experience to a new level with the newly updated Link Selector and more.

It's time for our latest feature release article, and we can't wait to show you what we've been working on!

Below, we’ll describe what Patterns is and walk through the set up for an app in Patterns that uses Web Scraper Cloud to acquire data and scan for product price differences on e-commerce websites. This app is a simple boilerplate template that can be extended to provide a more advances functionality. 

For the last few months, we have been working towards the Web Scraper 0.6.5 release and finally, it is here!

Selectors when it comes to CSS and jQuery refer to code you can use to interact with specific elements in the DOM. They are particularly useful when you want to style or act upon certain elements of the document without altering other elements. Through a careful use of selectors, you can manipulate a page exactly as intended or isolate the specific data you’re looking to scrape.

Amazon Athena lets you use SQL to interactively query any data you have within an Amazon S3 Bucket. With Web Scraper Cloud, you can automatically sync web scraping job results to an Amazon S3 Bucket where you can then use Athena to quickly query this data with little additional overhead. In this article, we’re going to guide you through syncing Web Scraper Cloud with an S3 bucket and using Athena to generate and perform calculations on table data. While we’re covering a specific use case, you can adapt this same process to any other data set to gather insights.

Cheerio is a node package that allows you to easily parse and extract elements from markup. Unlike a browser, Cheerio doesn’t produce a visual rendering, load external resources, execute Javascript code, or apply CSS. As a result, it’s much faster than other solutions when it comes to scraping web pages for valuable data.

In this article, we’re going to specifically cover how to set up a headless scraper with Cheerio and spoof browser headers to access a wider range of web pages.

We’ve all been hearing about the ongoing cases of companies failing lawsuits against web scraping activities. Worth mentioning are the cases of LinkedIn vs. HiQ on public data scraping and the case of Facebook Inc. vs. BrandTotal Ltd. on scraping data from Facebook. What is more interesting is that in one case it was ruled in favour of the data scraper, but in the other - of the company whose platform is being scraped. Seeing such cases with different outcomes raises questions, especially due to the recent General Data Protection Regulation (GDPR), on how lawful web scraping of publicly available data actually is? 

The moment has come that we are ready to release the most requested feature of all time! Creating pagination correctly has been one of the most frequently reported problems; however, that is about to change. The pagination selector is here and so are other additional updates that will make scraping with Web Scraper easier.

For discovery and greater understanding, questions are a must. Therefore, we have collected the most frequently asked questions by the public about different web scraping topics.

Most people could agree that observing a picture, a simple graph, or a pie chart is easier and more efficient than scrolling through blocks of text or trying to parse out useful insights from looking at numerous columns in a spreadsheet. This common knowledge is the basis of why visualization is so important.

Also, in a rapidly changing world, people seek ways of becoming more efficient, ways of increasing the time spent focusing on the insights, rather than pouring endless hours and effort into collecting the necessary assets to gain those end results that determine so much. This is why automatization nowadays is a key topic.

Just like a vehicle tire leaves an imprint in the mud which can be washed away only by a great storm or rainfall, an impression leaves an impact on a product which then becomes the formatting foundation of an opinion that can not be altered easily. For this simple reason, creating an interesting, positive, and captivating impression on your customers will go a long way for your business's success and overall reputation.

Web scraping is becoming a more widely known term. Most associate it with web data extraction, the most efficient and the simplest way of copying large chunks of information online; however, did you know that web scraping was born for a completely different purpose and it took almost two decades for it to transform into web scraping we are familiar with now?

With constant development process and continuous growth, we are happy to have released a new notification system as well as a data quality feature to allow further automatization optionality.

It is no lie that data is power in many ways. For different reasons and applications, different information available online can be used for gaining an advantage in various spheres of life, especially in business.

Nowadays, with almost everything being available online - many are looking for ways to start a new eCommerce business or automate an existing one. 

Imagine a fast-track to improve the efficiency of your new, already existing online store. CSV export from Web Scraper and import to Shopify can get you updating your product lists in only a few minutes. 

Data nowadays can be the driving fuel of a company. With the huge increase in technology and data, it has become more important than ever to retrieve, transform, and store that data correctly and effectively.

Scraping might get hard at times when you're dealing with website structures that frequently change or in general, are hard to scrape with just the point-and-click interface. No need to stress, with the knowledge of the CSS selector - any website structure can be overcome and any website - scraped. Here are the TOP 5 CSS selectors that we frequently use and might be of great use to you.

In the previous blogs, we have gone through the classical way of scraping and exposed a tip of how to potentially prevent sitemaps from breaking when an e-commerce site changes product placements in designated categories. Now we are going to take a look at another scraping method which is - scraping with the “Sitemap.xml Links” selector.

In the previous blog, we explained how to retrieve the necessary data the classical way by going through the categories, sub-categories, then the products, etc. It is the most primitive and intuitive way of gathering data with Web Scraper extension; however, sometimes a problem arises when the website layout is changed or the placement of various products is altered. For this reason. previously created sitemaps for that specific website may break, or stop working properly. 

With the enormous growth and development in technology, data being the main driver of modern and fast-growing companies, online business has evolved over the recent years. However, it comes as no surprise since ordering, reserving goods, and services online while not leaving the house is a huge time saver and accessible for everyone with a stable internet connection.

With cleaner data, we can begin to transform it. Data transformation can manifest in different forms. It can be clustering, merging, adding information, replacing strings, and so on. OpenRefine covers them all. 

We are happy to announce that Web Scraper 0.5.0 has been released! This release contains new features such as a new data selection UI engine, a new page load detection system, a welcome page, and a whole lot more!

Data transformation is a step for preparing data sets for AI training and analysis. Clean and transformed data is a vital part of precise and correct data reports and analysis. In this blog series part, we will look at what importance it is to detect and delete blank, inconsistent and duplicate data entries.

As the saying goes “garbage in, garbage out”. This can be associated with the idea of data analysis without data transformation beforehand. With bad, messy data, only lousy, chaotic analysis can be done. However, with data transformation, not only the process of analysis becomes easier but also the precision increases.

Have you ever been in a situation when you have two or more data sets of completely different structures? So different that it is impossible to analyze, manage, or integrate. It sounds like every professional’s worst nightmare.

If you are a journalist or a data enthusiast and you are in need of data, look no further. We, Web Scraper, offer a collaboration.

Nowadays, with increasing digitalization, where new technologies are constantly arising and computational power is available almost to everyone, the necessity for data has escalated drastically. 

Enormous amounts of data that traditional data-processing application software is not capable of dealing with, that is used for analyzing to reveal trends, patterns, associations of especially human behavior and interactions. The subject of Big Data and the usage has increased highly over recent years. With the immense development of technology and the large sets of data that are processed every day, the questions of how to take advantage of it is on everyone’s mind, regardless of the industry.

As possibly one of the wisest investments in big data future, data visualization can be a crucial tool for any business. With the continuous advancements in technology and the huge increase in the necessity for big data, visual representation of it can display and showcase many factors of a business that would have been hard to do without data visualization. In this blog, we are gonna explore ways of data representation with the help of a highly easy and versatile tool.

Of everything going on, we took interest in what people are most searching for on google search engine while the coronavirus outbreak.

Imagine a scenario where you invest an amount of money and there is a guaranteed profit! Hard? Impossible? Not at all! Let us provide you with a mathematically proven strategy, which, when applied correctly, actually provides such possibilities.

Imagine learning a function, which can work as various other functions; therefore, replacing them with only one and making work with big data easier. With the query function of Google Sheets that scenario is not that absurd because it provides exactly what described. Accessing, summarizing, filtering data from various spreadsheets with only one function - thanks to Google Sheets, the opportunity to do exactly that has been around for quite some time already.

We are happy to finally introduce a Parser feature for Web Scraper Cloud.

Usually, to post process data, a custom written script or extra time editing the data manually in a spreadsheet software would be the case; however, the Parser takes care and eases this process.

Have you ever planned a trip to a different state, city or even a country but did not know any places to have a delicious dinner at? We all know that Yellow Pages is the go-to site if we want to find restaurants in specific areas but it can often be frustrating to get boggled up with bottomless pages on it. Luckily, Web Scraper is here to solve this by allowing you to extract all the information you need from Yellow Pages. So you can enjoy great meals during your trip.

We are happy to announce that Web Scraper 0.4.0 has been released. This release contains a new selector, updates to other selectors and improved CSS selector generator. Starting from version 0.4.0 Web Scraper is also available in Firefox.