Extending Web Scraper Cloud with Patterns, a data app platform

Below, we’ll describe what Patterns is and walk through the set up for an app in Patterns that uses Web Scraper Cloud to acquire data and scan for product price differences on e-commerce websites. This app is a simple boilerplate template that can be extended to provide a more advances functionality. 

Patterns + Web Scraper Cloud — Price Intelligence App

Patterns provides a powerful interface for building data pipelines and visualising data scraped from Web Scraper Cloud. Many of these pipelines end with actions being triggered at certain thresholds. For instance, the example app we discuss below can be extended to power a pipeline to automatically increase/decrease the price of the comparable item in their Shopify account and sends a Slack alert to their pricing team.

URL: patterns graph

Patterns Graph

There are a wide variety of use-cases that the combination of Web Scraper Cloud and Patterns can solve for and we hope the below template provides the perfect jumping off point for your web scraping project!

What is Patterns?

Patterns is a data platform designed to be an easily deployable solution that:

• Managing the ingest, transformation, orchestration, visualisation, and automation layers of a data stack.
• Natively includes a database and compute instance, alternatively deploy Patterns on top of your existing data warehouse.
• An accessible, scalable, and highly expressive framework for building unified analytical and operational data systems.
• An easy-to-navigate UI that enables collaboration between technical and non-technical stakeholders.
• Includes a marketplace of solutions designed to enable code-replicability and decrease deployment timelines for popular use-cases.
• Built to solve problems for cross-functional data and business operations teams- focused on creating business value from data, solving the problem of stitching tooling and infrastructure together and providing an infinitely scalable solution for every stage of your data stack.

Most importantly, Patterns simple ecosystem empowers teams to stop using widgets, apps, and plug-ins (that each have their own subscription cost) to solve simple data management problems. Just clone an app, plug in your credentials, and deploy!

Setting up Web Scraper Cloud with Patterns is quick and simple:

Step 1. [Web Scraper Cloud] Clone a community sitemap or set up your own with the browser extension.

Step 2. [Web Scraper Cloud] Configure scheduler:

Web Scraper Cloud Scheduler

Step 3. [Web Scraper Cloud] Get a Web Scraper Cloud API key from here

Step 4. [Patterns] Clone the Ecommerce Price Monitoring App

Step 5. [Patterns] Click on the Get All Scraping Jobs for one Sitemap and enter the Web Scraper Cloud API key

Patterns Dashboard

Step 6. [Patterns] Press the play button ▶️ and watch your data flow in!