Build a company list from public directories

Public directories can provide the raw material for a company list, but downloading rows is only the beginning. A useful list needs defined coverage, a consistent schema, traceable sources, rules for resolving duplicate companies and locations, and a process for checking and refreshing the result.

This guide covers building an internal company dataset, not publishing an online directory. The aim is to turn records from several approved public sources into a list that sales operations, RevOps, market research, agency or data teams can explain and maintain.

Define the list before choosing a directory

Start with a list contract: a short specification that another team member could use to decide whether a company belongs in the dataset.

Record the following decisions before extraction:

Decision	What to define
Unit of data	Legal company, brand, branch, location or directory profile
Market	Industry, category, geography, organisation type and exclusions
Intended use	Market sizing, territory planning, account research or another approved purpose
Required fields	Fields that must be present before a record is accepted
Optional fields	Useful fields that may legitimately be absent
Prohibited fields	Personal or sensitive data the project should not collect
Coverage	Approved sources, categories, locations and search routes
Freshness	How recent each field needs to be for the intended decision

The unit of data is particularly important. Five branches of one company are useful for territory planning but misleading if the goal is to count distinct companies.

When both levels matter, create separate internal identifiers:

• company_id for the underlying organisation
• location_id for an office, shop, branch or facility

A directory's record ID should also be retained, but it identifies a listing within that source. It is not automatically a universal company identifier.

Select sources that contribute different evidence

One directory rarely represents an entire market. It may favour companies that actively maintain profiles, omit certain regions or use categories that do not match your definition.

Choose sources for a defined role rather than collecting several near-identical lists.

Source type	Useful contribution	Common limitation
Official company register	Legal name, identifier, registration status and registered address	May not describe current trading activity or market category
Industry or association directory	Evidence of specialisation or membership	Covers members, not necessarily the whole market
Local business directory	Trading locations, service categories and public business channels	Duplicate branches, broad categories and variable freshness
Supplier, dealer or partner locator	Evidence of a commercial relationship and regional coverage	Represents one network or vendor ecosystem
Software or review directory	Product category, market presence and source-specific reputation signals	Uses a platform-specific taxonomy
Company website	Corroboration of domain, locations and published business details	Poor as a market-wide discovery source

Before scraping, check whether the source offers an official API or bulk download that better suits the project. The Companies House API, for example, provides an official interface to public company information. Scraping is useful when approved records are available through accessible listing and detail pages but no suitable structured interface provides the required fields.

Create a source register containing the owner, source type, expected coverage, useful fields, available identifiers, access method, update pattern, known limits and internal approval status.

LinkedIn, social platforms and large sources behind login introduce different access, permission and identity issues and are outside this public-directory workflow.

Build a query matrix to expose coverage gaps

Directories often divide results by category, location and keyword. One broad search may stop at a page or result limit even when more relevant companies exist.

Turn the list contract into a reproducible query matrix. Each row should represent one route through a source.

Query key	Category	Geography	Search route	Validation check
hvac-lancashire	HVAC contractors	Lancashire	Category and county filter	Compare visible and extracted results
ventilation-leeds	Ventilation services	Leeds	Keyword and city search	Check category overlap
cooling-yorkshire	Commercial cooling	Yorkshire	Category landing page	Check pagination and result limits

Include synonyms where the directory's taxonomy requires them. “Industrial ventilation”, “air handling” and “commercial HVAC” may describe overlapping target companies without returning identical results.

Some overlap is useful. The same company appearing through two routes can reveal coverage gaps and supply evidence for later matching.

Attach the query_key to every extracted row. The source URL shows where a record appeared, while the query key explains how it was discovered. Without both, an unexpected gap is difficult to trace to a source, category, location or failed route.

Before running the full matrix, check whether the directory:

• shows a visible result count;
• stops after a fixed number of pages;
• changes the URL when filters are applied;
• loads results through JavaScript;
• requires scrolling or a Load more interaction;
• repeats promoted, featured or nearby records; or
• changes results across overlapping geographical terms.

Aim for the smallest repeatable query set that covers the defined market.

Check access and intended use separately

“Public” describes visibility. It does not settle every question about collection, storage, combination, redistribution or marketing use.

Review each source's terms, licence, access conditions and applicable rules before collection. Check robots.txt as part of the technical review, but do not mistake it for permission. RFC 9309 standardises the Robots Exclusion Protocol and explicitly distinguishes those crawl rules from access authorisation.

Privacy and direct-marketing obligations depend on the jurisdiction, data type, recipient and channel. A legal company name or general switchboard can raise different issues from a named employee, individual email or sole-trader record.

As a UK example, the Information Commissioner's Office guidance on collecting information and generating leads explains that personal information obtained from public sources still needs to be handled fairly, lawfully and transparently. Apply the rules relevant to the actual project and obtain specialist advice when needed.

An extraction tool can collect approved fields, but it cannot decide whether a source or downstream use is appropriate.

Design a schema that preserves the source evidence

Do not scrape each directory into an unrelated spreadsheet and plan to reconcile the columns later. Define a shared observation schema first.

Field group	Suggested fields
Provenance	source_name, source_record_id, source_url, query_key, observed_at, collection_run_id
Raw identity	company_name_raw, registry_id_raw, website_url_raw
Raw location	address_raw, city_raw, region_raw, postal_code_raw, country_raw
Raw business data	category_raw, phone_raw, status_raw
Normalised values	company_name_normalised, domain_normalised, address_normalised, phone_normalised
Resolution	company_id, location_id, match_method, match_confidence, review_status

Retain raw and normalised values together. If “North Coast Engineering Ltd.” becomes north coast engineering for matching, the published spelling still matters for audit and display.

Keep source-specific attributes outside the universal core. Ratings, memberships and incorporation status should not be collapsed into a generic score.

A robust model uses three connected tables:

1. Source observations: what each directory showed at a particular time.
2. Canonical companies and locations: the entities currently accepted for business use.
3. Match bridge: which observations relate to which company or location, including the evidence and review status.

This separation lets you correct a match without rewriting the source evidence.

Extract and test one source at a time

Build a separate sitemap for each source structure, even when all sources feed the same observation schema.

For a typical directory:

1. Identify the repeated wrapper containing one listing.
2. Extract the fields visible within that wrapper.
3. Follow the company link when required fields appear only on the detail page.
4. Configure numbered pagination, Next links, Load more behaviour or scrolling.
5. Retain the source record ID, source URL and query key.
6. Compare a limited export with the pages a person can see.

Web Scraper's AI Sitemap Wizard is a useful starting route for repeated directory records. Complex category navigation or interactions may require the visual builders and manual adjustment.

When fields are split between search results and profiles, follow the listing and detail-page workflow so fields from a company page remain attached to the correct listing. Optional or broken detail pages should produce empty fields or a recorded page outcome, not values shifted from a neighbouring company.

Test dense and sparse searches, early and later result pages, missing fields, multiple locations and alternative detail-page layouts. The pagination, load more and infinite scrolling guide covers the main loading patterns.

Use sitemap preview and validation before scaling. If a sitemap depends on scrolling, clicking or interaction-based pagination, it requires FullJS in Cloud rather than the Fast driver.

Rate limits, blocks and CAPTCHAs can make a source unreliable even when its selectors are correct. Proxies and retries may improve execution, but they do not establish permission or guarantee compatibility. If necessary, reduce the scope, replace the source or seek permission.

Merge and normalise observations before matching

Append each source export to a staging table. Do not send the rows directly to a CRM or overwrite the previous collection.

Confirm that fields with the same name have the same meaning, every row has provenance, and one row represents the declared unit of observation. Retain the extraction version when configuration changes could affect the output.

Normalisation should make values comparable, not decide that two rows represent the same entity. Useful transformations include:

• trimming whitespace and standardising casing and punctuation;
• parsing website URLs into normalised domains;
• splitting addresses into consistent components;
• storing phone country context and a normalised number;
• mapping source categories to an internal taxonomy; and
• keeping legal suffixes separately when producing a name-matching form.

Preserve raw values and taxonomy versions. Aggressive cleaning can erase distinctions between related legal entities, brands and branches.

Resolve companies with a matching hierarchy

Deduplication is an entity-resolution decision, not simply a “remove duplicate rows” operation.

The UK Office for National Statistics describes deterministic linkage as exact rule-based matching that can be used before probabilistic methods. That suggests a practical hierarchy for company records:

Tier	Evidence	Typical treatment	Main caveat
1	Same trusted registry identifier and jurisdiction	Automatic company match	Confirm it identifies the company, not another entity type
2	Same source and source record ID	Same source observation	Valid only within that directory
3	Same normalised domain with compatible name or geography	Strong company candidate	Groups, franchises and acquired brands may share or redirect domains
4	Same normalised name and full address	Strong location candidate	Several entities can occupy one address
5	Compatible name, postcode, phone and category	Composite candidate	Shared switchboards and generic names require care
6	Fuzzy name or address similarity	Manual or probabilistic review	Similar text alone is weak evidence

Do not use company name as a universal key. “ABC Services” can represent unrelated businesses, while one company may use a legal name, trading name and abbreviated brand.

Resolve company and location identity separately. Two rows with the same domain but different addresses may be branches of one company, not duplicate locations. Conversely, two companies with similar names at a shared office may remain separate entities.

Use states such as matched, probable and manual_review. Store the evidence and rule behind each decision.

Decide which value survives a conflict

Matching two observations does not determine which value should appear in the canonical record. Set field-level precedence rather than choosing one preferred source for the entire row.

For example:

• take a legal identifier or registration status from an approved official register;
• take specialisation evidence from a relevant industry directory;
• accept a domain from the company website or a corroborated directory link;
• select a branch address and phone from the best location-level source; and
• retain ratings, memberships and accreditations as dated, attributed observations.

Recency matters, but “newest wins” is not a complete rule. A newer aggregator record should not silently overwrite a trusted registry identifier.

Useful safeguards include:

• a blank value never overwrites a populated value without an explicit reason;
• conflicting trusted values create a review task;
• each accepted value retains its source and observation date;
• manual corrections remain separate from raw observations; and
• uncertain matches stay outside the released list.

Worked example: resolve a company without collapsing its branches

A sales operations team is building a list of commercial ventilation installers across northern England from a broad business directory, an industry association and a manufacturer's partner locator.

The business directory returns “NorthAir Services Ltd” in Leeds with northair.example, a local phone and a broad HVAC category. The association lists “NorthAir Services Limited” at a Manchester head office with the same domain and an industrial ventilation category. The partner locator shows authorised service locations in Leeds and Sheffield, both using the corporate domain but different addresses.

The shared domain and compatible names support one company, but not one location. The final model contains:

• one company_id for NorthAir Services;
• separate location_id values for Leeds, Manchester and Sheffield;
• the strongest available source for each accepted field;
• dated association and partner evidence; and
• all original observations linked through the match bridge.

If the Sheffield record disappears during the next partner-locator run, it enters review. The team first checks query coverage, extraction health and other sources rather than marking the branch closed.

Validate the list, not only the scrape

A completed job does not prove that the intended pages loaded, the selectors remained correct or the records satisfy the list contract. A successful response can still contain a challenge page, repeated content or empty fields, as explained in 200 OK but no data.

Validate at three levels.

Extraction health

Check records by source and query, pages reached, failed or empty pages, required-field population and unexpected changes from the previous successful run. Web Scraper Cloud's data-quality controls can monitor extraction-level measures. They do not perform cross-source entity resolution.

Dataset quality

The UK Government Data Quality Hub identifies accuracy, completeness, uniqueness, consistency, timeliness and validity as separate data-quality dimensions. Translate them into project-specific measures:

Dimension	Company-list measure
Completeness	Accepted records containing every required field
Uniqueness	Confirmed duplicate rate and unresolved candidate-match rate
Validity	Domains, phones, dates and identifiers in the expected format
Consistency	Conflicting values within or across matched sources
Timeliness	Age of the latest accepted observation
Accuracy	Stratified sample checked against source pages or an approved reference

There is no universal acceptable fill rate or duplicate rate. Define thresholds from the pilot and the consequences of an error.

Segment accuracy

Sample accepted, rejected and uncertain records across sources, regions and match tiers. Check segment fit, company-versus-branch classification and provenance.

Set a compact release gate with an owner, threshold and failure action for each critical measure. A failed page threshold might quarantine the full source run, while an unresolved match might quarantine only that company.

Refresh without erasing history

A maintainable list appends dated observations instead of overwriting the previous values in place.

For every refresh:

1. Run the same approved source and query coverage.
2. Compare extraction measures with the previous accepted run.
3. Append the new observations.
4. Match them to existing companies and locations.
5. Classify observations as new, changed, unchanged or missing.
6. Review ambiguous matches, conflicts and suspicious absences.
7. Publish a new accepted snapshot.

A missing observation does not prove that a company has closed. It might reflect a category change, expired membership, source redesign, incomplete pagination or access failure. Require stronger or corroborating evidence before making that decision.

Refresh cadence should follow field volatility and business need. Once a sitemap is validated, Web Scraper Cloud scheduling can run recurring jobs and completed dataset exports can deliver the results downstream.

Where Web Scraper fits

Web Scraper provides the extraction and delivery layer between approved public sources and the systems that validate and use the records.

Build and test one sitemap for each source in the browser extension, using the same core observation schema. Move tested sitemaps to Cloud for recurring execution, monitoring, retries and delivery.

Keep source approval, legal review, company and location matching, enrichment, contact verification, suppression, lead scoring, CRM assignment and outreach in the downstream workflow.

Start with one approved source, a tightly defined segment and a representative sample. Validate the full path from query coverage to canonical output before adding more directories. Web Scraper's lead generation workflow shows how tested directory sitemaps can support recurring public-company data collection while your team retains control of validation and downstream use.