Methodology

How a dataset is built

Every dataset on RobotsDataset is one niche in one place — dental clinics in Sofia, shelf prices across Bulgarian retail chains — collected by AI agents from primary public sources and served so that every value can be checked.

1. Primary public sources only

Registries and open data (for example the daily price filings of retail chains with the Consumer Protection Commission), the businesses' own websites, and Google Places for the place_id alone. We honour robots.txt and a per-domain rate limit; we never log in; we never store review text. A policy decides whether a source is allowed before any request is made.

2. Snapshot first, extraction second

The raw content of every page is stored with its hash before it is read. Extraction runs on the snapshot, never on the live site — so a value can be re-extracted years later from the same evidence. If the hash has not changed, nothing is extracted again.

3. Extraction and validation

Deterministic parsers wherever possible; a language model only for unstructured pages and only when the content changed. Output is validated against the dataset's JSON Schema; an invalid value goes to a review queue, not into the database.

4. Provenance on every value

No value without source_url, observed_at, confidence and the extractor version. History is bitemporal (valid_from / valid_to): you can ask what a price was on a given date. Every record page shows the source and date next to each value.

5. Freshness

Every attribute has a freshness SLA; /v1/{dataset}/status reports the median age of the data and changelog what changed. Jobs are idempotent: the same collection can run twice without duplicates.

6. What we do not do

• Personal data beyond the professional — names of doctors and owners only as published professionally.
• Review text, or ratings republished as our own.
• Silent changes: DELETE /v1/optout and the correction form have worked since day one.

Check for yourself

Open any record and follow the source link next to a value. If something does not match, write to webmaster@robotsdataset.com — an error with provenance is fixed from the source, not by hand.

How to cite the data: /cite. License: /legal/license.