# openwebdata.io — measurement scripts, for Bright Data compliance review

Prepared 14 September 2026 for Zora, Compliance Officer at Bright Data, after our
call the same morning. These are the actual files from my repository, unchanged.
Nothing has been removed except credentials, which have never been part of the
code (see *Credentials* below).

Contact: Sebastian Rantala, Bastinet (sole trader, Finnish business ID 2492063-7),
sebastian@bastinet.fi

---

## What the script does

Once a month, from one server that I administer myself in Finland, the script
sends a small fixed set of HTTP requests through each proxy provider's network to
the same public pages, and records whether the real page came back. Bright Data
would be one of the providers measured. The results are published on
openwebdata.io with the date, and the raw result files are public:
https://openwebdata.io/wp-content/uploads/owd/runs/

It is a measurement, not a collection pipeline. Nothing from the target pages is
kept.

## One run, step by step (`aja.mjs`)

1. Read the target list (`kohteet.json`) and the provider list. Providers come
   from a credentials file **outside** this folder and outside the repository.
2. For each provider, ask `https://api.ipify.org?format=json` five times to see
   the exit address, its owning network (ASN) and country. This is also the
   control: it must always succeed, so a failure elsewhere is the target site
   blocking the request rather than a broken connection.
3. For each target, send **five** requests through that provider (`TOISTOT`,
   default 5). One request per target proved too noisy to publish.
4. Classify each response (`tulkitse()`, line 143): does the response contain
   the marker that proves the real page came back (`vaadi` in `kohteet.json`),
   is it a block page, a captcha, a geo message, an HTTP error, or a timeout.
5. Write one line per target: pass count, the reason, the HTTP code, the median
   time and the number of bytes. **No page content is kept.** To be precise
   about the mechanics (`hae()`, line 109): curl writes the response to a
   temporary file, the script reads at most the first 200 kB of it in order to
   test the marker, and the file is deleted immediately afterwards, in the same
   function. Nothing from a target page is parsed, stored or published — the
   saved result file contains only the verdict, the reason, the timings and the
   byte count.
6. Save the result file, and publish it after a manual check.

`selain.mjs` is the same measurement through a real browser (Playwright,
Chromium, images and fonts blocked). `rotaatio.mjs` does nothing but ask for the
exit address 100 times in a row, to see how often the address changes; it never
touches a target site.

## Targets

`kohteet.json` contains nine entries, which is the list used for providers that
permit all of them. For the Bright Data account, **Google Search and LinkedIn are
excluded**, as agreed on the call. That exclusion is not a note in a document — it
is enforced in code:

- `rajoitukset.json` lists the excluded targets per provider, with the reason.
- `aja.mjs`, `eiSallittu()` (line 82) and the target loop (line 229) skip those
  targets **before any request is made** and record the row as `ei-sallittu`
  ("not permitted by provider"), which is published as `not permitted` rather
  than as a failed request. `selain.mjs` does the same.

The seven targets that would be requested through Bright Data:

| Target | URL | What it is |
|---|---|---|
| ipify | https://api.ipify.org?format=json | address lookup, control |
| Amazon | https://www.amazon.com/s?k=airpods | search results page |
| Zillow | https://www.zillow.com/homes/for_sale/ | listing page |
| Walmart | https://www.walmart.com/search?q=airpods | search results page |
| Best Buy | https://www.bestbuy.com/ | homepage |
| Indeed | https://www.indeed.com/jobs?q=data+engineer | job search results |
| Nike | https://www.nike.com/w/mens-shoes-nik1zy7ok | category page |

All seven are logged-off, publicly accessible pages. No logins, no cookies, no
accounts, no personal profiles, no content behind authentication.

## Volume

Five requests per target per run, so **35 requests per run**, once a month:
roughly 2–3 MB of traffic with plain requests, about 20 MB if the browser
variant is also run. A few gigabytes a year in total, not a month.

## What is published

Pass or fail, the reason, the HTTP code, the response time, the byte count, and
the exit address **masked to its /24 network** (`47.14.38.9` becomes
`47.14.38.x`), because those addresses belong to real households. The owning
network and carrier are kept, because that is what the measurement is about. The
masking is done by `tyokalut/julkaise-data.mjs` before anything is published;
happy to send that file too if it is useful.

## Credentials

The scripts read proxy credentials from a JSON file kept outside the repository
and outside this package (`salaisuudet/proxyt.json`, see `aja.mjs` line 37). No
credentials appear in any file here, and none are shared with anyone.

## A note on language

I work in Finnish, so the comments and some identifiers in the code are Finnish.
The logic is not affected. The words that appear most often:

| Finnish | English |
|---|---|
| `aja` / `ajo` | run |
| `selain` | browser |
| `rotaatio` | rotation |
| `kohde` / `kohteet` | target / targets |
| `palvelu` / `palvelut` | provider / providers |
| `pyyntö` | request |
| `toistot` | repeats |
| `tila` | verdict |
| `syy` | reason |
| `estetty` | blocked |
| `vajaa` | page returned without result content |
| `ei-sallittu` | not permitted by provider |
| `suora` | no proxy (baseline from the same server) |
| `osoite` | address |
| `tulkitse` | classify |
| `kuiva` | dry run, shows what it would do without sending anything |

I am happy to walk anyone on your team through the code on a call, or to answer
questions by email.

## How to run it

```
node aja.mjs --kuiva                  # dry run: prints what it would do, sends nothing
node aja.mjs                          # all configured providers, 5 repeats per target
node aja.mjs --vain brightdata        # one provider only
node selain.mjs --vain brightdata     # the browser variant
node rotaatio.mjs --vain brightdata   # address rotation only, no target sites
```