Skip to main content
← OptiTalent

Trust & transparency

OptiTalent-JobLibrary crawler

OptiTalent-JobLibrary reads public job ads on employers' own hiring boards, so people using OptiTalent can find jobs they can actually get. Every job links to the original ad, so applicants go back to the source.

01

How to recognise it

User agent: OptiTalent-JobLibrary/0.1 (+https://optitalent.cc). Every request comes from 46.250.243.223 and is signed with HTTP Message Signatures (Web Bot Auth): the Signature-Agent header is "https://api.optitalent.cc", and the public key is at https://api.optitalent.cc/.well-known/http-message-signatures-directory.

02

How it behaves

Each site's robots.txt and terms are read before it is added, and robots.txt is read again on every run: a path it disallows for us isn't requested, and its crawl-delay is kept.

About one request a second to a site. The public job-board APIs built for career sites (Greenhouse, Lever, Ashby) get a few a second, never faster than their crawl-delay. It runs once a day.

A 403 stops all requests to that host for the run. A 429 halves our rate and waits as Retry-After asks; if it comes again, the host stops for the run. It never retries around a block, passes challenges, or changes its user agent. It reads public job ads only: no accounts, no logins, no applicant data.

03

What it keeps

The job's title, employer, location, link and dates, fields derived from the ad (such as seniority, employment type and stated work-rights requirements), and a numeric search index computed from the ad.

The ad text, cleaned, with phone numbers and email addresses removed before it is stored. It is shown on that job's page on optitalent.cc, next to a link to the original ad, and used for our own search index. It is never sold or passed on, never used to train a model, and deleted 30 days after the ad closes.

04

Opting out

Block OptiTalent-JobLibrary in robots.txt; it takes effect on our next run, within a day. Or email privacy@optitalent.cc: we remove a source within two business days of a request, including everything kept from it.