Third-party Crawlers — Enable Indexing Without Losing Control
How to design third-party crawling and indexing for Work Identity using consent, scope, access, indexing, and revocation instead of all-or-nothing exposure.
Open indexing does not mean open everything
Search engines, AI crawlers, and partner crawlers have different purposes. One permission model for all of them is usually too coarse.
Step 1 — Start with consent
Users or organizations should know what data is indexable, by which type of crawler, and for what purpose.
Framework: Consent → Scope → Access → Indexing → Revocation
Consent — has the user agreed?
Scope — which entities or fields may the crawler read?
Access — through public web, tokens, signed endpoints, or scoped APIs?
Indexing — what may be stored in an index and for how long?
Revocation — what happens when permission changes?
Step 2 — Make scope precise
A public Profile may be indexed. A Project summary may be indexed. Private Evidence should not be. Selective Results should only be readable inside granted scope.
Step 3 — Separate crawler discovery from data access
A crawler may learn that an entity exists without being allowed to retrieve all of its content.
Step 4 — Give indexing a policy
Indexes can use refresh cycles, retention periods, deletion signals, and source timestamps to reduce stale data.
Step 5 — Make revocation real
If a user changes a Project from public to private, the system should stop future access and emit a signal so downstream indexes can update.
Step 6 — Preserve third-party provenance
Where technically possible, Forwork should know which crawler accessed which data, when, and under which permission scope.
Applying this on Forwork
A useful model: User Consent → Scope Registry → Access Policy → Crawl/Index Endpoint → Refresh/Revocation Signal. Without valid consent or scope, the endpoint does not expose data.
Conclusion
Do not only ask, “Should third parties be allowed to crawl Profiles?”
Ask: “Which third party, what may it read, for how long, under which permission, and how can that permission be revoked?”
Consent → Scope → Access → Indexing → Revocation.