Third-party Crawlers — Aktifkan indexing tanpa kehilangan control

Use different rules for different crawlers.

Make indexing visible to users.

Use five layers.

Control entities and fields.

Knowing an entity exists is not full access.

Use refresh, retention, and deletion signals.

Stop access when permissions change.

Track access where possible.

Use scoped crawl or index endpoints.

Consent → Scope → Access → Indexing → Revocation.