AI Digest — Specification¶

Implementation specification for the AI Digest feature.

Business Requirements¶

Requirement	Target
Reduction in manual curation time	80% vs. current process
Content personalization	Dual summaries per item (novice and expert)
Operational cost model	Serverless, pay-per-use
Data scalability	No infrastructure changes required as volume grows

Pipeline Steps¶

1. Trigger¶

Cloud Scheduler fires daily at 08:00 UTC. No manual intervention required for routine runs.

2. Data Collection¶

The LangChain agent queries the Perplexity API using a structured prompt:

"Fetch AI developments from [START_DATE] to [END_DATE]. Output JSON array with fields: "
"title, summary_novice, summary_expert, category_tags, impact, source_links, "
"additional_references, research_paper, github_code, website."

Time-range filters are applied to reduce hallucination risk by anchoring responses to dated sources.

3. Approval Workflow¶

graph LR
    A[Cloud Function] --> B[Slack Draft Message]
    B --> C{Reviewer Decision}
    C -->|Approved| D[Push to Firebase]
    C -->|Rejected| E[Notify Editors]
    E --> F[Escalate after 24h]

Approvers interact with the draft via Approve/Reject buttons in Slack. Rejections trigger editor notification and escalate automatically if not resolved within 24 hours.

4. CMS Integration¶

Approved digests are delivered to Payload CMS via webhook:

{
  "type": "ai_digest",
  "content": "<Firebase document>",
  "status": "approved"
}

Next.js rebuilds via getStaticProps on receipt of the webhook.

Tech Stack¶

Component	Technology	Purpose
Orchestration	LangChain.js + Cloud Run	Multi-step reasoning, JSON generation, retries
Serverless runtime	Google Cloud Functions (Node.js)	Slack workflows, CMS/Firebase integration
Data storage	Firebase Firestore	Structured digest items with 30-day TTL
Approval system	Slack Bolt SDK + Workflow Builder	Interactive approvals with conditional routing
CMS	Payload CMS (GCP-hosted)	Content management with webhook support
Frontend	Next.js	Static site generation with incremental builds
Event bus	Cloud Pub/Sub	Decoupled agent communication

Cost Model¶

Tactic	Implementation	Target saving
Cold start mitigation	Cloud Run min-instances=1; GCF 256 MB memory	40% faster execution
Resource scaling	Pub/Sub topic partitioning	30% lower messaging costs
Data lifecycle	Firestore TTL policies (30-day auto-delete)	70% storage reduction
Error handling	Exponential backoff for API retries	Reduced retry cost

Security¶

Concern	Approach
Secrets management	Google Secret Manager for all API keys
Access control	Least-privilege IAM roles on Cloud Functions
Data at rest	Firebase encryption at rest (default)
Audit trail	Firestore logs capture all approval decisions

Failure Recovery¶

Dead-letter queues — Unprocessed Pub/Sub messages are routed to Cloud Storage for inspection.

Slack alerts — Real-time notifications fire on Perplexity API failures, approval timeouts, and CMS sync errors.

Retry mechanism — The LangChain agent uses fallback tools and exponential backoff. Maximum three retries before the run is marked failed and an alert fires.

Architecture Rationale¶

The serverless-first approach was chosen to match the infrequent, bursty execution pattern of a daily pipeline. Running dedicated infrastructure for a once-daily job would carry unnecessary fixed cost. The Slack approval step was prioritized as a hard requirement — no content publishes without human sign-off, regardless of confidence score.

See Architecture for system diagrams.