Autonomous Client Acquisition Scraper
High-Performance Multi-Source Lead Harvester with Custom Intent-Relevancy Scoring and Headless Playwright Automation
Executive Summary
The Autonomous Client Acquisition Scraper & Control Center is an enterprise-grade, high-performance sourcing system designed to eliminate the exhausting operational friction of finding, prioritizing, and writing high-converting pitches to target prospects.
By orchestrating a robust multi-source crawling engine co-hosting lightweight RSS scanners, API parsers, and headless Playwright browsers, the platform autonomously aggregates warm leads from directories and forums. Integrated with a local **SQLModel relational schema**, a custom **0–100 prompt intent-scoring module**, and rapid LLM inference utilizing **Groq Llama-3-70B**, the pipeline automatically constructs highly-targeted, custom outreaches in **under 2 minutes** — bypassing manual research bottlenecks entirely.
Manual Sourcing Bottleneck
- • High Sourcing Overhead: Sifting through endless regional Craigslist feeds, Google Maps listings, and forums manually consumes 3-4 hours daily.
- • Relevancy Blindness: Manually reviewing hundreds of spam postings to identify the 5% that align with specific technical services.
- • Outreach Fatigue: Crafting high-converting, personalized value pitches based on the company's background causes writer fatigue.
- • Dashboard State Crashing: Running programmatic updates on Streamlit filter states during database queries often triggers layout API conflicts.
The Engineering Solution
- • Tri-Engine Sweep: Automates simultaneous crawling using HackerNews Algolia endpoints, RSS parsers, and stealth Playwright engines.
- • Intent-Scoring Core: Runs unstructured text through an AI evaluation module to assign a custom priority metric (0-100) instantly.
- • Groq AI Synthesizer: Integrates context-aware prompt templates to draft custom, conversion-optimized outreaches in seconds.
- • Safe Callback Interceptor: Implements custom `on_click` state overrides to update dashboard filters with zero interface exceptions.
Lead Sourcing System Architecture
Core Architecture Pillars
1. Multi-Source Scraping Core & Playwright Stealth Integration
To harvest leads without trigger blocks or CAPTCHA defenses across regional listings and physical business directories, we implemented distinct data pipelines:
- Playwright Harvester spins up headless Chromium instances with custom user-agents, randomized scroll intervals, and device dimension mimicking to safely scrape targeted physical local businesses.
- API and RSS Connectors fetch real-time listings from HackerNews (via Algolia REST APIs) and Craigslist feeds asynchronously with minimum overhead.
2. Asynchronous State Callbacks & Multitasking Workflows
Streamlit frontend applications normally re-run the entire script context from top-to-bottom on user interactions, which poses unique state sync challenges:
- Programmatic Selectbox Modification: Standard state changes on active select widgets normally trigger layout API exceptions. We resolved this by integrating custom `on_click` callback functions that update `st.session_state` parameters safely before elements are rendered.
- New Tab Badge Interception: Clicking dynamic
NeworDraftedworkflow status badges programmatically constructs URL query parameters and intercepts them via secure routing loops, launching the side-by-side pitch editor panel directly in a **new browser tab** and leaving the main prospects dashboard list completely intact.
3. Dynamic Lead Action Console & Side-by-Side Pitch Editor
Outbound sales managers require rapid context shifts without closing their primary search lists:
Managing Multi-threaded SQLModel Relational Databases
Because multiple background scrapers and dynamic multi-tab Streamlit instances write to the SQLite database simultaneously, we configured the connection engine defensively:
- Constructed SQLite database links using
connect_args={"check_same_thread": False}parameters. - Utilized a secure context manager session handler (`with Session(engine) as session`) that safely commits transactional updates and isolates operations in concurrent processing environments.
- Mapped database relations using SQLModel ORM structure, guaranteeing full schema safety while saving or editing OutreachDraft objects.
Database & Schema Design
The system operates a local SQLite database designed using SQLModel ORM structures, maintaining a strict 1-to-many relationship between acquired leads and synthesized pitches:
Lead (Table Schema)
ID: Integer (PK)- company string (Nullable)
- title string
- description string
- source "HackerNews" | "Craigslist" | "Maps"
- score integer (0–100)
- status "new" | "drafted" | "contacted" | "archived"
- createdAt DateTime
OutreachDraft (Schema)
ID: Integer (PK)- leadId [FK] ForeignKey (Lead.id)
- subject string
- body string (TEXT)
- createdAt DateTime
- updatedAt DateTime
Dynamic SQLite Indexing Strategy
- Table Indexes: Built SQL indexes on `Lead.score` and `Lead.status` columns to ensure database search filters inside the Streamlit grid run in **O(log N) complexity**, guaranteeing instantaneous rendering even with thousands of crawled leads.
- ForeignKey Cascading: Mapped cascading delete configurations so archiving/removing a Lead automatically cleans up the corresponding OutreachDraft record to keep storage requirements minimal.
Operational Lead Intent Valuation Tiers
To prioritize high-value client opportunities, the organic scoring module runs posting descriptions through strict parameters to group leads into separate operation workflow paths:
| Relevancy Tier | Score Range | Evaluation Rules | Action Strategy |
|---|---|---|---|
| High Relevancy | 80 – 100 | Direct tech stack match (e.g. Next.js, AI, automation) + urgent signals. | Auto-Generated Pitch |
| Medium Relevancy | 50 – 79 | Generic tech requirements, partial stack match, or low urgency listings. | On-Demand Draft |
| Low Relevancy | 0 – 49 | Out-of-scope work (e.g. basic data entry), non-technical listings, or spam. | Automatic Archive |
Key Benefits & Tangible Outcomes
92% Operational Hours Reclaimed
Reduces typical manual sourcing tasks to under 2 minutes, saving outbound agency managers 15-20 tedious hours every single week.
Top-of-Inbox Outbound Speed
Generates unique personalized value pitches within seconds of a post appearing online, placing outreaches at the top of client inboxes.
Unified Sourcing Ingestion
Aggregates HackerNews API parameters, Craigslist regional RSS feeds, and dynamic Google Maps local lists into a single consolidated dashboard.
0-Error Production Builds
Integrating dynamic suspense boundaries prevents Next.js compiler exceptions, leading to robust deployment builds and stable code structures.
Looking for a full-stack developer to build custom software?
Let's discuss how we can scale your operations and build high-performance systems for your business.
Get in Touch / View Resume →