Sales Intelligence Web Crawler

The Challenge

Sales teams need timely, accurate market signals to identify and prioritise leads. One key signal is company hiring activity—particularly whether roles are truly remote or just marketed as remote when they're actually hybrid. Job listings often self-report as "remote" to attract candidates, but this doesn't always reflect reality.

The Solution

I built a scalable web crawler that systematically discovers and analyses job listings across company websites, using LLMs to extract structured data and classify roles according to the client's specific definitions of remote work.

Technical Architecture

• Next.js: Full-stack application for crawler management, data viewing, and client-facing dashboards
• PostgreSQL: Storage for crawled data, classification results, and job listing history
• Railway: Infrastructure for deploying and scaling the crawler workloads
• LLM Integration: AI-powered parsing of unstructured job descriptions to determine true remote status

Key Capabilities

• Scalable Crawling: Processes thousands of company career pages efficiently
• Intelligent Classification: LLMs analyse job descriptions beyond simple keyword matching to determine genuine remote flexibility
• Custom Definitions: Classification rules configurable to each client's standards—not relying on what companies self-report
• Market Signals: Aggregates hiring trends to identify companies in growth phases

Results

This work directly contributed to securing the company's largest client. The accuracy of the remote classification—going beyond unreliable self-reported data—proved to be a key differentiator. This success opened new business opportunities and drove further investment in scaling the platform.