SGCarMart — Used Car Intelligence Platform
End-to-End Data Engineering Pipeline for Singapore's Used Car Market
Built an end-to-end data engineering pipeline for Singapore's used car market. The system scrapes approximately 15,000 listings from SGCarmart using a two-phase web scraper, processes the data through a Medallion architecture (Bronze → Silver → Gold), and serves it through an NL2SQL chatbot deployed on Hugging Face Spaces. The chatbot uses a 4-stage LangGraph pipeline with self-correction capabilities to translate natural language questions into SQL queries.
The Problem
Singapore's used car market lacks accessible, structured data for buyers and analysts. SGCarmart listings are unstructured and difficult to analyze at scale. Users need a way to ask natural language questions about car prices, depreciation, COE trends, and value comparisons without writing SQL.
The Solution
Designed a complete data platform with incremental web scraping, multi-stage data transformation, and an AI-powered NL2SQL interface. The system features SCD Type 2 history tracking, value scoring algorithms, and a 4-stage query pipeline with safety layers including schema locks, SQL allowlists, and self-correction with up to 3 retry iterations.
Key Features
Explore the main capabilities and functionality of this project
Medallion Architecture
Three-stage data pipeline: Bronze (raw HTML/RSC) → Silver (28 validated fields) → Gold (35 business columns with computed metrics)
NL2SQL Chatbot
Natural language to SQL translation with 4-stage pipeline and self-correction up to 3 retry iterations
Multi-turn Conversations
Maintains conversation history for contextual follow-up queries and iterative data exploration
Automated Pipeline
Daily scheduled GitHub Actions runner with Docker deployment on Hugging Face Spaces
Technical Challenges
Key challenges faced during development and how they were solved
Technology Stack
Technologies and tools used to build this project
Lessons Learned
Agentic AI is a powerful tool for building complex, self-correcting systems
OpenTelemetry provides comprehensive observability for monitoring and debugging agent traces
LangGraph is a versatile framework for building complex, self-correcting systems
Interested in Learning More?
I'd love to discuss this project in detail and share insights about the development process.