Autonomous Lane-Keeping RC Car

Built as part of the AIRE475 course at Abu Dhabi University, this is a fully autonomous RC car that navigates a two-lane track in real time, running entirely on a Raspberry Pi 5. The system uses a dual-stream pipeline, one for lane following using Bird's Eye View (BEV) transformation, polynomial fitting, and Kalman filtering, and another for obstacle detection using a NCNN-quantized YOLOv11n model. Both streams feed into a synchronized dual-PID controller managing steering and speed simultaneously. The car successfully navigated curved sections, handled dashed lane markings, and stopped within ±2cm of obstacles at 15+ FPS. A real-time Flask web dashboard enabled live tuning of PID gains during test runs.

Implementing advanced autonomous driving (robust lane detection, real-time object tracking, and multi-axis vehicle control) on a resource-constrained Raspberry Pi 5 that must run everything at 15+ FPS. The indoor arena added challenges: curved track sections, dashed lane markings, and a highly reflective floor that caused specular noise in the camera feed.

A hybrid computer vision pipeline combining classical image processing (BEV transformation, HSV masking, Top-Hat enhancement, Canny edge detection, polynomial fitting) with a NCNN-quantized YOLOv11n deep learning model. Two parallel perception streams feed into a dual-PID controller coordinated through a differential drive mixer. Kalman filters stabilize noisy polynomial coefficients and allow the car to coast through brief detection gaps.

1. 01Pi Camera captures raw frames in real time on the Raspberry Pi 5
2. 02Lane pipeline applies BEV homography, HSV white/yellow masking, Top-Hat enhancement, and Canny edge detection
3. 03Sliding window algorithm locates lane pixels, then fits quadratic polynomials to left and right lanes independently
4. 04Kalman filter smooths polynomial coefficients and predicts lane position between frames
5. 05Obstacle pipeline runs raw frames through NCNN-quantized YOLOv11n for real-time bounding box detection
6. 06Detected objects projected into BEV space, with the closest in-lane object designated as the Most Important Object (MIO)
7. 07Steering PID calculates lateral offset from lane center, outputs differential motor correction
8. 08Distance PID uses MIO distance to reduce speed and stop the car safely before obstacles
9. 09Both PID outputs mix into left/right PWM signals driving the differential motors via GPIO
10. 10Flask dashboard streams live camera + BEV feed and allows real-time slider-based PID tuning

• Dual-stream parallel pipeline: lane following and obstacle avoidance running concurrently
• Bird's Eye View (BEV) homographic transformation eliminating perspective distortion for accurate lane modeling
• NCNN-quantized YOLOv11n object detection achieving 15+ FPS on Raspberry Pi 5
• Kalman filter for smooth Kalman-filtered lane polynomial coefficients with predictive 'coasting' through occlusions
• Dual-PID control: steering PID for lateral lane centering, distance PID for longitudinal obstacle avoidance
• Most Important Object (MIO) identification via BEV ego-lane projection and distance estimation
• Real-time Flask web dashboard for live Kp/Ki/Kd tuning and camera feed monitoring during runs
• Stopping precision within ±2cm of detected in-lane obstacles
• HSV masking + Top-Hat morphological transform for robust lane detection under reflections and variable lighting

Built for AIRE475 (Self-Driving Cars) at Abu Dhabi University, supervised by Dr. Sajid Khawaja. The system achieved 15+ FPS inference on Raspberry Pi 5, successfully navigated curved track sections and dashed lane markings, maintained tracking through reflective flooring, and stopped within ±2cm of obstacles. Team: Housein Hassan Kahhoul, Omar Majed Saab, El Sayed Hesham Mowafi.