[2019] Vision Based Traffic Sensing and Control on FPGA - SOC Design

Vision-Based Adaptive Traffic Control System, a project by Team AbruTech, won Gold at NBQSA 2020 (National ICT Awards) organized by British Computer Society in Tertiary Student Category and has been nominated for APICTA (Asia Pacific) Awards to be held in Malaysia.​

​

A patent for our system is currently under review at NIPO and the system is being further developed with funding from World Bank via AHEAD into a product by a multidisciplinary team of engineers through the Enterprise (Business Linkage Cell) of the University of Moratuwa, together with RDA and SD&CC.​

​

We are forever indebted to our families of Tehara and Chinthana for hosting our team for weeks/months during strikes and study breaks, allowing us to work together. In addition, we thank our Supervisors: Prof. Rohan and Prof. Saman Bandara for their support and assistance.

Problem Statement

In most countries, traffic flow is controlled by traffic lights with pre-set timers. In Sri Lanka, this often causes congestion during peak hours as the system is not sensitive to the traffic levels in each lane of an intersection. To solve this, the traffic policemen usually turn off the lights and manually control the traffic during peak hours.

Solution

Team: Abarajithan, Rukshan, Tehara, Chinthana

We present a system-on-chip design that:

• Processes the video feed locally at edge through YOLOv2 (a 23-layer convolutional neural network for single shot object detection)
• Deduces traffic flow in each phase
• Suggests green times to the traffic lights 

Results

• The above videos demonstrate the object detection (with YOLOv2) and tracking (a custom-built algorithm that can be implemented in C without using any libraries) on test data (CNN was trained on images from a different road)
• The tracking algorithm is lightweight enough to run at thousands of FPS. The minimum FPS needed to reliably track vehicles travelling at 70 km/h (speed limit) is 3 FPS (as shown)
• Only the vehicles coming towards the camera are considered. The blue box signifies the vehicle being counted.

The timing diagram shows the convolution in Vivado simulation and the floating-point error

IOT dashboard for demonstration in SLIOT competition on Nvidia Jetson Nano logging data through MQTT

Project Tasks

The following tasks were (and are being) done.

1. Machine Learning

• Built four remotely powered, wirelessly data-collection device
• Collected, annotate and augment traffic images to create a Sri Lankan traffic dataset (1500 images)
• Built a numpy-based inference framework (keras-like) from scratch as the testbench
• Optimized the architecture of YOLOv2 object detection neural network for hardware implementation
• Trained YOLOv2 and TinyYOLO

2. FPGA Implementation

• Designed a resource-efficient hardware architecture for a CNN acceleration engine to implement YOLOv2 on FPGA
• Designed memory pipelines for high throughput data feeding
• Implementing and debugging the acceleration engine & memory pipeline

3. Object Tracking and Traffic Sensing

• Built a standalone (no libraries used) vehicle tracking algorithm
• Built vehicle counting and green-time allocation 
• Finalize algorithms and rewrite them to C (bare-metal on ARM side of ZYNQ FPGA.

4. Traffic Simulation and Testing

• Built a simulation model of Piliyandala bypass junction in VISSIM (industry grade traffic simulation software used by civil engineers to design intersections) to test traffic control algorithms

5. IOT Implementation

• Logging data to a central server through MQTT, when demonstrating the project in NVIDIA Jetson Nano

Methodology

1. Data Collection

Building and fixing the data collection device

2. Modifications to YOLOv2

• Fused batch normalization into convolution by modifying the weights and biases accordingly.
• Interchanged conv => leaky-relu => max-pool to conv => max-pool => leaky-relu to reduce power.
• Changed the output layer from 80 classes to 5 classes, by reusing weights of appropriate classes.
• Changed grid size from (13 x 13) to (12 x 8) and designed the sensing algorithm accordingly
• Trained with custom Sri Lankan Traffic Dataset
• Built a numpy-based inference framework and tested custom floating-point arithmetic (two types of float8), float16 and integer quantization.

3. CNN Accelerator Design

• Accelerator core v1.0 was designed to perform 12 of 3x3 convolutions in 9 clock cycles, using 9-muxes = 24, 3-muxes = 48, 16-bit registers = 144,  Multipliers = 3, Accumulators = 3.
• This was redesigned into core 2.0, which was 4 times faster, using five times fewer 3-muxes, zero 9-muxes, about 20 times fewer registers (for the same speed), with 100% utilization of all multipliers and adders.
• Currently building the caches and memory pipes to run the AXIS cores without stalling.

4. Object Tracking

• Built a custom lightweight tracking algorithm that can be implemented in C, without any libraries, so it can be run bare-metal (standalone) on the ZYNQ-PS side with minimal memory bandwidth (such that the ZYNQ-PL can use maximum bandwidth)
• Near 97% vehicle counting accuracy in the daytime, 85% accuracy in the night, rainy time, on test data (on a road the CNN has never seen before)
• Hoping to achieve near 100% counting accuracy in day, night and rain conditions through improvements.
• NOTE: Object detector (YOLOv2) has less accuracy. But tracking algorithm is designed to obtain  near 100% accuracy in vehicle counting and identification

5. Traffic Control Algorithm

• Designed and tested 8 algorithms based on density, bounding box count, flow...etc
• Currently working on eliminating traffic snake formation

Modeling the Piliyandala bypass junction in VISSIM - An industry-grade traffic simulation software

[2018] Custom Processor Optimized for Image Processing

The major challenge in designing a processor is the trade-off between the size of ISA, hardware complexity and user-friendliness. ABRUTECH is a unique custom processor highly optimized to manipulate matrices while preserving the functionalities of a generic processor. It has been designed to strike the delicate balance in the above tradeos. While having only 16 instructions, the ISA of ABRUTECH is crafted to be simple, yet highly powerful and is implemented using only 1000 logic elements.

This is demonstrated in the results section, with sample programs that are only 30-40 bytes long but are able to downsample and upsample any 512x512 image by any integer, detect edges, find prime numbers and Fibonacci numbers. . . etc.

Repository: github.com/abarajithan11/ABRUTECH_custom_image_processor

ISA

With only 16 instructions each of which execute in only 2.2 clocks cycles on average, our ISA and the complier allow the programmer to write programs quickly that take only 1.8 bytes of memory per instruction on average.

Our ISA was designed to incorporate advantageous features from both RISC and CISC instruction sets. As in RISC, each operation is designed to perform a specific task, especially the load and store operations are maintained strictly separate. However, unlike RISC, not all instructions are of equal length, but are either 1, 2 or 4 bytes long. Certain instructions are encoded, resulting in high code density. This allows building smaller programs that use the limited instruction memory efficiently while also allowing faster execution. However, this is balanced with maintaining moderate hardware level complexity in our system architecture.

 

Architecture

ABRUTECH works with an 8-bit wide, 262144-bit (512x512) deep data memory and an 8-bit wide 256-bit deep instruction memory, both of which can be loaded through UART. The system was coded in Verilog HDL using Intel Quartus II Prime and implemented successfully on an Altera de2-115 development board.

While being an 8-bit processor, (bus sizes and most register sizes being 8-bit), ABRUTECH features a 12-bit ALU and accumulator, which allows it to process calculations with intermediate steps that give results up to 4096, without causing an overow error. The ALU is also designed to perform round-o divisions (unlike the typical oor division), to improve accuracy.

Our processor also features a special module called Address Maker, which (optionally) allows the programmer to navigate a 512 x 512 matrix either row wise or column wise, without the need of a complex algorithm. This helps in implementing image processing algorithms such as downsampling, nearest neighbor upsampling, upsampling by bilinear interpolation.

Another special feature in ABRUTECH is a bank of shift registers, which help to perform linear convolution operations several times faster. Together with Address Maker, this allows the programmer to perform 2D convolution with a linearly decomposable kernel, such as Gaussian smoothing or edge detection, without losing a row and a column of data in the process, as with the traditional algorithms.

Compiler and Simulator

We also built a corresponding python based compiler program, which scans the excel sheet where ISA is specied and translates the algorithm written in the human language to an array of binary values, which are then sent to the instruction memory through UART. The compiler identifies syntax errors, which allows the programmer to write assembly code with ease, using our ISA.

In addition to the compiler, we also built a simulator software for ABRUTECH. The simulator can run the algorithm like the processor and show the values of registers and memory at each step, helping us debug an algorithm fast and remotely, without repeatedly loading it into the processor.

Algorithms

As detailed in the report, we devised brand new algorithms for downsampling and other tasks, which are mathematically justified (in the report) to have better accuracy and speed than traditional algorithms.

Hardware Debugging Features

The system itself is implemented with hardware debugging features, such as the ability to run the processor either at 1 Hz clock frequency, 10 MHz clock frequency or through a manual clock provided by a push button. We are able to see the currently fetched instruction and currently retrieved data on 7 segment displays and LED bulbs.

Our processor is also free of major hardware vulnerabilities, such as spectre and meltdown since we did not have the time to implement the branch prediction and speculative execution modules.

Results

While being optimized to process square images, the qualities of a generic processor are also preserved, which is presented in the section `Preservation of Genericity'. Algorithms to calculate the Fibonacci sequence and to find prime numbers less than 256 have been implemented and presented with results.

Implementation of each of these algorithms resulted in a sum of squared difference (SSD) error of zero, which shows the accuracy of our FPGA implementation.

Report - Processor

Report - Graphic Equalizer and Cache