[2019] Vision Based Traffic Sensing and Control on FPGA - SOC Design

Vision-Based Adaptive Traffic Control System, a project by Team AbruTech, won Gold at NBQSA 2020 (National ICT Awards) organized by British Computer Society in Tertiary Student Category and has been nominated for APICTA (Asia Pacific) Awards to be held in Malaysia.​

​

A patent for our system is currently under review at NIPO and the system is being further developed with funding from World Bank via AHEAD into a product by a multidisciplinary team of engineers through the Enterprise (Business Linkage Cell) of the University of Moratuwa, together with RDA and SD&CC.​

​

We are forever indebted to our families of Tehara and Chinthana for hosting our team for weeks/months during strikes and study breaks, allowing us to work together. In addition, we thank our Supervisors: Prof. Rohan and Prof. Saman Bandara for their support and assistance.

Problem Statement

In most countries, traffic flow is controlled by traffic lights with pre-set timers. In Sri Lanka, this often causes congestion during peak hours as the system is not sensitive to the traffic levels in each lane of an intersection. To solve this, the traffic policemen usually turn off the lights and manually control the traffic during peak hours.

Solution

Team: Abarajithan, Rukshan, Tehara, Chinthana

We present a system-on-chip design that:

• Processes the video feed locally at edge through YOLOv2 (a 23-layer convolutional neural network for single shot object detection)
• Deduces traffic flow in each phase
• Suggests green times to the traffic lights 

Results

• The above videos demonstrate the object detection (with YOLOv2) and tracking (a custom-built algorithm that can be implemented in C without using any libraries) on test data (CNN was trained on images from a different road)
• The tracking algorithm is lightweight enough to run at thousands of FPS. The minimum FPS needed to reliably track vehicles travelling at 70 km/h (speed limit) is 3 FPS (as shown)
• Only the vehicles coming towards the camera are considered. The blue box signifies the vehicle being counted.

The timing diagram shows the convolution in Vivado simulation and the floating-point error

IOT dashboard for demonstration in SLIOT competition on Nvidia Jetson Nano logging data through MQTT

Project Tasks

The following tasks were (and are being) done.

1. Machine Learning

• Built four remotely powered, wirelessly data-collection device
• Collected, annotate and augment traffic images to create a Sri Lankan traffic dataset (1500 images)
• Built a numpy-based inference framework (keras-like) from scratch as the testbench
• Optimized the architecture of YOLOv2 object detection neural network for hardware implementation
• Trained YOLOv2 and TinyYOLO

2. FPGA Implementation

• Designed a resource-efficient hardware architecture for a CNN acceleration engine to implement YOLOv2 on FPGA
• Designed memory pipelines for high throughput data feeding
• Implementing and debugging the acceleration engine & memory pipeline

3. Object Tracking and Traffic Sensing

• Built a standalone (no libraries used) vehicle tracking algorithm
• Built vehicle counting and green-time allocation 
• Finalize algorithms and rewrite them to C (bare-metal on ARM side of ZYNQ FPGA.

4. Traffic Simulation and Testing

• Built a simulation model of Piliyandala bypass junction in VISSIM (industry grade traffic simulation software used by civil engineers to design intersections) to test traffic control algorithms

5. IOT Implementation

• Logging data to a central server through MQTT, when demonstrating the project in NVIDIA Jetson Nano

Methodology

1. Data Collection

Building and fixing the data collection device

2. Modifications to YOLOv2

• Fused batch normalization into convolution by modifying the weights and biases accordingly.
• Interchanged conv => leaky-relu => max-pool to conv => max-pool => leaky-relu to reduce power.
• Changed the output layer from 80 classes to 5 classes, by reusing weights of appropriate classes.
• Changed grid size from (13 x 13) to (12 x 8) and designed the sensing algorithm accordingly
• Trained with custom Sri Lankan Traffic Dataset
• Built a numpy-based inference framework and tested custom floating-point arithmetic (two types of float8), float16 and integer quantization.

3. CNN Accelerator Design

• Accelerator core v1.0 was designed to perform 12 of 3x3 convolutions in 9 clock cycles, using 9-muxes = 24, 3-muxes = 48, 16-bit registers = 144,  Multipliers = 3, Accumulators = 3.
• This was redesigned into core 2.0, which was 4 times faster, using five times fewer 3-muxes, zero 9-muxes, about 20 times fewer registers (for the same speed), with 100% utilization of all multipliers and adders.
• Currently building the caches and memory pipes to run the AXIS cores without stalling.

4. Object Tracking

• Built a custom lightweight tracking algorithm that can be implemented in C, without any libraries, so it can be run bare-metal (standalone) on the ZYNQ-PS side with minimal memory bandwidth (such that the ZYNQ-PL can use maximum bandwidth)
• Near 97% vehicle counting accuracy in the daytime, 85% accuracy in the night, rainy time, on test data (on a road the CNN has never seen before)
• Hoping to achieve near 100% counting accuracy in day, night and rain conditions through improvements.
• NOTE: Object detector (YOLOv2) has less accuracy. But tracking algorithm is designed to obtain  near 100% accuracy in vehicle counting and identification

5. Traffic Control Algorithm

• Designed and tested 8 algorithms based on density, bounding box count, flow...etc
• Currently working on eliminating traffic snake formation

Modeling the Piliyandala bypass junction in VISSIM - An industry-grade traffic simulation software

[2018: Internship at CSIRO] Developing an End to End Pipeline for Machine Learning in Robotics

The Internship

In May 2018, I was chosen for a 6 month paid internship program at DATA61 CSIRO (The national science agency of Australia), in Brisbane, Australia. I was supervised by Nicholas Hudson (who previously worked as a Senior Roboticist in NASA Jet Propulsion Laboratory, Boston Dynamics and Google) and Dr. Navinda Kottege.

The main task assigned to me was to develop an end-to-end pipeline for robotics-related machine learning for future use in DATA61 CSIRO and demonstrate it by extending NVIDIA's trailnet (Resnet-18 based) neural network for autonomous navigation of indoor hallways and hill-climbing. For this I did the following tasks:

1. Built (with a partner) a 6-wheeled robot platform for data collection
2. Collected data (with a partner) into ROS bags
3. Wrote scripts to convert ROSbags into TF records on Bracewell (CSIRO's supercomputer)
   I needed to modify and build custom singularity docker containers to run this.
4. Create a 20-layer residual convolutional neural network with both Keras-Tensorflow and pure Tensorflow.
5. Build a pipeline to thoroughly and efficiently shuffle and prefetch data from TF records and train the model.
6. Optimize the trained model for embedded devices (Jetson TX2) with TensorRT (both pure C++ and python tf-trt)
7. Implement the inference engines as ROS nodes (C++ and python both)
8. Modify the motor controller ROS package of Roboclaw to avoid thread racing and deadlock issues and make the robot autonomously navigate hallways
9. Build models for hill climbing and experiment with them.

Following is a presentation I did to the audience of scientists from Robotics and Automation Group, DATA61, CSIRO about my pipeline.

Hallway Navigation

[2018] Custom Processor Optimized for Image Processing

The major challenge in designing a processor is the trade-off between the size of ISA, hardware complexity and user-friendliness. ABRUTECH is a unique custom processor highly optimized to manipulate matrices while preserving the functionalities of a generic processor. It has been designed to strike the delicate balance in the above tradeos. While having only 16 instructions, the ISA of ABRUTECH is crafted to be simple, yet highly powerful and is implemented using only 1000 logic elements.

This is demonstrated in the results section, with sample programs that are only 30-40 bytes long but are able to downsample and upsample any 512x512 image by any integer, detect edges, find prime numbers and Fibonacci numbers. . . etc.

Repository: github.com/abarajithan11/ABRUTECH_custom_image_processor

ISA

With only 16 instructions each of which execute in only 2.2 clocks cycles on average, our ISA and the complier allow the programmer to write programs quickly that take only 1.8 bytes of memory per instruction on average.

Our ISA was designed to incorporate advantageous features from both RISC and CISC instruction sets. As in RISC, each operation is designed to perform a specific task, especially the load and store operations are maintained strictly separate. However, unlike RISC, not all instructions are of equal length, but are either 1, 2 or 4 bytes long. Certain instructions are encoded, resulting in high code density. This allows building smaller programs that use the limited instruction memory efficiently while also allowing faster execution. However, this is balanced with maintaining moderate hardware level complexity in our system architecture.

 

Architecture

ABRUTECH works with an 8-bit wide, 262144-bit (512x512) deep data memory and an 8-bit wide 256-bit deep instruction memory, both of which can be loaded through UART. The system was coded in Verilog HDL using Intel Quartus II Prime and implemented successfully on an Altera de2-115 development board.

While being an 8-bit processor, (bus sizes and most register sizes being 8-bit), ABRUTECH features a 12-bit ALU and accumulator, which allows it to process calculations with intermediate steps that give results up to 4096, without causing an overow error. The ALU is also designed to perform round-o divisions (unlike the typical oor division), to improve accuracy.

Our processor also features a special module called Address Maker, which (optionally) allows the programmer to navigate a 512 x 512 matrix either row wise or column wise, without the need of a complex algorithm. This helps in implementing image processing algorithms such as downsampling, nearest neighbor upsampling, upsampling by bilinear interpolation.

Another special feature in ABRUTECH is a bank of shift registers, which help to perform linear convolution operations several times faster. Together with Address Maker, this allows the programmer to perform 2D convolution with a linearly decomposable kernel, such as Gaussian smoothing or edge detection, without losing a row and a column of data in the process, as with the traditional algorithms.

Compiler and Simulator

We also built a corresponding python based compiler program, which scans the excel sheet where ISA is specied and translates the algorithm written in the human language to an array of binary values, which are then sent to the instruction memory through UART. The compiler identifies syntax errors, which allows the programmer to write assembly code with ease, using our ISA.

In addition to the compiler, we also built a simulator software for ABRUTECH. The simulator can run the algorithm like the processor and show the values of registers and memory at each step, helping us debug an algorithm fast and remotely, without repeatedly loading it into the processor.

Algorithms

As detailed in the report, we devised brand new algorithms for downsampling and other tasks, which are mathematically justified (in the report) to have better accuracy and speed than traditional algorithms.

Hardware Debugging Features

The system itself is implemented with hardware debugging features, such as the ability to run the processor either at 1 Hz clock frequency, 10 MHz clock frequency or through a manual clock provided by a push button. We are able to see the currently fetched instruction and currently retrieved data on 7 segment displays and LED bulbs.

Our processor is also free of major hardware vulnerabilities, such as spectre and meltdown since we did not have the time to implement the branch prediction and speculative execution modules.

Results

While being optimized to process square images, the qualities of a generic processor are also preserved, which is presented in the section `Preservation of Genericity'. Algorithms to calculate the Fibonacci sequence and to find prime numbers less than 256 have been implemented and presented with results.

Implementation of each of these algorithms resulted in a sum of squared difference (SSD) error of zero, which shows the accuracy of our FPGA implementation.

Report - Processor

Report - Graphic Equalizer and Cache

[2018] Smart Lock: Assembly programming, Product Realization, Marketing

The objective of this project is to gain experience in designing and launching a brand new product into a market. Our product is the 'Smart Door Lock', a safe door lock with IOT features.

I am the Group Leader for the group of 10. Apart from management and coordination responsibilities, I was the main mechanical designer in version 1.0 and the person responsible for micro-controller programming in version 2.0

Version 1.0

We designed the version 1.0 of this product in 2016. Having limited knowledge of design theory, we ran into a few problems. Key product goals were:

• A device that can be fixed on top of any manual lock to make it smart.
• Can be locked and unlocked through an encrypted platform from anywhere in the world
• Fully manually overridable from both inside and outside using a key.

As we went through multiple design cycles, we were facing a major design crisis. The problem was a combination of the following

• High torque forces required to turn the shaft of the lock
• Servo torque limitations, 
• Servo angle limitations 
• Necessity to override it completely from inside and outside
• Infeasibility to keep the torque connection from the manual override and the actual lock permanently connected (since this would result in externally turning the servo during manual override, which can destroy the servo)

As the lead designer in the group, I came up with a neat solution that included multiple gear wheels and a gear shift system which can solve all the above problems. However, it resulted in increased design and manufacturing complexity to the level such that we cannot produce it easily in Sri Lanka. 

.

Smart lock v1.0 pitching presentation from Abarajithan Gnaneswaran

.

Version 1.5

Hence, we have currently redesigned the lock by eliminating a minor requirement (goal) to result in an excellent design that is far simpler and cheaper to manufacture. Our prototype in that semester was based on this new design and we have decided to proceed with that design.

Github Repository

https://github.com/abarajithan11/smart_door_lock

[2017] Bidirectional Maze Solver: Pointer manipulation for flipping, Computer Vision ideas

In November 2017, I formed a group with 3 others to compete in Sri Lanka Robotics Challenge 2017 (to be held on January 2018). According to the task, the two teams must produce two robots each, one stationary and one mobile, which should coordinate via any wireless method to complete the task.

My Responsibilities

I was appointed the group leader and also I took the responsibility of designing and coding the mobile robot, which is a line maze solver. I also refined the concept and made final design of the stationary robot (which should be able to shoot a ball turning 360 degrees in horizontal plane and to about 10 degrees in vertical plane).

The Mobile Robot: Innovative Ideas

Since the task of mobile robot is relatively easy to perform, I realized the key for winning is the speed. When I considered different ways to reduce the time taken, I noted two factors:

1. The robot takes some time to make a U-turn at nodes of the maze. Also, a U-turn is dangerous as since the sensor panel moves completely out of the line, there is a chance the robot might get disoriented.
2. By traditional always-take-left algorithm, the robot visits empty nodes (where boxes are not kept), wasting time.

1. Bidirectionality: Eliminating U Turns

I proposed the idea to make the robot fully bidirectional. While increasing the cost, this needed some redesign of the body from traditional designs and the need for some innovative methods to be used in coding.

a) C pointers and OOP programming for bidirectionality

I have learnt the wonders of pointer & reference manipulations in C++ language and I have learnt OOP concepts through Java & PHP programming. I decided to combine them as a solution.

As shown below, I made classes for motors and created objects for each motor, sonar, IR panel and color detector. Then I defined references *pMotorLF, *pMotorLR...etc for front left, front right...etc motors from the POV of the robot. The rest of the code would be written in terms of the pointers, not of the objects themselves. 

eg: pMotorLF->drive(-100)

Using a custom flip function I could change the objects to which the reference is pointing to and this would effectively flip the entire robot internally.

A function definition for the motor class.

Flip function definition. Calling this function would immediately software-flip the robot by 180 degrees

b) Hardware issues & solutions

Due to bidirectionality, we decided to place the relatively expensive main line sensor panel (8-qtr) at the exact center of the robot. This has the disadvantage that the the small lag between sensing and acting would make the robot act too late and veer off the line. It was solved by (c)

2. Variable Speeds & variable PID constants

To solve this, I came up with the idea of using two sets of speed configurations. I proposed that we could use a relatively cheap 3-qtr at exact center of each front and back lines. 

• If the robot senses a line on the front 3-qtr, he assumes a a higher speed and the PID constants associated (and tuned) with respect to that speed. 
• If not, robot assumes a lower speed and associated PID constants.

Using this method, the robot moves at high speeds along straight lines and at lower speeds near junctions and curvy lines, reducing the error.

3. Loophole in the rules - Workaround for hard coding

According to the task, few nodes of the maze graph are left empty, without any boxes. A traditional take-left robot would visit them, wasting time. The solution would be to hard code the maze, but according to rules, the positions of the boxes are changed in every round and hard coding is a crime. 

As a workaround, I proposed the idea of manually inputting the maze using buttons before the round (which was not restricted, a loophole). I wrote an algorithms to always take left, but avoid the i-th left, where i is a member of an array (where  the turns to be avoided are stored). The array is initialized empty and it can be filled by pressing buttons on top of the robot. 

Before a round, when the boxes are placed, I would take down the turns that need to be avoided and press buttons according to the programmed protocol to fill the array with those numbers. The robot would then simply avoid those turns, saving time.

Ideas on Computer Vision

To avoid empty nodes, I also proposed a solution with machine vision. By that time, we were taking a module: "Fundamentals of Image Processing and Computer Vision" and I had some experience with image processing methods. 

I proposed, we could 

• Mount a camera module on the stationary bot (then the view and the projective transformation is known in advance)
• Apply projective transform, crop to the arena and get ImIn
• Skeletonize the thresholded ImIn to get the path
• Finding unique blobs in the R,G,B layers separately to identify boxes
• Using array navigation to find which nodes to be avoided
• Transmit an array full of "turns to be avoided" to the mobile robot

I proposed the above to Kanchana, my colleague who is actively researching in computer vision field and he agreed to join our team.

Project Discontinued

The group was highly active and as I wrote the algorithms, rest of the group started to build the prototypes for the robot. However, due to academic workload, the hardware guy in the group decided to quit. He needed to focus on the web development project which we needed to do for another module. Without a hardware guy, I could not continue this and finish it before deadline. Therefore, sadly the project was abandoned.

Github Repository

github.com/abarajithan11/bidirectional_bot

[2017] FIR Filter Design for Given Specifications

Introduction

This project is undertaken with the objective of demonstrating the design process of a digital filter using the windowing method in conjunction with the Kaiser window to fit given requirements. The filter is then analyzed and verified. A comprehensive summery of the basic principles of filter design also have been revised as a part of this project report. Mathematical analysis has been carried out using Matlab 2014a software. It is submitted as the semester project for the module 'Digital Signal Processing'.

Magnitude Response of the designed bandstop filter

Objectives

• To test the conceptual understanding of Fourier series methods of designing FIR Filters
• To provide experience in designing a FIR Filter to fit the given specifications

My Responsibilities

This is an individual project

• Analyzing given specifications and building the mathematical model
• Applying the equations to write a Matlab program
• Visualizing the results from Matlab program.
• Documentation

Skills Acquired

• Matlab
• Digital Signal Processing Theories

Results

CLICK ON IMAGES TO ZOOM

Attenuation in stopband

Ripples on lower passband

DFT of a test excitation signal

DFT of the signal passed through my filter

DFT of the signal passed through an ideal filter

Full Documentation

http://bit.ly/FIR-Design-Aba

[2017] UART Implementation with FPGA

Introduction

This was completed as a semester group project for the module: 'Electronics III'. A UART Receiver - Transmitter was implemented in FPGA using RTL programming. Both RTL and Testbench code were written and debugged. The final setup was tested with digital oscilloscopes from laboratory for evaluation.

Objectives

• To test our ability in RTL programming.
• To make us familiar with implementation using FPGA

Documentation

[2017] Ring Finding Robot: with GPS following, obstacle avoiding, wall following, color detection, parallel alignment, robotic arm manipulation capabilities

Through this project we built Gollum, an outdoor mobile robot to perform GPS following, obstacle avoiding, wall following, color detection, parallel alignment and robotic arm manipulation.

I was responsible for designing the algorithm and programming the robot (1,200 lines of code)

Task:

The robot starts from within the starting circle of 1m in radius. Out of three pre-defined target locations, one location is chosen at random and give to us in the start. We input the target location using a keypad/set of 3 switches. The robot should the follow GPS and visit the target, which is 20m away. On the way, there is a "forest" of obstracles (30 cm cubes) which the robot should avoid. Target is marked by a circle of white line (3cm thickness) 2m in radius around the target point.

At the target point, lies a cylindar, 50cm in radius. The cyclindar has 40 cm wide entrance on one side.The robot should find the cylindar, move to meet it, follow the wall around it, find the entrance, move in. Inside the cylindar lies a 5 cm high, 40cm diameter plate that contains a metal ring og 6cm diameter anywhere in it. The robot should be able to search and pick up the ring with an arm.

Then then robot should return to the starting location and stop.

GitHub Repository:

https://github.com/abarajithan11/gollum_ring_finder

Behind the Scenes:

I took Gollum to one of my basic robotics workshop as an inspiration. Some school students mishandled him, he fell down and the perspex body was broken in half.

I had to bandage him with a piece of cloth to bring him back.

Reprogramming the parameters for the new version

Rebuilding gollum's body. This time with a cool black sticker

Bending perspex with a high current precision heating device

The mess of wires... need to be sorted.

Samith's boarding room. The hardware workshop.

[2017] Designing, Simulating and Building an HF Amplifier

[2016] Microprocessor programming, PCB Design, Enclosure Design to build an Alarm Clock