[2019] Vision Based Traffic Sensing and Control on FPGA - SOC Design

Vision-Based Adaptive Traffic Control System, a project by Team AbruTech, won Gold at NBQSA 2020 (National ICT Awards) organized by British Computer Society in Tertiary Student Category and has been nominated for APICTA (Asia Pacific) Awards to be held in Malaysia.​

​

A patent for our system is currently under review at NIPO and the system is being further developed with funding from World Bank via AHEAD into a product by a multidisciplinary team of engineers through the Enterprise (Business Linkage Cell) of the University of Moratuwa, together with RDA and SD&CC.​

​

We are forever indebted to our families of Tehara and Chinthana for hosting our team for weeks/months during strikes and study breaks, allowing us to work together. In addition, we thank our Supervisors: Prof. Rohan and Prof. Saman Bandara for their support and assistance.

Problem Statement

In most countries, traffic flow is controlled by traffic lights with pre-set timers. In Sri Lanka, this often causes congestion during peak hours as the system is not sensitive to the traffic levels in each lane of an intersection. To solve this, the traffic policemen usually turn off the lights and manually control the traffic during peak hours.

Solution

Team: Abarajithan, Rukshan, Tehara, Chinthana

We present a system-on-chip design that:

• Processes the video feed locally at edge through YOLOv2 (a 23-layer convolutional neural network for single shot object detection)
• Deduces traffic flow in each phase
• Suggests green times to the traffic lights 

Results

• The above videos demonstrate the object detection (with YOLOv2) and tracking (a custom-built algorithm that can be implemented in C without using any libraries) on test data (CNN was trained on images from a different road)
• The tracking algorithm is lightweight enough to run at thousands of FPS. The minimum FPS needed to reliably track vehicles travelling at 70 km/h (speed limit) is 3 FPS (as shown)
• Only the vehicles coming towards the camera are considered. The blue box signifies the vehicle being counted.

The timing diagram shows the convolution in Vivado simulation and the floating-point error

IOT dashboard for demonstration in SLIOT competition on Nvidia Jetson Nano logging data through MQTT

Project Tasks

The following tasks were (and are being) done.

1. Machine Learning

• Built four remotely powered, wirelessly data-collection device
• Collected, annotate and augment traffic images to create a Sri Lankan traffic dataset (1500 images)
• Built a numpy-based inference framework (keras-like) from scratch as the testbench
• Optimized the architecture of YOLOv2 object detection neural network for hardware implementation
• Trained YOLOv2 and TinyYOLO

2. FPGA Implementation

• Designed a resource-efficient hardware architecture for a CNN acceleration engine to implement YOLOv2 on FPGA
• Designed memory pipelines for high throughput data feeding
• Implementing and debugging the acceleration engine & memory pipeline

3. Object Tracking and Traffic Sensing

• Built a standalone (no libraries used) vehicle tracking algorithm
• Built vehicle counting and green-time allocation 
• Finalize algorithms and rewrite them to C (bare-metal on ARM side of ZYNQ FPGA.

4. Traffic Simulation and Testing

• Built a simulation model of Piliyandala bypass junction in VISSIM (industry grade traffic simulation software used by civil engineers to design intersections) to test traffic control algorithms

5. IOT Implementation

• Logging data to a central server through MQTT, when demonstrating the project in NVIDIA Jetson Nano

Methodology

1. Data Collection

Building and fixing the data collection device

2. Modifications to YOLOv2

• Fused batch normalization into convolution by modifying the weights and biases accordingly.
• Interchanged conv => leaky-relu => max-pool to conv => max-pool => leaky-relu to reduce power.
• Changed the output layer from 80 classes to 5 classes, by reusing weights of appropriate classes.
• Changed grid size from (13 x 13) to (12 x 8) and designed the sensing algorithm accordingly
• Trained with custom Sri Lankan Traffic Dataset
• Built a numpy-based inference framework and tested custom floating-point arithmetic (two types of float8), float16 and integer quantization.

3. CNN Accelerator Design

• Accelerator core v1.0 was designed to perform 12 of 3x3 convolutions in 9 clock cycles, using 9-muxes = 24, 3-muxes = 48, 16-bit registers = 144,  Multipliers = 3, Accumulators = 3.
• This was redesigned into core 2.0, which was 4 times faster, using five times fewer 3-muxes, zero 9-muxes, about 20 times fewer registers (for the same speed), with 100% utilization of all multipliers and adders.
• Currently building the caches and memory pipes to run the AXIS cores without stalling.

4. Object Tracking

• Built a custom lightweight tracking algorithm that can be implemented in C, without any libraries, so it can be run bare-metal (standalone) on the ZYNQ-PS side with minimal memory bandwidth (such that the ZYNQ-PL can use maximum bandwidth)
• Near 97% vehicle counting accuracy in the daytime, 85% accuracy in the night, rainy time, on test data (on a road the CNN has never seen before)
• Hoping to achieve near 100% counting accuracy in day, night and rain conditions through improvements.
• NOTE: Object detector (YOLOv2) has less accuracy. But tracking algorithm is designed to obtain  near 100% accuracy in vehicle counting and identification

5. Traffic Control Algorithm

• Designed and tested 8 algorithms based on density, bounding box count, flow...etc
• Currently working on eliminating traffic snake formation

Modeling the Piliyandala bypass junction in VISSIM - An industry-grade traffic simulation software

[2018: Internship at CSIRO] Developing an End to End Pipeline for Machine Learning in Robotics

The Internship

In May 2018, I was chosen for a 6 month paid internship program at DATA61 CSIRO (The national science agency of Australia), in Brisbane, Australia. I was supervised by Nicholas Hudson (who previously worked as a Senior Roboticist in NASA Jet Propulsion Laboratory, Boston Dynamics and Google) and Dr. Navinda Kottege.

The main task assigned to me was to develop an end-to-end pipeline for robotics-related machine learning for future use in DATA61 CSIRO and demonstrate it by extending NVIDIA's trailnet (Resnet-18 based) neural network for autonomous navigation of indoor hallways and hill-climbing. For this I did the following tasks:

1. Built (with a partner) a 6-wheeled robot platform for data collection
2. Collected data (with a partner) into ROS bags
3. Wrote scripts to convert ROSbags into TF records on Bracewell (CSIRO's supercomputer)
   I needed to modify and build custom singularity docker containers to run this.
4. Create a 20-layer residual convolutional neural network with both Keras-Tensorflow and pure Tensorflow.
5. Build a pipeline to thoroughly and efficiently shuffle and prefetch data from TF records and train the model.
6. Optimize the trained model for embedded devices (Jetson TX2) with TensorRT (both pure C++ and python tf-trt)
7. Implement the inference engines as ROS nodes (C++ and python both)
8. Modify the motor controller ROS package of Roboclaw to avoid thread racing and deadlock issues and make the robot autonomously navigate hallways
9. Build models for hill climbing and experiment with them.

Following is a presentation I did to the audience of scientists from Robotics and Automation Group, DATA61, CSIRO about my pipeline.

Hallway Navigation

[2018] Custom Processor Optimized for Image Processing

The major challenge in designing a processor is the trade-off between the size of ISA, hardware complexity and user-friendliness. ABRUTECH is a unique custom processor highly optimized to manipulate matrices while preserving the functionalities of a generic processor. It has been designed to strike the delicate balance in the above tradeos. While having only 16 instructions, the ISA of ABRUTECH is crafted to be simple, yet highly powerful and is implemented using only 1000 logic elements.

This is demonstrated in the results section, with sample programs that are only 30-40 bytes long but are able to downsample and upsample any 512x512 image by any integer, detect edges, find prime numbers and Fibonacci numbers. . . etc.

Repository: github.com/abarajithan11/ABRUTECH_custom_image_processor

ISA

With only 16 instructions each of which execute in only 2.2 clocks cycles on average, our ISA and the complier allow the programmer to write programs quickly that take only 1.8 bytes of memory per instruction on average.

Our ISA was designed to incorporate advantageous features from both RISC and CISC instruction sets. As in RISC, each operation is designed to perform a specific task, especially the load and store operations are maintained strictly separate. However, unlike RISC, not all instructions are of equal length, but are either 1, 2 or 4 bytes long. Certain instructions are encoded, resulting in high code density. This allows building smaller programs that use the limited instruction memory efficiently while also allowing faster execution. However, this is balanced with maintaining moderate hardware level complexity in our system architecture.

 

Architecture

ABRUTECH works with an 8-bit wide, 262144-bit (512x512) deep data memory and an 8-bit wide 256-bit deep instruction memory, both of which can be loaded through UART. The system was coded in Verilog HDL using Intel Quartus II Prime and implemented successfully on an Altera de2-115 development board.

While being an 8-bit processor, (bus sizes and most register sizes being 8-bit), ABRUTECH features a 12-bit ALU and accumulator, which allows it to process calculations with intermediate steps that give results up to 4096, without causing an overow error. The ALU is also designed to perform round-o divisions (unlike the typical oor division), to improve accuracy.

Our processor also features a special module called Address Maker, which (optionally) allows the programmer to navigate a 512 x 512 matrix either row wise or column wise, without the need of a complex algorithm. This helps in implementing image processing algorithms such as downsampling, nearest neighbor upsampling, upsampling by bilinear interpolation.

Another special feature in ABRUTECH is a bank of shift registers, which help to perform linear convolution operations several times faster. Together with Address Maker, this allows the programmer to perform 2D convolution with a linearly decomposable kernel, such as Gaussian smoothing or edge detection, without losing a row and a column of data in the process, as with the traditional algorithms.

Compiler and Simulator

We also built a corresponding python based compiler program, which scans the excel sheet where ISA is specied and translates the algorithm written in the human language to an array of binary values, which are then sent to the instruction memory through UART. The compiler identifies syntax errors, which allows the programmer to write assembly code with ease, using our ISA.

In addition to the compiler, we also built a simulator software for ABRUTECH. The simulator can run the algorithm like the processor and show the values of registers and memory at each step, helping us debug an algorithm fast and remotely, without repeatedly loading it into the processor.

Algorithms

As detailed in the report, we devised brand new algorithms for downsampling and other tasks, which are mathematically justified (in the report) to have better accuracy and speed than traditional algorithms.

Hardware Debugging Features

The system itself is implemented with hardware debugging features, such as the ability to run the processor either at 1 Hz clock frequency, 10 MHz clock frequency or through a manual clock provided by a push button. We are able to see the currently fetched instruction and currently retrieved data on 7 segment displays and LED bulbs.

Our processor is also free of major hardware vulnerabilities, such as spectre and meltdown since we did not have the time to implement the branch prediction and speculative execution modules.

Results

While being optimized to process square images, the qualities of a generic processor are also preserved, which is presented in the section `Preservation of Genericity'. Algorithms to calculate the Fibonacci sequence and to find prime numbers less than 256 have been implemented and presented with results.

Implementation of each of these algorithms resulted in a sum of squared difference (SSD) error of zero, which shows the accuracy of our FPGA implementation.

Report - Processor

Report - Graphic Equalizer and Cache

[2018] Smart Lock: Assembly programming, Product Realization, Marketing

The objective of this project is to gain experience in designing and launching a brand new product into a market. Our product is the 'Smart Door Lock', a safe door lock with IOT features.

I am the Group Leader for the group of 10. Apart from management and coordination responsibilities, I was the main mechanical designer in version 1.0 and the person responsible for micro-controller programming in version 2.0

Version 1.0

We designed the version 1.0 of this product in 2016. Having limited knowledge of design theory, we ran into a few problems. Key product goals were:

• A device that can be fixed on top of any manual lock to make it smart.
• Can be locked and unlocked through an encrypted platform from anywhere in the world
• Fully manually overridable from both inside and outside using a key.

As we went through multiple design cycles, we were facing a major design crisis. The problem was a combination of the following

• High torque forces required to turn the shaft of the lock
• Servo torque limitations, 
• Servo angle limitations 
• Necessity to override it completely from inside and outside
• Infeasibility to keep the torque connection from the manual override and the actual lock permanently connected (since this would result in externally turning the servo during manual override, which can destroy the servo)

As the lead designer in the group, I came up with a neat solution that included multiple gear wheels and a gear shift system which can solve all the above problems. However, it resulted in increased design and manufacturing complexity to the level such that we cannot produce it easily in Sri Lanka. 

.

Smart lock v1.0 pitching presentation from Abarajithan Gnaneswaran

.

Version 1.5

Hence, we have currently redesigned the lock by eliminating a minor requirement (goal) to result in an excellent design that is far simpler and cheaper to manufacture. Our prototype in that semester was based on this new design and we have decided to proceed with that design.

Github Repository

https://github.com/abarajithan11/smart_door_lock

[2017] DonateLK: A Web Based System for Donations

Introduction

DonateLK was developed as a semester project for the module: 'Modular Software Development'. It is a web-based market place which is designed to support consumer to consumer donations. Individuals who would like to donate (Donors) as well as those who are looking for donations (Recipients) can create donor and recipient accounts respectively. After verification from the administration those individuals can then send and receive items through Donate.lk warehouse without any compromise of privacy as explained below. 

Objectives

• To test our ability to produce software as a team, within the given deadline. 
• To provide hands-on experience in application of software engineering models (Iterative Model, Extreme Programming)
• To provide experience in division of work, teamwork and project management in large programming projects.

My Responsibilities

I was the team leader for a team of five. I was also the lead developer during the development of the module. I was responsible for:

• Writing 7 out of 8 classes in PHP
• Designing the Database
• Integration of PHP and the UI designed in CSS and HTML

NOTE: The number of commits to the github repository from my profile (abarajithan11) is low due to the fact my system was having trouble with interfacing with git due to some system preferences I have set at that time. Hence most of my commits were through my team member's accounts.

Functionality

Donate.lk is a web-based market place which is designed to support consumer to consumer donations. Individuals who would like to donate (Donors) as well as those who are looking for donations (Recipients) can create donor and recipient accounts respectively. After verification from the administration those individuals can then send and receive items through Donate.lk warehouse without any compromise of privacy as explained below.

Donors and recipients can register as users to the system. A recipient must provide a short summary about himself/herself in addition to details about his/her job, salary and financial information. A recipient account is first initiated into a waiting state. An admin must review the recipient’s account and verify that the short summery is true, before making the recipient account active. The details and summery of a recipient are only visible to an admin, while only the short summery is shown to a donor.

A donor can place information about his/her item in the system. A recipient can find items through a search feature and request up to five items. Multiple recipients can request one item. A donor can view all the items he/she has donated and the requests that have arrived to his/her items. By considering the summery of each recipient, the donor confirms one request out of many, per item. This marks the item as ‘confirmed by donor’

Then the donor is given with a unique request ID, which can be used as a transaction ID. The donor can attach this ID and send the item to the administrator by mail. Then the admin marks his item’s status as ‘sent by donor’.

Items sent by donor are shown to the admin in admin’s database. The admin then receives the item and marks it as ‘received by admin’. Then he/she sends it to the recipient, marking the item as ‘sent by admin’. The recipient can track the status of the item through recipient home page. When recipient receives the item, he/she marks it as ‘received by recipient’ and that ends a transaction.

This system ensures the privacy of information of both donors and recipients. The only communication is through their transaction ID, issued by the system anonymously.

Furthermore, the system can be used by the administration to view the status of all items in transit as well as the review, terminate or suspend all user accounts.

Skills Acquired

• PHP, CSS, HTML
• Division of work in a programming project
• Team management & leadership
• Familiarity with preparing SRS document

Full Documentation

https://github.com/ravindurashmin/MSD-project

Evaluation Copy

We have hosted our product online using a free hosting site.

You may test our system through that also.

Kindly note that since this is a free hosting service, the site

will be blocked from 1 AM to 5 AM everday. 

You can test on any other time.

donatelk.000webhostapp.com/

-------------Usernames & Passwords---------------

TYPE USERNAME PASSWORD

ADMIN : admin1 passpass

DONOR : aba1 pass

DONOR : rash1 pass

DONOR : chinthana1 pass

RECIPIENT : mayuka1 pass

User Interface

[2017] Bidirectional Maze Solver: Pointer manipulation for flipping, Computer Vision ideas

In November 2017, I formed a group with 3 others to compete in Sri Lanka Robotics Challenge 2017 (to be held on January 2018). According to the task, the two teams must produce two robots each, one stationary and one mobile, which should coordinate via any wireless method to complete the task.

My Responsibilities

I was appointed the group leader and also I took the responsibility of designing and coding the mobile robot, which is a line maze solver. I also refined the concept and made final design of the stationary robot (which should be able to shoot a ball turning 360 degrees in horizontal plane and to about 10 degrees in vertical plane).

The Mobile Robot: Innovative Ideas

Since the task of mobile robot is relatively easy to perform, I realized the key for winning is the speed. When I considered different ways to reduce the time taken, I noted two factors:

1. The robot takes some time to make a U-turn at nodes of the maze. Also, a U-turn is dangerous as since the sensor panel moves completely out of the line, there is a chance the robot might get disoriented.
2. By traditional always-take-left algorithm, the robot visits empty nodes (where boxes are not kept), wasting time.

1. Bidirectionality: Eliminating U Turns

I proposed the idea to make the robot fully bidirectional. While increasing the cost, this needed some redesign of the body from traditional designs and the need for some innovative methods to be used in coding.

a) C pointers and OOP programming for bidirectionality

I have learnt the wonders of pointer & reference manipulations in C++ language and I have learnt OOP concepts through Java & PHP programming. I decided to combine them as a solution.

As shown below, I made classes for motors and created objects for each motor, sonar, IR panel and color detector. Then I defined references *pMotorLF, *pMotorLR...etc for front left, front right...etc motors from the POV of the robot. The rest of the code would be written in terms of the pointers, not of the objects themselves. 

eg: pMotorLF->drive(-100)

Using a custom flip function I could change the objects to which the reference is pointing to and this would effectively flip the entire robot internally.

A function definition for the motor class.

Flip function definition. Calling this function would immediately software-flip the robot by 180 degrees

b) Hardware issues & solutions

Due to bidirectionality, we decided to place the relatively expensive main line sensor panel (8-qtr) at the exact center of the robot. This has the disadvantage that the the small lag between sensing and acting would make the robot act too late and veer off the line. It was solved by (c)

2. Variable Speeds & variable PID constants

To solve this, I came up with the idea of using two sets of speed configurations. I proposed that we could use a relatively cheap 3-qtr at exact center of each front and back lines. 

• If the robot senses a line on the front 3-qtr, he assumes a a higher speed and the PID constants associated (and tuned) with respect to that speed. 
• If not, robot assumes a lower speed and associated PID constants.

Using this method, the robot moves at high speeds along straight lines and at lower speeds near junctions and curvy lines, reducing the error.

3. Loophole in the rules - Workaround for hard coding

According to the task, few nodes of the maze graph are left empty, without any boxes. A traditional take-left robot would visit them, wasting time. The solution would be to hard code the maze, but according to rules, the positions of the boxes are changed in every round and hard coding is a crime. 

As a workaround, I proposed the idea of manually inputting the maze using buttons before the round (which was not restricted, a loophole). I wrote an algorithms to always take left, but avoid the i-th left, where i is a member of an array (where  the turns to be avoided are stored). The array is initialized empty and it can be filled by pressing buttons on top of the robot. 

Before a round, when the boxes are placed, I would take down the turns that need to be avoided and press buttons according to the programmed protocol to fill the array with those numbers. The robot would then simply avoid those turns, saving time.

Ideas on Computer Vision

To avoid empty nodes, I also proposed a solution with machine vision. By that time, we were taking a module: "Fundamentals of Image Processing and Computer Vision" and I had some experience with image processing methods. 

I proposed, we could 

• Mount a camera module on the stationary bot (then the view and the projective transformation is known in advance)
• Apply projective transform, crop to the arena and get ImIn
• Skeletonize the thresholded ImIn to get the path
• Finding unique blobs in the R,G,B layers separately to identify boxes
• Using array navigation to find which nodes to be avoided
• Transmit an array full of "turns to be avoided" to the mobile robot

I proposed the above to Kanchana, my colleague who is actively researching in computer vision field and he agreed to join our team.

Project Discontinued

The group was highly active and as I wrote the algorithms, rest of the group started to build the prototypes for the robot. However, due to academic workload, the hardware guy in the group decided to quit. He needed to focus on the web development project which we needed to do for another module. Without a hardware guy, I could not continue this and finish it before deadline. Therefore, sadly the project was abandoned.

Github Repository

github.com/abarajithan11/bidirectional_bot

[2017] FIR Filter Design for Given Specifications

Introduction

This project is undertaken with the objective of demonstrating the design process of a digital filter using the windowing method in conjunction with the Kaiser window to fit given requirements. The filter is then analyzed and verified. A comprehensive summery of the basic principles of filter design also have been revised as a part of this project report. Mathematical analysis has been carried out using Matlab 2014a software. It is submitted as the semester project for the module 'Digital Signal Processing'.

Magnitude Response of the designed bandstop filter

Objectives

• To test the conceptual understanding of Fourier series methods of designing FIR Filters
• To provide experience in designing a FIR Filter to fit the given specifications

My Responsibilities

This is an individual project

• Analyzing given specifications and building the mathematical model
• Applying the equations to write a Matlab program
• Visualizing the results from Matlab program.
• Documentation

Skills Acquired

• Matlab
• Digital Signal Processing Theories

Results

CLICK ON IMAGES TO ZOOM

Attenuation in stopband

Ripples on lower passband

DFT of a test excitation signal

DFT of the signal passed through my filter

DFT of the signal passed through an ideal filter

Full Documentation

http://bit.ly/FIR-Design-Aba

[2017] UART Implementation with FPGA

Introduction

This was completed as a semester group project for the module: 'Electronics III'. A UART Receiver - Transmitter was implemented in FPGA using RTL programming. Both RTL and Testbench code were written and debugged. The final setup was tested with digital oscilloscopes from laboratory for evaluation.

Objectives

• To test our ability in RTL programming.
• To make us familiar with implementation using FPGA

Documentation