[2019] Vision Based Traffic Sensing and Control on FPGA - SOC Design

Vision-Based Adaptive Traffic Control System, a project by Team AbruTech, won Gold at NBQSA 2020 (National ICT Awards) organized by British Computer Society in Tertiary Student Category and has been nominated for APICTA (Asia Pacific) Awards to be held in Malaysia.​

​

A patent for our system is currently under review at NIPO and the system is being further developed with funding from World Bank via AHEAD into a product by a multidisciplinary team of engineers through the Enterprise (Business Linkage Cell) of the University of Moratuwa, together with RDA and SD&CC.​

​

We are forever indebted to our families of Tehara and Chinthana for hosting our team for weeks/months during strikes and study breaks, allowing us to work together. In addition, we thank our Supervisors: Prof. Rohan and Prof. Saman Bandara for their support and assistance.

Problem Statement

In most countries, traffic flow is controlled by traffic lights with pre-set timers. In Sri Lanka, this often causes congestion during peak hours as the system is not sensitive to the traffic levels in each lane of an intersection. To solve this, the traffic policemen usually turn off the lights and manually control the traffic during peak hours.

Solution

Team: Abarajithan, Rukshan, Tehara, Chinthana

We present a system-on-chip design that:

• Processes the video feed locally at edge through YOLOv2 (a 23-layer convolutional neural network for single shot object detection)
• Deduces traffic flow in each phase
• Suggests green times to the traffic lights 

Results

• The above videos demonstrate the object detection (with YOLOv2) and tracking (a custom-built algorithm that can be implemented in C without using any libraries) on test data (CNN was trained on images from a different road)
• The tracking algorithm is lightweight enough to run at thousands of FPS. The minimum FPS needed to reliably track vehicles travelling at 70 km/h (speed limit) is 3 FPS (as shown)
• Only the vehicles coming towards the camera are considered. The blue box signifies the vehicle being counted.

The timing diagram shows the convolution in Vivado simulation and the floating-point error

IOT dashboard for demonstration in SLIOT competition on Nvidia Jetson Nano logging data through MQTT

Project Tasks

The following tasks were (and are being) done.

1. Machine Learning

• Built four remotely powered, wirelessly data-collection device
• Collected, annotate and augment traffic images to create a Sri Lankan traffic dataset (1500 images)
• Built a numpy-based inference framework (keras-like) from scratch as the testbench
• Optimized the architecture of YOLOv2 object detection neural network for hardware implementation
• Trained YOLOv2 and TinyYOLO

2. FPGA Implementation

• Designed a resource-efficient hardware architecture for a CNN acceleration engine to implement YOLOv2 on FPGA
• Designed memory pipelines for high throughput data feeding
• Implementing and debugging the acceleration engine & memory pipeline

3. Object Tracking and Traffic Sensing

• Built a standalone (no libraries used) vehicle tracking algorithm
• Built vehicle counting and green-time allocation 
• Finalize algorithms and rewrite them to C (bare-metal on ARM side of ZYNQ FPGA.

4. Traffic Simulation and Testing

• Built a simulation model of Piliyandala bypass junction in VISSIM (industry grade traffic simulation software used by civil engineers to design intersections) to test traffic control algorithms

5. IOT Implementation

• Logging data to a central server through MQTT, when demonstrating the project in NVIDIA Jetson Nano

Methodology

1. Data Collection

Building and fixing the data collection device

2. Modifications to YOLOv2

• Fused batch normalization into convolution by modifying the weights and biases accordingly.
• Interchanged conv => leaky-relu => max-pool to conv => max-pool => leaky-relu to reduce power.
• Changed the output layer from 80 classes to 5 classes, by reusing weights of appropriate classes.
• Changed grid size from (13 x 13) to (12 x 8) and designed the sensing algorithm accordingly
• Trained with custom Sri Lankan Traffic Dataset
• Built a numpy-based inference framework and tested custom floating-point arithmetic (two types of float8), float16 and integer quantization.

3. CNN Accelerator Design

• Accelerator core v1.0 was designed to perform 12 of 3x3 convolutions in 9 clock cycles, using 9-muxes = 24, 3-muxes = 48, 16-bit registers = 144,  Multipliers = 3, Accumulators = 3.
• This was redesigned into core 2.0, which was 4 times faster, using five times fewer 3-muxes, zero 9-muxes, about 20 times fewer registers (for the same speed), with 100% utilization of all multipliers and adders.
• Currently building the caches and memory pipes to run the AXIS cores without stalling.

4. Object Tracking

• Built a custom lightweight tracking algorithm that can be implemented in C, without any libraries, so it can be run bare-metal (standalone) on the ZYNQ-PS side with minimal memory bandwidth (such that the ZYNQ-PL can use maximum bandwidth)
• Near 97% vehicle counting accuracy in the daytime, 85% accuracy in the night, rainy time, on test data (on a road the CNN has never seen before)
• Hoping to achieve near 100% counting accuracy in day, night and rain conditions through improvements.
• NOTE: Object detector (YOLOv2) has less accuracy. But tracking algorithm is designed to obtain  near 100% accuracy in vehicle counting and identification

5. Traffic Control Algorithm

• Designed and tested 8 algorithms based on density, bounding box count, flow...etc
• Currently working on eliminating traffic snake formation

Modeling the Piliyandala bypass junction in VISSIM - An industry-grade traffic simulation software

[2018] Smart Lock: Assembly programming, Product Realization, Marketing

The objective of this project is to gain experience in designing and launching a brand new product into a market. Our product is the 'Smart Door Lock', a safe door lock with IOT features.

I am the Group Leader for the group of 10. Apart from management and coordination responsibilities, I was the main mechanical designer in version 1.0 and the person responsible for micro-controller programming in version 2.0

Version 1.0

We designed the version 1.0 of this product in 2016. Having limited knowledge of design theory, we ran into a few problems. Key product goals were:

• A device that can be fixed on top of any manual lock to make it smart.
• Can be locked and unlocked through an encrypted platform from anywhere in the world
• Fully manually overridable from both inside and outside using a key.

As we went through multiple design cycles, we were facing a major design crisis. The problem was a combination of the following

• High torque forces required to turn the shaft of the lock
• Servo torque limitations, 
• Servo angle limitations 
• Necessity to override it completely from inside and outside
• Infeasibility to keep the torque connection from the manual override and the actual lock permanently connected (since this would result in externally turning the servo during manual override, which can destroy the servo)

As the lead designer in the group, I came up with a neat solution that included multiple gear wheels and a gear shift system which can solve all the above problems. However, it resulted in increased design and manufacturing complexity to the level such that we cannot produce it easily in Sri Lanka. 

.

Smart lock v1.0 pitching presentation from Abarajithan Gnaneswaran

.

Version 1.5

Hence, we have currently redesigned the lock by eliminating a minor requirement (goal) to result in an excellent design that is far simpler and cheaper to manufacture. Our prototype in that semester was based on this new design and we have decided to proceed with that design.

Github Repository

https://github.com/abarajithan11/smart_door_lock

[2017] DonateLK: A Web Based System for Donations

Introduction

DonateLK was developed as a semester project for the module: 'Modular Software Development'. It is a web-based market place which is designed to support consumer to consumer donations. Individuals who would like to donate (Donors) as well as those who are looking for donations (Recipients) can create donor and recipient accounts respectively. After verification from the administration those individuals can then send and receive items through Donate.lk warehouse without any compromise of privacy as explained below. 

Objectives

• To test our ability to produce software as a team, within the given deadline. 
• To provide hands-on experience in application of software engineering models (Iterative Model, Extreme Programming)
• To provide experience in division of work, teamwork and project management in large programming projects.

My Responsibilities

I was the team leader for a team of five. I was also the lead developer during the development of the module. I was responsible for:

• Writing 7 out of 8 classes in PHP
• Designing the Database
• Integration of PHP and the UI designed in CSS and HTML

NOTE: The number of commits to the github repository from my profile (abarajithan11) is low due to the fact my system was having trouble with interfacing with git due to some system preferences I have set at that time. Hence most of my commits were through my team member's accounts.

Functionality

Donate.lk is a web-based market place which is designed to support consumer to consumer donations. Individuals who would like to donate (Donors) as well as those who are looking for donations (Recipients) can create donor and recipient accounts respectively. After verification from the administration those individuals can then send and receive items through Donate.lk warehouse without any compromise of privacy as explained below.

Donors and recipients can register as users to the system. A recipient must provide a short summary about himself/herself in addition to details about his/her job, salary and financial information. A recipient account is first initiated into a waiting state. An admin must review the recipient’s account and verify that the short summery is true, before making the recipient account active. The details and summery of a recipient are only visible to an admin, while only the short summery is shown to a donor.

A donor can place information about his/her item in the system. A recipient can find items through a search feature and request up to five items. Multiple recipients can request one item. A donor can view all the items he/she has donated and the requests that have arrived to his/her items. By considering the summery of each recipient, the donor confirms one request out of many, per item. This marks the item as ‘confirmed by donor’

Then the donor is given with a unique request ID, which can be used as a transaction ID. The donor can attach this ID and send the item to the administrator by mail. Then the admin marks his item’s status as ‘sent by donor’.

Items sent by donor are shown to the admin in admin’s database. The admin then receives the item and marks it as ‘received by admin’. Then he/she sends it to the recipient, marking the item as ‘sent by admin’. The recipient can track the status of the item through recipient home page. When recipient receives the item, he/she marks it as ‘received by recipient’ and that ends a transaction.

This system ensures the privacy of information of both donors and recipients. The only communication is through their transaction ID, issued by the system anonymously.

Furthermore, the system can be used by the administration to view the status of all items in transit as well as the review, terminate or suspend all user accounts.

Skills Acquired

• PHP, CSS, HTML
• Division of work in a programming project
• Team management & leadership
• Familiarity with preparing SRS document

Full Documentation

https://github.com/ravindurashmin/MSD-project

Evaluation Copy

We have hosted our product online using a free hosting site.

You may test our system through that also.

Kindly note that since this is a free hosting service, the site

will be blocked from 1 AM to 5 AM everday. 

You can test on any other time.

donatelk.000webhostapp.com/

-------------Usernames & Passwords---------------

TYPE USERNAME PASSWORD

ADMIN : admin1 passpass

DONOR : aba1 pass

DONOR : rash1 pass

DONOR : chinthana1 pass

RECIPIENT : mayuka1 pass

User Interface

[2017] Bidirectional Maze Solver: Pointer manipulation for flipping, Computer Vision ideas

In November 2017, I formed a group with 3 others to compete in Sri Lanka Robotics Challenge 2017 (to be held on January 2018). According to the task, the two teams must produce two robots each, one stationary and one mobile, which should coordinate via any wireless method to complete the task.

My Responsibilities

I was appointed the group leader and also I took the responsibility of designing and coding the mobile robot, which is a line maze solver. I also refined the concept and made final design of the stationary robot (which should be able to shoot a ball turning 360 degrees in horizontal plane and to about 10 degrees in vertical plane).

The Mobile Robot: Innovative Ideas

Since the task of mobile robot is relatively easy to perform, I realized the key for winning is the speed. When I considered different ways to reduce the time taken, I noted two factors:

1. The robot takes some time to make a U-turn at nodes of the maze. Also, a U-turn is dangerous as since the sensor panel moves completely out of the line, there is a chance the robot might get disoriented.
2. By traditional always-take-left algorithm, the robot visits empty nodes (where boxes are not kept), wasting time.

1. Bidirectionality: Eliminating U Turns

I proposed the idea to make the robot fully bidirectional. While increasing the cost, this needed some redesign of the body from traditional designs and the need for some innovative methods to be used in coding.

a) C pointers and OOP programming for bidirectionality

I have learnt the wonders of pointer & reference manipulations in C++ language and I have learnt OOP concepts through Java & PHP programming. I decided to combine them as a solution.

As shown below, I made classes for motors and created objects for each motor, sonar, IR panel and color detector. Then I defined references *pMotorLF, *pMotorLR...etc for front left, front right...etc motors from the POV of the robot. The rest of the code would be written in terms of the pointers, not of the objects themselves. 

eg: pMotorLF->drive(-100)

Using a custom flip function I could change the objects to which the reference is pointing to and this would effectively flip the entire robot internally.

A function definition for the motor class.

Flip function definition. Calling this function would immediately software-flip the robot by 180 degrees

b) Hardware issues & solutions

Due to bidirectionality, we decided to place the relatively expensive main line sensor panel (8-qtr) at the exact center of the robot. This has the disadvantage that the the small lag between sensing and acting would make the robot act too late and veer off the line. It was solved by (c)

2. Variable Speeds & variable PID constants

To solve this, I came up with the idea of using two sets of speed configurations. I proposed that we could use a relatively cheap 3-qtr at exact center of each front and back lines. 

• If the robot senses a line on the front 3-qtr, he assumes a a higher speed and the PID constants associated (and tuned) with respect to that speed. 
• If not, robot assumes a lower speed and associated PID constants.

Using this method, the robot moves at high speeds along straight lines and at lower speeds near junctions and curvy lines, reducing the error.

3. Loophole in the rules - Workaround for hard coding

According to the task, few nodes of the maze graph are left empty, without any boxes. A traditional take-left robot would visit them, wasting time. The solution would be to hard code the maze, but according to rules, the positions of the boxes are changed in every round and hard coding is a crime. 

As a workaround, I proposed the idea of manually inputting the maze using buttons before the round (which was not restricted, a loophole). I wrote an algorithms to always take left, but avoid the i-th left, where i is a member of an array (where  the turns to be avoided are stored). The array is initialized empty and it can be filled by pressing buttons on top of the robot. 

Before a round, when the boxes are placed, I would take down the turns that need to be avoided and press buttons according to the programmed protocol to fill the array with those numbers. The robot would then simply avoid those turns, saving time.

Ideas on Computer Vision

To avoid empty nodes, I also proposed a solution with machine vision. By that time, we were taking a module: "Fundamentals of Image Processing and Computer Vision" and I had some experience with image processing methods. 

I proposed, we could 

• Mount a camera module on the stationary bot (then the view and the projective transformation is known in advance)
• Apply projective transform, crop to the arena and get ImIn
• Skeletonize the thresholded ImIn to get the path
• Finding unique blobs in the R,G,B layers separately to identify boxes
• Using array navigation to find which nodes to be avoided
• Transmit an array full of "turns to be avoided" to the mobile robot

I proposed the above to Kanchana, my colleague who is actively researching in computer vision field and he agreed to join our team.

Project Discontinued

The group was highly active and as I wrote the algorithms, rest of the group started to build the prototypes for the robot. However, due to academic workload, the hardware guy in the group decided to quit. He needed to focus on the web development project which we needed to do for another module. Without a hardware guy, I could not continue this and finish it before deadline. Therefore, sadly the project was abandoned.

Github Repository

github.com/abarajithan11/bidirectional_bot

[2017] Basic Robotics Workshops for students of Over 20 schools

As with all developing countries, students of Sri Lanka are not exposed to the latest technologies. Students are usually exposed to programming (mostly on paper) in age 16 if only they chose an optional subject: ICT. To introduce the wonders of microprocessors and to unleash the limitless possibilities by untapping the creativity in our students, my team decided to conduct arduino robotics workshops around Sri Lanka.

From August 2017 to December 2017, our team organized and conducted six workshops for over 160 students from over 20 schools. Following is the slideshow I prepared for my presentation for students who are absolute beginners for programming and robotics.

Arduino Programming for Basic Robotics - University of Moratuwa from Abarajithan Gnaneswaran

The students were very enthusiastic. Being absolute beginners to programming, they quickly learnt to code within a matter of hours and started modified the for loop and while loops to do whatever they want by the end.

We faced several difficulties during the workshops:

• No funds were provided, we spent our own money on travel and our refreshments
• We couldn't obtain electronic items (breadboards, jumpers, arduino boards). We collected our own items and our friend's items.
• I took my own robot (which I built for evaluation for a module) to all the workshops for demonstration. It was mishandled and broken in half in the 4th workshop.

First workshop in Kanthale

Second workshop in Dambulla for over 40 students

Third workshop in Matale for over 30 students

Fourth Workshop in Galewela

Final Workshop in University of Moratuwa

My robot (Gollum) broken in half during 4th workshop

Following is a slideshow I created for a nationwide exhibition to explain PID concepts for newbies.

PID Control system for Dummies from Abarajithan Gnaneswaran

--

[2017] Feel Lanka 11.0 - Tourism Development with Foreign Volunteers

Focused on tourism development in unexplored parts of Sri Lanka, it was a six-week project with a team of foreign and local volunteers collaborating with leading companies and government institutions. We are a team of 15 foreign volunteers from Germany, China, Poland, Croatia, Ukraine, Lebanon, Canada..etc. and locals (students of University of Moratuwa).

My Responsibilities: I was the co-leader of a team of 13 foreign and 7 local volunteers; I coordinated relationships with executive directors of 8 companies, negotiated agreements; lead the team to excursions around the country to collect footage; oversaw production of 9 videos.

Outcome

Team and Sponsors

Our sponsors

International Team

Local Team

Letter of acknowledgement from our mentor

[2017] Ring Finding Robot: with GPS following, obstacle avoiding, wall following, color detection, parallel alignment, robotic arm manipulation capabilities

Through this project we built Gollum, an outdoor mobile robot to perform GPS following, obstacle avoiding, wall following, color detection, parallel alignment and robotic arm manipulation.

I was responsible for designing the algorithm and programming the robot (1,200 lines of code)

Task:

The robot starts from within the starting circle of 1m in radius. Out of three pre-defined target locations, one location is chosen at random and give to us in the start. We input the target location using a keypad/set of 3 switches. The robot should the follow GPS and visit the target, which is 20m away. On the way, there is a "forest" of obstracles (30 cm cubes) which the robot should avoid. Target is marked by a circle of white line (3cm thickness) 2m in radius around the target point.

At the target point, lies a cylindar, 50cm in radius. The cyclindar has 40 cm wide entrance on one side.The robot should find the cylindar, move to meet it, follow the wall around it, find the entrance, move in. Inside the cylindar lies a 5 cm high, 40cm diameter plate that contains a metal ring og 6cm diameter anywhere in it. The robot should be able to search and pick up the ring with an arm.

Then then robot should return to the starting location and stop.

GitHub Repository:

https://github.com/abarajithan11/gollum_ring_finder

Behind the Scenes:

I took Gollum to one of my basic robotics workshop as an inspiration. Some school students mishandled him, he fell down and the perspex body was broken in half.

I had to bandage him with a piece of cloth to bring him back.

Reprogramming the parameters for the new version

Rebuilding gollum's body. This time with a cool black sticker

Bending perspex with a high current precision heating device

The mess of wires... need to be sorted.

Samith's boarding room. The hardware workshop.

[2017] Dobbybot: A Battlebot under $32

Dobbybot in the hands of our skillful driver: Praga

For a battlebot competition, we decided to build a bot under 30$. Clad in stainless steel plates for armour and a flipper arm powered by 12 kgcm servo motors, production cost was 32$. 

The Design

I was responsible for structural design. I made multiple mechanical designs and finally used my 3-wedge sided, very low profile (7 cm high) version. The robot competed with other robots that cost 300$ and won the 3rd place. My wedged design with low center of gravity and almost zero ground clearance was commended for its uniqueness and novelty by the judging panel.

The frame of the robot

Scaled drawing of how the parts fit together

Making

We took the robot 100 km away from the university, to the hometown of Chinthana for welding. We got the stainless steel plate free from his relative.

 Designs and Documents

http://bit.ly/DobbyBot