A deep learning model that can perdict sub-pixel movement from 3d optical flow
EKF SLAM On Turtlebot3 from Scratch
EKF SLAM|Robotics Sensing & Navigation|Machine Learning|Differential Drive Control
Implemented an Extended Kalman Filter on Turtlebot3 in both known and unknown data association and used machine learning to detect landmarks from laserscan
ROS, Gazebo, Git, Linux, OpenCV, SLAM MoveIt, Artificial Intelligence, Motion Planning, Feedback Control
Programming
C++, C，Python, MATLAB
Mechanical Engineering
3D printing, Soldering, Laser Cutting, Solidworks, AutoDesk Inventor
About Me
Brief background introduction
I recently graduated from University of California, Irvine with a bachelor degree in mechanical engineering. During my undergraduate years, I learned various areas in mechanical engineering such as thermaldynamics, mechanical design, fluidmechanics, control and etc. I also conducted in developing a medical device which I was responsible for mechanical design. I wish to strength my ability to integrate mechanical, electrical and programming skills so I decide to dig in robotics field. Currently, I am pursuing a master degree of robotics at Northwestern University. My interest includes motion control, SLAM, artificial Intelligence, machine learning and motion planning.
Contact me
Mingqing Yuan
Master of Science in Robotics @ Northwestern University
3D optical flow
Brief Description
This is my final project at Northwestern University. The project is focused on using a rgbd camera(Intel Realsense D435i) to capture an object with random motion. By capturing the pointcloud at consecutive timestamps and giving a groundtruth motion, the deep learning model is able to learn the sub-piexl motion(less than 5mm). The whole project is simulation based using ROS and Gazebo.
1.Setup simulation environment 2.Post-processing raw pointcloud data 3.Given random gaussian motion 4.Capture pointcloud data and corresponding groundtruth motion 5.Training Deep learning model 6.Evaluation of the model
Part 1. Setup simulation environment
The simulation is all done in Gazebo. The realsense camera is pointing at the small cube which is the moving object. The giant cube behind acts like a static wall. The idea here is when the small cube is moving which means its pointcloud is also shifting, whereas the wall will not move and act like a ground reference. It is important for the deep learning model to learn the motion.
Part 2. Post-processing raw pointcloud data
In this step, I used PCL library for post-processing raw pointcloud data. I used passfilters, voxelgrid filter to filter out unnecessary point clouds and only focuses on ROI(region of interest).
Part 3. Given random gaussian motion
I wrote a ROS simulation node which subscribe the states of the cube from gazebo world then publish a motion with gaussian noise added to it. The motion is very small(<5mm) since I want to achieve sub-pixel movement prediction.
Part 4. Capture pointcloud data and corresponding groundtruth motion
In this step, I firstly saved pointcloud data and corresponding motion data into rosbag and then convert it to pcd files and motion.txt file. Aligining the timestamp is key here, I used python for extracting files names and timestamps then did a matching for pcd files and corresponding motion in the same timestamp
Part 5. Training Deep learning model
The deep learning model adopted the thinking of scene flownet3d. It consist of set conv2d layers, flow-embedding layers and set upconv2d layers. set conv2d layers is used for grouping pointclouds based on a specific radius. flow-embedding layers learns to aggregrate both feature similarites and spatial relationship to produce embeddings that encode point motions. the set upconv2d layers are used to propapage and refine the embedding in a informed way. In the original flownet3d, it uses rgb color as a learning feature and it calculates the geometric differences between two pointcloud. However, my proposed approch does not rely on the rgb color and geometric difference. Insted, I give a groundturth motion for a pair of pointcloud and see if the model can learn such motion.
6.Evaluation of the model
I generate around 15000 pointcloud from simulations and I use 13000 files for training and rest for testing. During evaluation, I use two accuarcy metrics. 1.Error within 10mm. 2. Error within 5mm. The final accuracy reaches 78% for errors within 10mm but only 52% for errors within 5mm. Even though the result is not very ideal, it can be improved by creating a larger dataset and tunning some hyperparameters. For more information, please visit my github page.
This is my implementation of Extended Kalman Filter SLAM on Turtlebot3. The blue path is the real(ground truth) path, the red one is the EKF slam path and the green one is the odometery path. The blue circles are tubes from simulator, green circles are landmarks sensed by customized laserscan and the yellow circles are estimated locations of landmark from EKF SLAM.
Turtle Simulator
This is a customized simultor for turtlebot3. In this simulator, the turtlebot is represented by a green sphere. Tubes are shown in blue color at given location. When the robot moves close enough to the tubes, the fake laserscan can detect them and publish the red tubes with some noises. The fake laserscan uses circle-line intersection algorithm to detect the tubes.
EKF SLAM
EKF SLAM is a class of algorithms which utilizes the extended Kalman filter (EKF) for simultaneous localization and mapping (SLAM). Typically, EKF SLAM algorithms are feature based, and use the maximum likelihood algorithm for data association. This algorithm can be divided into two parts:State update and measurement update. The state estimate is the same for both. The robot state used a probabilistic motion model. Xi bar is the state estimate at time t, g is the motion model, u is the body twist at time t and epsilion is the motion noise: \begin{equation} \hat{\xi_t^-} = g(\hat{\xi}_{t-1}, u_t, \epsilon) \end{equation} Next, propagate the uncertainty using linearized transition model: \begin{equation} \hat{\Sigma_t^-} = g'(\hat{\xi}_{t-1}, u_t, \epsilon) \ \hat{\Sigma}_{t-1} \ g'(\hat{\xi}_{t-1}, u_t,\epsilon)^T + \bar{Q} \end{equation} The process noise is a diagonal matrix corresponding to the gaussian noise for the robot pose(x,y,theta) \begin{equation}\bar{Q} =\begin{bmatrix} Q & 0_{3*2n} \\ 0_{2n*3} & 0_{2n*2n} \end{bmatrix}\end{equation} The algorithm can be found below.
The laserscan is clustered by many points using a distance threshold. To find circles in cluster, I implemented the circle fitting algorithm. The algorithm can be found below.
Known Data Association assumes we known the relative positions of landmarks in the robot frame. For each measurement, i is assocaited with landmark j, then compute the Kalman gain K. \begin{equation} K_i = \Sigma_t^- h_j' \ (\xi_t)^T (h_j' \ (\xi_t) \Sigma_t^- \ h_j'(\xi_t)^T + R)^{-1} \end{equation} Next, use the measurement model to predict the state given current belief and updated posterior state \begin{equation}\hat{z_t}^i = h_j(\hat{\xi_t^-}) \end{equation} \begin{equation} \hat{\xi_t} = \hat{\xi_t^-} + K_i(z_t^- - \hat{z_t^i}) \end{equation} Lastly, compose the posterior covariance. \begin{equation} \Sigma_t = (I - K_i h_j'(\xi_t)) \Sigma_t^- \end{equation}
Unknown Data Association
This data association is achieved by using the mahalanobis distance. Iterates over the number of the observed landmarks N where k is the landmark number. \begin{equation} \psi_k = h_k’(\hat{\xi_t}^-) \Sigma_t^- \ h_k’(\hat{\xi_t}^-)^T + R \end{equation} Then, calculate the predicted measurement above Next, compute the Mahalanobis distance and append to a vector of measurements \begin{equation} d_k = (z_i - \hat{z_k}) \psi_k (z_i - \hat{z_k})^T \end{equation} Let d_star be the minimum mahalonbis distance and be the landmark index corresponding to the minimimum distance. If d_star is smaller than the lower-bound Mahalanobis threshold, then j is the index used in the measurement update. If it is greater than the upper-bound Mahalanobis threshold, then j = N and increase N by 1. The algorithm can be found below.
The diff_drive class performs the calculation for turtlebot's odometery. The purpose of this class is to convert wheel velocities to a body twist Vb and then integrate twist to get pose. The full explaination can be found below.
This is my individual winter project at Northwestern University. The goal of the project is to build a mobile robotic system equipped with perception, localization, mapping and navigation abilities in both simulation and real world. The robot is capable of exploring an unknown environment autonomously and mapping the world. It can also detect pedestrians and classify different objects with lidar and camera. Based on the changing the environment, the robot should give responses accordingly.
Hardware Dependencies
Clearpath Jackal Robot, Velodyne-16 Lidar, Intel Realsense Camera D435i
Software Dependencies
Gazebo, Slam Toolbox, Costmap2D, hdl people tracking, NumPy, OpenCV(4.5.1), YOLOv4-tiny
Project Breakdown
1.SLAM(Simultaneous Localization and Mapping) 2.Frontier Exploration 3. People Tracking 4.Real-time Object Detecion
Part 1. SLAM
SLAM(Simultaneous Localization and Mapping) technology allows the robot to map an unknown environment while keeps tracking its own location. I used slam toolbox package to accomplish the task.Below is a SLAM flowchart for Jackal.
Demo
Part 2. Frontier Exploration
Frontier exploration is a fast and advanced algorithm for robot to explore in an unknown environment and map the world. This algorithm finds every existed frontiers and robot will move to the closest frontier based on its distance to the centroid of frontier. A detailed explaination for my implementation of the algorithm can be found at my github page below.
Demo
As shown in the demo, robot will move to the centroid of the closest frontier and constantly updating the map. All found frontiers are represented in blue; If the robot dose not move to the target point in a time limit, then the frontier will turn red and robot will move to the next frontier point.
Part 3. People Tracking
For people tracking feature, I used hdl_people tracking package which can detect walking pedestrains by extracting information from the pointcloud data then displayed as markerarray in Rviz.
Demo
Part 4. Real-time Object Detection
This part of the project is also my final project for COMP_ENG 395:Connected and Autonomous Vehicles:Challenges and Design which I thorougly explained how I implemented YOLO using OpenCV and evaluated the performance of YOLOv3, YOLOv4 and YOLOv4-tiny models then tested on the Jackal. Based on my test result, I found YOLOv4-tiny can reach approximately 30FPS in ROS whereas the other two models only have 3FPS. Even though YOLOv4-tiny has less accuracy compared with the other two models, it is still good enough to classify most objects. Besides, FPS is extreamly important in real time detection for autonomous vehicle so I decided to use YOLOv4-tiny on Jackal. My full analysis paper can be found at my github page below.
Demo
Future Work
1. Add a boundary feature to the automapping so I can specify an area for the robot to explore instead of trying to map the whole environment(Not realistic if given a very large environment). 2. Implement lidar - camera fusion so the detected objects can be shown in the map and use GPU for handling object detection task. 3. Estimate the distance between the camera and objects using the depth information from the realsense camera. For more information, please visit my github page below.
This is a setup instruction for how to bring Clearpath Jackal to ROS Noetic which is the latest version of ROS. Since many packages used for Jackal are not officially released for ROS noetic, this instruction will guide you how to build and set up jackal from scratch.
Process Architecture
1. Installing Ubuntu 20.04 LTS 2. Setting up the wireless network 3. Installing ROS Noetic 4. Building packages and dependencies on Jackal 5. Building Jackal packages on remote pc 6. Setting up ROS between Jackal and remote PC 7. Final setup for Jackal specifics
1. Installing Ubuntu 20.04 LTS
The first step is to install Ubuntu 20.04 LTS on Jackal. I recommended to prepare a new SSD and a bootable USB then follow the official Ubuntu setup from the website.
2. Setting up the wireless network
The second step is to connect Jackal to Wifi. This step is not strictly necessary but it is very convient if the Jackal connect to the Wifi directly. A detailed instruction can be found in my githhub page bbelow.
3. Installing ROS Noetic
Install ROS Noetic on both of your remoet pc and Jackal. Please follow the official ROS setup instruction.
4. Building packages and dependencies on Jackal
Since many packages for Jackal written by Clearpath aren't released for ROS noetic so they need to be built from source. Building those packages is a strightforward process and it can be done by cloning source code into your own workspace then run catkin_make. See my gthub page for all needed packages and detailed instruction.
5. Building Jackal packages on remote pc
Similar to the previous step, user need to build unpublished packages by building source code in workspace.
6. Setting up ROS between Jackal and remote PC
To let your Jackal and PC share a ROS master, you need to set up ROS_MASTER_URI and ROS_HOSTNAME environment on both Jackal and remote pc.
7. Final setup for Jackal specifics
Those are key steps to get Jackal up and running by setting up all undocumented intricates implemented by Clearpath on a Jacakl image. User need to add udev rules and install package for VLP-16. Please see my github page below for detailed instructions.
This is a default final project "Jack/Dice in the box" for ME314 Machine Dynamics at Northwestern University. The goal of this project is to simulate a dice bouncing inside of a moving box as shown in the demo.
Modeling System
In this project, the system contains two bodies:A dice and a box.The dice has a length of 1, mass of 1 and box has a length of 6, mass of 6.Both bodies have rotational inertia equals 1(Assuming the center of the mass is the center of the geometry).The dice has a configuration(x_dice, y_dice, theta_dice) and the box has a configuration(x_box, y_box, theta_box).The system involves 3 frames:world_frame, box_frame and dice_frame.And there are two external forces apply to the box in x and y direction which cause the box to move along a trajectory.The dice falls from the center of the box and should experience several impacts with the box.Also, the dice is contrainted to the inner area of the box and the moving trajectory of the box should not affect by the dice.
Technical Outline
1. Draw a diagram to show the transformation of frame. 2. Calculate all the transformation of frame. 3. Calculate Lagrangian, Eular-Lagrange equations with the existence of external forces and constraints. 4. Wirte function for updating impact equations 5. Modify simulation function and run simulation 6. Plot the trajectory of dice/box versus time. For more detailed information and explanation, please visit my github page below.
This is the final project for the cource ME 449 Robotic Manipulation at Northwestern University. It is also the capstone project for the course "Modern Robotics" on Cousera.The goal of the project is to wirte a program that enables KUKA youBot to finish a pick and place task in V-Rep simulation scene.
Procedures for motion planning
1. Plan a trajectory as desired trajectory for the end-effector to follow 2. Use the previous desired trajectory in feedfoward+PI to calculate the kinematics of the youBot 3. Use the calculated kinematics to drive the youBot 4. Save the result as a .csv file and conduct simulation in V-rep scene 6.
Strategy for completing tasks
There are three tasks and I need to choose a controller and find the best configuration setup. For task 'Best', the main goal is to minimize the end effector error so I choose to use Feedfoward + PI controller. With KP = 20, KI = 0.5, this controller presents a very good result that the end effector error becomes very small as time goes by and the KUKA youBot finishes the task smoothly and accurately in the simulation. For task 'Overshoot', I need to find an end effector error with large overshoot value. With KP = 2, KI = 0.001, there is a overshoot between 9 - 14s. For task 'NewTask', I can customized a target location for the robot to go and I used the same strategy as the task 'Best'. With KP = 20, KI = 0.1, the end effector error is very small and the robot finishes the task very well in the simulation. For more detailed information, please visit my github page below.
This is the final team project for the class ME495:Embeded System In Robotics at Northwestern University. In this project, we programmed a Rethink Baxter robot to sort bottles and cans located in front of it, and drop them into separate recycle bins. We used computer vision to detect and locate a couple of randomly placed bottles and cans, and used MoveIt to control the robot.
Team Members
Jake Ketchum Kailey Smith Yael Ben Shalom Mingqing Yuan Chris Aretakis
System Architecture
Computer Vision(Yael and Jake):We use the realsense depth camera to capture image and use openCV to detect and classify the objects in the image.After the classification process,the programs saves the location and classification information which will be used in manipulation part. Manipulation(Kailey, Mingqing and Chris):We use MoveIt! to control the motion of Baxter. After getting the location and classification information from CV, we call a customized service to return those information as list so Baxter can follow the planned trajectory.
General Procedures of motion
1. Move to the home position where the robot is safely above all objects. 2. For the current item in the list, display either the can image or the bottle image, depending on the classification. 3. Next, move to the object's (x,y) corrdinate at a safe z height away. This is the same height for bottles and cans. 4. Then move down to the appropriate perch height, depending on classification. (For example, the robot arm will be position further away from the table for bottle, since those are taller than cans). 5. Once safely at the perch height, move down so that the object is in between the grippers. 6.Grasp the object. 7.Move back up to the "safe" position as step 3. 8. Move back to the home position. This step was added to ensure predictable behavior of the robot arm. 9. Depending on the object's classification, move to the appropriate bin. Also, display the recylcing image. 10. Once over the bin, open the grippers and drop the object. Show that the object has been recycled with the bin image. 11. Repeat for all objects found. For more detailed infomation, please visit my github page.
Result
A good calibration of camera ensures we can get a precise location of bottles and cans relative to the robot. After getting the locations, robot can move to each one of them and pick the objects. One deficit is we hard-coded the z distance from the gripper to cans/bottles. Even though it works well after many testings, it would be better to calculate the distance from the robot gripper camera.
Team Contributions
Jake Ketchum:Aligned the Realsense Camera to Baxter frame and wrote code for classifying object. Kailey Smith:Integrated the camera and manipulation node. Modified code and did test at Lab. Yael Ben Shalom:Calculated an approximate offset for the realsense camera and wrote code for classfiying objects and rosunit test. Mingqing Yuan:Laid out the preliminary code for the manipulation of the baxter and did initial movement testing. Chris Aretakis:Laid out pseudo code for motion and designed a gripper for Baxter.
This is a RRT challenge in MSR Hackathon at Northwestern University. Rapidly-Exploring Random Tree (RRT) is a fundamental path planning algorithm in robotics. It consists of a set of vertices, which represent configurations in some domain D and edges, which connect two vertices. The algorithm randomly builds a tree in such a way that, as the number of vertices n increases to inifinity, the vertices are uniformly distributed across the domain D in R^n space. For more information, please visit my github page below.
This is the final project for course MAE 172 Robot Design at UC Irvine. This is a customized project from the scratch and fulfills the course requirement.
Motivation
1.Many games need card shuffling are popular 2.Cards shuffled by hand are easy to be curved 3.No product with acceptable price for family market 4.Card dealing by hand takes a lot of time and makes errors.
Design description
The mechanism of this design is very straightforward. In the beginning, the user turns on the machine then the rack and pinion elevator will lift the poker case approximately 55mm up. Next, the user put one deck of poker into the case. While IR sensors detect cards, two rollers connected to motors on the poker case begin to roll in opposite directions until after the IR sensor detects there are no cards in the poker case continuously 12 times. After distributing cards onto the left and right platform, the elevator goes down and motors beneath two platforms start working and drive cards back to the poker case. When all cards are dispensed back to the poker case, the elevator rises to the highest point and shakes 0.5 seconds to the cards to make them into a deck so the user can easily remove cards. When all cards are removed and IR sensors detect nothing, the elevator goes down to the original position. One complete shuffling process takes approximately 50 seconds. Moreover, we have a switch for the purple LED lights, which can make the shuffler look cool at night and users can play cards in the dark. For more detailed explanation, please view my github page below.
This is the final project for the course MAE106 mechanical system labratory at UC Irvine and the task is to design a wheel robot and participate a robot contest.
Mechanical Design
Our mechanical design goals were to build a robot that was stable, efficient and has a low fraction. To achieve stability, we wanted to make sure the error is small, so we employed the CNC router to cut pur wood plates. For wood selection, we chose plywood instead of the one given by school because it is lighter, smoother and has a better structure for CNC cutting. Besides, we cut two polycarbonate plates for the installation of power transmission. Moreover, we expected collision with other robots, so we tried to lower the center of mass of our robot by reducing the length of vertical plates.
Software design
For software design, we first want an accurate heading reading. We had heard that the magnetometer would be interfered by many factors, since the magnetic field of the earth is quite weak compared to the magnets, solenoids, and other electronics, so we decided to also add a gyroscope to eliminate sudden, and incorrect directionschange. We bought a LSM9DS1 chip — a 9DOF IMU with a magnetometer, an accelerometer and a gyro. To code this chip, we modified the existing library for this Adafruit LSM9DS1, so the heading is calculated within the library functions. Calibrations were added into the sensor reading functions, along with a simple low pass filter. The magnetometer read just fine, but the gyro, after integration, always have a random shift, the accelerometer after two integrations only got worse. We then tried comparing heading readings both from the magnetometer and the gyro using a Madgwick filter, the shift is kind of eliminated. Initially, it still has a low frequency noise that puts a plus or minus 10 degrees, but after adjusting the low pass filter of the magnetometer and gyro readings, the noise is much smaller. For more detailed information, please view my github page below.
