Task Planning and Execution for Block Rearrangement

As an attempt to create a modular system for our lab to implement task planning to rearrange real world scenes (table-top), I architected this ROS2-based pick-and-place system that can detect objects in the scene (instance segmentation) and plan a sequence of pick-and-place operations to rearrange them according to the desired arrangement.

We heavily leverage MoveIt for trajectory planning and execution for kinematic chains and ROS2 UR driver for control of the UR10 robot arm. For checking the feasibility of higher level planning (pick, place, etc.), we use the MoveIt Task Constructor to generate the control trajectories for planned actions. We also pre-train a Mask R-CNN model to reliably detect all the relevant objects in the scene. Since we use PDDL planners, we also perform predicate grounding based on the poses of the detected objects. Any task planning pipeline can be used to come up with a sequence of actions for the given grounded predicates from the detections in the scene.

Benchmarking Antipodal Grasping Pipelines for Grocery Store Applications

Application of manipulation in grocery store applications typically involve accessing items from shelves that pose constraints on approach angle of grasps. To shed light on efficiency of constrained-grasping approaches compared to popular grasping approaches, we implement and compare ROS2 grasping pipeline leveraging CAPGrasp (Continuous Approach-constrained grasp generation) against pipelines using GGCNN (Generative Grasping CNN). The experiments are performed in Gazebo-simulated environments.

Learning Policy for Navigation under Uncertainty

Sensor and actuation uncertainty pose challenges in deployment of robots in real-world, task-critical scenarios without human intervention. One such scenario involve navigation of needles in blood tissues during surgical operations. This problem can be modeled as a POMDP (Partially Observable Markov Decision Process) In this project, we model a noisy gradient grid-world environment with robot actuation noise as a POMDP and evaulate various online POMDP solvers (POMCP - Partially Observable Monte Carlo Tree Search, DESPOT - Determinized Sparse Partially Observable Tree). The policies learned by these solvers are evaluated in terms of computation-time and path-time.

Traffic Control using Reinforcement Learning

For an 24-hour intercollegiate competition (Hackerspace), we highlight the traffic congestion problem prevalent with static-timer based traffic control and propose a solution using deep reinforcement learning (Deep Q Network). We integrated existing work on RL-based traffic control with lane density estimation from camera footage and pitch the subsequent results through a SUMO-based traffic simulation (eventual winners of the competition).

3D Traffic Scene Perception and Rendering in Blender

We implement a full self-driving (FSD) visualization system for autonomous vehicles, similar to Tesla's FSD visualization, where we process dashcam footage to detect traffic objects and render them in Blender as accurately as possible. Our pipeline uses YOLO and DETIC to detect traffic objects in the scene, combined with Marigold (Stable Diffusion-based) for depth estimation. For comprehensive FSD analytics, we extract optical flow using RAFT, traffic light status, lane markings, and pedestrian pose estimation - all essential components for FSD visualization and decision-making in self-driving cars based on the perceived traffic state.

Vision-based Grasping of Unknown Objects using Top Surface

For this class project, we opt for traditional grasping methods based on point cloud processing. The main idea is to retrieve the contours of the target objects and use them to locate points on the boundaries of the contours.
To acheive this, we detect the table surface using RANSAC and remove this major plane. This leaves us with point clouds of objects on the table. After thresholding these point clouds to obtain the top surface, we use alpha shapes to obtain an approximation of the contours of the top surface. By locating the points on the contour closest to the centroid and its opposite, we return these as grasp points for further grasp evaluation.

Comparative Study of Controller Design for Quadrotor Target Interception

In this MATLAB-simulation based project, we design two controllers fixed-gain LQR (Linear Quadratic Regulator) and non-linear MPC (Model Predictive Control) for a quadrotor tasked with intercepting a rogue UAV in restricted airspace. Given the quadrotor dynamics, we design the controllers for fast response along with disturbance rejection upon capturing target. The results are evaluated in terms of these metrics for various UAV trajectories.

OpenManipulatorX - Object Manipulation and Position Control

The objective of this project is to learn about the different components involved in manipulation control. Initially, we derive forward, inverse kinematics of the robot, along with the parametric Jacobian matrix to implement simple cartesian control. Next, we tune the gains of our PID controller to acheive nominal error response in position control of the arm. Finally, we test the performance of our position control for various input joint angles.