Pick-and-Place Robot with Computer Vision (See Description)

This video showcases a first-time setup of my robot's pick-and-place algorithm, which uses a 3D camera to locate chess pieces within the robot's work area. This project integrates the kinematics and controls of a three degree-of-freedom robot, image segmentation techniques to separate the objects of interest from the background, and an Intel Realsense 435 "stereo" camera to locate objects in 3D space.
Equipment: Robot kinematics and controls were derived within the MATLAB computation environment and implemented within the Arduino IDE. Image processing and segmentation were implemented in MATLAB with a Windows desktop computer.
How it works:
1. An image is taken of the robot's work area
2. The user defines a region of interest (first time setup only)
3. The image is segmented to separate the chess pieces from the rest of the image
4. The user can interactively set the origin, which is needed for coordinate transformation between the camera and robot (first time setup only)
5. The user can verify the accuracy by clicking on different parts of the robot work area grid and comparing the output coordinates to the grid's coordinates (first time setup only)
6. The path planning algorithm moves the robot to pick-and-place the chess piece onto a per-determined location
The current system has an issue with missing depth information because of the orientation of the camera with respect to the robot's work area. To simplify the rotation matrix that is used to transform the object coordinates between the camera and robot's reference frames, the camera is orientated such that its coordinate axes are orthogonal to the robot's reference frame (the origin of the robot's work area).
This top-down view doesn't capture the depth information from the middle of taller objects or objects that are further away from the camera, meaning the centroid (marked in red) actually represents a point on the work area passing through the chess piece further above the object. This is why you see the king get thrown around during the second pickup sequence!.
Music:
Risk of Rain Soundtrack
"Chanson d'automne..."
Chris Christodoulou