(1)Plane detection and desk segmentation

In this step, we are looking for a way to segment the near-desk region, since only the objects on the desk are what we are interested.

Down sampling

The 1 frame is down sampled by the leaf size of 1 cm, in order to make the following process faster.

Normal based plane detection

Detecting all planes using normal-based plane detection.

Desk segmentation

Looking for the plane whose centroid is nearest to the camera as the desk.

(2)Transform matrix calculation

Calculating the centroid and normal of the desk plane. Computing the transform matrix, in order to make the coordinate's origin (0, 0, 0) at the centroid of the desk, and the x-y plane on the desk, and the +z axis is vertical to the desk on the above side. This transformation will be applied in every frame in the future.

(3)Pass through the near-desk region

This region will be segmented as the interested area in every future frame. The region was selecting by calculating the max-min value of the desk (x-min, x-max, y-min, y-max, z-min, z-max), and then segmenting the area of (x-min, x-max), (y-min, y-max), (z-min, z-max+0.25). That means, the 0.25m area above the desk will be considered and segmented.

However, in the case which the desk is connected with the wall, sometimes part of the wall will be in this area due to some errors in reality. So we can segment the area of (x-min, x-max), (y-min, y-max-0.02), (z-min, z-max+0.25) to avoid the clustering error.

Details

(1)Octree establishment

About Octree

Octree is a hierarchical tree data structure for representing point cloud data (to separate the 3D space in to voxels, and to use the voxels which contain the points to represent the cloud).

Point Cloud Module Octree Wikipedia of Octree

Octree of the Desk Cloud

The octree of the segmented is build for all the first 11 visible frames. The parameter, the leaf length is set to 5 cm.

(2)Octree based change detection

Old method

In point cloud libaray, the Octree change detection function can detect the changes between two point clouds. The new voxels, which do not appear in another point cloud can be detected and stored.

The change detection will be applied in the first 11 frames of the desk region, frame by frame. The voxels which exist in all 11 frames will be considered as the model structre.

The points extracted from the 11th frame based on the octree structure is considered as the model's point cloud.

The resolution of the octree is chosen as 0.05(leaf size of 5cm).

Reason

The method of automatically selecting the resolution is to be developed.

(3)Clustering the objects on the desk

Euclidean clustering is applied on the near-desk region. The distance threshold is set as 2 cm. That is, the points in the neighborhood 2cm of the object will all be considered as part of the object.

The clustering method will be optimized since two objects can be connected to each other

(4)Storing the origin infomation of each object

The information, like centroid coordinates and the bounding box, of all the objects in the model are stored.

Centroid

The method to get the centroid of the objects is the function getCentroid in the PCL library.

Bounding box

Version1.Line detection based Bounding box(already given up)

Evaluation

Document

Version2.Normal vector based Bounding box with improvement

The normal vector of the front plane is detected. Based on this, the box will be rotated parallel to the x-y axis.

After that, the maximum and minimum value of the coordinates can represent the bounding box. In order to make it more accurate, some methods are being developed.

tests with 1 frame

tests with added-up 11 frames

Plane detection based Bounding box is to be tested

Entropy based method may make the cloud of the clusters better?--To be discussed

tests with entropy