First, we take a set of input images from an environment that is not restricted by illumination. These images contain different views and poses of hair. We perform semi-automatic segmentation on these images to separate the hair from the background, and then use PMVS, an advanced multi-view stereo algorithm, to reconstruct a point cloud. We usually use 30 to 50 input images for reconstruction.
Then, we filter the reconstructed point cloud and the normal vector estimated by PMVS since they may be noisy. We smooth them using Moving Least Squares (MLS): for each point, we fit the optimal plane to its nearby weighted neighbor points. The normal vector of the plane and the projection of the point on the plane are used to update the original normal vector and position of the point. We use a standard deviation of 2 mm to get reasonable filtering results.
Next, we calculate the direction field on the filtered point cloud. This direction field represents the 3D direction of each point. We use an image-based approach that exploits the orientation information in the input image. We first perform orientation filtering on each input image to obtain the 2D orientation of each pixel. Then, for each point in the point cloud, we find its corresponding pixel in the input image and the corresponding pixels of its neighbor points in the input image. We use the direction of these pixels to calculate the 3D direction of the points in the point cloud. This direction is non-directional and only indicates the direction but does not distinguish between left and right. We used a PCA-based method that can infer a 3D orientation from multiple 2D orientations.
Detailed: Extract the direction information of each pixel from the input image. This information can help the network reconstruct the shape and fluidity of the hair.
The process of growing hair segments on a point cloud is as follows: for each point in the point cloud, we find its next neighbor point based on its direction, and then connect them into a line segment. This line segment is a hair part of the segment. We repeat this process until we encounter one of the following situations: 1) no suitable neighbor point is found; 2) the direction of the neighbor point is too different from the direction of the current point; 3) the neighbor point is already occupied by other hair segments. In this way, we can grow a set of local hair segments on the point cloud, which can be approximately represented by a curve.
The process of covering hair segments with strips is as follows: for each hair segment, we cover it with a strip of fixed width and thickness. The direction and length of this strip are determined by the data of the hair segment. decided. Then, we cluster each strip and divide them into several groups according to the similarity of their position and orientation. Each group contains some adjacent strips, which can be approximated by a curve. express. In this way, we can overlay the point cloud with ribbons, revealing locally consistent hair bundle structures that reflect the texture and flow of the hair and can also be associated with guide hairs for animation.
- The process of discovering missing connections between ribbons goes like this: Due to occlusions and missing data, there may be some gaps between the ribbons that affect the integrity and continuity of the hair bundles. To solve this problem, we tried fitting arcs to the hair segments covered by the strips to see if we could make connections at the gaps. For each strip, we find an endpoint of it, and then find an endpoint of another strip within its neighborhood to see if they can be connected by an arc. We used a method based on the least squares method to calculate the parameters of the arc, including the center, radius and angle. We also define a measure of the fitting error that depends on the difference in distance and orientation of the arc and hair segments. If the fitting error is less than a threshold, we consider that there is a reasonable connection between the two strips, and we encode this connection in a connection graph for subsequent analysis and optimization.
- The process of directional analysis of ribbons goes like this: the orientation of ribbons can be indeterminate due to the 180 degree ambiguity of the orientation field, which can lead to inconsistent orientations of hair bundles Or unreasonable. To solve this problem, we perform a global orientation analysis on the ribbons and use a Markov Random Field (MRF) formula to optimize the direction of each ribbon. Our goal is to make the ribbon orientation as compatible as possible with local cues for connections and ribbon orientation in the connection graph. The connections in the connection diagram represent the topological relationships between strips, and they should satisfy some geometric constraints. For example, the directions of adjacent strips should be smooth rather than abruptly turning. Local cues for the ribbon direction represent the shape characteristics of the ribbon, and they can be extracted from the data in the point cloud, such as the curvature, length, and direction changes of the ribbon. We used a graph cut-based method to solve this MRF formula and obtain the optimal direction of each strip. In this way, we can ensure that the direction of the hair bundles is consistent and reasonable.
- The process of connecting ribbons into hair bundles is as follows: After analyzing the connections and directions, we can connect the ribbons into complete hair bundles. The goal is for each hair strand to start at the scalp and end at the ends of the hair without breakage or overlap. We use a method based on depth-first search to traverse all the nodes in the connection graph, that is, the endpoints of the ribbon, and then connect them into a continuous curve based on their connections and directions. This curve is a Hair bundles. We repeat this process until all nodes have been traversed, or the preset number of hair bundles is reached. This way we can generate a complete set of hair strands from the strips, which reflect the structure and style of the hair, but also serve as guide hairs for use in animations and simulations.