for Robust Facial Landmark Localization Chanmi Yu Computer Vision Lab

Motivation: Facial Image Analysis Where I am looking at? Where is my face? => Face Detection Am I smiling? Am I wearing sun-glasses ? Facial Landmark Detection (or Face Alignment) 2

A Naïve method: Sliding Window Eye detector Nose detector Mouth detector Chin detector Input Shape constraint Running detectors independently Goal: Detecting facial points Running detectors jointly 3

Proposed work: Exemplar-based Graph Matching (EGM) Bigger circle -> Larger weight Efficient candidate point generation Affine-invariant shape constraint Optimal inference 4

Proposed System Key contribution 1 SVM Training Training Images Similar Exemplars Shape Constraints Detectors QP Testing Image Sliding window Response Map Candidates Points RANSAC Inference Result Key contribution 2 Belhumeur et al. CVPR, 2011.

Landmark Detector Training 29 Landmarks 858 training images 3 positive patches (randomly rotated) per image 6 negative patches per image Root SIFT + SVM Regression Positive Negative

Slide window: Reduce the Search Area Scan on a smaller area.

Candidates points generation RANSAC-like RANSAC 과 비슷한 방법으로 한다.!! Transformation matrix P. N. Belhumeur, D. W. Jacobs, D. J. Kriegman, and N. Kumar. “Localizing parts of faces using a consensus of exemplars”. In CVPR, 2011. 8

Affine-invariant Shape Constraints Coordinate matrix   i : exemplar(total m) c: landmark(total k)

Inference Correspondence matrix: k=4, n=8 case Cost matrix: Weight matrix: Coordinate matrix:

Experiments LFPW bioID Helen:illumination, 등의 challenge가 많아서 더 차이가 컸다 Training, Testing Testing Testing Better performance than Belhumeur LFPW bioID Helen:illumination, 등의 challenge가 많아서 더 차이가 컸다 일부 이미지에서는 training set이 덜 다양해서 exemplar가 제한적이었다 정량적인 결과는 나타내지 않고 정성적인것만 나타냈다. 꽤 괜찮은편이지만 잘 안맞는거는 exemplar때문이라고 한다. 0.02 error = 1 pixel Worse performance due to limited exemplars #training images: 858 #testing images: 228 #testing images: 1520 11

Discussions Affine-invariant shape constraint learned from multiple exemplars Optimal landmark configuration obtained by solving an LP-based graph matching problem Applicability in other domains such as body part detection and human pose estimation If the exemplars are limited, performance would be worse. Detector and Graph has different effect on each landmarks. 코 부분에 에러가 생기기 쉽다. -> detector보다 shape constraint에서 더 높은 영향 1~8 eyebrow부분은 detector가 더 큰 영향 잘 안된 이미지들이 왜 안됐는지 설명 새로운 아이디어 제시 12

Error 0.48 Test image SVM Training Training Images Similar Exemplars Haar-like feature SVM LFPW dataset (858 images) Test image SVM Training Training Images Similar Exemplars Shape Constraints Detectors QP Testing Image Sliding window Response Map Candidates Points RANSAC Inference Result Error 0.48 Result 13

Results and Evaluation 14

Thank you