Face-Voice Matching using Cross-modal Embeddings

Summary

The author using one audio do two pictrues classification by collect data from MS-celeb-1M and voxceleb. These results are very similar to human performance.

摘要 (中文)

Research Objective

Background and Problems

• Background

  • Speaker diarization, which identifies “who spoke when,” is an essential process in dialogue systems or robotics services.
  • There are two major groups of speaker diarization techniques: one uses the direction of arrival (DOA) of audio signals [13, 35, 44] and another uses lip motion [36].

• previous methods brief introduction

  • Nevertheless, there are cases when neither DOA-based nor lip motion-based speaker diarization can be used. Let us assume a situation when someone is speaking behind you.
  • only one study was proposed very recently for forcedchoice tasks [29]. The authors trained neural network-based classifiers that take an input of two voices (faces) as candidates and one face (voice) as a query, and output which candidate has the same identity as the query.

• Problem Statement

  • not state in introduction

main work

1. The method consists of two branches: a face branch and a voice branch. The face branch extracts 100-dimensional vectors from an input image using ResNet50 convolutional neural network architecture [12] and four fully connected layers. The voice branch first extracts an ivector [5] using a pre-trained acoustic model. The i-vector is then converted into 100-dimensional vectors by four fully connected layers.
2. There have two datasets. One is a big dataset, One is a age range to 18-30.
3. We proposed a cross-modal embedding-based face-voice matching model, which extracts features from face images and voices and optimizes the distance metric of the features extracted from two modalities.

loss:

Related work

• Human capability for face-voice association:

Method(s)

• Methods one : Study on Human Performance

Conclusion

Reference(optional)

Arouse for me