Computer vision and the psychology of clothing

When meeting a person we try to make inferences about “the kind of person they are”. One set of clues that we (sometimes inaccurately) use are clothes worn by the target person. In this context, we answered three questions in a new research string. First, what are the most common clothing-based inferences made by laypeople and researchers? Second, how much agreement is their typically between different people making inferences on the same set of clothing? And third, can machines recognize subtle social cues in clothing and predict the inferences that actual humans would draw based on specific clothes?

We find that the most common psychological inferences pertain to the broad dimensions of happiness, sexual interest, intelligence, trustworthiness, and confidence. Interestingly however, different people often make different inferences based on the same clothes. In fact, it is generally more informative to know who does the judging than how the clothes look like.

Accordingly, computer vision models can repredict human inferences only medium well, but still better than asking one or even two actual human raters to provide inferences.

In this research, we also provide a dataset of 5000 annotated clothing images and point out interesting associations between clothing characteristics and psychological inferences (e.g., extreme effects of wearing silver).

If you are interested in the details, please read the postprint here: Publication in Journal of Computational Social Science