Computational Analysis of Behavior and Mental Disorders

Synchrony is observed in countless phenomena and its discovery is central to answering many computer vision and pattern recognition problems. A basic research problem has remained surprisingly underexplored: Given a set of sequences, how can we discover subsets in which every pair of sequences satisfies a synchrony criterion? This problem naturally arises in many domains. For example, in a video, only movement in a subset of pixels is synchronized in time, and their identification can lead to discovering events of interest (e.g., facial expressions).

We develop methods to detect and quantify synchrony in behavioral, biological, and other data types. Our tools can be used in many domains including computer vision, brain connectivity analysis, financial analysis, genetic analysis, among many others.

Temporal phase estimation for 15 MMI sequences. For each sequence, we show (i) the input set i.e., the motion of all pixels over time; (ii) the motion of synchronized pixels discovered with SyncRef, (iii) the ground truth (GT) temporal phase label (solid green lines) vs. the label predicted as the mean of synchronized pixels (dashed blue lines); and (iv) expression at apex.

Brain regions with high synchrony. (a) Regions are colored based on the number of times that they were included in the synchronized set. (b) Regions are assigned to the seven known functional systems of the brain. Numbers and colors indicate the number of times that the regions of these systems were included in the largest synchronized set.

Sariyanidi E., Zampella C. J., Bartley K. G., Herrington J., Satterthwaite T. D., Schultz R. T., Tunç B., Discovering Synchronized Subsets of Sequences: A Large Scale Solution, IEEE Conference on Computer Vision and Pattern Recognition, Accepted for publication, 2020