A Unified Semiotics Framework for Spatial and Non-Spatial Brain Network Data Visualization

We have designed a semiotics approach for the design and evaluation of visualizations of spatial and non-spatial data. Semiotics is the study of symbols and their compositions. Motivated by scientists' increasing difficulty in acquiring knowledge from their data with increasing complexity and heterogeneity, we designed a semiotics framework to provide a unified way to study visualizations. Our semiotics approach expands upon Bertin's semiology to understand how and why visualization works. This approach can help visualization designers study visual symbols and their compositions in both two-dimensional (2D) and three-dimensional (3D) visualizations.;We claim three major contributions. Bertin's semiology classified a set of retinal properties for data presentation: position on a 2D plane, size, color, value, orientation, texture grain and shape. A mark with variations on these retinal properties is a symbol that encodes information. Our first contribution is that we have extended Bertin's 2D semiology to 3D visualizations and added shading. We have demonstrated the practicality of this extension (1) by describing existing techniques using this approach; (2) by conducting empirical studies on the effectiveness of visual encoding and shading methods for brain connectivity visualizations, including both 2D network and 3D tractography data visualizations; and (3) by a focused study on color variations. Our second contribution: using this approach as the theoretical foundation, we have demonstrated its use through two ranking studies on a set of visual symbols and their compositions. From these studies, we provided a series of design recommendations drawn upon our empirical study results to help visualization designers make more informed choices on visualization designs. Last but not least, our work expands the knowledge of visualization design by adding our ranking of visual variable effectiveness to existing ranking studies.;Human brain imaging research consists of multi-modal and heterogeneous data analyses and we applied our approach to study brain imaging data visualizations. We have conducted three user studies on 2D functional network and 3D tract data visualizations.;In the first study, we ranked and compared four retinal properties for quantitative comparison of aggregated values and four shading methods for 3D spatial structure discriminations. The results indicate that hue-varying iso-luminance color and monotonic-luminance color maps are the most effective methods for encoding quantities compared to size and texture. Shading with color encoding tract orientations was the most effective for 3D spatial structure discriminations compared to halo, depth-dependent halo and ambient occlusion lighting. The second study focused on 2D network visualizations and studied visualization performance with nine mark types (area, hue, lightness, angle, slope, length, shape, density, and texture) combined with three positioning types (projection, circular, and matrix). The results show that area and size are the most effective retinal properties. Among the three positioning methods, circular was always among the best. Projection was good when tasks demanded symmetry or proximity and matrix was good when placing marks in close proximity benefited the tasks. The last study was on color due to color being most effective in 3D dense data visualization as discovered from the first study. We studied six types of color maps (gray scale, black-body, diverging, iso-luminant, extended black-body and cool-warm) for quantitative value aggregations and four types of color mapping of spatial structure (all gray, absolute color, eigenmap embedding, and Boy's surface embedding). The results suggest that a monotonic luminance color map with a moderate amount of hue variations is the best and that orientation encoding using Boy's surface embedding provides the highest accuracy.;In conclusion, our semiotics framework has led us to design a set of novel experiments to understand how visualization works. It contributes to the understanding of complex quantitative visual aggregation and spatial discrimination tasks for 3D brain connectivity visualizations as well as a novel ranking of visual variables for showing quantitative data on 2D networks.