| Traditional whiteboards offer a convenient, although static, environment for solving complex problems and exploring designs. Combined with software capable of understanding and reacting to sketches in real time, computational whiteboards could enable written equations to be captured and solved, state machines to be executed and debugged, and designs to be explored dynamically. Performing computation on the whiteboard requires working interactively with user inputs that are inherently noisy, whose interpretations are ambiguous, in a space frequently shared for many separate purposes. Computers capable of understanding and reacting to handwritten sketches have been the subject of intense Human-Computer Interface and Machine Learning research for over 50 years now, and while many pieces of the solution exist in isolation, a real roadblock to progress in the area is the lack of a unifying framework enabling modularity, composability, and multiple interpretations over the same data. This dissertation explores a novel approach to the way computing is structured within a free-form, digital whiteboard environment.;To inform our design decisions, we begin with a characterization of whiteboard traffic and an investigation into workloads that whiteboard-sized devices are likely to face in the future. Building on these findings, we present a software architecture for general purpose sketch recognition and interaction. Our whiteboard framework provides support for managing and coordinating shared resources, scheduling communication, dynamically controlling the allocation and deallocation of application code, and regularizing the integration of sketch-based components through a minimal interface. Finally, as an evaluation of our design, we provide a set of solutions to various sketch recognition problems through the implementation of applications developed within our framework.;Throughout this thesis, we describe the software realities of sketch-based applications that have led to our current design, the recurring patterns of application behavior that we encountered and how they are supported, the implementation details of our experimental framework, and demonstrate the first general purpose sketch recognition system capable of handling multiple, simultaneous applications. |
