Pascal Rupp
Zero-Shot Scatterplot Annotation
[
thesis]
Information
- Publication Type: Bachelor Thesis
- Workgroup(s)/Project(s):
- Date: September 2026
- Date (Start): February 2026
- Date (End): August 2026
- Matrikelnummer: 12323804
- First Supervisor: Manuela Waldner
- Keywords: dimensionality reduction, unstructured data, scatterplot annotation
Abstract
Scatter plots of image embeddings group visually similar images into clusters. To understand what a cluster represents, the individual images must be examined and their commonalities interpreted. This paper investigates whether a user-specified vocabulary of expected concepts can be interactively and automatically placed into an existing scatter plot using a pre-trained vision-language model. The approach utilizes the shared high-dimensional embedding space of Contrastive Language-Image Pretraining (CLIP) for images and text. A text label is encoded in the same way as an image and projected into the existing two-dimensional layout of the image embeddings using the adapted Uniform Manifold Approximation and Projection (UMAP) model. Since both CLIP encoders were trained to share the same high-dimensional space, the position of the label in the resulting scatter plot provides an approximate answer to the question of where to find images that match that word—without any classifier training and without a single labeled example. A simple concept creation interface allows users to define and customize their own annotation vocabulary and immediately place it on the scatter plot. The approach is evaluated on the Canadian Institute for Advanced Research-100 (CIFAR- 100) and Fashion-Modified National Institute of Standards and Technology database (Fashion-MNIST) datasets. Across all 100 CIFAR-100 classes, the projected labels are, on average, about as close to their target cluster as a typical image from that class itself. However, accuracy varies significantly: classes that are clearly distinct visually and semantically are labeled reliably, while classes that overlap in CLIP’s own embedding space—such as several person-related categories—are not reliably recognized. The system also generalizes to vocabulary that was never used to label the dataset, while remaining within the interactive time budget (approximately 21 ms per label after a one-time setup). The results show that projecting user-defined text directly onto an image scatter plot is a fast and practical way to make its clusters interpretable without labeled training data—with reliability limited by the underlying vision-language model rather than by the projection mechanism.Additional Files and Images
Weblinks
No further information available.BibTeX
@bachelorsthesis{rupp-2026-zsa,
title = "Zero-Shot Scatterplot Annotation",
author = "Pascal Rupp",
year = "2026",
abstract = "Scatter plots of image embeddings group visually similar
images into clusters. To understand what a cluster
represents, the individual images must be examined and their
commonalities interpreted. This paper investigates whether a
user-specified vocabulary of expected concepts can be
interactively and automatically placed into an existing
scatter plot using a pre-trained vision-language model. The
approach utilizes the shared high-dimensional embedding
space of Contrastive Language-Image Pretraining (CLIP) for
images and text. A text label is encoded in the same way as
an image and projected into the existing two-dimensional
layout of the image embeddings using the adapted Uniform
Manifold Approximation and Projection (UMAP) model. Since
both CLIP encoders were trained to share the same
high-dimensional space, the position of the label in the
resulting scatter plot provides an approximate answer to the
question of where to find images that match that
word—without any classifier training and without a single
labeled example. A simple concept creation interface allows
users to define and customize their own annotation
vocabulary and immediately place it on the scatter plot. The
approach is evaluated on the Canadian Institute for Advanced
Research-100 (CIFAR- 100) and Fashion-Modified National
Institute of Standards and Technology database
(Fashion-MNIST) datasets. Across all 100 CIFAR-100 classes,
the projected labels are, on average, about as close to
their target cluster as a typical image from that class
itself. However, accuracy varies significantly: classes that
are clearly distinct visually and semantically are labeled
reliably, while classes that overlap in CLIP’s own
embedding space—such as several person-related
categories—are not reliably recognized. The system also
generalizes to vocabulary that was never used to label the
dataset, while remaining within the interactive time budget
(approximately 21 ms per label after a one-time setup). The
results show that projecting user-defined text directly onto
an image scatter plot is a fast and practical way to make
its clusters interpretable without labeled training
data—with reliability limited by the underlying
vision-language model rather than by the projection
mechanism.",
month = sep,
address = "Favoritenstrasse 9-11/E193-02, A-1040 Vienna, Austria",
school = "Research Unit of Computer Graphics, Institute of Visual
Computing and Human-Centered Technology, Faculty of
Informatics, TU Wien ",
keywords = "dimensionality reduction, unstructured data, scatterplot
annotation",
URL = "https://www.cg.tuwien.ac.at/research/publications/2026/rupp-2026-zsa/",
}