Information
- Publication Type: Master Thesis
- Workgroup(s)/Project(s): not specified
- Date: 2026
- TU Wien Library: AC18105606
- Second Supervisor: Johannes Eschner
- Open Access: yes
- First Supervisor: Manuela Waldner
- Pages: 67
- Keywords: concept discovery, human-in-the-loop, unstructured data
Abstract
Large image collections often contain recurring visual concepts. Without predefined classes or taxonomies, however, they are difficult to explore. Automatically generated groups do not necessarily reflect the concepts or granularity relevant to a particular analysis. This thesis therefore introduces Semi-Automatic Concept Discovery (SaCoDi), a human-AI workflow that clusters and labels image collections while allowing users to split or merge clusters. SaCoDi is implemented as a Python prototype. Its command-line interface was used for controlled experiments, while a graphical interface supports cluster inspection and refinement. Three labeling strategies are compared: deterministic WordNet labeling, direct labeling by a multimodal language model, and hybrid language-model abstraction of WordNet candidates. Experiments on STL-10, PACS, and Food-101 evaluate label quality, the effects of content and visual domain, refinement operations, and runtime. The direct multimodal strategy achieves the highest weighted coverage, exclusivity, and semantic compatibility in every dataset configuration and the highest or tied-highest class compatibility. The WordNet strategy is reproducible and inspectable but can select overly broad or incorrect terms. The hybrid strategy can derive more suitable abstractions but does not consistently match the direct strategy's specificity. All initial outputs are structurally valid. Local reclustering yields the highest class purity in four selected split cases, while hybrid and visual merges balance coverage and exclusivity differently. Semantic labeling dominates runtime, and local edits are cheaper than complete recomputation. The evaluation uses single-author semantic judgments, no user study, and curated classification benchmarks. The findings therefore support SaCoDi's technical and methodological feasibility, not its effectiveness for end users.
Additional Files and Images
Weblinks
BibTeX
@mastersthesis{wolf-2026-sacodi,
title = "SaCoDi: Semi-Automatic Concept Discovery Utilizing
Multimodal Models and Interactive Refinement",
author = "Dominik Wolf",
year = "2026",
abstract = "Large image collections often contain recurring visual
concepts. Without predefined classes or taxonomies, however,
they are difficult to explore. Automatically generated
groups do not necessarily reflect the concepts or
granularity relevant to a particular analysis. This thesis
therefore introduces Semi-Automatic Concept Discovery
(SaCoDi), a human-AI workflow that clusters and labels image
collections while allowing users to split or merge clusters.
SaCoDi is implemented as a Python prototype. Its
command-line interface was used for controlled experiments,
while a graphical interface supports cluster inspection and
refinement. Three labeling strategies are compared:
deterministic WordNet labeling, direct labeling by a
multimodal language model, and hybrid language-model
abstraction of WordNet candidates. Experiments on STL-10,
PACS, and Food-101 evaluate label quality, the effects of
content and visual domain, refinement operations, and
runtime. The direct multimodal strategy achieves the highest
weighted coverage, exclusivity, and semantic compatibility
in every dataset configuration and the highest or
tied-highest class compatibility. The WordNet strategy is
reproducible and inspectable but can select overly broad or
incorrect terms. The hybrid strategy can derive more
suitable abstractions but does not consistently match the
direct strategy's specificity. All initial outputs are
structurally valid. Local reclustering yields the highest
class purity in four selected split cases, while hybrid and
visual merges balance coverage and exclusivity differently.
Semantic labeling dominates runtime, and local edits are
cheaper than complete recomputation. The evaluation uses
single-author semantic judgments, no user study, and curated
classification benchmarks. The findings therefore support
SaCoDi's technical and methodological feasibility, not its
effectiveness for end users.",
pages = "67",
address = "Favoritenstrasse 9-11/E193-02, A-1040 Vienna, Austria",
school = "Research Unit of Computer Graphics, Institute of Visual
Computing and Human-Centered Technology, Faculty of
Informatics, TU Wien",
keywords = "concept discovery, human-in-the-loop, unstructured data",
URL = "https://www.cg.tuwien.ac.at/research/publications/2026/wolf-2026-sacodi/",
}
thesis