Re:Draw - Context Aware Translation as a Controllable Method for Artistic Production

Information

• Publication Type: Conference Paper
• Workgroup(s)/Project(s):
  • Advanced Visual and Geometric Computing for 3D Capture, Display, and Fabrication
• Date: August 2024
• ISBN: 978-1-956792-04-1
• Publisher: International Joint Conferences on Artificial Intelligence
• Location: Jeju Island
• Lecturer: Joao Afonso Cardoso
• Event: 33rd International Joint Conference on Artificial Intelligence (IJCAI 2024)
• DOI: 10.24963/ijcai.2024/842
• Booktitle: Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI-24)
• Pages: 9
• Conference date: 3. August 2024 – 9. August 2024
• Pages: 7609 – 7617
• Keywords: Application domains: Images, movies and visual arts, Application domains: Computer Graphics and Animation, Methods and resources: AI systems for collaboration and co-creation, Methods and resources: Machine learning, deep learning, neural models, reinforcement learning, Theory and philosophy of arts and creativity in AI systems: Social (multi-agent) creativity and human-computer co-creation

Abstract

We introduce context-aware translation, a novel method that combines the benefits of inpainting and image-to-image translation, respecting simultaneously the original input and contextual relevance – where existing methods fall short. By doing so, our method opens new avenues for the controllable use of AI within artistic creation, from animation to digital art. As an use case, we apply our method to redraw any hand-drawn animated character eyes based on any design specifications – eyes serve as a focal point that captures viewer attention and conveys a range of emotions; however, the labor-intensive na- ture of traditional animation often leads to compromises in the complexity and consistency of eye design. Furthermore, we remove the need for production data for training and introduce a new character recognition method that surpasses existing work by not requiring fine-tuning to specific productions. This proposed use case could help maintain consistency throughout production and unlock bolder and more detailed design choices without the production cost drawbacks. A user study shows contextaware translation is preferred over existing work 95.16% of the time.

Additional Files and Images

Weblinks

• Entry in reposiTUm (TU Wien Publication Database)
• DOI: 10.24963/ijcai.2024/842

BibTeX

@inproceedings{cardoso-2024-r-c,
  title =      "Re:Draw - Context Aware Translation as a Controllable Method
               for Artistic Production",
  author =     "Joao Afonso Cardoso and Francesco Banterle and Paolo Cignoni
               and Michael Wimmer",
  year =       "2024",
  abstract =   "We introduce context-aware translation, a novel method that
               combines the benefits of inpainting and image-to-image
               translation, respecting simultaneously the original input
               and contextual relevance – where existing methods fall
               short. By doing so, our method opens new avenues for the
               controllable use of AI within artistic creation, from
               animation to digital art. As an use case, we apply our
               method to redraw any hand-drawn animated character eyes
               based on any design specifications – eyes serve as a focal
               point that captures viewer attention and conveys a range of
               emotions; however, the labor-intensive na- ture of
               traditional animation often leads to compromises in the
               complexity and consistency of eye design. Furthermore, we
               remove the need for production data for training and
               introduce a new character recognition method that surpasses
               existing work by not requiring fine-tuning to specific
               productions. This proposed use case could help maintain
               consistency throughout production and unlock bolder and more
               detailed design choices without the production cost
               drawbacks. A user study shows contextaware translation is
               preferred over existing work 95.16% of the time.",
  month =      aug,
  isbn =       "978-1-956792-04-1",
  publisher =  "International Joint Conferences on Artificial Intelligence",
  location =   "Jeju Island",
  event =      "33rd International Joint Conference on Artificial
               Intelligence (IJCAI 2024)",
  doi =        "10.24963/ijcai.2024/842",
  booktitle =  "Proceedings of the Thirty-Third International Joint
               Conference on Artificial Intelligence (IJCAI-24)",
  pages =      "9",
  pages =      "7609--7617",
  keywords =   "Application domains: Images, movies and visual arts,
               Application domains: Computer Graphics and Animation,
               Methods and resources: AI systems for collaboration and
               co-creation, Methods and resources: Machine learning, deep
               learning, neural models, reinforcement learning, Theory and
               philosophy of arts and creativity in AI systems: Social
               (multi-agent) creativity and human-computer co-creation",
  URL =        "https://www.cg.tuwien.ac.at/research/publications/2024/cardoso-2024-r-c/",
}