demonstration

Towards Generating UI Design Feedback with LLMs

Authors:

Peitong Duan,

Jeremy Warner,

Bjoern HartmannAuthors Info & Claims

UIST '23 Adjunct: Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

Article No.: 70, Pages 1 - 3

https://doi.org/10.1145/3586182.3615810

Published: 29 October 2023 Publication History

Get Access

Abstract

Feedback on user interface (UI) mockups is crucial for the design process, and designers often seek and leverage feedback to improve their UIs. However, human feedback is not always readily available. Given the recent emergence of LLMs, which have been shown to be proficient in rule-based reasoning, we explore the potential of LLMs to provide feedback automatically. In particular, we investigate automating heuristic evaluation, which currently entails a human expert assessing how well a UI adheres to a given set of design guidelines. We build an LLM-based heuristic evaluation plugin for Figma, which designers can use to evaluate their UI mockups. The plugin queries the LLM with the guidelines and a JSON representation of the UI mockup and then renders the identified guideline violations as constructive suggestions for design improvements. Future work is needed to study what types of usability problems can be successfully identified by LLM-driven heuristic evaluation.

Supplemental Material

ZIP File

Supplemental File

Download
266.64 MB

References

[1]

Zoya Bylinskii, Nam Wook Kim, Peter O’Donovan, Sami Alsheikh, Spandan Madan, Hanspeter Pfister, Fredo Durand, Bryan Russell, and Aaron Hertzmann. 2017. Learning Visual Importance for Graphic Designs and Data Visualizations. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 57–69. https://doi.org/10.1145/3126594.3126653

Digital Library

Google Scholar

[2]

Tsung-Hsiang Chang, Tom Yeh, and Robert C. Miller. 2010. GUI Testing Using Computer Vision. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 1535–1544. https://doi.org/10.1145/1753326.1753555

Digital Library

Google Scholar

[3]

Chun-Fu (Richard) Chen, Marco Pistoia, Conglei Shi, Paolo Girolami, Joseph W. Ligman, and Yong Wang. 2017. UI X-Ray: Interactive Mobile UI Testing Based on Computer Vision(IUI ’17). Association for Computing Machinery, New York, NY, USA, 245–255. https://doi.org/10.1145/3025171.3025190

Digital Library

Google Scholar

[4]

Chunggi Lee, Sanghoon Kim, Dongyun Han, Hongjun Yang, Young-Woo Park, Bum Chul Kwon, and Sungahn Ko. 2020. GUIComp: A GUI Design Assistant with Real-Time, Multi-Faceted Feedback. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376327

Digital Library

Google Scholar

[5]

Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshyvanyk. 2018. Automated Reporting of GUI Design Violations for Mobile Apps. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE ’18). Association for Computing Machinery, New York, NY, USA, 165–175. https://doi.org/10.1145/3180155.3180246

Digital Library

Google Scholar

[6]

D Royce Sadler. 1989. Formative assessment and the design of instructional systems. Instructional science 18 (1989), 119–144.

Google Scholar

[7]

Eldon Schoop, Xin Zhou, Gang Li, Zhourong Chen, Bjoern Hartmann, and Yang Li. 2022. Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 36, 21 pages. https://doi.org/10.1145/3491102.3517497

Digital Library

Google Scholar

[8]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arxiv:2201.11903 [cs.CL]

Google Scholar

[9]

Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey P Bigham. 2021. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 275, 15 pages. https://doi.org/10.1145/3411764.3445186

Digital Library

Google Scholar

Cited By

View all

Chin JChin JLee SLee SPark CPark CYeoun MYeoun M(2024)Proposal of User Interface Based on Heavy User Usage Analysis in LLM ServiceArchives of Design Research10.15187/adr.2024.08.37.4.28737:4(287-313)Online publication date: 31-Aug-2024
https://doi.org/10.15187/adr.2024.08.37.4.287
Lu QFang JYao ZYang YLyu SMi HYao L(2024)Large Language Model Agents Enabled Generative Design of Fluidic Computation InterfacesAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686351(1-3)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686351
Dix ALuyten KMayer SPalanque PPanizzi ESpano LZiegler J(2024)Second Workshop on Engineering Interactive Systems Embedding AI TechnologiesCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3662837(103-107)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3662837

Index Terms

Towards Generating UI Design Feedback with LLMs
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. Heuristic evaluations
    2. Interactive systems and tools
  2. Interaction design
    1. Interaction design process and methods
      1. User interface design

Recommendations

Generating Automatic Feedback on UI Mockups with Large Language Models
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

Feedback on user interface (UI) mockups is crucial in design. However, human feedback is not always readily available. We explore the potential of using large language models for automatic feedback. Specifically, we focus on applying GPT-4 to automate ...
Akin: Generating UI Wireframes From UI Design Patterns Using Deep Learning
IUI '21 Companion: Companion Proceedings of the 26th International Conference on Intelligent User Interfaces

During the User interface (UI) design process, designers use UI design patterns for conceptualizing different UI wireframes for an application. This paper introduces Akin, a UI wireframe generator that allows designers to chose a UI design pattern and ...
UI Fin: a process-oriented interface design tool
IUI '09: Proceedings of the 14th international conference on Intelligent user interfaces

Even though over the years a multitude of user interface design tools have been created, designers in practice find themselves limited to a small set of realistic options. These options include interface builders that are attached to development ...

Comments

Information & Contributors

Information

Published In

UIST '23 Adjunct: Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

October 2023

424 pages

ISBN:9798400700965

DOI:10.1145/3586182

Editors:
Sean Follmer
Stanford University, USA
,
Jeff Han,
Jürgen Steimle
Saarland University, Germany
,
Nathalie Henry Riche
Microsoft Research, USA

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 October 2023

Check for updates

Author Tags

Qualifiers

Demonstration
Research
Refereed limited

Conference

UIST '23

Sponsor:

UIST '23: The 36th Annual ACM Symposium on User Interface Software and Technology

October 29 - November 1, 2023

CA, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 355 of 1,733 submissions, 20%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
719
Total Downloads

Downloads (Last 12 months)562
Downloads (Last 6 weeks)32

Reflects downloads up to 20 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Chin JChin JLee SLee SPark CPark CYeoun MYeoun M(2024)Proposal of User Interface Based on Heavy User Usage Analysis in LLM ServiceArchives of Design Research10.15187/adr.2024.08.37.4.28737:4(287-313)Online publication date: 31-Aug-2024
https://doi.org/10.15187/adr.2024.08.37.4.287
Lu QFang JYao ZYang YLyu SMi HYao L(2024)Large Language Model Agents Enabled Generative Design of Fluidic Computation InterfacesAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686351(1-3)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686351
Dix ALuyten KMayer SPalanque PPanizzi ESpano LZiegler J(2024)Second Workshop on Engineering Interactive Systems Embedding AI TechnologiesCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3662837(103-107)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3662837

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

Supplemental Material

References

Cited By

Index Terms

Recommendations

Generating Automatic Feedback on UI Mockups with Large Language Models

Akin: Generating UI Wireframes From UI Design Patterns Using Deep Learning

UI Fin: a process-oriented interface design tool

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations