Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

Contenu

Titre

Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

Date de soumission

26 mars 2023 à 08:21:52 +00:00

Créateur

Chenfei Wu

Shengming Yin

Weizhen Qi

Xiaodong Wang

Zecheng Tang

Nan Duan

Est référencé par

BB33LBRM

Résumé

ChatGPT is attracting a cross-field interest as it provides a language interface with remarkable conversational competency and reasoning capabilities across many domains. However, since ChatGPT is trained with languages, it is currently not capable of processing or generating images from the visual world. At the same time, Visual Foundation Models, such as Visual Transformers or Stable Diffusion, although showing great visual understanding and generation capabilities, they are only experts on specific tasks with one-round fixed inputs and outputs. To this end, We build a system called \textbf{Visual ChatGPT}, incorporating different Visual Foundation Models, to enable the user to interact with ChatGPT by 1) sending and receiving not only languages but also images 2) providing complex visual questions or visual editing instructions that require the collaboration of multiple AI models with multi-steps. 3) providing feedback and asking for corrected results. We design a series of prompts to inject the visual model information into ChatGPT, considering models of multiple inputs/outputs and models that require visual feedback. Experiments show that Visual ChatGPT opens the door to investigating the visual roles of ChatGPT with the help of Visual Foundation Models. Our system is publicly available at \url{https://github.com/microsoft/visual-chatgpt}.

Date

8 mars 2023

doi

10.48550/arXiv.2303.04671

uri

http://arxiv.org/abs/2303.04671

Titre abrégé

Visual ChatGPT

Source

arXiv.org

is compiled by

Lucky Semiosis

is in semantic relation with

computer science

Computer Vision and Pattern Recognition

Complexité

540

Date de modification

8 septembre 2023 à 06:54:19 +00:00

Détails de la complexité

Physique,1,,,,,39,39

Physique,2,,,,,46,92

Actant,2,,,,,7,14

Concept,1,,,,,38,38

Concept,2,,,,,46,92

Rapport,1,1,Physique,Concept,properties,38,38

Rapport,1,1,Physique,Physique,values,38,38

Rapport,1,1,Physique,Actant,dcterms:creator,6,6

Rapport,2,2,Actant,Concept,properties,34,68

Rapport,2,2,Actant,Physique,values,34,68

Rapport,1,1,Physique,Actant,cito:isCompiledBy,1,1

Rapport,1,1,Physique,Concept,skos:semanticRelation,2,2

Rapport,2,2,Concept,Concept,properties,6,12

Rapport,2,2,Concept,Physique,values,6,12

Rapport,1,1,Physique,Physique,media,2,2

Rapport,2,2,Physique,Concept,properties,4,8

Rapport,2,2,Physique,Physique,values,4,8

Rapport,1,1,Physique,Physique,uri,2,2

Totaux de la complexité

Physique,2,1,2,85,131

Actant,1,2,2,7,14

Concept,2,1,2,84,130

Rapport,13,1,2,177,265

Existence,18,1,2,353,540

437

Date de modification

8 janvier 2024 à 10:57:46 +00:00

Détails de la complexité

dimension,niv (sujet),niv objet,sujet,objet,prédicat,nb,c

Physique,1,,,,,19,19

Physique,2,,,,,47,94

Actant,1,,,,,1,1

Actant,2,,,,,7,14

Concept,1,,,,,12,12

Concept,2,,,,,43,86

Rapport,1,1,Actant,Concept,properties,12,12

Rapport,1,1,Actant,Physique,values,18,18

Rapport,1,1,Actant,Physique,owner,1,1

Rapport,2,1,Actant,Physique,oa:hasSource,2,4

Rapport,2,2,Actant,Concept,properties,41,82

Rapport,2,2,Actant,Physique,values,41,82

Rapport,1,1,Actant,Actant,dcterms:creator,6,6

Rapport,1,1,Actant,Actant,cito:isCompiledBy,1,1

Rapport,1,1,Actant,Concept,skos:semanticRelation,2,2

Rapport,1,1,Actant,Physique,media,2,2

Rapport,1,1,Actant,Physique,uri,1,1

Totaux de la complexité

Physique,2,1,2,66,113

Actant,2,1,2,8,15

Concept,2,1,2,55,98

Rapport,11,1,2,127,211

Existence,17,1,2,256,437,

Collections

Zotero