CM3leon: A Multimodal Chameleon in Generative AI

CM3leon, a groundbreaking multimodal AI model capable of generating both text and graphics.

2 min read

Recently, there has been a resurgence of interest in generative AI models due to advances in natural language processing and systems that generate graphics from text input. A single foundation model that can create both text and graphics, CM3leon (pronounced "chameleon"), has been revealed by a recent Meta-analysis. 

CM3leon is the first multimodal model trained using a recipe modified from text-only language models, including a second multitask supervised fine-tuning (SFT) stage and a large-scale retrieval-augmented pre-training stage.

The CM3Leon architecture uses a decoder-only transformer, similar to standard text-based models. Processing and generating text and graphics distinguishes CM3Leon from other systems. CM3leon offers cutting-edge performance for text-to-image generation despite being trained with five times less computing than preceding transformer-based techniques.

The versatility and strength of autoregressive models are combined with the effectiveness and economy of training and inference in CM3leon. The CM3 model satisfies the causal masked mixed-modal model requirements since it can produce text and image sequences depending on any given text and image sequence. This is a significant improvement compared to earlier models, which could only do one of these tasks. 

The researchers demonstrate how CM3leon may significantly improve performance on tasks including image caption production, visual question answering, text-based editing, and conditional image generation by using large-scale multitask instruction tuning to both picture and text generation. The team has included an independently trained super-resolution stage to generate higher-resolution images from the initial model outputs.

The characteristics of CM3leon allow image production tools to create more logical imagery that closely matches input instructions. Here are some examples of the various tasks that CM3leon can complete using just one model: 

  • Creation and alteration of images with text guidance
  • Text-to-image
  • Image editing with text guidance
  • Text-based exercises
  • Object-oriented image editing
  • Object-to-image
  • Segmentation-to-image
  • Results of super-resolution

In the long run, models like CM3leon might help foster innovation and improve applications in the metaverse. In the future, we are excited to release more models and push the limits of multimodal language models.