CM3leon by Meta
UnknownMeta research foundation model for efficient multimodal text-to-image and image-to-text generation.
工具信息
什么是 CM3leon by Meta？
CM3leon by Meta is a single causal masked mixed-modal foundation model that performs both text-to-image and image-to-text generation. It is trained with a retrieval-augmented pre-training and multitask supervised fine-tuning recipe adapted from text-only language models, enabling generation of sequences of text and images conditioned on arbitrary prior image and text content.
核心功能
Text-to-image generation
Generates coherent images from text prompts, including highly compositional structures and complex objects, achieving state-of-the-art FID of 4.88 on zero-shot MS-COCO.
Image-to-text generation
Produces text outputs such as captions and answers from image inputs as part of the same multimodal model.
Text-guided image editing
Edits images according to textual instructions while understanding both text and visual content, such as changing sky color.
Visual question answering
Answers questions about image content through multitask instruction tuning.
Image caption generation and long-form captioning
Generates image captions and long-form descriptions, with zero-shot performance competitive against larger models.
Conditional and multitask image generation
Supports conditional image generation and super-resolution within a single model via large-scale multitask supervised fine-tuning.
Retrieval-augmented pre-training and efficient training
Uses large-scale retrieval-augmented pre-training and multitask SFT; trained with five times less compute than previous transformer-based methods while remaining inference-efficient.
使用场景
- #1
Complex compositional image synthesis
Creating detailed images from multi-constraint prompts, such as a potted cactus with sunglasses and a hat or a raccoon anime character preparing for battle.
- #2
Text-based image editing
Modifying existing images via natural language instructions while preserving global shapes and local details.
- #3
Vision-language understanding tasks
Performing image captioning, long-form captioning, and visual question answering with a single model.
- #4
Multimodal sequence generation
Generating mixed sequences of text and images conditioned on arbitrary prior image and text content.
定价方案
Research model announcement only; no pricing, commercial product, public access, API, or licensing details provided.
