INSUBCONTINENT EXCLUSIVE:
The landscape of vision design pre-training has actually gone through substantial development, particularly with the increase of Large
Typically, vision designs operated within repaired, predefined paradigms, however LLMs have introduced a more flexible method, unlocking new
ways to leverage pre-trained vision encoders
This shift has actually prompted a reevaluation of pre-training methods for vision designs to better line up with multimodal applications.In
a brand-new paper Multimodal Autoregressive Pre-training of Large Vision Encoders, an Apple research team presents AIMV2, a household of
vision encoders that uses a multimodal autoregressive pre-training strategy
Unlike standard methods, AIMV2 is developed to anticipate both image patches and text tokens within an unified series
This combined objective makes it possible for the design to excel in a series of jobs, such as image recognition, visual grounding, and
multimodal understanding.The crucial development of AIMV2 lies in its capability to generalize the unimodal autoregressive framework to a
By treating image patches and text tokens as a single sequence, AIMV2 combines the prediction process for both methods
This approach enhances its capability to understand complicated visual and textual relationships.The pre-training process of AIMV2 involves
a causal multimodal decoder that first forecasts image spots, followed by the generation of text tokens in an autoregressive manner
This basic yet reliable style provides several benefits: Simplicity and Efficiency: The pre-training procedure does not need large batch
sizes or complex inter-batch interaction, making it simpler to implement and scale.Alignment with LLM Multimodal Applications: The
architecture naturally incorporates with LLM-driven multimodal systems, enabling smooth interoperability.Denser Supervision: By extracting
learning signals from every image spot and text token, AIMV2 accomplishes denser guidance compared to conventional discriminative goals,
assisting in more efficient training.The architecture of AIMV2 is centered on the Vision Transformer (ViT), a well-established model for
Nevertheless, the AIMV2 group presents crucial adjustments to boost its efficiency: Constrained Self-Attention: A prefix attention mask is
used within the vision encoder, allowing bidirectional attention during reasoning without additional adjustments.Feedforward and
Normalization Upgrades: The SwiGLU activation function is made use of as the feedforward network (FFN), while all normalization layers are
These choices are inspired by the success of similar techniques in language modeling, causing improved training stability and
efficiency.Unified Multimodal Decoder: A shared decoder manages the autoregressive generation of image spots and text tokens at the same
time, further strengthening AIMV2s multimodal capabilities.Empirical examinations reveal the outstanding abilities of AIMV2
The AIMV2-3B encoder achieves 89.5% accuracy on ImageNet-1k utilizing a frozen trunk, showing its potential for high-performance image
AIMV2 regularly exceeds cutting edge contrastive models, such as CLIP and SigLIP, in multimodal image understanding across diverse
benchmarks.One of the crucial factors to this success is AIMV2s capability to fully make use of the knowing signals from all input tokens
This dense supervision method enables more efficient training with fewer samples compared to other self-supervised or vision-language
pre-trained models.AIMV2 represents a significant step forward in the advancement of vision encoders
By unifying image and text prediction under a single multimodal autoregressive structure, AIMV2 accomplishes remarkable efficiency
throughout a broad series of tasks
Its simple pre-training procedure, integrated with architectural improvements like SwiGLU and RMSNorm, ensures scalability and adaptability
As vision models continue to scale, AIMV2 provides a blueprint for more effective, flexible, and merged multimodal learning systems.The code
is available on tasks GitHub
The paper Multimodal Autoregressive Pre-training of Large Vision Encoders is on arXiv.Author: Hecate He|Editor: Chain ZhangLike this: ...
