beitv2_large_patch16_224.in1k_ft_in22k模型卡片

一个BEiT-v2图像分类模型。使用自监督的掩蔽图像建模（MIM）通过OpenAI CLIP B/16 teacher中的VQ-KD编码器作为视觉词汇化器进行ImageNet-1k的训练。在ImageNet-22k上进行微调。

模型细节

• 模型类型：图像分类 / 特征主干
• 模型统计数据：
  • 参数（M）：325.8
  • GMACs：61.6
  • 激活数（M）：63.5
  • 图像尺寸：224 x 224
• 论文：
  • BEiT v2：基于向量量化的视觉词汇掩蔽图像建模： https://arxiv.org/abs/2208.06366
  • 一张图像等于16x16个词：规模化图像识别的Transformer： https://arxiv.org/abs/2010.11929v2
• 数据集：ImageNet-22k
• 原始来源： https://github.com/microsoft/unilm/tree/master/beit2

模型使用

图像分类

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model('beitv2_large_patch16_224.in1k_ft_in22k', pretrained=True)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # unsqueeze single image into batch of 1

top5_probabilities, top5_class_indices = torch.topk(output.softmax(dim=1) * 100, k=5)

图像嵌入

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model(
    'beitv2_large_patch16_224.in1k_ft_in22k',
    pretrained=True,
    num_classes=0,  # remove classifier nn.Linear
)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # output is (batch_size, num_features) shaped tensor

# or equivalently (without needing to set num_classes=0)

output = model.forward_features(transforms(img).unsqueeze(0))
# output is unpooled, a (1, 197, 1024) shaped tensor

output = model.forward_head(output, pre_logits=True)
# output is a (1, num_features) shaped tensor

模型对比

在timm model results 中探索此模型的数据集和运行时度量。

引用

@article{peng2022beit,
  title={Beit v2: Masked image modeling with vector-quantized visual tokenizers},
  author={Peng, Zhiliang and Dong, Li and Bao, Hangbo and Ye, Qixiang and Wei, Furu},
  journal={arXiv preprint arXiv:2208.06366},
  year={2022}
}

@article{dosovitskiy2020vit,
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and  Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  journal={ICLR},
  year={2021}
}

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/huggingface/pytorch-image-models}}
}