vit_medium_patch16_gap_384.sw_in12k_ft_in1k模型卡

一种用于图像分类的视觉变换器（ViT）模型。这是Ross Wightman在timm中使用的一种特定变体，具有令牌全局平均池化。在ImageNet-12k上进行了预训练，并在ImageNet-1k上进行了微调。

配方详情：

• 基于Swin Transformer的训练/预训练配方，进行了修改（与DeiT和ConvNeXt配方相关）
• 使用AdamW优化器、梯度裁剪、EMA权重平均化
• 采用余弦学习率调度和预热

模型详情

• 模型类型：图像分类/特征骨干
• 模型统计数据：
  • 参数数量（百万）：39.0
  • GMACs：22.0
  • 激活数量（百万）：32.1
  • 图像尺寸：384 x 384
• 论文：
  • 一幅图像相当于16x16个单词：用于图像识别的变换器规模： https://arxiv.org/abs/2010.11929v2
• 数据集：ImageNet-1k
• 预训练数据集：ImageNet-12k
• 原始论文： https://github.com/huggingface/pytorch-image-models

模型用途

图像分类

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model('vit_medium_patch16_gap_384.sw_in12k_ft_in1k', pretrained=True)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # unsqueeze single image into batch of 1

top5_probabilities, top5_class_indices = torch.topk(output.softmax(dim=1) * 100, k=5)

图像嵌入

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model(
    'vit_medium_patch16_gap_384.sw_in12k_ft_in1k',
    pretrained=True,
    num_classes=0,  # remove classifier nn.Linear
)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # output is (batch_size, num_features) shaped tensor

# or equivalently (without needing to set num_classes=0)

output = model.forward_features(transforms(img).unsqueeze(0))
# output is unpooled, a (1, 576, 512) shaped tensor

output = model.forward_head(output, pre_logits=True)
# output is a (1, num_features) shaped tensor

模型比较

在timm model results 中探索此模型的数据集和运行时指标。

引用

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/huggingface/pytorch-image-models}}
}

@article{dosovitskiy2020vit,
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and  Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  journal={ICLR},
  year={2021}
}