vit_medium_patch16_gap_240.sw_in12k 模型卡片

一种视觉转换器（ViT）图像分类模型。这是一个特定于 timm 结构的变体，使用标记的全局平均汇聚。由 Ross Wightman 在 timm 中使用下面描述的建模模板在 ImageNet-12k 上进行训练。

训练细节:

• 基于 Swin Transformer 的训练 / 预训练流程，并进行了修改（与 DeiT 和 ConvNeXt 流程相关）
• 使用 AdamW 优化器，梯度裁剪，EMA 权重平均
• 使用余弦 LR 计划和预热

模型细节

• 模型类型: 图像分类 / 特征骨干网络
• 模型统计:
  • 参数数量（M）: 44.4
  • GMACs: 8.6
  • 激活数（M）: 12.6
  • 图像尺寸: 240 x 240
• 论文:
  • 一张图像等于 16x16 个单词：大规模图像识别的 transformer 模型: https://arxiv.org/abs/2010.11929v2
• 数据集: ImageNet-12k
• 原始引用: https://github.com/huggingface/pytorch-image-models

模型用法

图像分类

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model('vit_medium_patch16_gap_240.sw_in12k', pretrained=True)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # unsqueeze single image into batch of 1

top5_probabilities, top5_class_indices = torch.topk(output.softmax(dim=1) * 100, k=5)

图像嵌入

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model(
    'vit_medium_patch16_gap_240.sw_in12k',
    pretrained=True,
    num_classes=0,  # remove classifier nn.Linear
)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # output is (batch_size, num_features) shaped tensor

# or equivalently (without needing to set num_classes=0)

output = model.forward_features(transforms(img).unsqueeze(0))
# output is unpooled, a (1, 225, 512) shaped tensor

output = model.forward_head(output, pre_logits=True)
# output is a (1, num_features) shaped tensor

模型比较

在 timm 中探索此模型的数据集和运行时指标: model results .

引用

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/huggingface/pytorch-image-models}}
}

@article{dosovitskiy2020vit,
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and  Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  journal={ICLR},
  year={2021}
}