vit_relpos_base_patch16_clsgap_224.sw_in1k 模型卡片

Vision Transformer (ViT) 图像分类模型。这是 timm 中 ViT 架构的特定变体，采用了相对位置嵌入、类别标记和通过全局平均池化来得到最终表示的方法。在 timm 中，使用下述模板的 ImageNet-1k 数据进行训练。

推导细节：

• 基于 Swin Transformer 训练/预训练方法，进行了修改（与 DeiT 和 ConvNeXt 方法相关）
• 使用 AdamW 优化器、梯度裁剪、EMA 权重平均
• 采用余弦学习率表和预热

模型细节

• 模型类型：图像分类/特征骨干
• 模型统计信息：
  • 参数（百万）：86.4
  • GMACs：16.9
  • 激活函数（百万）：17.7
  • 图像尺寸：224 x 224
• 论文：
  • Swin Transformer V2：扩展容量和分辨率： https://arxiv.org/abs/2111.09883
  • 一张图片 equals 16x16 词：规模化图像识别的 Transformer： https://arxiv.org/abs/2010.11929v2
• 数据集：ImageNet-1k
• 原始链接： https://github.com/huggingface/pytorch-image-models

模型用途

图像分类

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model('vit_relpos_base_patch16_clsgap_224.sw_in1k', pretrained=True)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # unsqueeze single image into batch of 1

top5_probabilities, top5_class_indices = torch.topk(output.softmax(dim=1) * 100, k=5)

图像嵌入

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model(
    'vit_relpos_base_patch16_clsgap_224.sw_in1k',
    pretrained=True,
    num_classes=0,  # remove classifier nn.Linear
)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # output is (batch_size, num_features) shaped tensor

# or equivalently (without needing to set num_classes=0)

output = model.forward_features(transforms(img).unsqueeze(0))
# output is unpooled, a (1, 197, 768) shaped tensor

output = model.forward_head(output, pre_logits=True)
# output is a (1, num_features) shaped tensor

模型比较

在 timm 中探索该模型的数据集和运行时指标： model results

引用

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/huggingface/pytorch-image-models}}
}

@inproceedings{liu2021swinv2,
  title={Swin Transformer V2: Scaling Up Capacity and Resolution}, 
  author={Ze Liu and Han Hu and Yutong Lin and Zhuliang Yao and Zhenda Xie and Yixuan Wei and Jia Ning and Yue Cao and Zheng Zhang and Li Dong and Furu Wei and Baining Guo},
  booktitle={International Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2022}
}

@article{dosovitskiy2020vit,
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and  Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  journal={ICLR},
  year={2021}
}