vit_srelpos_small_patch16_224.sw_in1k 模型卡片

一种视觉变压器（ViT）图像分类模型。这是ViT架构的一个timm特定变体，具有共享的相对位置嵌入，没有类别令牌，并通过令牌的全局平均池化得到最终表示。在timm中使用下面描述的模板从ImageNet-1k上训练得到。

训练细节：

• 基于Swin Transformer训练/预训练的方法进行修改（涉及到DeiT和ConvNeXt的方法）
• AdamW优化器，梯度裁剪，EMA权重平均化
• 余弦LR调度与热身

模型细节

• 模型类型：图像分类 / 特征主干
• 模型统计：
  • 参数数量（百万）：22.0
  • GMACs：4.2
  • 激活层数量（百万）：8.5
  • 图像尺寸：224 x 224
• 论文：
  • Swin Transformer V2: 扩大容量和分辨率： https://arxiv.org/abs/2111.09883
  • 一个图像相当于16x16的单词：用于大规模图像识别的变压器： https://arxiv.org/abs/2010.11929v2
• 数据集：ImageNet-1k
• 原始链接： https://github.com/huggingface/pytorch-image-models

模型用法

图像分类

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model('vit_srelpos_small_patch16_224.sw_in1k', pretrained=True)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # unsqueeze single image into batch of 1

top5_probabilities, top5_class_indices = torch.topk(output.softmax(dim=1) * 100, k=5)

图像嵌入

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model(
    'vit_srelpos_small_patch16_224.sw_in1k',
    pretrained=True,
    num_classes=0,  # remove classifier nn.Linear
)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # output is (batch_size, num_features) shaped tensor

# or equivalently (without needing to set num_classes=0)

output = model.forward_features(transforms(img).unsqueeze(0))
# output is unpooled, a (1, 196, 384) shaped tensor

output = model.forward_head(output, pre_logits=True)
# output is a (1, num_features) shaped tensor

模型比较

在timm中探索此模型的数据集和运行时指标： model results 。

引用

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/huggingface/pytorch-image-models}}
}

@inproceedings{liu2021swinv2,
  title={Swin Transformer V2: Scaling Up Capacity and Resolution}, 
  author={Ze Liu and Han Hu and Yutong Lin and Zhuliang Yao and Zhenda Xie and Yixuan Wei and Jia Ning and Yue Cao and Zheng Zhang and Li Dong and Furu Wei and Baining Guo},
  booktitle={International Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2022}
}

@article{dosovitskiy2020vit,
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and  Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  journal={ICLR},
  year={2021}
}