Quiz-10 (2024.12.05) // 범위: 11wk-1 까지

Author

최규빈

Published

December 5, 2024

항목	허용 여부	비고
강의노트 참고	허용	수업 중 제공된 강의노트나 본인이 정리한 자료를 참고 가능
구글 검색	허용	인터넷을 통한 자료 검색 및 정보 확인 가능
생성 모형 사용	허용	인공지능 기반 도구(GPT 등) 사용 가능

import os
os.environ["WANDB_MODE"] = "offline"

import pandas as pd
import numpy as np
import datasets 
import transformers
import torch
import torch.utils
import PIL.Image
import io

/home/cgb3/anaconda3/envs/hf/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

1. `imdb` – 40점

아래의 코드를 실행하여 model, imdb 을 불러오라.

imdb = datasets.load_dataset("guebin/imdb-tiny")
tokenizer = transformers.AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
model = transformers.AutoModelForSequenceClassification.from_pretrained(
    "distilbert/distilbert-base-uncased", num_labels=2
)
def preprocess_function(examples):
    return tokenizer(examples["text"], truncation=True)
imdb = imdb.map(preprocess_function,batched=True)

Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

???에 적절한 collate_fn 을 설계하여 아래의 코드를 완성하라.

trainer = transformers.Trainer(
    model=model,
    data_collator=???,
    train_dataset=imdb['train']
)
trainer.train()

제약사항

DataCollatorWithPadding을 사용하지 말고 수동구현 할 것
batch_maker를 이용한 검증코드를 제시할 것

(풀이1)

trainer = batch_maker = transformers.Trainer(
    model=model,
    data_collator=lambda x: x,
    train_dataset=imdb['train']
)

single_batch = next(iter(batch_maker.get_eval_dataloader(imdb['train'])))
def collate_fn(single_batch):
    collated_data = dict()
    collated_data['input_ids'] = torch.nn.utils.rnn.pad_sequence([torch.tensor(dct['input_ids']) for dct in single_batch]).t()    
    collated_data['attention_mask'] = torch.nn.utils.rnn.pad_sequence([torch.tensor(dct['attention_mask']) for dct in single_batch]).t()
    collated_data['labels'] = torch.tensor([dct['label'] for dct in single_batch])
    return collated_data
model.to("cpu")
model(**collate_fn(single_batch))

SequenceClassifierOutput(loss=tensor(0.6894, grad_fn=<NllLossBackward0>), logits=tensor([[ 0.0484,  0.0165],
        [ 0.0187,  0.0715],
        [ 0.0006,  0.0235],
        [ 0.0460,  0.0185],
        [ 0.0360,  0.0242],
        [ 0.0357,  0.0588],
        [ 0.0615, -0.0133],
        [ 0.0319,  0.0157]], grad_fn=<AddmmBackward0>), hidden_states=None, attentions=None)

trainer.data_collator = collate_fn

trainer.train()

[6/6 00:01, Epoch 3/3]

Step	Training Loss

TrainOutput(global_step=6, training_loss=0.4297721783320109, metrics={'train_runtime': 1.4264, 'train_samples_per_second': 21.033, 'train_steps_per_second': 4.207, 'total_flos': 3663551494080.0, 'train_loss': 0.4297721783320109, 'epoch': 3.0})

(풀이2)

batch_maker = transformers.Trainer(
    model=model,
    data_collator=lambda x: x,
)
_batched_data = batch_maker.get_eval_dataloader(imdb['train'])
batched_data = list(_batched_data)
single_batch = batched_data[-1]
def collate_fn(single_batch):
    collated_data = dict()
    collated_data['input_ids'] = torch.nn.utils.rnn.pad_sequence([torch.tensor(dct['input_ids']) for dct in single_batch]).t()    
    collated_data['attention_mask'] = torch.nn.utils.rnn.pad_sequence([torch.tensor(dct['attention_mask']) for dct in single_batch]).t()
    collated_data['labels'] = torch.tensor([dct['label'] for dct in single_batch])
    return collated_data
model.to("cpu")
model(**collate_fn(single_batch))

SequenceClassifierOutput(loss=tensor(0.2261, grad_fn=<NllLossBackward0>), logits=tensor([[ 0.6386, -0.8341],
        [ 0.6473, -0.6306]], grad_fn=<AddmmBackward0>), hidden_states=None, attentions=None)

trainer = transformers.Trainer(
    model=model,
    data_collator=collate_fn,
    train_dataset=imdb['train']
)
trainer.train()

[6/6 00:01, Epoch 3/3]

Step	Training Loss

TrainOutput(global_step=6, training_loss=0.09632491072018941, metrics={'train_runtime': 1.265, 'train_samples_per_second': 23.716, 'train_steps_per_second': 4.743, 'total_flos': 3663551494080.0, 'train_loss': 0.09632491072018941, 'epoch': 3.0})

2. `Food101` – 60점

(1)번 20점, (2)번 40점

(1) 아래의 코드를 실행하여 food_pandas 를 불러오라.

food_pandas = pd.read_parquet("hf://datasets/guebin/food101-tiny/data/train-00000-of-00001.parquet")

food_pandas를 활용하여 아래와 같은 food를 만들어라.

food

Dataset({
    features: ['image', 'label', 'PILImage'],
    num_rows: 10
})

food[0]['PILImage']

(풀이)

food = datasets.Dataset.from_pandas(food_pandas)
food = food.map(lambda dct: {'PILImage': PIL.Image.open(io.BytesIO(dct['image']['bytes']))})
food

Map: 100%|██████████████████████████████| 10/10 [00:00<00:00, 170.21 examples/s]

Dataset({
    features: ['image', 'label', 'PILImage'],
    num_rows: 10
})

(2) 아래의 코드를 실행하여 model과 image_processor를 불러오라.

model = transformers.AutoModelForImageClassification.from_pretrained(
    "google/vit-base-patch16-224-in21k",
    num_labels=101
)
image_processor = transformers.AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k")

Some weights of ViTForImageClassification were not initialized from the model checkpoint at google/vit-base-patch16-224-in21k and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Fast image processor class <class 'transformers.models.vit.image_processing_vit_fast.ViTImageProcessorFast'> is available for this model. Using slow image processor class. To use the fast image processor class set `use_fast=True`.

image_processor 를 이용하여 (1)에서 만든 food의 이미지를 전처리하고, ???에 적절한 data_collator 혹은 collate_fn 을 설계하여 아래의 코드를 완성하라.

trainer = transformers.Trainer(
    model=model,
    data_collator=???,
    train_dataset=food
)
trainer.train()

제약사항

food에 추가적인 전처리를 하지 말 것. (food.map(), food.with_transform()을 하지 말 것)
image_processor 이외의 방법으로 전처리 하지 말 것. (즉 torchvision.transform의 기능을 이용하지 말 것)
batch_maker를 이용한 검증코드를 제시할 것

(풀이1)

trainer = batch_maker = transformers.Trainer(
    model = model,
    data_collator = lambda x: x,
    train_dataset = food,
)
batch_maker.args.remove_unused_columns = False
single_batch = list(batch_maker.get_eval_dataloader(food))[-1]
def collate_fn(single_batch):
    collated_data = dict() 
    collated_data['pixel_values'] = torch.tensor(np.stack(image_processor([o['PILImage'] for o in single_batch])['pixel_values']))
    collated_data['labels'] = torch.tensor([o['label'] for o in single_batch])
    return collated_data
model.to("cpu")
model(**collate_fn(single_batch))

ImageClassifierOutput(loss=tensor(2.3029, grad_fn=<NllLossBackward0>), logits=tensor([[-1.2726e-01, -3.3700e-01,  1.7147e-02, -1.8227e-01, -2.3199e-01,
         -9.9364e-02,  2.2974e+00, -1.0705e-01,  5.6145e-02, -1.8476e-01,
         -1.4030e-01, -8.0887e-02, -8.3965e-03,  4.8124e-02,  5.5113e-02,
         -1.1146e-01,  7.4657e-02,  2.4175e-02, -8.6416e-02, -2.8180e-01,
         -1.7828e-01, -6.4924e-02, -1.3666e-02,  1.1117e-02, -2.2681e-01,
         -2.2888e-01,  4.7380e-02, -2.8418e-02, -1.4658e-01, -1.5298e-02,
         -1.4828e-01, -9.1727e-02,  1.2207e-01, -1.4936e-01, -2.2425e-02,
         -1.9131e-01, -1.6596e-01,  4.5441e-03,  6.5683e-02, -3.7738e-02,
         -5.9682e-02, -6.4176e-03, -1.8336e-01, -2.0530e-01, -1.4324e-01,
         -1.4007e-01, -1.2768e-01, -1.8246e-01, -8.9799e-02,  1.6769e-04,
         -1.1082e-01, -7.8334e-02, -9.1374e-02, -7.9874e-02, -1.5870e-01,
         -6.0640e-02, -3.6169e-01, -1.9152e-01, -2.2351e-01, -1.0452e-01,
          1.3898e-01, -8.4339e-02, -5.6506e-02, -2.3305e-01,  1.0445e-01,
         -2.2902e-01, -2.7481e-01, -9.5906e-03, -1.4605e-01, -1.6736e-01,
         -3.6123e-03, -2.0731e-01, -7.4135e-02,  1.1395e-02, -1.0359e-01,
          1.1315e-01,  4.8628e-02, -6.9685e-02,  5.7619e-02, -1.8000e-01,
         -6.1588e-02, -2.2751e-01, -5.4042e-02, -1.5877e-01, -9.8506e-02,
         -1.0731e-02,  4.0640e-02, -1.7046e-01,  3.3500e-02, -6.9026e-02,
         -1.0761e-01, -7.0151e-02, -7.2786e-02, -5.1693e-02, -2.5389e-01,
         -3.1437e-02, -9.6297e-02, -1.0820e-01, -1.7233e-01, -8.6750e-02,
         -5.1112e-02],
        [-1.3249e-01, -2.8871e-01, -7.2212e-02, -1.1958e-01, -1.1806e-01,
         -1.5619e-01,  2.3591e+00, -7.0754e-02,  2.3708e-02, -6.9813e-02,
         -7.2404e-02, -2.2769e-01,  7.4557e-02, -4.0510e-02,  1.5176e-02,
         -1.3797e-01,  1.5358e-02,  4.9052e-02, -1.4520e-01, -7.8577e-02,
         -2.4480e-01, -1.4358e-01, -1.1630e-01, -3.3120e-02, -1.1248e-01,
         -6.5139e-02, -3.9346e-02, -1.6220e-01, -1.2314e-02,  4.6165e-02,
         -4.0994e-02, -3.6899e-02,  2.6281e-02, -2.0647e-01,  9.9815e-02,
         -2.4997e-01, -1.6528e-01, -2.6163e-02,  4.0938e-03, -1.4413e-02,
          6.7258e-03, -1.6554e-01, -1.6084e-01, -2.2127e-01,  3.4782e-02,
         -3.1506e-02, -1.5634e-01, -2.0848e-01, -1.1443e-01, -1.2029e-02,
         -1.1736e-01, -5.0776e-02, -6.3749e-02, -6.2940e-02, -1.1540e-02,
          3.6494e-02, -2.2582e-01, -1.8537e-02, -2.0378e-01, -1.8006e-01,
          1.6186e-01, -5.4517e-02, -1.7961e-01, -1.8446e-01, -3.3175e-02,
         -1.1115e-01, -1.4888e-01, -1.3875e-02, -4.4281e-02, -1.7475e-01,
         -6.7390e-03, -2.9468e-01,  2.4270e-02, -9.8443e-03, -1.1411e-01,
          5.8463e-02,  2.6745e-03, -9.5164e-02,  6.0501e-02, -8.8094e-02,
         -3.6848e-02, -2.1409e-01, -2.8960e-02, -2.2914e-01, -1.1546e-01,
         -7.4867e-02, -7.0005e-02, -2.2299e-02,  1.1858e-01, -1.5664e-01,
         -1.3949e-01,  3.8144e-02, -1.2400e-01, -4.6237e-02, -2.4283e-01,
         -8.1280e-02, -4.4803e-02, -3.6446e-02, -1.3107e-01, -3.6944e-02,
          5.9170e-02]], grad_fn=<AddmmBackward0>), hidden_states=None, attentions=None)

trainer.data_collator = collate_fn
trainer.train()

[6/6 00:01, Epoch 3/3]

Step	Training Loss

TrainOutput(global_step=6, training_loss=3.661321004231771, metrics={'train_runtime': 1.5054, 'train_samples_per_second': 19.928, 'train_steps_per_second': 3.986, 'total_flos': 2326822460928000.0, 'train_loss': 3.661321004231771, 'epoch': 3.0})

(풀이2)

batch_maker = transformers.Trainer(
    model = model,
    data_collator = lambda x: x,
    train_dataset = food,
    args = transformers.TrainingArguments(
        output_dir="asdf",
        remove_unused_columns=False
    )
)
single_batch = list(batch_maker.get_eval_dataloader(food))[-1]
def collate_fn(single_batch):
    collated_data = dict() 
    collated_data['pixel_values'] = torch.tensor(np.stack(image_processor([o['PILImage'] for o in single_batch])['pixel_values']))
    collated_data['labels'] = torch.tensor([o['label'] for o in single_batch])
    return collated_data
model.to("cpu")
model(**collate_fn(single_batch))

ImageClassifierOutput(loss=tensor(3.2432, grad_fn=<NllLossBackward0>), logits=tensor([[-1.8510e-01, -1.9999e-01, -9.5537e-02, -9.0041e-02, -2.2867e-01,
         -1.2575e-02,  1.3140e+00, -1.2839e-01, -3.0236e-02, -2.5974e-01,
         -7.4725e-02,  6.1918e-03, -1.6538e-02,  1.6465e-01, -2.6724e-02,
         -2.7049e-02,  4.3340e-02,  8.9233e-02,  9.9222e-02, -1.5289e-01,
         -1.2641e-01,  1.3453e-01, -8.2606e-02, -5.3130e-02, -2.6672e-01,
         -2.0686e-01,  1.1107e-02, -9.7890e-02, -3.2137e-02,  2.3901e-02,
         -1.7500e-01, -8.0506e-02,  1.8472e-01, -1.3049e-01,  2.8494e-02,
         -9.1536e-02, -1.0137e-01,  1.9131e-02,  1.5325e-01, -5.3265e-02,
         -1.0158e-01,  1.5654e-01, -2.4823e-01, -1.2814e-01, -2.1389e-01,
         -5.3225e-02, -4.7214e-02, -3.0080e-01, -1.1903e-01, -5.0342e-02,
         -7.7965e-02,  6.4922e-02,  1.8810e-02, -1.1921e-02, -1.4526e-01,
         -1.2815e-01, -3.7510e-01, -1.7370e-01, -2.6114e-01,  3.4536e-02,
          1.0942e-01,  5.5211e-02, -7.6012e-02, -1.3587e-01,  2.5610e-01,
         -6.7371e-02, -2.7418e-01,  6.2201e-02, -2.2526e-01, -1.4086e-01,
          1.3544e-01, -1.8168e-01, -3.8061e-02,  6.6375e-02,  6.5439e-02,
          5.8745e-02,  6.2100e-02, -6.2408e-02,  1.1016e-02, -1.4570e-01,
         -1.0031e-01, -5.0556e-02,  2.8897e-02, -1.6445e-01,  3.3521e-02,
         -4.1956e-02,  2.4169e-01, -9.8579e-02,  5.9701e-02,  9.3557e-02,
          8.9737e-03, -1.3610e-01,  1.8038e-02, -5.8732e-02, -1.1990e-01,
         -9.3936e-02,  3.9783e-02, -5.5422e-02, -1.5393e-01, -1.3591e-01,
          4.8549e-02],
        [-6.2273e-02, -1.7226e-01, -1.2308e-01, -7.5453e-02, -1.3770e-01,
         -1.1669e-01,  1.4039e+00, -1.5833e-02, -8.2333e-02, -2.4664e-02,
          5.7223e-03, -2.5468e-01,  1.6655e-01,  6.2704e-02,  5.9537e-02,
         -1.1380e-01, -8.5781e-03,  1.6243e-02, -8.5453e-02,  5.9695e-03,
         -2.5368e-01, -8.4907e-02, -1.6856e-01, -2.9378e-02, -1.2118e-01,
          9.7930e-02, -1.4896e-01, -6.1749e-02,  5.5475e-02, -5.5201e-02,
          5.0549e-02, -3.4004e-02,  1.0318e-01, -1.4305e-01,  1.7079e-01,
         -2.0896e-01, -1.1980e-01, -1.0171e-02,  8.3601e-02,  1.8135e-02,
          8.0280e-02, -4.6555e-02, -1.8812e-01, -2.7424e-01,  4.4743e-02,
         -3.5335e-02, -9.2994e-02, -3.1070e-01, -2.1141e-01, -7.7412e-02,
         -7.0491e-02,  1.3877e-03,  6.9164e-02, -4.1210e-02,  1.2504e-03,
         -9.4582e-02, -1.6719e-01, -7.4593e-02, -1.9046e-01, -6.6895e-02,
          1.4930e-01, -6.8632e-03, -1.2952e-01, -7.0757e-02, -8.3189e-02,
         -3.2319e-02,  2.5300e-02,  4.6809e-02, -5.2601e-04, -1.1607e-02,
         -1.4187e-01, -1.4581e-01, -2.7281e-02,  4.8160e-02, -8.8708e-02,
          1.2457e-01,  2.5711e-02, -7.2490e-02, -2.3467e-02, -2.6480e-02,
          1.9758e-02, -1.4388e-01, -3.6046e-02, -1.9946e-01, -9.2800e-03,
         -1.5308e-01,  1.0609e-01,  1.5982e-02,  1.0204e-01, -7.5755e-02,
         -3.0089e-02,  7.9821e-02, -4.8105e-04,  6.3365e-03, -1.4550e-01,
          2.7974e-02,  8.6050e-02, -2.4519e-02, -1.2191e-01, -2.8178e-02,
          7.5331e-02]], grad_fn=<AddmmBackward0>), hidden_states=None, attentions=None)

trainer = transformers.Trainer(
    model = model,
    data_collator = collate_fn,
    train_dataset = food,
    args = transformers.TrainingArguments(
        output_dir="asdf",
        remove_unused_columns=False
    )
)
trainer.train()

[6/6 00:01, Epoch 3/3]

Step	Training Loss

TrainOutput(global_step=6, training_loss=2.512084643046061, metrics={'train_runtime': 1.3329, 'train_samples_per_second': 22.507, 'train_steps_per_second': 4.501, 'total_flos': 2326822460928000.0, 'train_loss': 2.512084643046061, 'epoch': 3.0})

1. imdb – 40점

2. Food101 – 60점

1. `imdb` – 40점

2. `Food101` – 60점