- 데이터 분석을 하고 싶음.
- 어려울 것이라 예상되는 점.
(1) 안 돌아감. import 부터 잘 안될 것임.(2) 이해가 안될걸??(3) 이게 진짜 어려움. 이걸 잘하기 위해서는 파이썬의 기본 문법이 강해야함.(4) 돌아는 가지만 결과가 좋지 않을 것임.- 소망: 아래가 되었으면..
- 마음가짐: 컨셉을 잘 잡아야함..
느리지만 정확하게 해결하는것은 옳지 않다. 모든 분석은 시간싸움임. 느려지는 순간 이미 뒤쳐진다. (마라톤을 하는게 아님. 100m 달리기임)
- 저는 컴공과 교수가 아니에요.. (그럴만한 실력도 없어요) 그냥 먹고살기 위해서 눈치껏 코딩을 할 뿐입니다..
- 좋은 공부재료 (데이터+분석방법) 를 잘 선정해야함. 이게 사실 어려워요..
- ref: https://huggingface.co/docs/transformers/tasks/sequence_classification
- 허깅페이스
- Before you begin, make sure you have all the necessary libraries installed:
pip install transformers datasets evaluate accelerate- 저게 뭐하는 코드냐??
질문:
pip install transformers datasets evaluate accelerate는 뭐하는 코드야??
질문: 코랩에서 저 명령어를 실행하면 어디엣 설치되는거야? 내 윈도우 노트북에 설치되어 있는거야??
질문: 그러면 코랩 끄면 다시 설치해야해?
from datasets import load_dataset/home/cgb3/anaconda3/envs/hf/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdmfrom datasets import load_dataset 의 의미는 dataset 모듈안에 있는 여러함수들 중에서 load_dataset 이라는 함수만 불러온다는 의미이다.import dataset 을 한뒤 dataset.load_dataset 와 같은 형식으로 사용해도 무방하지만, 메모리 관리면에서 필요한 함수만 콕 찝어서 불러오는 from datasets import load_dataset 이 더 유리하다.load_dataset 함수의 사용방법복습: https://guebin.github.io/PP2024/posts/14wk-2.html
- load_dataset 이 뭐지?
type(load_dataset)functionset(dir(load_dataset)) & {'__call__'}{'__call__'}load_dataset(???) 와 같은 방식으로 쓰면 되겠군..- load_dataset은 어떻게 쓰지??
load_dataset?Signature:
load_dataset(
path: str,
name: Optional[str] = None,
data_dir: Optional[str] = None,
data_files: Union[str, Sequence[str], Mapping[str, Union[str, Sequence[str]]], NoneType] = None,
split: Union[str, datasets.splits.Split, NoneType] = None,
cache_dir: Optional[str] = None,
features: Optional[datasets.features.features.Features] = None,
download_config: Optional[datasets.download.download_config.DownloadConfig] = None,
download_mode: Union[datasets.download.download_manager.DownloadMode, str, NoneType] = None,
verification_mode: Union[datasets.utils.info_utils.VerificationMode, str, NoneType] = None,
ignore_verifications='deprecated',
keep_in_memory: Optional[bool] = None,
save_infos: bool = False,
revision: Union[str, datasets.utils.version.Version, NoneType] = None,
token: Union[bool, str, NoneType] = None,
use_auth_token='deprecated',
task='deprecated',
streaming: bool = False,
num_proc: Optional[int] = None,
storage_options: Optional[Dict] = None,
trust_remote_code: bool = None,
**config_kwargs,
) -> Union[datasets.dataset_dict.DatasetDict, datasets.arrow_dataset.Dataset, datasets.dataset_dict.IterableDatasetDict, datasets.iterable_dataset.IterableDataset]
...load_dataset() 와 같은 형식으로는 쓸 수 없겠음. –> 확인해보니까 진짜 쓸 수 없음.load_dataset('asdf') 와 같은 방식으로 하나의 위치인자를 문자열로 전달해야 겠네.. (나머지는 알아서 디폴트값으로 전달되겠지..)path라는 변수에 저장되겠지??path에 뭘 써야하지?? –> 생각1: load_dataset은 아마 데이터를 불러오는 함수일텐데, 이 데이터가 저장되는 경로를 path에 명시하라는거 아닌가? // 생각2: 아니면 불러올 데이터가 저장된 path인가? –> GPT문의imdb = load_dataset("imdb")IMDB 탐색imdb- 이제 imdb가 뭔지 살펴보자..
- 뜯어보자..
imdb # 데이터인듯DatasetDict({
train: Dataset({
features: ['text', 'label'],
num_rows: 25000
})
test: Dataset({
features: ['text', 'label'],
num_rows: 25000
})
unsupervised: Dataset({
features: ['text', 'label'],
num_rows: 50000
})
})- 딕셔너리는 아님
type(imdb)datasets.dataset_dict.DatasetDict- 그런데 거의 딕셔너리 비슷한 느낌으로 일단 사용되는것 같음.
imdb.keys()dict_keys(['train', 'test', 'unsupervised'])imdb['train']Dataset({
features: ['text', 'label'],
num_rows: 25000
})imdb['test']Dataset({
features: ['text', 'label'],
num_rows: 25000
})imdb['unsupervised']Dataset({
features: ['text', 'label'],
num_rows: 50000
})- imdb는 딕셔너리 같은 것이고, imdb['train']와 같은 명령어로 세부항목에 접근가능함. 즉 아래의 구조임.
imdb['train'] \(\subset\) imdbimdb['test'] \(\subset\) imdbimdb['unsupervised'] \(\subset\) imdbimdb['train']- imdb['train']을 살펴보자..
imdb['train']Dataset({
features: ['text', 'label'],
num_rows: 25000
})- 이건 딕셔너리 처럼 안되는 것 같네..
imdb['train'].keys()AttributeError: 'Dataset' object has no attribute 'keys'imdb['train']['features']KeyError: "Column features not in the dataset. Current columns in the dataset: ['text', 'label']"- 그럼 imdb['train']는 어떻게 쓰라는 것이냐?? \(\to\) 쓸만한 기능이 있을지 dir로 체크 \(\to\) __getitem__이 있음.. \(\to\) imdb['train'][0] 를 써볼 용기가 생김..
set(dir(imdb['train'])) & {'__getitem__'}{'__getitem__'}imdb['train'][0]{'text': 'I rented I AM CURIOUS-YELLOW from my video store because of all the controversy that surrounded it when it was first released in 1967. I also heard that at first it was seized by U.S. customs if it ever tried to enter this country, therefore being a fan of films considered "controversial" I really had to see this for myself.<br /><br />The plot is centered around a young Swedish drama student named Lena who wants to learn everything she can about life. In particular she wants to focus her attentions to making some sort of documentary on what the average Swede thought about certain political issues such as the Vietnam War and race issues in the United States. In between asking politicians and ordinary denizens of Stockholm about their opinions on politics, she has sex with her drama teacher, classmates, and married men.<br /><br />What kills me about I AM CURIOUS-YELLOW is that 40 years ago, this was considered pornographic. Really, the sex and nudity scenes are few and far between, even then it\'s not shot like some cheaply made porno. While my countrymen mind find it shocking, in reality sex and nudity are a major staple in Swedish cinema. Even Ingmar Bergman, arguably their answer to good old boy John Ford, had sex scenes in his films.<br /><br />I do commend the filmmakers for the fact that any sex shown in the film is shown for artistic purposes rather than just to shock people and make money to be shown in pornographic theaters in America. I AM CURIOUS-YELLOW is a good film for anyone wanting to study the meat and potatoes (no pun intended) of Swedish cinema. But really, this film doesn\'t have much of a plot.',
'label': 0}- imdb['train'][1], imdb['train'][-1], imdb['train'][:2] 등을 실행해보자..
imdb['train'][:5]{'text': ['I rented I AM CURIOUS-YELLOW from my video store because of all the controversy that surrounded it when it was first released in 1967. I also heard that at first it was seized by U.S. customs if it ever tried to enter this country, therefore being a fan of films considered "controversial" I really had to see this for myself.<br /><br />The plot is centered around a young Swedish drama student named Lena who wants to learn everything she can about life. In particular she wants to focus her attentions to making some sort of documentary on what the average Swede thought about certain political issues such as the Vietnam War and race issues in the United States. In between asking politicians and ordinary denizens of Stockholm about their opinions on politics, she has sex with her drama teacher, classmates, and married men.<br /><br />What kills me about I AM CURIOUS-YELLOW is that 40 years ago, this was considered pornographic. Really, the sex and nudity scenes are few and far between, even then it\'s not shot like some cheaply made porno. While my countrymen mind find it shocking, in reality sex and nudity are a major staple in Swedish cinema. Even Ingmar Bergman, arguably their answer to good old boy John Ford, had sex scenes in his films.<br /><br />I do commend the filmmakers for the fact that any sex shown in the film is shown for artistic purposes rather than just to shock people and make money to be shown in pornographic theaters in America. I AM CURIOUS-YELLOW is a good film for anyone wanting to study the meat and potatoes (no pun intended) of Swedish cinema. But really, this film doesn\'t have much of a plot.',
'"I Am Curious: Yellow" is a risible and pretentious steaming pile. It doesn\'t matter what one\'s political views are because this film can hardly be taken seriously on any level. As for the claim that frontal male nudity is an automatic NC-17, that isn\'t true. I\'ve seen R-rated films with male nudity. Granted, they only offer some fleeting views, but where are the R-rated films with gaping vulvas and flapping labia? Nowhere, because they don\'t exist. The same goes for those crappy cable shows: schlongs swinging in the breeze but not a clitoris in sight. And those pretentious indie movies like The Brown Bunny, in which we\'re treated to the site of Vincent Gallo\'s throbbing johnson, but not a trace of pink visible on Chloe Sevigny. Before crying (or implying) "double-standard" in matters of nudity, the mentally obtuse should take into account one unavoidably obvious anatomical difference between men and women: there are no genitals on display when actresses appears nude, and the same cannot be said for a man. In fact, you generally won\'t see female genitals in an American film in anything short of porn or explicit erotica. This alleged double-standard is less a double standard than an admittedly depressing ability to come to terms culturally with the insides of women\'s bodies.',
"If only to avoid making this type of film in the future. This film is interesting as an experiment but tells no cogent story.<br /><br />One might feel virtuous for sitting thru it because it touches on so many IMPORTANT issues but it does so without any discernable motive. The viewer comes away with no new perspectives (unless one comes up with one while one's mind wanders, as it will invariably do during this pointless film).<br /><br />One might better spend one's time staring out a window at a tree growing.<br /><br />",
"This film was probably inspired by Godard's Masculin, féminin and I urge you to see that film instead.<br /><br />The film has two strong elements and those are, (1) the realistic acting (2) the impressive, undeservedly good, photo. Apart from that, what strikes me most is the endless stream of silliness. Lena Nyman has to be most annoying actress in the world. She acts so stupid and with all the nudity in this film,...it's unattractive. Comparing to Godard's film, intellectuality has been replaced with stupidity. Without going too far on this subject, I would say that follows from the difference in ideals between the French and the Swedish society.<br /><br />A movie of its time, and place. 2/10.",
'Oh, brother...after hearing about this ridiculous film for umpteen years all I can think of is that old Peggy Lee song..<br /><br />"Is that all there is??" ...I was just an early teen when this smoked fish hit the U.S. I was too young to get in the theater (although I did manage to sneak into "Goodbye Columbus"). Then a screening at a local film museum beckoned - Finally I could see this film, except now I was as old as my parents were when they schlepped to see it!!<br /><br />The ONLY reason this film was not condemned to the anonymous sands of time was because of the obscenity case sparked by its U.S. release. MILLIONS of people flocked to this stinker, thinking they were going to see a sex film...Instead, they got lots of closeups of gnarly, repulsive Swedes, on-street interviews in bland shopping malls, asinie political pretension...and feeble who-cares simulated sex scenes with saggy, pale actors.<br /><br />Cultural icon, holy grail, historic artifact..whatever this thing was, shred it, burn it, then stuff the ashes in a lead box!<br /><br />Elite esthetes still scrape to find value in its boring pseudo revolutionary political spewings..But if it weren\'t for the censorship scandal, it would have been ignored, then forgotten.<br /><br />Instead, the "I Am Blank, Blank" rhythymed title was repeated endlessly for years as a titilation for porno films (I am Curious, Lavender - for gay films, I Am Curious, Black - for blaxploitation films, etc..) and every ten years or so the thing rises from the dead, to be viewed by a new generation of suckers who want to see that "naughty sex film" that "revolutionized the film industry"...<br /><br />Yeesh, avoid like the plague..Or if you MUST see it - rent the video and fast forward to the "dirty" parts, just to get it over with.<br /><br />'],
'label': [0, 0, 0, 0, 0]}imdb['train'][-1]{'text': 'The story centers around Barry McKenzie who must go to England if he wishes to claim his inheritance. Being about the grossest Aussie shearer ever to set foot outside this great Nation of ours there is something of a culture clash and much fun and games ensue. The songs of Barry McKenzie(Barry Crocker) are highlights.',
'label': 1}- imdb['train'][0] 을 구체적으로 살펴보자.
print(imdb['train'][0]['text'])
print(imdb['train'][0]['label'])I rented I AM CURIOUS-YELLOW from my video store because of all the controversy that surrounded it when it was first released in 1967. I also heard that at first it was seized by U.S. customs if it ever tried to enter this country, therefore being a fan of films considered "controversial" I really had to see this for myself.<br /><br />The plot is centered around a young Swedish drama student named Lena who wants to learn everything she can about life. In particular she wants to focus her attentions to making some sort of documentary on what the average Swede thought about certain political issues such as the Vietnam War and race issues in the United States. In between asking politicians and ordinary denizens of Stockholm about their opinions on politics, she has sex with her drama teacher, classmates, and married men.<br /><br />What kills me about I AM CURIOUS-YELLOW is that 40 years ago, this was considered pornographic. Really, the sex and nudity scenes are few and far between, even then it's not shot like some cheaply made porno. While my countrymen mind find it shocking, in reality sex and nudity are a major staple in Swedish cinema. Even Ingmar Bergman, arguably their answer to good old boy John Ford, had sex scenes in his films.<br /><br />I do commend the filmmakers for the fact that any sex shown in the film is shown for artistic purposes rather than just to shock people and make money to be shown in pornographic theaters in America. I AM CURIOUS-YELLOW is a good film for anyone wanting to study the meat and potatoes (no pun intended) of Swedish cinema. But really, this film doesn't have much of a plot.
0- imdb['train'][1] 을 구체적으로 살펴보자.
print(imdb['train'][1]['text'])
print(imdb['train'][1]['label'])"I Am Curious: Yellow" is a risible and pretentious steaming pile. It doesn't matter what one's political views are because this film can hardly be taken seriously on any level. As for the claim that frontal male nudity is an automatic NC-17, that isn't true. I've seen R-rated films with male nudity. Granted, they only offer some fleeting views, but where are the R-rated films with gaping vulvas and flapping labia? Nowhere, because they don't exist. The same goes for those crappy cable shows: schlongs swinging in the breeze but not a clitoris in sight. And those pretentious indie movies like The Brown Bunny, in which we're treated to the site of Vincent Gallo's throbbing johnson, but not a trace of pink visible on Chloe Sevigny. Before crying (or implying) "double-standard" in matters of nudity, the mentally obtuse should take into account one unavoidably obvious anatomical difference between men and women: there are no genitals on display when actresses appears nude, and the same cannot be said for a man. In fact, you generally won't see female genitals in an American film in anything short of porn or explicit erotica. This alleged double-standard is less a double standard than an admittedly depressing ability to come to terms culturally with the insides of women's bodies.
0- imdb['train'][100] 을 구체적으로 살펴보자.
print(imdb['train'][100]['text'])
print(imdb['train'][100]['label'])Terrible movie. Nuff Said.<br /><br />These Lines are Just Filler. The movie was bad. Why I have to expand on that I don't know. This is already a waste of my time. I just wanted to warn others. Avoid this movie. The acting sucks and the writing is just moronic. Bad in every way. The only nice thing about the movie are Deniz Akkaya's breasts. Even that was ruined though by a terrible and unneeded rape scene. The movie is a poorly contrived and totally unbelievable piece of garbage.<br /><br />OK now I am just going to rag on IMDb for this stupid rule of 10 lines of text minimum. First I waste my time watching this offal. Then feeling compelled to warn others I create an account with IMDb only to discover that I have to write a friggen essay on the film just to express how bad I think it is. Totally unnecessary.
0영어: Terrible movie. Nuff Said.
These Lines are Just Filler. The movie was bad. Why I have to expand on that I don’t know. This is already a waste of my time. I just wanted to warn others. Avoid this movie. The acting sucks and the writing is just moronic. Bad in every way. The only nice thing about the movie are Deniz Akkaya’s breasts. Even that was ruined though by a terrible and unneeded rape scene. The movie is a poorly contrived and totally unbelievable piece of garbage.
OK now I am just going to rag on IMDb for this stupid rule of 10 lines of text minimum. First I waste my time watching this offal. Then feeling compelled to warn others I create an account with IMDb only to discover that I have to write a friggen essay on the film just to express how bad I think it is. Totally unnecessary.
한글: 끔찍한 영화. 더 할 말 없음. 이 줄들은 그냥 채우기일 뿐이다. 영화가 나빴다. 내가 왜 이걸 더 길게 써야 하는지 모르겠다. 이건 이미 내 시간 낭비다. 나는 그저 다른 사람들에게 경고하고 싶었을 뿐이다. 이 영화를 피하라. 연기도 형편없고, 대본도 완전히 멍청하다. 모든 면에서 나쁜 영화다. 영화에서 유일하게 좋은 것은 데니즈 아카야의 가슴뿐이었다. 하지만 그것조차도 끔찍하고 불필요한 강간 장면 때문에 망쳤다. 이 영화는 조잡하게 만들어졌고, 전혀 믿을 수 없는 쓰레기다. 이제 IMDb에 대한 불만을 좀 말하겠다. 10줄 이상 써야 한다는 이 어리석은 규칙 때문에 말이다. 먼저 이 쓰레기를 보면서 내 시간을 낭비했다. 그리고 다른 사람들에게 경고하려고 IMDb 계정을 만들었더니, 영화에 대해 내가 얼마나 나쁘게 생각하는지를 표현하려면 이 따위 에세이를 써야 한다는 걸 알게 되었다. 완전히 불필요하다.
- imdb['train'][-1] 을 구체적으로 살펴보자.
print(imdb['train'][-1]['text'])
print(imdb['train'][-1]['label'])The story centers around Barry McKenzie who must go to England if he wishes to claim his inheritance. Being about the grossest Aussie shearer ever to set foot outside this great Nation of ours there is something of a culture clash and much fun and games ensue. The songs of Barry McKenzie(Barry Crocker) are highlights.
1영어: The story centers around Barry McKenzie who must go to England if he wishes to claim his inheritance. Being about the grossest Aussie shearer ever to set foot outside this great Nation of ours there is something of a culture clash and much fun and games ensue. The songs of Barry McKenzie(Barry Crocker) are highlights.
한글: 이 이야기는 유산을 상속받기 위해 영국으로 가야 하는 배리 맥켄지에 초점을 맞추고 있습니다. 배리 맥켄지는 이 위대한 호주를 떠난 사람 중 가장 거친 호주 농부로, 문화적 충돌이 일어나고 다양한 재미와 사건들이 이어집니다. 배리 맥켄지(배리 크로커)가 부르는 노래들이 이 영화의 하이라이트입니다.
- 요약: imdb['train'] 에는 여러개의 영화평이 있고, 각각 긍정평가와 부정평가를 담고 있음.
len(imdb['train'])25000imdb['train'][24999]{'text': 'The story centers around Barry McKenzie who must go to England if he wishes to claim his inheritance. Being about the grossest Aussie shearer ever to set foot outside this great Nation of ours there is something of a culture clash and much fun and games ensue. The songs of Barry McKenzie(Barry Crocker) are highlights.',
'label': 1}imdb['train'][-1]{'text': 'The story centers around Barry McKenzie who must go to England if he wishes to claim his inheritance. Being about the grossest Aussie shearer ever to set foot outside this great Nation of ours there is something of a culture clash and much fun and games ensue. The songs of Barry McKenzie(Barry Crocker) are highlights.',
'label': 1}imdb['test']- imdb['train'] 과 비슷함..
imdb['test'][0]{'text': 'I love sci-fi and am willing to put up with a lot. Sci-fi movies/TV are usually underfunded, under-appreciated and misunderstood. I tried to like this, I really did, but it is to good TV sci-fi as Babylon 5 is to Star Trek (the original). Silly prosthetics, cheap cardboard sets, stilted dialogues, CG that doesn\'t match the background, and painfully one-dimensional characters cannot be overcome with a \'sci-fi\' setting. (I\'m sure there are those of you out there who think Babylon 5 is good sci-fi TV. It\'s not. It\'s clichéd and uninspiring.) While US viewers might like emotion and character development, sci-fi is a genre that does not take itself seriously (cf. Star Trek). It may treat important issues, yet not as a serious philosophy. It\'s really difficult to care about the characters here as they are not simply foolish, just missing a spark of life. Their actions and reactions are wooden and predictable, often painful to watch. The makers of Earth KNOW it\'s rubbish as they have to always say "Gene Roddenberry\'s Earth..." otherwise people would not continue watching. Roddenberry\'s ashes must be turning in their orbit as this dull, cheap, poorly edited (watching it without advert breaks really brings this home) trudging Trabant of a show lumbers into space. Spoiler. So, kill off a main character. And then bring him back as another actor. Jeeez! Dallas all over again.',
'label': 0}len(imdb['test'])25000imdb['unsupervised']- imdb['unsupervised'] 를 살펴보자.
imdb['unsupervised'][3]{'text': "Being that the only foreign films I usually like star a Japanese person in a rubber suit who crushes little tiny buildings and tanks, I had high hopes for this movie. I thought that this was a movie that wouldn't put me to sleep. WRONG! Starts off with a bang, okay, now she's in training, alright, she's an assassin, I'm still with you, oh, now she's having this moral dilemma and she can't decide if she loves her boyfriend or her controller, zzzzz.... Oh well, back to Gamera!",
'label': -1}imdb['train'], imdb['test'] 와 비슷해 보이지만 살짝 다름.- 혹시 imdb['unsupervised'][??] 의 모든 라벨값이 모두 -1인가?
set([l['label'] for l in imdb['unsupervised']]){-1}set([l['label'] for l in imdb['train']]){0, 1}set([l['label'] for l in imdb['test']]){0, 1}imdb['train'],Dataset({
features: ['text', 'label'],
num_rows: 25000
})imdb['test']Dataset({
features: ['text', 'label'],
num_rows: 25000
})- 느낌상 imdb['unsupervised']는 imdb['train'] 와 imdb['test'] 에서 text를 합치고, label은 모두 -1 로 바꾼 자료가 아닐까? 하는 의심이 들었음. —> 아니었음..
txt0 = imdb['unsupervised'][0]['text']
txt0'This is just a precious little diamond. The play, the script are excellent. I cant compare this movie with anything else, maybe except the movie "Leon" wonderfully played by Jean Reno and Natalie Portman. But... What can I say about this one? This is the best movie Anne Parillaud has ever played in (See please "Frankie Starlight", she\'s speaking English there) to see what I mean. The story of young punk girl Nikita, taken into the depraved world of the secret government forces has been exceptionally over used by Americans. Never mind the "Point of no return" and especially the "La femme Nikita" TV series. They cannot compare the original believe me! Trash these videos. Buy this one, do not rent it, BUY it. BTW beware of the subtitles of the LA company which "translate" the US release. What a disgrace! If you cant understand French, get a dubbed version. But you\'ll regret later :)'[l for l in imdb['train'] if l['text'] == txt0][][l for l in imdb['test'] if l['text'] == txt0][][l for l in imdb['unsupervised'] if l['text'] == txt0][{'text': 'This is just a precious little diamond. The play, the script are excellent. I cant compare this movie with anything else, maybe except the movie "Leon" wonderfully played by Jean Reno and Natalie Portman. But... What can I say about this one? This is the best movie Anne Parillaud has ever played in (See please "Frankie Starlight", she\'s speaking English there) to see what I mean. The story of young punk girl Nikita, taken into the depraved world of the secret government forces has been exceptionally over used by Americans. Never mind the "Point of no return" and especially the "La femme Nikita" TV series. They cannot compare the original believe me! Trash these videos. Buy this one, do not rent it, BUY it. BTW beware of the subtitles of the LA company which "translate" the US release. What a disgrace! If you cant understand French, get a dubbed version. But you\'ll regret later :)',
'label': -1}]- 요약:
imdb는 imdb['train'], imdb['test'], imdb['unsupervised'] 로 나누어져 있음.imdb['train'], imdb['test'] 에는 각각 (text,label)와 같은 형식으로 정보가 저장되어 있음. 여기에서 label은 0 혹은 1의 값을 가지는데 0은 부정, 1은 긍정을 의미함.imdb['unsupervised'] 에도 조사해보니 각각 (text,label)와 같은 형식으로 정보가 저장되어 있었지만, 여기에서 label값은 모두 -1의 값만 있었음. 따라서 사살상 imdb['unsupervised']는 text에 대한 정보만 있다고 생각해도 무방. 그 text가 영화에 대한 긍정평가인지 부정평가인지 분류가 되어있지 않은 상태.- 외우세요: train, test, unsupervised 와 같은 단어는 매우 중요한 단어니까 일단 눈여겨보세요
- 왜 데이터가 imdb['train'], imdb['test'], imdb['unsupervised'] 와 같이 나누어져 있는가?
imdb['train'], imdb['test'] 는 지도학습 모델을 배우기 위한 예제데이터이고, imdb['unsupervised']는 비지도학습모델을 배우기 위한 예제데이터임.imdb['train'], imdb['test'] 에만 관심을 가질 예정임.- 지도학습이란? (이 예제에 한정하여 설명)
f를 찾는 일f라는 함수가 존재하여 아래와 같은 동작이 가능하도록 해야함.f("영화가 너무 재미없어요")
> "부정평가입니다"
f("영화, 괜찮은데요??")
> "긍정평가입니다"
f("배우들 연기 진짜 잘함. 영상미도 있음. 그런데 스토리 때문에 망했네.")
> "부정평가입니다"- 이러한 함수 f를 우리가 잘 정의한다면 좋겠음. (가능한가??)
- 대충 아래와 같은 과정을 거친다고 생각하면 편리함.
정보(숫자,텍스트,이미지,…) \(\to\) 숫자 \(\to\) 계산 \(\to\) 계산된숫자 \(\to\) 정보
- 예를들면 아래와 같은 방식이 가능
긍정단어 = {'좋아', '재미있었음', '잘생김', '예뻐', '연기훌륭함'}
부정단어 = {'지루해', '재미없었음', '비추천'}txt0 = "남주가 너무 잘생김 여주도 예뻐 영화가 중간에 조금 지루해 그래도 아무튼 재미있었음"긍정단어수 = sum([l in 긍정단어 for l in txt0.split(' ')])
부정단어수 = sum([l in 부정단어 for l in txt0.split(' ')])
긍정평가일확률 = 긍정단어수/(긍정단어수+부정단어수)
부정평가일확률 = 부정단어수/(긍정단어수+부정단어수)
긍정평가일확률,부정평가일확률(0.75, 0.25)def f(text):
긍정단어수 = sum([l in 긍정단어 for l in text.split(' ')])
부정단어수 = sum([l in 부정단어 for l in text.split(' ')])
긍정평가일확률 = 긍정단어수/(긍정단어수+부정단어수)
부정평가일확률 = 부정단어수/(긍정단어수+부정단어수)
if 긍정평가일확률 > 0.5:
return 긍정평가일확률,"긍정평가"
else:
return 부정평가일확률,"부정평가"f("남주가 너무 잘생김 여주도 예뻐 영화가 중간에 조금 지루해 그래도 아무튼 재미있었음")(0.75, '긍정평가')- 당연히 현재는 많은 문제점이 있음.
f("남주가 너무 잘생김. 여주도 예뻐. 영화가 중간에 조금 지루해 그래도 아무튼 재미있었음.")(1.0, '부정평가')f("캐스팅 좋아 여주인공 특히 예뻐 배우들 연기훌륭함 그렇지만 스토리때문에 나는 개인적으로 비추천")(0.75, '긍정평가')- 다행인점: 좋은 f를 만들기 위해서 우리가 고민할 필요 없음. (똑똑한 사람들이 다 만들어 놓음. 그리고 만들고 있음.)
f 를 한땀한땀 설계. 초고수가 밤새 코딩해서 진짜 잘 맞추는 f를 한번에 제시.f 를 대충 설계. 거의 멍청이 수준의 f임. 그런데 데이터를 줄수록 f가 점점 똑똑해짐. 나중에는 다 맞춤. –> 인공지능???- 초고수가 f를 직접 설계하던 시대에는 별로 문제가 없었음. 그런데 최근 컴퓨터가 데이터를 보고 f를 스스로 수정하기 시작하면서 이상한 방식으로 f가 수정되는 경우가 보고되고 있음. f가 똑똑해 보이는데, 사실 멍청한 상태..
- 최규빈 교수의 착각
- 이 상황을 살짝 말만 바꿔볼게요.
f 가 올바르게 동작해야함.- train data / test data
imdbDatasetDict({
train: Dataset({
features: ['text', 'label'],
num_rows: 25000
})
test: Dataset({
features: ['text', 'label'],
num_rows: 25000
})
unsupervised: Dataset({
features: ['text', 'label'],
num_rows: 50000
})
})- 학생들 입장에서 생각해본다면??
{15000문항: 50%맞음, 10000문항: 45%맞음}{15000문항: 90%맞음, 10000문항: 85%맞음}{15000문항: 100%맞음, 10000문항: 70%맞음} <— 이런 경험 있어요??- 이럴경우 아래와 같이 상황이 정리된다.
- 학생이 자가진단용으로 빼둔 10000개의 문항을 보통 validation set 이라고 부른다. 따라서 아래와 같이 정리 가능하다.
- train / validation / test 에 대한 용어는 엄밀하지 않게 사용되는 경우가 많아 그때그때 상황에 맞게 알아서 해석해야 한다.
- 느낌: 아래가 가장 정확한 설명임
헷갈리는 이유는 더 좋은 훈련을 위한 목적과 올바르게 학습됨을 평가하기 위한 목적이 무 자르듯이 구분되지 않기 때문.
- 이상한 분류법