기사를 크롤링하여 워드클라우드를 만들어보자.

기사를 크롤링하여 WORDCLOUD를 만들어보자.¶

기사를 크롤링하여 Wordcloud를 만들어 보겠습니다. 먼저 저는 조선일보의 기사를 이용하였습니다. 전체적인 구조는 크롤링을 하고 의미없는 값들을 정리한 후 기사들을 모두 단어로 쪼개서 wordcloud를 만들어 보았습니다.

In [1]:

from urllib.request import urlopen
from bs4 import BeautifulSoup     
from wordcloud import WordCloud, STOPWORDS
import numpy as np
from PIL import Image
import nltk
from konlpy.tag import Kkma
from konlpy.corpus import kobill

In [3]:

url_base="http://news.chosun.com/site/data/html_dir/2019/12/01/20191201"
last='.html'

title=[]
news_list=[]
for i in range(3000):
    
    index=str(i)
    if len(index)==1:
        index='0000'+index
    elif len(index)==2:
        index='000'+index
    elif len(index)==3:
        index='00'+index
    else: 
        insex='0'+index
    try:
        page=urlopen(url_base+index+last)
        soup = BeautifulSoup(page, 'html.parser') 
        title.append(soup.find('h1').get_text())
        news_list.append(soup.find_all(id="news_body_id",itemprop="articleBody")[0].get_text())
    except:
        pass

In [4]:

import re
news_raw_delete=[]
b=[]
c=[]
d=[]
for i in range(len(news_list)):
    b.append(re.sub('....기자|스포츠조선....|스포츠조선....|\[OSEN.......]|...무단전재.........|ssu08....................................|TV조선|...................com|▲.*|좋아요..|Copy..............................|...osen...|Copyrig........|....sportschosun.com|....Chosun.com|제휴안내구독신청','',news_list[i]))

    c.append(re.sub('[-=+,#/\?:^$.@*\"※~&%ㆍ!』\\‘|\(\)\[\]\<\>`\'…》]', '', b[i]))

    d.append(re.sub('\s|입력','',c[i]))
    
    
    news_raw_delete.append(d[i])
#b=re.sub('...................com|▲.*|좋아요..|Copy..............................','',news_list[5])
#b=re.sub('[-=+,#/\?:^$.@*\"※~&%ㆍ!』\\‘|\(\)\[\]\<\>`\'…》]', '', a)
#re.sub('\s|입력','',b)

In [5]:

len(title)

Out[5]:

449

449개의 기사를 크롤링 했습니다 :)

In [27]:

news_raw_delete[0]

Out[27]:

'201911302351갈길바쁜맨체스터시티가뉴캐슬원정에서무승부를거뒀다맨시티는30일한국시간밤영국뉴캐슬의세인트제임스파크에서열린20192020시즌잉글리시프리미어리그EPL14라운드뉴캐슬과경기에서22무승부를거뒀다맨시티는승점1을추가해29점으로2위로올라섰지만1경기덜치른3위레스터와승점동률로불안한상황이다뉴캐슬은541포메이션가동했다전방에조엘링턴중원에생막시맹셸비헤이든알미론이나섰다빌렘스더밋클라크페르난데스만퀴요가수비를지켰고골문은두브라브카가책임졌다원정팀맨시티는433으로맞섰다스털링제주스마레즈가스리톱다비드실바귄도간데브라위너중원을책임졌다멘디페르난지뉴스톤스워커가수비를구성했다골키퍼는에데르송양팀은전반부터치열하게맞붙었다전반21분맨시티가선제골로앞서갔다스털링이실바와패스를주고받으며박스안으로진입했다스털링의깔끔한마무리로맨시티는10으로앞서갔다불과4분후맨시티는실점했다전반25분빌렘스가알미론과원투패스로수비를뚫었다빌렘스는오른발슈팅으로맨시티골문을열었다스코어는11동점상황에서맞은후반에도맨시티는쉽사리리드를잡지못했다맨시티는후반25분실바와마레즈를빼고포덴과베르나르두실바를투입했다뉴캐슬은조엘링턴을빼고게일을투입해반격했다후반35분맨시티가드디어골을터뜨렸다뉴캐슬수비가멘디의크로스를걷어내자데브라위너가강력한슈팅으로골망을흔들었다하지만뉴캐슬은다시동점골을넣었다후반43분셸비가환상적인중거리슈팅으로경기를원점으로돌렸다맨시티의파상공세가계속됐지만뉴캐슬은두브라브카의선방에힘입어동점을22무승부를지켰다raul16kr'

In [12]:

news_join = ','.join(news_raw_delete)
#article_join

In [28]:

from konlpy.tag import Twitter; t = Twitter()
tokens_ko = t.nouns(news_join)
tokens_ko[0]

C:\Users\Wonkim\Anaconda33\lib\site-packages\konlpy\tag\_okt.py:16: UserWarning: "Twitter" has changed to "Okt" since KoNLPy v0.4.5.
  warn('"Twitter" has changed to "Okt" since KoNLPy v0.4.5.')

Out[28]:

'맨체스터'

In [14]:

import requests, datetime

In [15]:

import os, sys, time

In [16]:

ko = nltk.Text(tokens_ko)

In [17]:

print(len(ko.tokens))           # returns number of tokens (document length)
print(len(set(ko.tokens)))   # returns number of unique tokens
ko.vocab()    

40540
7780

Out[17]:

FreqDist({'경기': 494, '안양': 222, '감독': 220, '리그': 216, '울산': 216, '선수': 207, '팬': 196, '위': 181, '전자랜드': 160, '모습': 155, ...})

In [18]:

import matplotlib.pyplot as plt
import platform

path = "c:/Windows/Fonts/malgun.ttf"
from matplotlib import font_manager, rc
if platform.system() == 'Darwin':
    rc('font', family='AppleGothic')
elif platform.system() == 'Windows':
    font_name = font_manager.FontProperties(fname=path).get_name()
    rc('font', family=font_name)
else:
    print('Unknown system... sorry~~~~') 
    
%matplotlib inline

In [19]:

plt.figure(figsize=(12,6))
ko.plot(50)
plt.show()

In [29]:

stop_words = ['를','의','개','전','점','해','위','등','말','승','로','팀','앞','번','곡','명','더','씨','것','날','등','때','관련','위해','위','더','건','이번']

ko = [each_word for each_word in ko if each_word not in stop_words]
ko[0]

Out[29]:

'맨체스터'

In [25]:

ko = nltk.Text(ko)

plt.figure(figsize=(12,6))
ko.plot(50)     # Plot sorted frequency of top 50 tokens
plt.show()

In [26]:

data = ko.vocab().most_common(150)

wordcloud = WordCloud(font_path='../datahandling/NanumBrush.ttf',
                      relative_scaling = 0.2,
                      background_color='white',
                      ).generate_from_frequencies(dict(data))
plt.figure(figsize=(12,8))
plt.imshow(wordcloud)
plt.axis("off")
plt.show()

크롤링 한 기사들이 스포츠 관련 기사가 많았는지 스포츠 관련 단어들의 빈도 수가 높게 나오는 결과가 나왔네요 :)