如果我講不到這裡 😨，可以看下列文章：¶

現在在這裡

3.1. 為何點擊某個按鈕後，即使網頁內容變了，網址還是一樣？¶

有時是兩種東西造成的

form (表單)
javascript (可以用修改html)（未來web課程才會細講）

Form (表單) 可以讓你輸入內容，而當你點擊下方的 Submit按鈕，通常會有另一隻Javascript程式將表單中的內容透過 Request(POST)發送給遠端伺服器，而遠端伺服器將新的內容回傳之後，Javascript 再將網頁中的內容直接修改，因此網址不會有變化。

有些則是當你點擊按鈕後，他一樣會發個 request，但他會將你導向到別的網址(像火車時刻表)

也就是說我們透過觀察開發人員工具中的"Network"來查看他發送了什麼request，再依樣畫葫蘆發一樣的request給對方，通常我們就能得到我們要的資料。

Example 3.1¶

使用 Post 來取得 7/20 22:00 出發，台南到台北的高鐵車號

備註：高鐵有 [API](https://ptx.transportdata.tw/PTX)，以後有需要請直接使用API不要這樣直接爬，這裏只是拿來做 post 的練習。

打開開發人員工具的Network頁籤，觀察點擊"立即查詢"後會發佈什麼 Request

In [58]:

import requests
form_data = {
    'startStation': '9c5ac6ca-ec89-48f8-aab0-41b738cb1814',
    'endStation': '977abb69-413a-4ccf-a109-0272c24fd490',
    'theDay': '2018/07/20',
    'timeSelect': '22:00',
    'waySelect': 'DepartureInMandarin',
}
url = 'https://m.thsrc.com.tw/tw/TimeTable/SearchResult'
response = requests.post(url, data=form_data)
print(login_response.status_code)

In [66]:

soup = BeautifulSoup(response.text, 'html.parser')
# print(response.text)
tags = soup.find_all('a','ui-block-a')
print(tags)

[<a class="ui-block-a" data-ajax="false" href="/tw/TimeTable/TrainInfo/0696"><div>0696</div></a>, <a class="ui-block-a" data-ajax="false" href="/tw/TimeTable/TrainInfo/0294"><div>0294</div></a>]

3.2. 那怎麼寫一個可以抓登入後的頁面資料的爬蟲？¶

先想想. . .
hint: 也是Request

最簡單的方式是，跟剛才一樣觀察登入時會發的Request，再用程式發一樣的內容。
等登入成功後，通常對方會給你個cookie(用來暫存資料)，你只要將cookie存下來，下次 GET 網頁時一併將這個cookie給對方即可。

Example 3.2¶

模擬登入來取得資料，確認登入後收到的html資料裡有 Logout 字樣

注意，多數網站不會想讓開發者能用機器人來登入，所以才會有驗證碼或什麼Receptra(我不是機器人)。
因此請先確認 /robots.txt 中有無限制使用權限，不過這個網站沒有放。

In [55]:

form_data = {
    'username': 'Neo',
    'password': 'quote*2018*some_static_word'
}
url = 'http://quotes.toscrape.com/login'
login_response = requests.post(url,data=form_data)
print(login_response.status_code)
print(login_response.cookies.get_dict())

# session = requests.Session()
# session.post(url, form_data)
# r = session.get('http://quotes.toscrape.com/')
# r.text

200
{'session': 'eyJ1c2VybmFtZSI6Ik5lbyJ9.Dio9Ng.UDw5cSFXPlLVbuVNhCF4IqhUaro'}

In [45]:

url = 'http://quotes.toscrape.com/'
response = requests.get(url,cookies=login_response.cookies) 
print(response.text)

<!DOCTYPE html>
<html lang="en">
<head>
	<meta charset="UTF-8">
	<title>Quotes to Scrape</title>
    <link rel="stylesheet" href="/static/bootstrap.min.css">
    <link rel="stylesheet" href="/static/main.css">
</head>
<body>
    <div class="container">
        <div class="row header-box">
            <div class="col-md-8">
                <h1>
                    <a href="/" style="text-decoration: none">Quotes to Scrape</a>
                </h1>
            </div>
            <div class="col-md-4">
                <p>
                
                    <a href="/logout">Logout</a>
                
                </p>
            </div>
        </div>
    

<div class="row">
    <div class="col-md-8">

    <div class="quote" itemscope itemtype="http://schema.org/CreativeWork">
        <span class="text" itemprop="text">“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”</span>
        <span>by <small class="author" itemprop="author">Albert Einstein</small>
        <a href="/author/Albert-Einstein">(about)</a> - <a href="http://goodreads.com/author/show/9810.Albert_Einstein">(Goodreads page)</a>
        </span>
        <div class="tags">
            Tags:
            <meta class="keywords" itemprop="keywords" content="change,deep-thoughts,thinking,world" /    > 
            
            <a class="tag" href="/tag/change/page/1/">change</a>
            
            <a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>
            
            <a class="tag" href="/tag/thinking/page/1/">thinking</a>
            
            <a class="tag" href="/tag/world/page/1/">world</a>
            
        </div>
    </div>

    <div class="quote" itemscope itemtype="http://schema.org/CreativeWork">
        <span class="text" itemprop="text">“It is our choices, Harry, that show what we truly are, far more than our abilities.”</span>
        <span>by <small class="author" itemprop="author">J.K. Rowling</small>
        <a href="/author/J-K-Rowling">(about)</a> - <a href="http://goodreads.com/author/show/1077326.J_K_Rowling">(Goodreads page)</a>
        </span>
        <div class="tags">
            Tags:
            <meta class="keywords" itemprop="keywords" content="abilities,choices" /    > 
            
            <a class="tag" href="/tag/abilities/page/1/">abilities</a>
            
            <a class="tag" href="/tag/choices/page/1/">choices</a>
            
        </div>
    </div>

    <div class="quote" itemscope itemtype="http://schema.org/CreativeWork">
        <span class="text" itemprop="text">“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”</span>
        <span>by <small class="author" itemprop="author">Albert Einstein</small>
        <a href="/author/Albert-Einstein">(about)</a> - <a href="http://goodreads.com/author/show/9810.Albert_Einstein">(Goodreads page)</a>
        </span>
        <div class="tags">
            Tags:
            <meta class="keywords" itemprop="keywords" content="inspirational,life,live,miracle,miracles" /    > 
            
            <a class="tag" href="/tag/inspirational/page/1/">inspirational</a>
            
            <a class="tag" href="/tag/life/page/1/">life</a>
            
            <a class="tag" href="/tag/live/page/1/">live</a>
            
            <a class="tag" href="/tag/miracle/page/1/">miracle</a>
            
            <a class="tag" href="/tag/miracles/page/1/">miracles</a>
            
        </div>
    </div>

    <div class="quote" itemscope itemtype="http://schema.org/CreativeWork">
        <span class="text" itemprop="text">“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”</span>
        <span>by <small class="author" itemprop="author">Jane Austen</small>
        <a href="/author/Jane-Austen">(about)</a> - <a href="http://goodreads.com/author/show/1265.Jane_Austen">(Goodreads page)</a>
        </span>
        <div class="tags">
            Tags:
            <meta class="keywords" itemprop="keywords" content="aliteracy,books,classic,humor" /    > 
            
            <a class="tag" href="/tag/aliteracy/page/1/">aliteracy</a>
            
            <a class="tag" href="/tag/books/page/1/">books</a>
            
            <a class="tag" href="/tag/classic/page/1/">classic</a>
            
            <a class="tag" href="/tag/humor/page/1/">humor</a>
            
        </div>
    </div>

    <div class="quote" itemscope itemtype="http://schema.org/CreativeWork">
        <span class="text" itemprop="text">“Imperfection is beauty, madness is genius and it&#39;s better to be absolutely ridiculous than absolutely boring.”</span>
        <span>by <small class="author" itemprop="author">Marilyn Monroe</small>
        <a href="/author/Marilyn-Monroe">(about)</a> - <a href="http://goodreads.com/author/show/82952.Marilyn_Monroe">(Goodreads page)</a>
        </span>
        <div class="tags">
            Tags:
            <meta class="keywords" itemprop="keywords" content="be-yourself,inspirational" /    > 
            
            <a class="tag" href="/tag/be-yourself/page/1/">be-yourself</a>
            
            <a class="tag" href="/tag/inspirational/page/1/">inspirational</a>
            
        </div>
    </div>

    <div class="quote" itemscope itemtype="http://schema.org/CreativeWork">
        <span class="text" itemprop="text">“Try not to become a man of success. Rather become a man of value.”</span>
        <span>by <small class="author" itemprop="author">Albert Einstein</small>
        <a href="/author/Albert-Einstein">(about)</a> - <a href="http://goodreads.com/author/show/9810.Albert_Einstein">(Goodreads page)</a>
        </span>
        <div class="tags">
            Tags:
            <meta class="keywords" itemprop="keywords" content="adulthood,success,value" /    > 
            
            <a class="tag" href="/tag/adulthood/page/1/">adulthood</a>
            
            <a class="tag" href="/tag/success/page/1/">success</a>
            
            <a class="tag" href="/tag/value/page/1/">value</a>
            
        </div>
    </div>

    <div class="quote" itemscope itemtype="http://schema.org/CreativeWork">
        <span class="text" itemprop="text">“It is better to be hated for what you are than to be loved for what you are not.”</span>
        <span>by <small class="author" itemprop="author">André Gide</small>
        <a href="/author/Andre-Gide">(about)</a> - <a href="http://goodreads.com/author/show/7617.Andr_Gide">(Goodreads page)</a>
        </span>
        <div class="tags">
            Tags:
            <meta class="keywords" itemprop="keywords" content="life,love" /    > 
            
            <a class="tag" href="/tag/life/page/1/">life</a>
            
            <a class="tag" href="/tag/love/page/1/">love</a>
            
        </div>
    </div>

    <div class="quote" itemscope itemtype="http://schema.org/CreativeWork">
        <span class="text" itemprop="text">“I have not failed. I&#39;ve just found 10,000 ways that won&#39;t work.”</span>
        <span>by <small class="author" itemprop="author">Thomas A. Edison</small>
        <a href="/author/Thomas-A-Edison">(about)</a> - <a href="http://goodreads.com/author/show/3091287.Thomas_A_Edison">(Goodreads page)</a>
        </span>
        <div class="tags">
            Tags:
            <meta class="keywords" itemprop="keywords" content="edison,failure,inspirational,paraphrased" /    > 
            
            <a class="tag" href="/tag/edison/page/1/">edison</a>
            
            <a class="tag" href="/tag/failure/page/1/">failure</a>
            
            <a class="tag" href="/tag/inspirational/page/1/">inspirational</a>
            
            <a class="tag" href="/tag/paraphrased/page/1/">paraphrased</a>
            
        </div>
    </div>

    <div class="quote" itemscope itemtype="http://schema.org/CreativeWork">
        <span class="text" itemprop="text">“A woman is like a tea bag; you never know how strong it is until it&#39;s in hot water.”</span>
        <span>by <small class="author" itemprop="author">Eleanor Roosevelt</small>
        <a href="/author/Eleanor-Roosevelt">(about)</a> - <a href="http://goodreads.com/author/show/44566.Eleanor_Roosevelt">(Goodreads page)</a>
        </span>
        <div class="tags">
            Tags:
            <meta class="keywords" itemprop="keywords" content="misattributed-eleanor-roosevelt" /    > 
            
            <a class="tag" href="/tag/misattributed-eleanor-roosevelt/page/1/">misattributed-eleanor-roosevelt</a>
            
        </div>
    </div>

    <div class="quote" itemscope itemtype="http://schema.org/CreativeWork">
        <span class="text" itemprop="text">“A day without sunshine is like, you know, night.”</span>
        <span>by <small class="author" itemprop="author">Steve Martin</small>
        <a href="/author/Steve-Martin">(about)</a> - <a href="http://goodreads.com/author/show/7103.Steve_Martin">(Goodreads page)</a>
        </span>
        <div class="tags">
            Tags:
            <meta class="keywords" itemprop="keywords" content="humor,obvious,simile" /    > 
            
            <a class="tag" href="/tag/humor/page/1/">humor</a>
            
            <a class="tag" href="/tag/obvious/page/1/">obvious</a>
            
            <a class="tag" href="/tag/simile/page/1/">simile</a>
            
        </div>
    </div>

    <nav>
        <ul class="pager">
            
            
            <li class="next">
                <a href="/page/2/">Next <span aria-hidden="true">&rarr;</span></a>
            </li>
            
        </ul>
    </nav>
    </div>
    <div class="col-md-4 tags-box">
        
            <h2>Top Ten tags</h2>
            
            <span class="tag-item">
            <a class="tag" style="font-size: 28px" href="/tag/love/">love</a>
            </span>
            
            <span class="tag-item">
            <a class="tag" style="font-size: 26px" href="/tag/inspirational/">inspirational</a>
            </span>
            
            <span class="tag-item">
            <a class="tag" style="font-size: 26px" href="/tag/life/">life</a>
            </span>
            
            <span class="tag-item">
            <a class="tag" style="font-size: 24px" href="/tag/humor/">humor</a>
            </span>
            
            <span class="tag-item">
            <a class="tag" style="font-size: 22px" href="/tag/books/">books</a>
            </span>
            
            <span class="tag-item">
            <a class="tag" style="font-size: 14px" href="/tag/reading/">reading</a>
            </span>
            
            <span class="tag-item">
            <a class="tag" style="font-size: 10px" href="/tag/friendship/">friendship</a>
            </span>
            
            <span class="tag-item">
            <a class="tag" style="font-size: 8px" href="/tag/friends/">friends</a>
            </span>
            
            <span class="tag-item">
            <a class="tag" style="font-size: 8px" href="/tag/truth/">truth</a>
            </span>
            
            <span class="tag-item">
            <a class="tag" style="font-size: 6px" href="/tag/simile/">simile</a>
            </span>
            
        
    </div>
</div>

    </div>
    <footer class="footer">
        <div class="container">
            <p class="text-muted">
                Quotes by: <a href="https://www.goodreads.com/quotes">GoodReads.com</a>
            </p>
            <p class="copyright">
                Made with <span class='sh-red'>❤</span> by <a href="https://scrapinghub.com">Scrapinghub</a>
            </p>
        </div>
    </footer>
</body>
</html>

3.3 為何 "檢查原始碼內的東西跟開發人員工具看到的不一樣" ？¶

先想想. . . . . . .

因為這些網頁在瀏覽器收到 html後，還要用Javascript跑一下才會得到你在開發人員工具看到的最終版本 (例如像臉書那樣，只要滑鼠不斷向下捲動就會不斷跑出新內容的網站)

那要怎麼爬 "html內的東西都是後來程式(JavaScript)產生的，因此光用request爬不到什麼東西的網站。"¶

先想想

想辦法模擬瀏覽器收到html後做的動作 💪¶

為此我們需要一個開發用的瀏覽器驅動程式 (Chrome Driver)

..還要一個可以操作這個驅動程式的python套件 (Selenium)

設定 Chrome driver¶

下載 Chrome driver
- Windows 選 win_32 那個版本即可
解壓縮後，放到目錄
- Windows: python 安裝目錄 (需再設定環境變數) 參考教學
- OSX: /usr/local/bin

Mac 參考指令¶

cp /Downloads/chromedriver /usr/local/bin
chmod +x /usr/local/bin/chromedriver

關於 Selenium¶

你在瀏覽器用滑鼠做的事（像點擊按鈕），都可以用 Selenium 來做。

python + Selenium + Chrome driver + Chrome = 你可以寫一隻程式來模擬操作網站的行為 (Selenium 原始的用途其實是拿來測試網頁功能用的)

安裝 Selenium¶

pip install selenium

Example 3.3¶

請取得 Data Science and Artificial Intelligence Practice 課程網站中，2/27的Description的內容
(Course Introduction and Basics of...)

In [1]:

from bs4 import BeautifulSoup
from selenium import webdriver

try:
    url = 'https://netdbncku.github.io/dsai/2018/'
    driver = webdriver.Chrome()  # '選擇使用 Chrome 來開啟網頁'，如果要用safari就要載 safari的driver
    driver.get(url)
    content = driver.page_source 
finally:
    print('ok')
#     driver.close()  # 關閉一個 Tab
    driver.quit()     # 關閉整個瀏覽器

ok

In [2]:

soup = BeautifulSoup(content, 'html.parser')
tag = soup.find(attrs={'data-title': 'Description'})
print(tag)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-2-dedc37234e2f> in <module>()
----> 1 soup = BeautifulSoup(content, 'html.parser')
      2 tag = soup.find(attrs={'data-title': 'Description'})
      3 print(tag)

NameError: name 'BeautifulSoup' is not defined

每次都會打開瀏覽器很麻煩？ (又很慢)

使用 Headless Chrome 加速運行 (不顯示瀏覽器視窗)¶

In [2]:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

try:
    url = 'https://netdbncku.github.io/dsai/2018/'
    chrome_options = Options()
    chrome_options.add_argument("--headless")  # 選擇 headless mode
    chrome_options.add_argument('window-size=375x812') # 選擇視窗大小
    # 告訴網站你是用 Safari開(雖然不是)
    chrome_options.add_argument(
        '--user-agent=Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1')
    driver = webdriver.Chrome(chrome_options=chrome_options)
    driver.get(url)

    content = driver.page_source
finally:
    print('ok')
#     driver.close()
    driver.quit()

ok

Exercise 3.3 (不計分)¶

取得蝦皮拍賣中，"機車" 搜尋結果的第一個商品名稱

In [47]:

from bs4 import BeautifulSoup
from selenium import webdriver

try:
    url = 'https://shopee.tw/search/?keyword=%E6%A9%9F%E8%BB%8A'
    driver = webdriver.Chrome()  # '選擇使用 Chrome 來開啟網頁'
    driver.get(url)
    content = driver.page_source 
finally:
    print('ok')
#     driver.close()
    driver.quit()

ok

In [50]:

soup = BeautifulSoup(content, 'html.parser')
tag = soup.find(class_='shopee-item-card__text-name').text
print(tag)

APM赤鬼封體蠟180g 超強撥水鍍膜蠟 汽機車蠟 消光烤漆可用 前擋風玻璃撥水適用 可濕上濕下無油影 抗刮耐磨防靜電

延伸學習¶

反爬蟲
- 你操作(GET)太快太頻繁了
- 擋你IP
- 輸入驗證碼
- "我不是機器人"
- class 或 id 隨機產生
- ...
反反爬蟲
- GET 中間加時間間隔
- 換IP (Proxy)
- 圖像辨識驗證碼 (OCR)
- ...應該暫時無解
加速爬蟲運行
- 分布式
- Asyncio

反爬蟲跟反反爬蟲一行人的戰爭依然會持續下去...

[X-Village] Lesson 09 - Crawler (& API)¶

By 呂伯駿 (Po Chun)¶