利用requests跟BeautifulSoup获取网页内容

对于下面这个形式的html，java中用jsoup来处理，python里面可以用requests+BeautifulSoup来处理

html源码

<html>
    <header>
        <title>测试</title>
    </header>
    <body>
        <div class="post-preview">
            <a href="https://127.0.0.1:90/archives/rrrr">
                <h2 class="post-title">
                    标题1
                </h2>
                <div class="post-content-preview">
                    摘要1
                </div>
            </a>
            <p class="post-meta">
                信息1
            </p>
        </div>
        <hr>
        <div class="post-preview">
            <a href="https://127.0.0.1:90/archives/rrrr4">
                <h2 class="post-title">
                    标题4
                </h2>
                <div class="post-content-preview">
                    摘要4
                </div>
            </a>
            <p class="post-meta">
                信息4
            </p>
        </div>
        <hr>
        <div class="post-preview">
            <a href="https://127.0.0.1:90/archives/rrrr2">
                <h2 class="post-title">
                    标题2
                </h2>
                <div class="post-content-preview">
                    摘要2
                </div>
            </a>
            <p class="post-meta">
                信息2
            </p>
        </div>
        <hr>
        <div class="post-preview">
            <a href="https://127.0.0.1:90/archives/rrrr3">
                <h2 class="post-title">
                    标题3
                </h2>
                <div class="post-content-preview">
                    摘要3
                </div>
            </a>
            <p class="post-meta">
                信息3
            </p>
        </div>
        <hr>
        <ul class="pager">
            <li class="next">
                <a href="https://127.0.0.1:90/page/2">Older Posts →</a>
            </li>
        </ul>
    </body>
</html>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

python 代码

import requests
from bs4 import BeautifulSoup

links = []
index = 1
isEnd = False

while not isEnd:
    url = f"http://127.0.0.1:90/page/{index}"
    response = requests.get(url, timeout=30)
    soup = BeautifulSoup(response.text, "html.parser")
    content = soup.select(".post-preview a")

    for link in content:
        linkHref = link["href"]
        links.append(linkHref)

    contentEnd = soup.select(".next a")
    if len(contentEnd) == 0:
        isEnd = True

    index += 1

print(join(links))

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

上次更新: 2024/01/07, 07:44:52

← python开发学习为知笔记记录了我的职业生涯→