[问题] 新手爬虫(被挡的问题)

楼主: etudiant (weiwei)   2022-10-09 22:41:29
板上的大大们好,小弟又来请教问题了,最近在爬群众募资平台的资料,但很常会中间好
几页爬不到东西,过一阵子又有了,想请问大家是什么问题...不确定是不是跟下图这个
检查网络连线的有关,有时候我换页点很快也会遇到QQ 如果有关的话,想请问是否有解
决的办法,谢谢!
https://i.imgur.com/GdWEijn.jpg
附上我的程式码:
(当初逻辑上是先去外面的页面抓完每页的id名称再套进去网址去找每个项目的资讯,之
后再转成Excel)
import requests
import bs4
import time
import random
import pandas as pd
collect_title=[]
collect_category=[]
collect_goal=[]
collect_final=[]
collect_people=[] #空列表之后存资料
def get_page_info(URL):
headers = {'cookie':'age_checked_for=12925;','user-agent': 'Mozilla/5.0 (W
indows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.
0.0.0 Safari/537.36'}
#含第九页一项18禁的cookie
response = requests.get(URL,headers=headers)
soup = bs4.BeautifulSoup(response.text,"html.parser")
data=soup.find_all('div','text-black project text-neutral-600 mx-4 mb-4 pb
-12 relative h-full')
for t in data:
href = t.find("a","block").attrs["href"] #取a标签下href后面的网址
link ="https://www.zeczec.com"+href
response_2 = requests.get(link,headers=headers)
soup_2 = bs4.BeautifulSoup(response_2.text,"html.parser") #解析取下的
网址中的网页内容
main_info = soup_2.find_all('div','container lg:my-8')
for i in main_info:
#category = i.find('a','underline text-neutral-600 font-bold inlin
e-block').text.strip()
category = i.find_all('a','underline text-neutral-600 font-bold in
line-block')[1].text.strip()
title = i.find('h2', 'text-xl mt-2 mb-1 leading-relaxed').text.str
ip()
final_cash = i.find('div','text-2xl font-bold js-sum-raised whites
pace-nowrap leading-relaxed').text.strip()
goal_cash = i.find('div','text-xs leading-relaxed').text.strip()
people = i.find('span','js-backers-count').text.strip()
final='类别:{} 标题:{} 目标:{} 实际:{} 赞助人数:{}'.format(categor
y,title,goal_cash[6:],final_cash[3:],people)
print(final)
collect_category.append(category)
collect_title.append(title)
collect_goal.append(goal_cash[6:])
collect_final.append(final_cash[3:])
collect_people.append(people) #丢入collect列表
time.sleep(2)
for i in range(1,13,1):
print("第"+str(i)+"页")
URL="https://www.zeczec.com/categories?category=1&page="+str(i)+"&type=0"
get_page_info(URL)
delay_time=[3,7,8,5]
delay=random.choice(delay_time)
time.sleep(delay)
print(len(collect_goal)) #计算抓了几笔
#print(collect_final)
#print(collect_people)
col1 = "类别"
col2 = "标题"
col3 = "目标金额"
col4 = "实际金额"
col5 = "赞助人数"
data = pd.DataFrame({col1:collect_category,col2:collect_title,col3:collect_goa
l,col4:collect_final,col5:collect_people}) #三栏放list
data.to_excel('音乐.xlsx', sheet_name='sheet1', index=False)
遇到问题的画面:
突然好几页不能抓到这样,不知道是requests 还是太频繁吗?
https://i.imgur.com/PaN0z1N.jpg
作者: surimodo (好吃棉花糖)   2022-10-09 23:59:00
就是太频繁阿 time.sleep 数字调高一点
作者: cocoaswifty (coco)   2022-10-09 23:59:00
你最后不就说出答案了

Links booklink

Contact Us: admin [ a t ] ucptt.com