这是我第一次做网络抓取。我想做的是获得所有可能的财产信息(位置、价格等)。
所以我现在用谷歌搜索收集到的信息是这样的:
import requests
from bs4 import BeautifulSoup
from pprint import pprint as pp
rootURL = 'https://www.jawitz.co.za/'
response = requests.get(rootURL)
html = response.content
soup = BeautifulSoup(html,'lxml')
dropdown_list = soup.select(".primary .child-pages a")
cityLinks = [rootURL + dropdown_list_value['href'] for dropdown_list_value in dropdown_list]
# params for our request
params = {"province": "Western Cape",
"suburb": "Van Riebeeck Park",
"region": "Worcester",
"id": 929,
"property_type": "Apartment"}
for city in cityLinks: # Looping each city from the Apartment drop down list
with requests.Session() as s:
r= s.get(city)
soup = BeautifulSoup(r.content)
# getting some information should go here
现在,我真的不确定如何检索信息,因为在这个网站上,在检查了资源后,我偶然发现了一些隐藏的输入。关于如何进行这项工作有什么建议吗?我对如何访问资源中可用的信息有点困惑。
提前谢谢!