I'm trying to get specific data from a website, I manage to get the HTML tree and process it, but for some reason the data I require within a tag is missing, I noticed that the tag is filled by a Javascript call and so, it loads a second after the page is loaded.
The site HTML structure I'm trying to get is this
<span>
共
<a href="javascript:void(0);" class="font-12-b ml-5 mr-5 js-search-resultnumber" id="filterDealerCount">
2
</a>
家经销商满足条件
</span>
And the python function im using is this:
import lxml.html as parser
import urllib.request as web
def getData(url):
pageContent = web.urlopen(url).read()
doc = parser.fromstring(pageContent)
alphabet = doc.find_class('text js-search-brand')
htmlCarDealers = []
for letter in alphabet:
htmlCarDealers.append(letter.find_class(''))
urlCarDealers = []
for letter in htmlCarDealers:
tmp = []
for dealer in letter:
tmp.append([dealer.text_content(), dealer.values()[1]])
urlCarDealers.append(tmp)
countCarDealers = []
cleanUrl = url.replace("/china", "")
for letter in urlCarDealers:
for dealer in letter:
dealerUrl = cleanUrl + dealer[1][:dealer[1].index('?')]
pageDealer = web.urlopen(dealerUrl).read()
dealerDoc = parser.fromstring(pageDealer)
size = dealerDoc.get_element_by_id('filterDealerCount').text_content()
countCarDealers.append([dealer[0], size])
return countCarDealers
My interested web page is "http://dealer.autohome.com.cn/china" and I'm trying get every brand and dealers in this country.
Aucun commentaire:
Enregistrer un commentaire