samedi 25 juin 2016

Javascript data in Python [on hold]


I'm trying to get specific data from a website, I manage to get the HTML tree and process it, but for some reason the data I require within a tag is missing, I noticed that the tag is filled by a Javascript call and so, it loads a second after the page is loaded. The site HTML structure I'm trying to get is this <span> 共 <a href="javascript:void(0);" class="font-12-b ml-5 mr-5 js-search-resultnumber" id="filterDealerCount"> 2 </a> 家经销商满足条件 </span> And the python function im using is this: import lxml.html as parser import urllib.request as web def getData(url): pageContent = web.urlopen(url).read() doc = parser.fromstring(pageContent) alphabet = doc.find_class('text js-search-brand') htmlCarDealers = [] for letter in alphabet: htmlCarDealers.append(letter.find_class('')) urlCarDealers = [] for letter in htmlCarDealers: tmp = [] for dealer in letter: tmp.append([dealer.text_content(), dealer.values()[1]]) urlCarDealers.append(tmp) countCarDealers = [] cleanUrl = url.replace("/china", "") for letter in urlCarDealers: for dealer in letter: dealerUrl = cleanUrl + dealer[1][:dealer[1].index('?')] pageDealer = web.urlopen(dealerUrl).read() dealerDoc = parser.fromstring(pageDealer) size = dealerDoc.get_element_by_id('filterDealerCount').text_content() countCarDealers.append([dealer[0], size]) return countCarDealers My interested web page is "http://dealer.autohome.com.cn/china" and I'm trying get every brand and dealers in this country.

Aucun commentaire:

Enregistrer un commentaire