How can I scrape Lowes
Hi everyone, I've been stuck with this problem for a couple of days now but I can't figure out how I can scrape pages from Lowes. From what I've seen I think this page is dynamically created(but I'm still not sure) so I've been struggling with actually getting this data. Originally I tried using selectors to get the job done, but that didn't return any results: Here's my code:
-*- coding: utf-8 -*-
import scrapy from ..items
import LowesspiderItem class LowesSpider(scrapy.Spider):
name = 'lowes'
allowed_domains = ['lowes.com']
start_urls = ['https://www.lowes.com/pd/ZLINE-KITCHEN-BATH-Ducted-Red-Matte-Wall-Mounted-Range-Hood-Common-42-Inch-Actual-42-in/1001440644']
def parse(self, response):
items = response.css('.grid-container')
for product in items:
item = LowesspiderItem() #get product price
productPrice = product.css('.art-pd-price::text').getall()
item["productPrice"] = productPrice
yield item
This would return an empty field back to me so like this '[]'
From what I learned online, the way to know if a page is dynamically created then it should be blank when disabling javascript, but the webpage I'm scraping turned out fine.
Here's the url I'm scraping: https://www.lowes.com/pd/ZLINE-KITCHEN-BATH-Professional-Deep-Recessed-6-Burners-Convection-Stainless-Steel-Common-36-in-Actual-36-in/1000525095
When disabling JavaScript/Cache:
and when I inspect element I can find the actual text for the price too:
So I'm confused of wether I can just use standard CSS selectors or not? I tried out Selenium but realized that it would not be feasible to scrape this website, and I was unable to get json data from the website, so any help would be appreciated!