An introduction to my senior project where I explain what steps I will take to create a dashboard that will give dealership’s an advantage over their competitors.
All over the world there are different car dealerships advertising the same vehicle. There could be multiple Ford dealerships in the same 100 mile radius that are owned by different companies and When buying a new car, you are interested in two things: which option is the most convenient and which option will give you the better deal. If it is worth driving to an adjacent city to save $2,000 on a new truck it is worth the gas and time to drive there.
I am on a mission to collect inventory data from as many dealerships as I can, build them into one main data set, and create an interactive dashboard to show off what kind of power this data can give a competitor.
Unfortunately, there isn’t one massive database that holds every dealer’s daily inventory. Neither is there an API for every dealership in the nation. This process requires grit and a lot of experimentation.
You could manually go through each dealership’s website and record each vehicle in an excel sheet… but then you would have to come back and do that all over again the next day because their inventory changed. Instead you can write a web scraping script to grab all of the data from a single website in less than a minute.
Web scraping is the art of sending code to search through a website and collect specific information based off of rules that you set in place.
There are many tools that can be used to web scrape, but for this project I will be using Python.
My goal is to create a single script that will call all of the web scrapes and then build a single data set from all of them. This will be the first part of the ETL (Extract, Transform, & Load) process.
I aim to do all of the transforming of the data inside python so that when I load the data is will be ready for analysis.
After I have all ready I will load the data into PowerBI, which is where I will be creating my dashboard that will allow me to search and compare all of the competition for a single vehicle.
PowerBI is Microsoft’s new application for dashboarding. You use to just create all of your graphs and analytics in Excel. Now we have this beautiful application that can link to any data source you can think of and handle all of the data modeling, transformation, and analytics all in itself.
You can even use PowerBI for webscraping. However, it doesn’t allow the flexibility and speed that Python has. I would only use PowerBI for web scraping if I were scraping only one website. PowerBI takes a lot of power to run and if it is having to do all of the transforming and scraping of data, it will slow you down. Especially if you do not have a high-end processing computer.
By the end of this semester I hope to have enough websites scraped that I will be able to end the semester with a powerful report that will show me all of the competition for all vehicles of one make.
I would also like to involve parallel programming in my model to scrape multiple websites at the same time. This will speed up the testing time by a long shot.
Once I collect enough data, I would like to run a machine learning model that predicts how much I should sell a vehicle to beat my competitors but to also make a profit.
Future posts to look forward to:
Is web scraping legal?
Which tools work great for web scraping and how do you find them?
Follow up with the progress made on this project.
For attribution, please cite this work as
Sant (2021, Feb. 17). Data Science with Keaton: Getting the Edge on Competitors: Dealership Dashboard. Retrieved from https://keatonjsant.github.io/posts/2021-02-17-getting-the-edge-on-competitors-dealership-dashboard/
BibTeX citation
@misc{sant2021getting,
author = {Sant, Keaton},
title = {Data Science with Keaton: Getting the Edge on Competitors: Dealership Dashboard},
url = {https://keatonjsant.github.io/posts/2021-02-17-getting-the-edge-on-competitors-dealership-dashboard/},
year = {2021}
}