Custom Web Scraping & Email Script
The end goal for this project was to use my knowledge of web scraping and implement a program to gather data that interests me. One of the use cases that really caught my eye for this project, was to implement a program that gathers housing data, saves a spreadsheet, and emails those results.
Project Goals & Steps to Complete
The main goals for this project were to create an application that scrapes data from the internet and saves that data to a csv file. I decided to take it a step further and email the data from the csv file and schedule this to run daily. My goal for this project was to create a web scraping application that gathers housing data that met my search criteria, save the desired information from what was scraped to a csv, and email the data from the csv file.
Project Challenges
The initial scope of the project was not too challenging to complete. I was able to create a web scraping application that gathered home data for the search criteria that I had set. For the initial scope of the project, the most challenging part was refreshing my memory on how to gather desired information from what was scraped. Using the inspect tool from the Chrome developer tools, I was able to identify the data and layout of the web pages I was gathering data off. Another small challenge was transforming the lists created from the data, into a csv file. For that, I was able to use the Pandas documentation and save the output from the lists to a csv file.
Many of the challenges I faced came when trying to create an email from the csv data. Initially I wanted to create two separate lists from the csv data, a list for new homes listed on the current day and a separate list for homes that were previously listed, but the price had changed. Ultimately, I was able to create code that worked successfully; however, the code was more complicated than it needed to be. For this, I did not use Pandas and opted for using the csv library and creating lists from built in Python functions. When trying to email the data, I would encounter an error where the email would not be sent. After some troubleshooting, I decided to scale back the scope of the email and only send an email for all the data from the original csv file. This data did not include new listings or homes that have a new price.
After some troubleshooting, it appears the issue I may have been running into was with emojis being included in branches of code for new homes and price changes. I plan on revisiting the initial scope of the email portion I had set for the project, after spending more time working with other projects.
Project Takeaways
This was a very fun project to complete! I skipped ahead in the 100 Days of Code curriculum since this was a project I was really looking forward to completing! In the near future, I do plan on being in the house hunting market and gathering this information is going to come in very handy when that time comes. Additionally, working on this project touched on tools that would be helpful in automating parts of my personal and professional life such as automating emails, saving data and gathering data.