Simple web scraper¶
Note:
Check out http://datascience.ibm.com/blog/markdown-for-jupyter-notebooks-cheatsheet/
Introduction to Jupyter: https://www.youtube.com/watch?v=Rc4JQWowG5I
# Modules or libraries get imported here in Python, Julia, and R #
import re
import requests
from bs4 import BeautifulSoup
import time
Load a webpage into memory and parse it using Beautiful Soup¶
lyriclink = 'https://www.nytimes.com/'
#lyriclink = 'https://www.washingtonpost.com/'
webpg = requests.get(lyriclink)
soup = BeautifulSoup(webpg.content,"lxml")
#soup.findAll(text=re.compile('Irma'))
count=0
keyword = 'Trump'
for lyrics in soup.findAll(string=re.compile(keyword)):
count+=1
print("The Word "+ keyword + " showed up",count,"times!!")
The Word Trump showed up 20 times!!
For homework, modify the variables lyriclink and the keyword to scrape a different website for a different word then upload the modified code to the In-Class Assignment for Week01 on Brightspace.
print(soup.prettify())
The New York Times - Breaking News, US News, World News and Videos Skip to content Skip to site indexIn [ ]:New York Times - Top Stories
Top Stories
Inside the Trump-Harris Debate Prep: Method Acting, Insults, Hard Questions
Vice President Kamala Harris is camped out at a hotel. Donald Trump is being peppered by aides. Both view Tuesday’s debate as a crucial moment.
7 min read
In an interview, Hillary Clinton gave advice on debating Donald Trump: “He can be rattled.”
3 min read
Kamala Harris’s combative debate style will get its biggest test against Donald Trump.
7 min read
Donald Trump Claims Kamala Harris’s Rallies Are Smaller. We Counted.
We took photographs, captured 360-degree video and counted attendees at six recent rallies to analyze which claims on crowd sizes held weight.
The Heritage Foundation has been pushing misinformation about noncitizen voters into social media feeds.
7 min read
Israel Strikes School Turned Shelter in Series of Attacks in Gaza
Israel said it launched a strike against Hamas militants operating from a school compound. Medics said several people sheltering there were killed.
3 min read
Family of American Slain in West Bank Demands an Independent Inquiry
With witnesses and Palestinian officials accusing Israeli soldiers of killing the woman, “an Israeli investigation is not adequate,” her family said.
3 min read
Japan Tries to Reclaim Its Clout as a Global Tech Leader
Japanese chip companies are tapping billions of dollars and collaborating with foreign firms as part of new government policies that look outward.
5 min read
The prospect that President Biden will block a merger between Nippon Steel and U.S. Steel is stirring discontent in Japan.
4 min read
Ukrainian Forces Block Russian Advance on a Key Eastern Town
Russia’s drive toward Pokrovsk has stalled along one part of the frontline, but its troops continue to advance in other parts of eastern Ukraine.
3 min read
Using ruins as his canvas, Gamlet Zinkivskyi has captured life in wartime Ukraine in harshly beautiful paintings.
5 min read
Iran sent short-range ballistic missiles to Russia, according to U.S. and European officials.
4 min read
LIVE
Jessica Pegula Takes On Aryna Sabalenka in the Women’s Final
Pegula, one of two Americans in the U.S. Open finals, is looking to win her first Grand Slam title.
From The Athletic
At the U.S. Open, ushers who barely make minimum wage direct New York’s well-heeled crowds.
5 min read
DealBook spoke with the U.S.T.A. chief about challenges faced by the business of tennis, and more.
8 min read
What to Watch and ReadGlynnis MacNicol
I Love the Kids in My Life. And I’m Raising None of Them.
4 min read
Featuring Allan Lichtman
He Predicted Trump in 2016, Biden in 2020 and Now…
Megan K. Stack
Blaming a Parent, Again, for Failed Gun Laws
5 min read
Nicholas Kristof
Forget the Yacht. The Best Travel Is on Foot, Through Wilderness.
4 min read
Jessica Grose
Can ‘The Secret Lives of Mormon Wives’ Change a Conservative Religious Culture?
4 min read
Tim Donahue
High Schoolers Need to Do Less So That They Can Do Better
5 min read
Eric Klinenberg
The Killer Climate Disaster That Has No Name
5 min read
Letters From Our Readers
Making End-of-Life Decisions
4 min read
In Case You Missed It
More NewsWhat We Know About the Investigations Involving Eric Adams’s Top Aides
The administration of New York City’s mayor was thrown into turmoil as federal inquiries reached his inner circle.
4 min read
Spy Chiefs of the C.I.A. and MI6 Convene, on a Couch and for a Crowd
Appearing together publicly for the first time, the heads of the U.S. and British intelligence services discussed Ukraine and Gaza.
3 min read
Well
Games
Daily puzzles
