Simple web scraper¶
Note:
Check out http://datascience.ibm.com/blog/markdown-for-jupyter-notebooks-cheatsheet/
Introduction to Jupyter: https://www.youtube.com/watch?v=Rc4JQWowG5I
# Modules or libraries get imported here in Python, Julia, and R #
import re
import requests
from bs4 import BeautifulSoup
import time
Load a webpage into memory and parse it using Beautiful Soup¶
lyriclink = 'https://www.nytimes.com/'
#lyriclink = 'https://www.washingtonpost.com/'
webpg = requests.get(lyriclink)
soup = BeautifulSoup(webpg.content,"lxml")
#soup.findAll(text=re.compile('Irma'))
count=0
keyword = 'biden'
for lyrics in soup.findAll(string=re.compile(keyword)):
count+=1
print("The Word "+ keyword + " showed up",count,"times!!")
The Word biden showed up 2 times!!
For homework, modify the variables webpg and the keyword to scrape a different website for a different word then upload the modified code to the In-Class Assignment for Week01 on Brightspace.
print(soup.prettify())
The New York Times - Breaking News, US News, World News and Videos Skip to content Skip to site indexIn [ ]:In Escalation, Adams Says Migrant Crisis ‘Will Destroy New York City’
At a town hall meeting, Mayor Eric Adams of New York said that he did not “see an ending” to the migrant crisis and renewed his push for federal help.
5 min read
The governor of Texas has sent migrants to other states, but they’re a fraction of the number of migrants boarding buses.
4 min read
‘The Devil They Know’: McConnell’s Health Issues Worry Democrats
Senator Mitch McConnell and his usual opponents are aligned on Ukraine, spending and impeachment as he faces mounting health scrutiny.
5 min read
Former Speaker Nancy Pelosi said she had no concerns about Senator Dianne Feinstein’s ability to serve.
3 min read
Peter Navarro Is Convicted of Contempt of Congress Over Jan. 6 Subpoena
Mr. Navarro is the second top adviser to former President Trump to be found guilty of contempt for defying the Jan. 6 committee.
5 min read
The prosecutor Fani Willis accused Jim Jordan of trying to obstruct her case against Donald Trump and his allies.
3 min read
Gov. Ron DeSantis and Vivek Ramaswamy called the Proud Boys’ sentences “excessive” and “wrong.”
3 min read
How Covid Affects the Heart
Three years into the pandemic, the short- and long-term risks are becoming more clear.
4 min read
Covid Cases Continue to Rise, but Experts Remain Optimistic
An uptick in hospitalizations and deaths persists, but the numbers are relatively low and new vaccines are around the corner.
4 min read
Ukrainians Embrace Cluster Munitions, but Are They Helping?
The weapons, banned by most countries over human rights concerns, are “not a magic wand,” but some Ukrainian troops say they are making a difference.
5 min read
Russian officials said air defenses shot down two drones in and around one of the country’s largest military hubs.
Mattel’s Windfall From ‘Barbie’
The company’s approach has paid off to a degree that even the C.E.O. could hardly have believed possible.
8 min read
Toyota has dominated sales of hybrid cars, but has been slow to sell all-electric vehicles.
7 min read
Shares in Apple fell following reports that China would bar government workers from using iPhones.
7 min read
David French
The Contagious Corruption of Ken Paxton
6 min read
Frank Bruni
Old Is Workable. Depravity Is a Dead End.
9 min read
Nicholas Kristof
One of the Best Things We Can Do for Our Health Is Free — and Fun
7 min read
Stephanie Muravchik and Jon A. Shields
Republicans in Wyoming See Clearly What’s Happening
5 min read
Michelle Cottle, Ross Douthat, Carlos Lozada and Lydia Polgreen
The Woke Burnout Is Real — and Politics Is Catching Up
Andy Hong
How Does an Older Person See the City? Take a Look.
Pamela Paul
The Problem With ‘Elites’ May Not Be What You Think It Is
5 min read
Craig Singleton
Under the Radar, China Has a Military Mission
5 min read
In Case You Missed It Top picks from The Times, recommended for you More NewsDanny Masterson Is Sentenced to 30 Years to Life in Prison for 2 Rapes
The “That ’70s Show” actor was found guilty in May of raping two women in the early 2000s.
3 min read
2 Senators Propose Bipartisan Framework for A.I. Laws
The effort by Senators Richard Blumenthal and Josh Hawley is the latest move by lawmakers to catch up with the rapidly evolving technology.
2 min read
Well From the Book ReviewNew York Times Games Daily word and visual games, plus more.