While writing the above article, I was wondering what the most watched videos of 33C3 were. This information is not easily accessible, so let's extract it from the individual video pages.
First, let's find all the links to the single episodes.
import requests
r = requests.get('https://media.ccc.de/c/33c3')
r
<Response [200]>
Let's parse the result:
from bs4 import BeautifulSoup
soup = BeautifulSoup(r.text, 'html.parser')
Now, let's build the link list:
links = []
for item in soup.select("h3 > a"):
links.append("https://media.ccc.de" + item.attrs['href'])
len(links)
144
links[0]
'https://media.ccc.de/v/33c3-8428-33c3_closing_ceremony'
Let's download each link content:
contents = {}
for link in links:
r = requests.get(link)
contents[link] = r
Let's check that each request went fine:
all([item.status_code == 200 for item in contents.values()])
True
Now, let's extract the number of views from each webpage.
def metadata(content):
"Returns metadata from url content."
soup = BeautifulSoup(content, 'html.parser')
tag = soup.find('ul', class_='metadata')
meta = dict(zip(["duration", "date", "views", "description"],
[item.text.strip() for item in tag.find_all('li')]))
meta['description'] = str(tag.find('a'))
meta['views'] = int(meta['views'])
meta['title'] = soup.title.text.replace('\n', '')
return meta
first = list(contents.values())[0].text
metadata(first)
{'date': '2016-12-28', 'description': '<a href="https://fahrplan.events.ccc.de/congress/2016/Fahrplan/events/8293.html">\nfahrplan.events.ccc.de\n</a>', 'duration': '61 min', 'title': 'C3TV -Netzpolitik in Österreich', 'views': 813}
Let's build a dataframe with this data:
import pandas as pd
all_metadata = []
for item in contents:
meta = metadata(contents[item].text)
meta['video_link'] = '<a href="{}">video link</a>'.format(item)
all_metadata.append(meta)
df = pd.DataFrame(all_metadata)
df = df.set_index(df.pop('title'))
df.columns
Index(['date', 'description', 'duration', 'video_link', 'views'], dtype='object')
df['description'] = df.description.apply(lambda s: s.replace('\n', ''))
df.head()
date | description | duration | video_link | views | |
---|---|---|---|---|---|
title | |||||
C3TV -Netzpolitik in Österreich | 2016-12-28 | <a href="https://fahrplan.events.ccc.de/congress/2016/Fahrplan/events/8293.html">fahrplan.events.ccc.de</a> | 61 min | <a href="https://media.ccc.de/v/33c3-8293-netzpolitik_in_osterreich">video link</a> | 813 |
C3TV -Predicting and Abusing WPA2/802.11 Group Keys | 2016-12-27 | <a href="https://fahrplan.events.ccc.de/congress/2016/Fahrplan/events/8195.html">fahrplan.events.ccc.de</a> | 60 min | <a href="https://media.ccc.de/v/33c3-8195-predicting_and_abusing_wpa2_802_11_group_keys">video link</a> | 708 |
C3TV -Formal Verification of Verilog HDL with Yosys-SMTBMC | 2016-12-28 | <a href="https://fahrplan.events.ccc.de/congress/2016/Fahrplan/events/7922.html">fahrplan.events.ccc.de</a> | 52 min | <a href="https://media.ccc.de/v/33c3-7922-formal_verification_of_verilog_hdl_with_yosys-smtbmc">video link</a> | 235 |
C3TV -The Zcash anonymous cryptocurrency | 2016-12-30 | <a href="https://fahrplan.events.ccc.de/congress/2016/Fahrplan/events/8330.html">fahrplan.events.ccc.de</a> | 31 min | <a href="https://media.ccc.de/v/33c3-8330-the_zcash_anonymous_cryptocurrency">video link</a> | 611 |
C3TV -The Global Assassination Grid | 2016-12-27 | <a href="https://fahrplan.events.ccc.de/congress/2016/Fahrplan/events/8425.html">fahrplan.events.ccc.de</a> | 63 min | <a href="https://media.ccc.de/v/33c3-8425-the_global_assassination_grid">video link</a> | 4498 |
Let's now export this to HTML:
pd.set_option('display.max_colwidth', -1)
from IPython.display import HTML
HTML(df.sort_values(by='views', ascending=False).to_html(escape=False))
date | description | duration | video_link | views | |
---|---|---|---|---|---|
title | |||||
C3TV -Console Hacking 2016 | 2016-12-28 | fahrplan.events.ccc.de | 53 min | video link | 21020 |
C3TV -Fnord-Jahresrückblick | 2016-12-29 | fahrplan.events.ccc.de | 99 min | video link | 14297 |
C3TV -Shut Up and Take My Money! | 2016-12-27 | fahrplan.events.ccc.de | 30 min | video link | 12451 |
C3TV -SpiegelMining – Reverse Engineering von Spiegel-Online | 2016-12-29 | fahrplan.events.ccc.de | 58 min | video link | 12162 |
C3TV -Where in the World Is Carmen Sandiego? | 2016-12-28 | fahrplan.events.ccc.de | 59 min | video link | 8630 |
C3TV -How Do I Crack Satellite and Cable Pay TV? | 2016-12-27 | fahrplan.events.ccc.de | 62 min | video link | 7880 |
C3TV -Copywrongs 2.0 | 2016-12-28 | fahrplan.events.ccc.de | 61 min | video link | 7042 |
C3TV -Nintendo Hacking 2016 | 2016-12-27 | fahrplan.events.ccc.de | 61 min | video link | 6073 |
C3TV -33C3 Opening Ceremony | 2016-12-27 | fahrplan.events.ccc.de | 15 min | video link | 6005 |
C3TV -Die Sprache der Populisten | 2016-12-28 | fahrplan.events.ccc.de | 59 min | video link | 5884 |
C3TV -Security Nightmares 0x11 | 2016-12-30 | fahrplan.events.ccc.de | 70 min | video link | 5877 |
C3TV -The Untold Story of Edward Snowden’s Escape from Hong Kong | 2016-12-29 | fahrplan.events.ccc.de | 65 min | video link | 5602 |
C3TV -Dieselgate – A year later | 2016-12-28 | fahrplan.events.ccc.de | 40 min | video link | 4977 |
C3TV -Nicht öffentlich. | 2016-12-28 | fahrplan.events.ccc.de | 65 min | video link | 4766 |
C3TV -Check Your Police Record! | 2016-12-27 | fahrplan.events.ccc.de | 31 min | video link | 4594 |
C3TV -The Global Assassination Grid | 2016-12-27 | fahrplan.events.ccc.de | 63 min | video link | 4498 |
C3TV -Machine Dreams | 2016-12-29 | fahrplan.events.ccc.de | 62 min | video link | 4201 |
C3TV -Methodisch inkorrekt! | 2016-12-30 | fahrplan.events.ccc.de | 113 min | video link | 4192 |
C3TV -33C3 Closing Ceremony | 2016-12-30 | fahrplan.events.ccc.de | 46 min | video link | 3955 |
C3TV -A look into the Mobile Messaging Black Box | 2016-12-28 | fahrplan.events.ccc.de | 63 min | video link | 3879 |
C3TV -Bonsai Kitten waren mir lieber - Rechte Falschmeldungen in sozialen Netzwerken | 2016-12-27 | fahrplan.events.ccc.de | 34 min | video link | 3649 |
C3TV -Gone in 60 Milliseconds | 2016-12-28 | fahrplan.events.ccc.de | 33 min | video link | 3584 |
C3TV -33C3 Infrastructure Review | 2016-12-30 | fahrplan.events.ccc.de | 62 min | video link | 3099 |
C3TV -Software Defined Emissions | 2016-12-27 | fahrplan.events.ccc.de | 60 min | video link | 3015 |
C3TV -3 Years After Snowden: Is Germany fighting State Surveillance? | 2016-12-28 | fahrplan.events.ccc.de | 64 min | video link | 2868 |
C3TV -Es sind die kleinen Dinge im Leben | 2016-12-28 | fahrplan.events.ccc.de | 32 min | video link | 2859 |
C3TV -Eine kleine Geschichte der Parlamentsschlägerei | 2016-12-29 | fahrplan.events.ccc.de | 26 min | video link | 2853 |
C3TV -The Ultimate Game Boy Talk | 2016-12-30 | fahrplan.events.ccc.de | 61 min | video link | 2826 |
C3TV -Build your own NSA | 2016-12-29 | fahrplan.events.ccc.de | 60 min | video link | 2758 |
C3TV -Make Wi-Fi fast again | 2016-12-27 | fahrplan.events.ccc.de | 57 min | video link | 2707 |
C3TV -Der 33 Jahrerückblick | 2016-12-28 | fahrplan.events.ccc.de | 26 min | video link | 2527 |
C3TV -Bootstraping a slightly more secure laptop | 2016-12-27 | fahrplan.events.ccc.de | 47 min | video link | 2221 |
C3TV -Einführung zu Blockchains | 2016-12-28 | fahrplan.events.ccc.de | 61 min | video link | 2215 |
C3TV -Exploiting PHP7 unserialize | 2016-12-27 | fahrplan.events.ccc.de | 44 min | video link | 2181 |
C3TV -The 12 Networking Truths | 2016-12-30 | fahrplan.events.ccc.de | 32 min | video link | 2172 |
C3TV -Deploying TLS 1.3: the great, the good and the bad | 2016-12-28 | fahrplan.events.ccc.de | 61 min | video link | 2120 |
C3TV -Kampf dem Abmahnunwesen | 2016-12-27 | fahrplan.events.ccc.de | 33 min | video link | 2103 |
C3TV -The Transhumanist Paradox | 2016-12-30 | fahrplan.events.ccc.de | 33 min | video link | 2094 |
C3TV -A Data Point Walks Into a Bar | 2016-12-27 | fahrplan.events.ccc.de | 32 min | video link | 2053 |
C3TV -Hochsicherheits-Generalschlüssel Marke Eigenbau | 2016-12-29 | fahrplan.events.ccc.de | 64 min | video link | 2004 |
C3TV -Shining some light on the Amazon Dash button | 2016-12-29 | fahrplan.events.ccc.de | 53 min | video link | 1930 |
C3TV -Hacking the World | 2016-12-28 | fahrplan.events.ccc.de | 26 min | video link | 1927 |
C3TV -Lockpicking in the IoT | 2016-12-28 | fahrplan.events.ccc.de | 60 min | video link | 1835 |
C3TV -Surveilling the surveillers | 2016-12-30 | fahrplan.events.ccc.de | 53 min | video link | 1775 |
C3TV -What could possibly go wrong with |
2016-12-27 | fahrplan.events.ccc.de | 55 min | video link | 1746 |
C3TV -Dissecting modern (3G/4G) cellular modems | 2016-12-30 | fahrplan.events.ccc.de | 58 min | video link | 1686 |
C3TV -Law Enforcement Are Hacking the Planet | 2016-12-27 | fahrplan.events.ccc.de | 31 min | video link | 1654 |
C3TV -Durchmarsch von Rechts | 2016-12-29 | fahrplan.events.ccc.de | 60 min | video link | 1646 |
C3TV -Woolim – Lifting the Fog on DPRK’s Latest Tablet PC | 2016-12-28 | fahrplan.events.ccc.de | 63 min | video link | 1642 |
C3TV -ATMs how to break them to stop the fraud | 2016-12-29 | fahrplan.events.ccc.de | 54 min | video link | 1636 |
C3TV -Recount 2016: An Uninvited Security Audit of the U.S. Presidential Election | 2016-12-28 | fahrplan.events.ccc.de | 60 min | video link | 1567 |
C3TV -Von Kaffeeriechern, Abtrittanbietern und Fischbeinreißern | 2016-12-28 | fahrplan.events.ccc.de | 27 min | video link | 1551 |
C3TV -Routerzwang und Funkabschottung | 2016-12-27 | fahrplan.events.ccc.de | 33 min | video link | 1531 |
C3TV -International exchange of tax information | 2016-12-28 | fahrplan.events.ccc.de | 28 min | video link | 1514 |
C3TV -Reverse engineering Outernet | 2016-12-27 | fahrplan.events.ccc.de | 52 min | video link | 1498 |
C3TV -How physicists analyze massive data: LHC + brain + ROOT = Higgs | 2016-12-28 | fahrplan.events.ccc.de | 31 min | video link | 1490 |
C3TV -You can -j REJECT but you can not hide: Global scanning of the IPv6 Internet | 2016-12-28 | fahrplan.events.ccc.de | 46 min | video link | 1463 |
C3TV -Everything you always wanted to know about Certificate Transparency | 2016-12-27 | fahrplan.events.ccc.de | 60 min | video link | 1427 |
C3TV -Berechnete Welt | 2016-12-29 | fahrplan.events.ccc.de | 31 min | video link | 1418 |
C3TV -Talking Behind Your Back | 2016-12-29 | fahrplan.events.ccc.de | 59 min | video link | 1414 |
C3TV -radare demystified | 2016-12-29 | fahrplan.events.ccc.de | 62 min | video link | 1383 |
C3TV -Retail Surveillance / Retail Countersurveillance | 2016-12-30 | fahrplan.events.ccc.de | 33 min | video link | 1360 |
C3TV -A world without blockchain | 2016-12-28 | fahrplan.events.ccc.de | 42 min | video link | 1347 |
C3TV -Irren ist staatlich | 2016-12-30 | fahrplan.events.ccc.de | 32 min | video link | 1344 |
C3TV -Geolocation methods in mobile networks | 2016-12-27 | fahrplan.events.ccc.de | 30 min | video link | 1334 |
C3TV -The Fight for Encryption in 2016 | 2016-12-27 | fahrplan.events.ccc.de | 60 min | video link | 1310 |
C3TV -Zwischen Technikbegeisterung und kritischer Reflexion: Chaos macht Schule | 2016-12-27 | fahrplan.events.ccc.de | 62 min | video link | 1303 |
C3TV -Pegasus internals | 2016-12-27 | fahrplan.events.ccc.de | 29 min | video link | 1300 |
C3TV -The Moon and European Space Exploration | 2016-12-30 | fahrplan.events.ccc.de | 32 min | video link | 1284 |
C3TV -The Universe Is, Like, Seriously Huge | 2016-12-30 | fahrplan.events.ccc.de | 32 min | video link | 1234 |
C3TV -Dissecting HDMI | 2016-12-29 | fahrplan.events.ccc.de | 63 min | video link | 1219 |
C3TV -The Clash of Digitalizations | 2016-12-28 | fahrplan.events.ccc.de | 31 min | video link | 1162 |
C3TV -In Search of Evidence-Based IT-Security | 2016-12-28 | fahrplan.events.ccc.de | 30 min | video link | 1136 |
C3TV -Visiting The Bear Den | 2016-12-28 | fahrplan.events.ccc.de | 59 min | video link | 1125 |
C3TV -Building a high throughput low-latency PCIe based SDR | 2016-12-27 | fahrplan.events.ccc.de | 60 min | video link | 1099 |
C3TV -Downgrading iOS: From past to present | 2016-12-28 | fahrplan.events.ccc.de | 52 min | video link | 1081 |
C3TV -A New Dark Age | 2016-12-29 | fahrplan.events.ccc.de | 57 min | video link | 1045 |
C3TV -An Elevator to the Moon (and back) | 2016-12-29 | fahrplan.events.ccc.de | 61 min | video link | 1026 |
C3TV -Intercoms Hacking | 2016-12-29 | fahrplan.events.ccc.de | 40 min | video link | 1019 |
C3TV -No USB? No problem. | 2016-12-28 | fahrplan.events.ccc.de | 57 min | video link | 1005 |
C3TV -Untrusting the CPU | 2016-12-27 | fahrplan.events.ccc.de | 61 min | video link | 998 |
C3TV -Corporate surveillance, digital tracking, big data & privacy | 2016-12-30 | fahrplan.events.ccc.de | 62 min | video link | 996 |
C3TV -Warum in die Ferne schweifen, wenn das Ausland liegt so nah? | 2016-12-30 | fahrplan.events.ccc.de | 60 min | video link | 990 |
C3TV -What's It Doing Now? | 2016-12-28 | fahrplan.events.ccc.de | 32 min | video link | 971 |
C3TV -Interplanetary Colonization | 2016-12-30 | fahrplan.events.ccc.de | 33 min | video link | 966 |
C3TV -Million Dollar Dissidents and the Rest of Us | 2016-12-29 | fahrplan.events.ccc.de | 53 min | video link | 961 |
C3TV -Anthropology for kids - What is privacy? | 2016-12-27 | fahrplan.events.ccc.de | 25 min | video link | 941 |
C3TV -Privatisierung der Rechtsdurchsetzung | 2016-12-30 | fahrplan.events.ccc.de | 36 min | video link | 939 |
C3TV -A Story of Discrimination and Unfairness | 2016-12-27 | fahrplan.events.ccc.de | 34 min | video link | 916 |
C3TV -Genetic Codes and what they tell us – and everyone else | 2016-12-30 | fahrplan.events.ccc.de | 32 min | video link | 864 |
C3TV -Stopping law enforcement hacking | 2016-12-28 | fahrplan.events.ccc.de | 64 min | video link | 856 |
C3TV -The DROWN Attack | 2016-12-27 | fahrplan.events.ccc.de | 55 min | video link | 853 |
C3TV -On Smart Cities, Smart Energy, And Dumb Security | 2016-12-30 | fahrplan.events.ccc.de | 45 min | video link | 834 |
C3TV -Netzpolitik in Österreich | 2016-12-28 | fahrplan.events.ccc.de | 61 min | video link | 813 |
C3TV -Hacking Reality | 2016-12-28 | fahrplan.events.ccc.de | 60 min | video link | 798 |
C3TV -Eavesdropping on the Dark Cosmos | 2016-12-30 | fahrplan.events.ccc.de | 33 min | video link | 789 |
C3TV -Hebocon | 2016-12-30 | fahrplan.events.ccc.de | 23 min | video link | 786 |
C3TV -Searchwing - Mit Drohnen leben retten | 2016-12-28 | fahrplan.events.ccc.de | 61 min | video link | 771 |
C3TV -Beyond Virtual and Augmented Reality | 2016-12-30 | fahrplan.events.ccc.de | 33 min | video link | 770 |
C3TV -The High Priests of the Digital Age | 2016-12-30 | fahrplan.events.ccc.de | 32 min | video link | 768 |
C3TV -Building Custom Pinball Machines | 2016-12-29 | fahrplan.events.ccc.de | 63 min | video link | 768 |
C3TV -How do we know our PRNGs work properly? | 2016-12-29 | fahrplan.events.ccc.de | 58 min | video link | 738 |
C3TV -Wheel of Fortune | 2016-12-28 | fahrplan.events.ccc.de | 36 min | video link | 727 |
C3TV -Decoding the LoRa PHY | 2016-12-29 | fahrplan.events.ccc.de | 64 min | video link | 725 |
C3TV -Make the Internet Neutral Again | 2016-12-27 | fahrplan.events.ccc.de | 31 min | video link | 719 |
C3TV -On the Security and Privacy of Modern Single Sign-On in the Web | 2016-12-28 | fahrplan.events.ccc.de | 64 min | video link | 719 |
C3TV -The woman behind your WiFi | 2016-12-29 | fahrplan.events.ccc.de | 31 min | video link | 716 |
C3TV -Predicting and Abusing WPA2/802.11 Group Keys | 2016-12-27 | fahrplan.events.ccc.de | 60 min | video link | 708 |
C3TV -Tapping into the core | 2016-12-28 | fahrplan.events.ccc.de | 31 min | video link | 691 |
C3TV -Lightning Talks Day 2 | 2016-12-28 | fahrplan.events.ccc.de | 120 min | video link | 677 |
C3TV -Keys of Fury | 2016-12-27 | fahrplan.events.ccc.de | 25 min | video link | 643 |
C3TV -Virtual Secure Boot | 2016-12-30 | fahrplan.events.ccc.de | 51 min | video link | 636 |
C3TV -The Zcash anonymous cryptocurrency | 2016-12-30 | fahrplan.events.ccc.de | 31 min | video link | 611 |
C3TV -Von Alpakas, Hasenbären und Einhörnern – Über Anerkennungskultur | 2016-12-30 | fahrplan.events.ccc.de | 32 min | video link | 588 |
C3TV -The Nibbletronic | 2016-12-27 | fahrplan.events.ccc.de | 20 min | video link | 563 |
C3TV -Netzpolitik in der Schweiz 2016 | 2016-12-28 | fahrplan.events.ccc.de | 62 min | video link | 544 |
C3TV -No Love for the US Gov. | 2016-12-29 | fahrplan.events.ccc.de | 53 min | video link | 544 |
C3TV -Lasers in the sky (with asteroids) | 2016-12-29 | fahrplan.events.ccc.de | 35 min | video link | 536 |
C3TV -Understanding the Snooper’s Charter | 2016-12-30 | fahrplan.events.ccc.de | 36 min | video link | 535 |
C3TV -Datenschutzgrundverordnung: Rechte für Menschen, Pflichten für Firmen & Chancen für uns | 2016-12-30 | fahrplan.events.ccc.de | 31 min | video link | 514 |
C3TV -Space making/space shaping | 2016-12-28 | fahrplan.events.ccc.de | 33 min | video link | 509 |
C3TV -Edible Soft Robotics | 2016-12-30 | fahrplan.events.ccc.de | 29 min | video link | 503 |
C3TV -Lightning Talks Day 4 | 2016-12-30 | fahrplan.events.ccc.de | 109 min | video link | 503 |
C3TV -Rebel Cities | 2016-12-30 | fahrplan.events.ccc.de | 33 min | video link | 502 |
C3TV -Welcome to the Anthropocene? | 2016-12-28 | fahrplan.events.ccc.de | 28 min | video link | 502 |
C3TV -Technologien für und wider Digitale Souveränität | 2016-12-29 | fahrplan.events.ccc.de | 60 min | video link | 501 |
C3TV -PUFs, protection, privacy, PRNGs | 2016-12-30 | fahrplan.events.ccc.de | 31 min | video link | 493 |
C3TV -Haft für Whistleblower? | 2016-12-28 | fahrplan.events.ccc.de | 34 min | video link | 470 |
C3TV -Code BROWN in the Air | 2016-12-29 | fahrplan.events.ccc.de | 37 min | video link | 464 |
C3TV -Saving the World with (Vegan) Science | 2016-12-30 | fahrplan.events.ccc.de | 31 min | video link | 435 |
C3TV -What We Can Learn about Creativity from 3D Printing | 2016-12-30 | fahrplan.events.ccc.de | 28 min | video link | 426 |
C3TV -From Server Farm to Data Table | 2016-12-30 | fahrplan.events.ccc.de | 32 min | video link | 425 |
C3TV -SpinalHDL : An alternative hardware description language | 2016-12-29 | fahrplan.events.ccc.de | 45 min | video link | 422 |
C3TV -Lightning Talks Day 3 | 2016-12-29 | fahrplan.events.ccc.de | 122 min | video link | 405 |
C3TV -Syrian Archive | 2016-12-28 | fahrplan.events.ccc.de | 30 min | video link | 350 |
C3TV -Memory Deduplication: The Curse that Keeps on Giving | 2016-12-29 | fahrplan.events.ccc.de | 59 min | video link | 322 |
C3TV -Ethics in the data society | 2016-12-30 | fahrplan.events.ccc.de | 35 min | video link | 321 |
C3TV -Making Technology Inclusive Through Papercraft and Sound | 2016-12-29 | fahrplan.events.ccc.de | 56 min | video link | 316 |
C3TV -Community | 2016-12-30 | fahrplan.events.ccc.de | 62 min | video link | 309 |
C3TV -Liberté, Égalité, Fraternité... and privacy ?! | 2016-12-29 | fahrplan.events.ccc.de | 64 min | video link | 254 |
C3TV -Do as I Say not as I Do: Stealth Modification of Programmable Logic Controllers I/O by Pin Control Attack | 2016-12-29 | fahrplan.events.ccc.de | 60 min | video link | 251 |
C3TV -The Economic Consequences of Internet Censorship | 2016-12-30 | fahrplan.events.ccc.de | 31 min | video link | 237 |
C3TV -Formal Verification of Verilog HDL with Yosys-SMTBMC | 2016-12-28 | fahrplan.events.ccc.de | 52 min | video link | 235 |
C3TV -Hacking collective as a laboratory | 2016-12-30 | fahrplan.events.ccc.de | 29 min | video link | 186 |
So there we have it: all talks sorted by views at the time of writing! Have fun watching!