The Making of Jared’s Disc Golf Power Rankings, Part 2

Man I learned a lot doing this project. So… more stuff I learned (or re-learned), when making my disc golf power rankings happen (part 1 is here)…

So I have a list of lists, how do I get the name of list instead of its contents?

So, for some reason I wanted to make a list of lists of the participants of tournaments. And sometimes I wanted to use the names of the list. The example given in the stack overflow question where this was addressed was:

x= ["a","b"]
y= ["c","d"]
z= [x,y]

print z[0] gives the contents of x. How can the output be “x”? The answer… it can’t. Use a dictionary. That’s what I did.

But then how do I iterate through that dictionary?

I just totally forgot this and had to look it up. Thanks for the reminder Real Python…

for key, value in dict.items():
    ...

How to get a value from a Pandas DataFrame and not the index and object type?

To be honest, I’ve had this problem throughout my code. I keep solving it by trial and error. Here’s a helpful stack overflow discussion about it.

How can I see more rows or columns of my dataframe in my jupyter notebook?

When my notebook shows my bigass dataframes it compacts them using “…”. But I want to see more!!! Song Hui Ming showed me how! And you can even change the display area!

# set up display area to show dataframe in jupyter qtconsole

pd.set_option('display.height', 1000)
pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)

df

How do you share a pandas dataframe in your jupyter natebook as an image?

So I wanted to show readers of this amazing blog how I got the numbers I did. To do that is actually pretty easy (I found out from Predictive Hacks). Just import dataframe image as dfi and…

import pandas as pd
import dataframe_image as dfi
 
disc_golf = pd.DataFrame({'A': ['Jared','Sam, 'Ben', 'Tim'],
                   'B':['Awesome','Ok','Mediocre','Talks']})
 
dfi.export(disc_golf, 'disc_golf_facts.png')

Voila! disc_golf_facts is a fact in the same directory as the notebook!

Oh, crap. I have to install dataframe_image in my jupyter notebook first. I just need to…

!pip3 install dataframe_image

right?

NO! DEAR GOD DON’T DO THAT!

Why not?

Just don’t. Ok. I don’t know. Shivangi says so. It’s probably bad for the environment or something. She says do this instead…

import sys
!conda install --yes --prefix {sys.prefix} dataframe_image

It worked for me and no animals were harmed.

How do I save a response from Requests to a text file?

While scraping, I found I kept connecting to websites to get the same info. I wanted to save the response into a nice html file to parse later. Not hard but really helpful. Thanks to JGC on Stack Overflow…

# Request the HTML for this web page:
response = requests.get("https://www.discgolf.org/jared_wins_again")
with open("jared.html", "w") as f:
    f.write(response.text)

There it is. All the html on the local file jared.html.

So I have this html file… how do I make Beautiful Soup read my amazing html file?

CasualDemon’s answer on Stack Overflow has worked for me…

 with open("jared.html") as fp:
     soup = BeautifulSoup(fp, 'html.parser')
    
 names_t = soup.find_all('td', class_ = "views-field views-field-nothing player-name")
 names = []
 for entry in names_t:
     name = entry.get_text()
     names.append(name.strip('\n').strip())

Or whatever you wanna do with your soup…

Even repeatedly parsing these html docs is getting repetitive. How do I save my dataframe to a database? Like an SQLite one?

This is super easy thanks to the legendary Byron Dolon…

Import sqlite. Kinda like…

import sqlite3

You’re welcome. Then create a connection variable (term?) to the database. Mine might be called “disc_golf.sqlite”…

conn = sqlite3.connect('disc_golf.sqlite')

Still with me? Then all you do is shove everything into a table in your database. My table is called “legends.”

df.to_sql(‘Legends’, con=conn, if_exists=’replace’, index=False)

Fill in some parameters and that’s it. For if_exists there are more options. And index shmindex. Check the pandas documentation for details.

So I have this database, how do I make a table into a dataframe?

It’s pretty easy I found out here…

disc_golf_df = pd.read_sql_query("SELECT * FROM Legends", conn)

“Conn” is the same type of variable as above. And bam! disc_golf_df is a dataframe from the database.

This is the best notebook ever! Can I share my jupyter notebook on Github?

Yeah. Duh. Just share the file like any other. Didn’t really need to look that up but did. At… here.

And that’s if for now. I’m learning more every time I play around with this data. Part 3 is probably in the near future…