One Year of Python

It was Black Friday of 2015. O’Reilly put on a sale of their programming ebooks and I was finally ready to take the plunge and learn Python. I bought three books:

I then signed up for a Coursera class, Python for Everybody, taught by Dr. Charles Severance and started the class. I was ready to do this. I needed a hobby. I had a problem to solve.

Then real life got in the way. A few months earlier, we started building a new house. In January it was time to sell our house, which meant hours of work. Then in February, we moved.

I put learning Python on the back burner. Before I knew it, it was July, and another six months had gone by. It was now fantasy football season and that was the problem I had to solve. I needed a program that would keep track of all football statistics and standings and automatically calculate each player’s points. It was time.

I re-started the Coursera course and spent the time. I was easily spending twenty hours a week reading the course materials, watching the videos and doing the homework.

I confirmed what I knew about myself: I learn best by doing, not just reading or watching videos. The books I had bought were helpful, but just sitting down and reading them, trying to follow along and do the exercises was difficult. Python for Everybody on Coursera was great.

I finished that and moved on to Python Jumpstart by Building 10 Apps by Michael Kennedy, which I had purchased in early 2016 via a Kickstarter campaign. I’m almost done with that a year after I started this journey.

Learning to code in Python is hard. I don’t have a background in computer science and with some of the concepts that the books and courses teach I just don’t have the base knowledge necessary. This sometimes makes it harder and takes longer to understand the concepts. I’m lucky that my wife has worked professionally as a programmer in multiple languages, including Java and SQL. But I drive her crazy when I ask her questions about concepts I clearly don’t understand. I use the wrong terminology or fail to grasp what I’ve been taught.

I don’t know how much I’ve retained from the classes and books. I’m trying to build my application in parallel with my learning. I’m convinced the only way I’m going to learn is to build something, which is a piece of advice most often found online for people aspiring to learn programming. I’m constantly hitting up Google and Stack Overflow when I get stuck. I’ll copy bits and pieces of code from these search results and I’m always doubting whether I understand what I’m copying. I’ve signed up for multiple newsletters and bookmarked dozens of websites with articles on how to learn, code snippets, programming challenges and more. I’m overwhelmed with the concepts I’m learning and I know I don’t understand, let alone use, these concepts.

But I’m going to keep trying. The only way I’ll learn is by building something. The code will be ugly. It will break. And I’ll keep updating it until it works and as I learn more, I’ll make it more elegant.

Here’s to another year.

Importing Team Data into NFLPool

Last weekend I discovered how to pretty print the five JSON files I get from MySportsFeeds. This was helpful to understand just how much data is nested within each file. I also spent a good chunk of the weekend writing in a notebook. I mostly did some data modeling on what each table in the database should store and what their primary keys would be. I also captured things I need to research and started breaking the project into chunks. As I tweeted out over the weekend:

Monday was a holiday so I did the first four courses of Python Jumpstart. I took a break and went back to the JSON files I had worked with. My goal was to build with what should be the easiest table and pull the team data out. This is a dictionary that includes the team name (Texans), city (Houston), abbreviation (HOU) and id (64). The ID number is supplied in the JSON feed and is unique, so I will use that as the primary key. There will be two more columns in the table for conference and division, but I wanted to deal with that later.

I wrote a for loop to try and pull out each team’s information. I quickly got stuck and nothing was working. At one point, the loop I had written worked, but only pulled out the data for the first ranked team. I showed my wife my code and she pointed out that it wasn’t iterating in a loop.

I was stuck for two nights working on this after dinner. I finally stepped back and modified my pretty print Python program and started breaking down all of the information in the JSON file again. I figured out what was a list and what was a dictionary and what was nested where. (It looks like I didn’t commit this to the git repo, oops! Will have to fix that.)

After doing this last night, I found the list I needed to work with. I then re-wrote my for loop and I was able to iterate through all 16 teams in the AFC:

for afc_team_list in teamlist:

afc_team_name = data["conferenceteamstandings"]["conference"][0]["teamentry"][x]["team"]["Name"]

afc_team_city = data["conferenceteamstandings"]["conference"][0]["teamentry"][x]["team"]["City"]

afc_team_id = data["conferenceteamstandings"]["conference"][0]["teamentry"][x]["team"]["ID"]

afc_team_abbr = data["conferenceteamstandings"]["conference"][0]["teamentry"][x]["team"]["Abbreviation"]

print((afc_team_name),",",(afc_team_city),",",(afc_team_id),",",(afc_team_abbr))

x = x + 1

I then copied and pasted and did it again for the NFC. I did try, unsuccessfully, to modify the conference list – “conference”0 – so I could just write one for loop instead of one for each of the two conferences. But it was working, so I’ll leave it for now. (I’m sure my code is ugly, but hey, I’m just starting).

After that it was all about writing the SQL insert statements to put this into a SQLite3 database. (For now, later it will go into MySQL). That took me a an hour, but at the end, I got it working and was even able to add the conference name to each row.

Next up, I need to take the data in the Division standings JSON file. In it is stored the division name for each division in a conference: AFC/AFC-East. I’ll need to write a for loop to grab it, slice it to remove the “AFC/“ and then stick that in the Division field for each team in the Teams table. I’ll also need to stop dropping and re-creating the table each time I insert data, but it’s working.

Progress!

Building the NFLPool webapp – Starting with JSON

I’m glad I started with the Python for Everybody specialization at Coursera before jumping into Python Jumpstart by building 10 Python Apps by Michael Kennedy. Mr. Kennedy moves fast. I’ve completed the first four apps and it’s good to get a refresher on the information I learned in Python for Everybody.

I also spent part of the weekend sketching in a notebook. I did some brainstorming about the database design I’ll need for NFLPool. I learned one of the bigger differences between MySQL and Postgresql is that MySQL does not have the ability to use foreign keys but MySQL is much faster. The lack of foreign keys may make the design a bit tougher, but more on that later in a different blog post.

I also sketched out some ideas for the functions I’m going to need to write so I’m not writing the same bit of code over and over again. From there, I created a to-do list of things to start working through. I find this whole process of building an app overwhelming. I never thought I’d be using paper and pencil so much, but I’ve found it helpful to break this into smaller chunks and attack them one at a time.

Then I started working on the import process for the JSON. This quickly derailed as I realized just how many stats MySportsFeeds captures from an NFL game. That quickly turned in to writing a JSON pretty print statement so I could see how the five different JSON files nested their dictionaries.

I currently download five JSON files every Tuesday via a cron job with all the statistics. I know my app won’t be ready for the 2016 season and my hope is by having 17 weeks of data, I can re-create the season to test my app to make sure it’s scoring each player correctly as we move through the season week by week. When I download the JSON via curl, it includes all the web headers, such as:

HTTP/1.1 200 OK
Date: Wed, 21 Sep 2016 12:16:07 GMT
Server: Apache-Coyote/1.1
Cache-Control: must-revalidate, no-store, s-maxage=0, max-age=0, private
Access-Control-Allow-Headers: Origin, Content-Type, Accept, Accept-Encoding, Accept-Language, Authorization
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Content-Encoding: gzip
Access-Control-Allow-Methods: GET, OPTIONS
Content-Type: application/json
Set-Cookie: JSESSIONID=B7548F2309747418749B5421282A5E08; Path=/leaguemanager-web/; HttpOnly
Vary: User-Agent
Connection: close
Transfer-Encoding: chunked

And then the JSON starts right after that with curly braces. I was proud of myself as I wrote an if statement to load the file, read the lines, and load the JSON when finding the curly braces. Then I wrote code to first print out all the statistics categories (commented out below) and pretty print all the JSON:

import json
import pprint
import os

#Open the JSON file that includes headers


#Change the name of the file to open to match the query below:
with open('json/20160921-division-team-standings.json') as file:
    alltext = file.readlines()  #Put each line into a list

# division-team-standings.json
for lines in alltext:
    if lines.startswith('{'):
        rawdata = lines
        data = json.loads(rawdata)
#        for stat_categories in data["divisionteamstandings"]["division"][0]["teamentry"][0]["stats"]:
#            pprint.pprint(stat_categories)   #Print all the categories in "stats"
        pprint.pprint(data)  #Print the JSON

I had five files to review and I just manually changed the code to the file I wanted and had a code block for each of the files. I know I probably should have just wrote a function, but I was in the zone. (My code probably isn’t very Pythonic either, but I have to start somewhere on this journey). I also know that when it comes time to build the real app I’ll be loading the JSON across the network and not from a local file, but future Paul gets to deal with that.

I also spent some time playing around with the nflgame and mlbgame Python modules. I need to spend some more time with them and I’ll share some thoughts on those in another blog post.

Next class up: Python Jumpstart by Building 10 Apps

I’ve completed the Python For Everybody course taught by Dr. Charles Severance at the University of Michigan on Coursera. All that’s left is the capstone project to put into practice what I’ve learned, but as I’m doing this to learn Python and not for the official certificate, I’m going to skip it. The course is taught in Python 2.7 and I want to shift to Python 3.x.

Python For Everybody was great. The pace and the exercises were perfect for the class. I wish I had realized sooner that there were additional exercises in the textbook that were not part of the required Coursera class. The fourth class, Python and Databases, was intense. The speed of the class was accelerated with teaching you SQL and how Python connects to databases (SQLlite specifically). The homework was much more simple in Python for Databases compared to the first three sessions. You usually had to only make some minor changes in the SQL syntax to complete the grade.

The two things I’m going to need to focus on to have success in building the two apps I want to build are dictionaries (from importing statistics via JSON) and databases. If I walked away from one thing from the Python for Databases class is that I’m going to need to spend some time with paper and pencil and plan my information architecture and database models if I’m going to be successful.

Next I’m going to start Python Jumpstart by Building 10 Apps by Michael Kennedy of the Talk Python podcast. I supported the Kickstarter earlier this year and am excited now that I hope I have enough of a base understanding of Python to tackle this. This will be taught in Python 3.x (yay!) and I’m hoping now that I have that base knowledge, building these apps along with the tutorials included will give the practice I need to later build a real app. It’s also going to go into a little more detail than what I’ve learned so far on list comprehension (which makes my head hurt), BeautifulSoup for web scraping, and Classes.

I also supported Mr. Kennedy’s next Kickstarter, Python for Entrepreneurs. This also has me excited as the second phase of building my fantasy sports app will be deploying it on the web. The description looks perfect for what I’ll need, in addition to learning the web framework Pyramid:

You will learn to build and design your web app

This course will teach you how to build a data-driven web application in Python.

We will:

• Build our web app with the Pyramid web framework, "the Python web framework that supports your decisions, by artisans for artisans."

Create and connect to our database using SQLAlchemy, the most popular data access layer in Python

• Learn the core elements of web design including CSS and front-end frameworks such as Bootstrap.

Time to get to work.

Web Scraping and Python

I’m flying along in the Coursera course Python for Everybody, from the University of Michigan taught by Dr. Charles Severance. I’ve completed the first two of four courses which give you an introduction to Python.

I’m now on the third course, Using Python to Access Web Data. This and the fourth course focused on databases, are the two key foundations for the web app I want to build. I just finished Chapter 12, which introduces the BeautifulSoup library for scraping web pages. This is going to be huge – I’ll be able to scrape ESPN to find which MLB or NFL teams lead their divisions or leading in the wild card races.

Being on vacation this week, I’ve been able to complete a few chapters and am now a couple weeks ahead of schedule. I’m tempted to pause and see if I can take what I’ve learned with BeautifulSoup and actually write some small Python programs to actually scrape and print the results. It might be good practice to reinforce what I’ve learned.

The next two chapters are key as well. XML and then the one I’m most looking forward to: JSON. I’ve already signed up for a developer account with MySportsFeeds and am receiving JSON data for player stats, teams and conference standings. I’ve spoken in the past with one of their lead developers and they don’t currently keep statistics for wildcard or playoff standings, so I’m going to need to use BeautifulSoup in my app to get those. I’ll also need to make a decision if I’m going to use that JSON data for player stats and query against it myself or just use the nflgame or nfldb libraries that have already been built. The biggest challenge their is that both of those libraries are written in Python 2.7 and I really want to write my apps in Python 3.x.

I know I’m getting ahead of myself. Every time I learn something that will be applicable to the app I want to build and I talk to my wife about it, she tells me to slow down. My mind is always racing with how I can apply what I’m learning and how it will affect the architecture of the app. Some people say the best way to learn a programming language is to build something and learn as you go. I can’t wait to put all this Python learning to practice.