Skip to content

2020

  • in python
  • 4 min read

Displaying the Album Information

In my last blog post, I was able to pass the folder to Discogs to get a list of albums, and then pick one at random and give me information back about the album picked. That information was returned as a dictionary and now it is time to wire it up to the Chameleon template in Pyramid that will display the HTML.

Here's an example of the JSON:

{'release_title': 'Massey Fucking Hall', 'release_uri': 'https://www.discogs.com/Japandroids-Massey-Fucking-Hall/release/16118824', 'artist_name': 'Japandroids', 'artist_url': 'https://api.discogs.com/artists/1473872', 'release_date': '2020-06-26', 'discogs_main_id': 1829048, 'discogs_main_url': 'https://api.discogs.com/masters/1829048', 'main_release_date': 2020, 'release_image_uri': 'https://img.discogs.com/BVmRNi_A5_Vm8X0ntzUDh8figvE=/fit-in/300x300/filters:strip_icc():format(jpeg):mode_rgb():quality(90)/discogs-images/R-16118824-1604177632-7503.jpeg.jpg', 'genres': ['Rock']}
In the controller that handles the routing for the page I have:

```@view_config(route_name="play", renderer="silversaucer:templates/play/play.pt") def play(_):

album_release_id = RandomRecordService.get_folder_count(2162484)
print(album_release_id)
release_data = RandomRecordService.get_album_data(album_release_id)
print(release_data)

return {"release_info": release_data}

It was way easier than expected. (And if you know me, you know that the random record picked above is perfect, as I'm a huge Japandroids fan.) The first thing I did was connect the image URI that the Discogs API passes in the Chameleon template: ${release_info.release_title} That worked, which was awesome. The whole image is shown, and I don't have to worry about parsing any of the URL. I quickly added the values to show the *Artist*, *Album*, and *Release Date* information:

Artist: ${release_info.artist_name}

Album: ${release_info.release_title}

Released: ${release_info.release_date}

\ 

It looks like this:

![Japandroids - Massey Fucking Hall](japandroids.png)

Looking at Debbie Gibson's *Electric Youth*, the *Genre* key in the dictionary returns a list:
'genres': ['Electronic', 'Pop'] 
With some trial and error I was able to display the list, but it showed up as code, looking like:
`['Electronic', 'Pop']`.  No one wants to see that.  The challenge is that Chameleon templates use something like a 
shorthand for Python code.  I revisited two of my former projects where I would iterate over a list and show the 
results, but after lots of trial and error I felt like I was going backwards.  Some more search engine queries and 
Stack Overflow searching I did what I always do when I get stuck:  I asked my wife.  It took her about 15 minutes to 
figure out how to loop over the list and show it in bullets:

  • ``` I was so close - she pointed out the mistake I made with having two `tal:repeat` methods in the template. That’s what’s hard about the trial and error method of finding the solution, especially as I wasn’t writing down the methods that kind of worked. (Oops. Usually I have a note going in Bear to capture things like this, but sometimes you get in the zone and just keep trying things as you get closer. Documentation is good!) Lastly, I changed the image size of the album image returned from Discogs, shrinking it from 400x400 to 300x300 and it seems to fit in the Bootstrap container better. You can see the difference with Debbie Gibson's *Electric Youth* below at 400x400 and Japandroids at 300x300 above. ![Debbie Gibson - Electric Youth](debbie-gibson.png) Now I have lots of cleanup to do. I need to turn the text into links, for example if you click on the artist, Japandroids, it takes you to their page on Discogs. I also need to fix how it justifies and draws the bullets. I think almost all of this is done in CSS and I don't know CSS at all... (I had to cheat to make the text color white to show up on the page already.) Random thoughts and musings: * Every time the page loads, it refreshes the API call. So if you click a link to the artist page on Discogs, for example, and then click the back button, that album data has been replaced. Not sure how to work around that, though I have some ideas. Probably a good question in IRC with the Pyramid team. * I like Chameleon templates. Not only is it the first template language I've learned, I've looked at Jinja templates and they look harder to use. They're used much more widely and that seems to lead to better documentation, but I don't have plans to change from Chameleon. * I had more thoughts, but I'm finishing up this blog post almost a week later as my internet went out. * The *Release Date* returned from Discogs has the same problem I've already talked about for the future "On This Day" feature I want to add. It could be just the year or the full release date, you never know what you're going to get. Look at the two screenshots above! I don't know if I want to add all the functionality to return the "correct" release date yet. I'm worried about how long the API calls take to load the page. (Even if it's just for me and I know how long it takes, I don't want it taking long.)
    • in python
    • 2 min read

    Iterating Through the Folder Dictionary

    After taking a break and coming back to it, I finally was able to iterate through the dictionary containing the folders I’ve sorted my records in to on Discogs.

    Dictionaries are a basic object in Python, but my struggles show that to be good - at anything - takes practice. And I haven’t been practicing. Thank goodness for Stack Overflow!

    I embedded all kinds of print statements trying to figure out how to play with the JSON from Discogs.

    After lots of trial and error, Google searches and Stack Overflow searches, I’ve got it.

    After making the API call to Discogs, we get the JSON back:

    record_json = response.json()

json_data = record_json
json_folders = json_data[”folders”]

    json_folders is a list, so we add the last line above. Now I had to make a for loop to iterate through that list - which is a list of dictionaries.

    The if statement looks to see if folder - which we passed from the controller when the user clicked which kind of album they want a random answer to, matches the folder ID found in the JSON.

    for get_folder_id in json_folders:
    print(json_folders)
    print(len(json_folders))

    if get_folder_id[”id”] == folder:

        lp_count = get_folder_id[”count”]
        print(”Folder passed = “, folder, ”LP Count is :”, lp_count)
        # folder_test = 2162484
        print(folder, lp_count)

    Voila! From there, I get a random number between one and the count found in the JSON. That number is then passed to the pagination request to get the album ID.

    Next up: Either fix the pagination method to reduce the if / else statement or move on to getting the album release information. I also need to remove a ton of print statements and clean up and remove all the methods I don’t have to use after refactoring this. Progress!

    • in python
    • 2 min read

    Silver Saucer Random Play Progress

    I’ve made some good progress over the last few days on being able to pick an album at random out of a given folder and get information about it.

    All of the work has focused on the back-end. Once I know I’m getting the right data out of Discogs, I’ll hook up the front-end to display it, but for now, this has been a little more complex than I expected. (It always is, isn’t it?)

    One thing that’s jumped out at me was I don’t understand how people do TDD (Test Driven Development). The amount of trial and error I do, even on “simple” things like building the right URL for the API call is ridiculous. I know it’s because I’m a novice at Python and I don’t do this for a living all day every day, but I have a lot of respect for those that do TDD. One of my original goals was to do this in TDD, but I have my excuses.

    You can tell I’m a novice by how procedural my code is. Do this, then do that. It was working well in a very procedural manner, so yesterday I refactored it all. Yup, I ripped it up and started over. Well, not from scratch. I could visualize in my head how to make the code more Pythonic and I was able to re-use some of the code, but most of it was refactored in the play_service. A good example is going back to the folder example. I had hard coded each folder ID and each folder had it’s own method to get a random album release ID and its data.

    I’ve been able to take the folder selected by the user (LP, EP or single) and just pass variable that to the method in play_service.py. This allowed me to use the same method to get the release ID information and the method doesn’t care if it’s a single or full album.

    I still need to fix the loop I’m iterating through in the JSON, but I’m really close. This is one of the downsides to being a hobbyist - so much I don’t remember, including basic things, like how to iterate through a dictionary.

    Practice makes perfect and I needs lots of practice. I told myself I need to think of some more projects to work on, but one project at a time. In the meantime, I should sign up for something like Python Morsels and keep up with it.

    This weekend I’m hoping to fix looping over the list in the dictionary and also fix API call that deals with the pagination of results from Discogs. You can see what a mess it is in the play_service.py file and I know there has to be a better way to programmatically determine the pagination than a long if / else statement. (And if I ever get over a 1000 records it will break! Or my wife will break me when I have that many…)

    • in python
    • 3 min read

    Discogs Authentication

    I was ready this morning to write a new blog post chronicling all the problems I had yesterday trying to authenticate against the Discogs API. After finding the motivation to sit down and code for a few hours, I ended the day with almost nothing done. I could access the parts of my collection that were public, but the parts that required authentication I couldn’t figure out.

    I sat down with my copy of coffee this morning, re-read the Discogs API authentication documentation, and voila! It just clicked. Sometimes you just need a good night’s sleep.

    You have two options for authentication when building an app with the Discogs API: full Oauth that allows users to login and validate their login, or pass a user token as part of the API query. Since my app is just going to use my Discogs information, the user token is perfect. Both authentication options give you options for a higher rate limit and the ability to return an image - and this last one will be important later.

    I thought I would tackle what would be the easiest part first - have the app pick a random album for me to play.

    It was a lot harder than I expected, but I’ve made good progress.

    Now that I could get back details of my collection via authentication, I had Discogs return a list of all the folders I’ve organized my music into. I put each of the following into its own folder, as when I hit the random button, I don’t want it to pick a 45 or 7” record with only two songs on it. I have 10 folders, with 0 and 9 included by default:

    • 0 - all
    • 1 - 10” (10” records, mostly EPs)
    • 2 - 12” (12” singles or EPs)
    • 3 - 7” (7” or 45 rpm singles)
    • 4 - Autographed (currently empty, but you can have a release in more than one folder)
    • 5 - Cassette
    • 6 - CD
    • 7 - Digital
    • 8 - LP
    • 9 - Uncategorized

    The JSON returned looks like this:

    {
    "folders": [
        {
            "id": 0,
            "name": "All",
            "count": 799,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/0"
        },
        {
            "id": 2162486,
            "name": "10\"",
            "count": 19,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2162486"
        },
        {
            "id": 2198941,
            "name": "12\"",
            "count": 30,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2198941"
        },
        {
            "id": 2162483,
            "name": "7\"",
            "count": 76,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2162483"
        },
        {
            "id": 2162485,
            "name": "Autographed",
            "count": 0,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2162485"
        },
        {
            "id": 2162487,
            "name": "Cassette",
            "count": 1,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2162487"
        },
        {
            "id": 2162488,
            "name": "CD",
            "count": 6,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2162488"
        },
        {
            "id": 2198943,
            "name": "Digital",
            "count": 1,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2198943"
        },
        {
            "id": 2162484,
            "name": "LP",
            "count": 666,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/2162484"
        },
        {
            "id": 1,
            "name": "Uncategorized",
            "count": 0,
            "resource_url": "https://api.discogs.com/users/prcutler/collection/folders/1"
        }
    ]
}

    I then got stuck for a while before I realized I should be using the folder IDs in the resource_url and not the JSON number of 0-9.

    That - plus authentication - are challenges for someone like me still learning. But progress is being made. Lots of trial and error with string concatenation to get the URLs and tokens right. The next thing to solve for: Discogs only returns 100 items in a list, making you interact with pagination to go deeper. I’ll need to do this to take the random number generated of while album to play to get its data, including the image to show, when a random album is chosen.

    • in python
    • 1 min read

    Pre-Commit Git Hooks

    It was a pleasant surprise recently to get an email from Michael Kennedy at Talk Python. As a previous buyer of one of the Talk Python Everything Bundle which includes a number of different Python training courses, I was eligible for the “Pro” Talk Python podcast subscription, which doesn’t included ads.

    Going through the backlog, I added a bunch of episodes, and the timing of Episode 282: Pre-Commit Framework was perfect. Just as I’m getting back into working on my projects, now is a perfect time to set up Black to help format my code. The documentation was easy to follow as well as other walk throughs on the web, and now every time I check in code, it runs black to format it Pythonically, isort to automatically sort my imports in each of my files, and I’ll be adding flake8 this weekend, too.

    Thanks again for the Pro podcast subscription, Talk Python!

    • in python
    • 4 min read

    The one piece of data Discogs doesn’t have

    As a record collector, Discogs is invaluable. What started as a crowd sourced website to catalog electronic music, it now has a catalog of almost every piece of recorded music. They’ve also added robust e-commerce abilities so it makes buying and selling easy - whether it’s for something popular or a rare record only made in Greece.

    It can take a little work to figure out which record you have, especially if it’s something popular like Fleetwood Mac’s Rumours, which has literally hundreds of different releases, some of which come down to figuring out the engraving on the inside of the record to figure out which pressing plant made it.

    The other thing that makes Discogs great is they have a Developer API available, making it easy to interact with all of the data they have stored programmatically. All of the data is available in JSON and there is also a Python library available, making it even easier.

    The amount of data about each album and each of its variants is amazing. I won’t go through all of it and you can get an overview here , but it includes almost everything you could probably think of off the top of your head.

    Each album release has a “master” release, that represent a set of similar releases. If an album is released this Friday, for example, and comes out on CD, cassette and vinyl, the master release would catalog each of these individually. Each individual one is a “release”. (And here’s to hoping Discogs changes the word master sometime in the near future).

    Here is the master release page for Pearl Jam’s Ten, released in 1991:

    Pearl Jam Ten Master Release

    You can also see there are 251 different versions (or releases) of Ten on different formats. The list is currently sorted by chronological year of release and if we scroll all the way down to 2009, we can find the repress I own on vinyl. You can click through each individual release to see how it’s unique or find that particular version for sale. Clicking through we see:

    Pearl Jam Vinyl Repress Release

    There is one thing I am looking for to build one part of my app: What date was an album released? Neither the master release or the release of the album I own supplies anything but the year, not the full date, which is what I want to know to celebrate an album’s anniversary.

    That’s by design. If you check out Discogs' Submission Guidelines , only the year is required. If you know the actual date, great, if you know the year and month, you can also input that.

    I also found a forum post from a few years ago, which I can’t find now, where the Discogs developers stated they wouldn’t change the release date field to require a full year / month / day and they didn’t want to change the database schema to support that (which I don’t blame them).

    But that doesn’t help me in building a web app that shows which albums were released today in history.

    In addition to my pandemic induced loss of motivation, this has blocked me from moving forward. I have a couple of potential ideas to move forward.

    The first is interfacing with the MusicBrainz API in addition to Discogs. MusicBrainz is a full open and free music database licensed under Creative Commons. They even link back to Discogs to make it easy to link releases. My thought is if Discogs returns a date field that doesn’t include the month and day, to then query MusicBrainz to see if they have it. I’m a little worried about responsiveness in making two queries like that, plus dealing with two different JSON schemas and matching.

    The second idea may work for the majority of releases, but there is a catch at the end. Doing a random check of albums, when you look at a master release, the very first release chronologically almost always has the full date attached. I could query the API, and slice the list returned and use the date provided in the first entry in the list. But I checked one random album I had that I knew is an obscure release (Maggie’s Dream if you’re curious) and there is no date.

    So I think what I’m going to do - as I need to get going and do something - is to build out and test the second option. This keeps me with just one API to learn in the short term and I can probably build a query pretty easily that will sort all of my releases by chronological order (using the trick above) that will show me how many albums don’t have a full release date. 10%? More? Less? Might as well build it out and get something done!

    • in python
    • 1 min read

    Stupid Pandemic

    The COVID-19 pandemic has changed a lot of things for a lot of people. One of the things it’s done to me is made it harder to focus on my coding projects. (Part of that is I’m very deadline driven, and well, no deadlines on personal projects…)

    Looking back on this year, I’ve actually accomplished a lot, but just not on the coding side. I’m just getting back to it and I have two projects I’m going to make progress on. The first one is my music centric web app built with Pyramid that interacts with my Discogs profile via the Discogs API. The second project involves a Raspberry Pi, but I’m much better at the coding side of that than I am on the hardware side. (And that’s saying something with my novice coding skills).

    • in python
    • 3 min read

    Introducing Silver Saucer

    I’ve started my next Python project: Silver Saucer, which will be hosted at silversaucer.com. I originally bought the domain about ten years ago when I was thinking of going into business for myself, which never happened and I hung onto the domain name because I liked it. It was inspired by Neil Gaiman’s poem, The Day The Saucers Came, which I also have a framed art print of:

    {{< figure src="day-the-saucers-came.jpg" title="The Day the Saucers Came" link="day-the-saucers-came-original.jpeg" >}}

    Once upon a time, I was also a big fan of The X-Files, so it seems fitting.

    With the pandemic here, I’ve had a little time to work on some long dormant projects, this being one of them. I’m stuck on another Python project (more because of the hardware than the coding), so I thought I’d find some time to work on this.

    I have a few different goals of things I want to learn and practice:

    Pyramid

    It’s my third project using the Pyramid web framework. I don’t have an urge to learn Flask or another framework - I would like to get better at Pyramid, which I know a little. I also really like the Pyramid community. The few times I’ve become stuck and asked for help in the Pyramid IRC channel, they’ve been both welcoming and helpful. My first two Pyramid projects were based on the first training Talk Python offered for Pyramid, which used a package called pyramid_handlers which is no longer the recommend way to build a web app in Pyramid. I’m doing it the recommended way this time, using a class based approach.

    Bootstrap and CSS

    I know a little of HTML, enough to get by. But CSS and Bootstrap, not so much. I’ve already integrated a Bootstrap theme and tweaked it where it’s almost working, but I’m just hacking at it - I don’t really know what I’m doing and it’s something I want to get better at. I should probably find some good HTML / CSS tutorials and go through those.

    Discogs API Integration

    The goal for Silversaucer.com is to integrate with the Discogs API). Discogs.com is a website that lets you catalog your record collection and also includes community features and a marketplace where you can buy and sell records (or CDs or almost any kind of media). There are two things I want to build using the Discogs API:

    Play a random record

    I know that Discogs already has a feature on their website where you can have it randomly choose an item in your collection and if you shake the mobile app it will also show you a random item in your collection. I want to take that to the next level and sort by type (record, 45, CD, etc.).

    On this day

    The second things I want to build is a page that shows all records released for today’s date. This one is going to be more complicated and I’ll share more in a separate blog post.

    Testing

    I’ve blogged about it before, but I struggle to learn pytest. I’m going to continue to try and learn more about software testing, starting with this app.

    Infrastructure

    I’ve already hooked up Silver Saucer to Azure Pipelines to automatically do continuous integration. I want to integrate pytest and code coverage next. After that I may look into continuous delivery, but there’s is a lot about Azure that I don’t know.

    Plenty of things to learn and keep me busy during these interesting times. Here's a teaser for making it to the end:

    {{< figure src="silversaucer.png" title="Silver Saucer screenshot" link="silversaucer.png" >}}

    • in python
    • 2 min read

    Another Reason to Love Open Source

    I've been using free and open source software (FOSS) for over twenty years and I've involved in with different FOSS projects for ten to fifteen years (on and off). Thare are so many different aspects of the open source community that I love (community being a big one to start with) and here's another example.

    I'm trying to build a new website using Python and the Pyramid framework. I want to interact with APIs on the Discogs website. Discogs is a website and community that allows you to catalog your music and also buy and sell music. For me, this is all about my record collection. I'll be posting more about what I want to do, but right now I'm just trying to learn what my options are with the Discogs API and how to connect and interact with their API. I'm struggling a little bit to get authentication working.

    I'm also using Postman, an application that makes it easy to test different APIs and see how they return data. I first learned about Postman through the Talk Python trainings (another reason to love the Talk Python trainings) and it was a core tool in building NFLPool and MLBPool to understand how MySportsFeeds showed sports data to integrate into those two apps.

    If I'm struggling to authenticate and login to the Discogs API, I'm never going to get any results.

    And here is why I love open source: Github user leopuleo has a Discogs-Postman repository where he has taken the time to not only put together almost 60 different ways to use the Discogs API, it also includes documentation and tests!

    Postman Discogs

    Postman Discogs Tests

    This is all licensed under the MIT License, giving you the ability to take it and modify as you need to. leopuleo didn't have to do this, nor did he have to share it, or give it a license that just gives it away and let's you do anything you want with it. This is just one reason why open source is awesome - someone builds something, shares it with the world, and even better, let's you build on top of it.

    Open source and free software rock.

    • in Arcade
    • 2 min read

    RetroPie Update

    I hooked up the Raspberry Pi to test it out. I did an update and it broke the boot up process and now wants the username and password. Think I remember that? Nope! Of course I don’t remember the password. So I downloaded a new RetroPie image which has led me down a rabbit hole of MAME configuration. The buttons on the controller don’t seem to be mapped correctly.

    I did downloaded a new romset from the Internet Archive for MAME 0.209. Doing further research, I tried both the mame4all emulator and libretro. Both had unexpected results. This led me into further research and it looks like I want to use a MAME 0.78 romset and the mame2003 emulator. (This is due to RetroPie using the libretro emulation engine as it's primary emulator). So I downloaded another 50GB from the Internet Archive and I’ll try out those ROMs to see if the buttons map correctly on the knockoff PS3 controller I’m using for testing. It’s interesting that the joystick buttons are different based on what emulator I’m using.

    I did briefly consider not using the Raspberry Pi 3B+ for the arcade cabinet and switching to a PC. Reading some forums and Reddit posts, a PC is always recommended for MAME. All I really want to do is play classic 80s arcade games that should mostly work with the Raspberry Pi. A PC is mostly tempting to add the Dolphin emulator so I can play Super Mario Strikers for the Nintendo Gamecube, but I don’t know if that’s worth the expense or the trouble. The Pi should more than handle the classic arcade games, Sega Genesis and Nintendo games I want to play.

    But doing all the research is fun!