Skip to content

2019

  • in python
  • 2 min read

Azure DevOps Python SQLite Issue - Part 4

It turns out that all the problems I was having for about ten days in trying to learn Azure Pipelines for continuous integration that I briefly touched on in Part 2 had nothing to do with me. There was a bug in Azure’s hosted images of Ubuntu used in the CI pipeline that did not include SQLite, as Python lists it as an optional module.

It’s good to know I’m not crazy and I did set it up right and this answers the question of why my builds would work on Python 3.6 but not Python 3.7. Kudos to Microsoft for the detailed post-mortem outlining the problem, what they did to fix it, and what they’ll do in the future to avoid something like this happening again.

In other news, I was distracted for a week to help a co-worker out with my first hardware project. I wrote a few lines of code to turn on and off a small pump that is controlled by a Raspberry Pi. I even wrote a test for it!

Lastly, I almost have CodeCov.io integrated into my CI. I make a commit, Azure Pipelines builds it and runs the tests, and the coverage report generated from the build is uploaded to CodeCov.io.

Next I’ll finally start going through some of Talk Python training courses and use a TDD approach to write tests for what I’m learning.

  • in python
  • 6 min read

Finishing Setup (with Dependabot and Pytest reporting) and Project Goals - Part 3

Using Dependabot to manage Python dependencies

All I want is to start coding. I want to learn how to write a Pyramid app the correct way and also to start learning pytest. There's just one more thing to do and one to tweak in Azure and then it's time for the fun part - the actual learning and coding.

First, I’m going to setup Dependabot to help manage my Python dependencies.

Last year I bought a Python bundle on Humble Bundle that included a one year subscription to Pyup.io. I’ve been using it for both NFLPool and MLBPool2 as it’s free for open source projects (both of those projects are licensed under a MIT license). I haven’t decided what license the content on SilverSaucer.com, but knowing me, it will probably be open source. But on the off chance it’s not, I’m going to try out Dependabot.

Github recently purchased Dependabot and made it free. Visit Dependabot, sign in with your Github profile and link which repositories you want Dependabot to help manage.

That’s it - Dependabot will keep an eye on your requirements.txt and send you Pull Requests when it finds a new version of a Python module for you to merge.

As I’ve integrated with Azure Pipelines for continuous integration, it builds the pull request and tells me if it failed or succeeded, which is nice to know before I merge it:

Dependabot Details

But of the five pull requests Dependabot created, three of the five failed to build in Azure Pipelines:

Dependabot failed builds

And it’s the exact same error as I ran into in part 2 - the builds in Python 3.6 work and they don’t in 3.7 due to some kind of (and I’m guessing) SQLAlchemy / pysqlite3 problem. (Log)

After even more troubleshooting, and if I had any hair (which I don’t, thankfully) I would have pulled it out. I finally gave up and just merged them all. It built locally and I figured future me could deal with it. And then all the builds worked again. I have no idea why. I don’t know how real developers do this stuff. But it’s working now, even with a couple smaller commits to update the README.

But when it does work, it feels great. I had updated the README, which kicked off a build, and since I hooked up Slack to Azure Pipelines, I saw this:

Azure working build

So now it’s setup just like the Test-Driven Development with Python says it should be. From Chapter 24:

“As our site grows, it takes longer and longer to run all of our functional tests. If this continues, the danger is that we’re going to stop bothering. Rather than let that happen, we can automate the running of functional tests by setting up a “Continuous Integration” or CI server. That way, in day-to-day development, we can just run the FT that we’re working on at that time, and rely on the CI server to run all the tests automatically and let us know if we’ve broken anything accidentally. The unit tests should stay fast enough that we can keep running them every few seconds.”

Excerpt From: Harry Percival. “Test-Driven Development with Python.” Apple Books.

Additional Tasks in Azure Pipelines

The Azure Pipeline docs are pretty great, including different things to configure based on the programming language you’re using.

For Python projects in Azure Pipelines, there are two tasks to add: 1. Publish Test Results 2. Publish Code Coverage

I have a feeling these will come in handy later.

I’ve only played around with the HTML report code coverage generates a couple of times. Looking at Azure DevOps, it looks kind of cool how they integrate testing into your dashboard.

In your project’s dashboard on Azure DevOps, go to Test Plans -> Runs:

Azure Test Plans / Runs

Choose which test results you want to look at it and there are pretty reports for you to view, and if you keep scrolling down on the right hand side it will give you different outcomes for review. It might save me a few clicks from the manual HTML code coverage builds, so I have that going for me.

Pytest Results

Project Goals

Now it’s finally time to start coding. I have two goals:

  1. Build a web app with the Pyramid web framework the correct way. My first two projects used a module called pyramid_handlers to manage views. This is an outdated way of writing code with Pyramid and I need to learn the modern way.
  2. Use Test Driven Development methodology to finally learn how to write tests using pytest. I’m going to write the test and then the code.

I’m not quite sure how I’m going to do this yet. I think it’s going to be a lot of jumping around a couple different Talk Python training courses. (If you have a rudimentary knowledge of Python, I highly recommend Talk Python’s training courses).

The course Building data-driven web apps with Pyramid and SQLAlchemy contains the knowledge on Pyramid and also includes a chapter on testing. The recently launched #100DaysOfWeb in Python has a bunch of chapters that will be useful in my quest, including Pyramid, Selenium, and Unit Testing Web Apps. That doesn’t even include things I want to eventually learn, like a Javascript introduction, CSS and more. (Yes, I saw the recent Jetbrains survey results and 40%+ of Python developers are using Flask for web development- but I’d like to get good at Pyramid before I switch to a different framework. And I really, really like the Pyramid community. The same could be said about my Python knowledge before diving into Javascript).

So while I’m going to kind of Frankenstein my courses, I’m going to do it using TDD. I’m not going to build a PyPI clone like Building data-driven web apps with Pyramid and SQLAlchemy teaches. Silversaucer.com is just an online playground for me to keep learning, and the first thing I’m going to build is a Randomizer for my vinyl record collection using the Discogs API. I’m going to work through the two courses above and I’ll also be using the following books to learn more about testing:

I’ll keep blogging what I learn and where I get stuck. I need to learn testing, especially if I want to continue maintaining and improving NFLPool and MLBPool2. And learning how to code is fun, in a frustrating kind of way (at times).

  • in python
  • 2 min read

Setting up Azure Pipelines - Part 2

In Part 1, I covered the challenges I had in setting up my SSH key with Azure Pipelines to work with my existing Github repository, which contains a new Pyramid project without any customization (yet).

Now that Azure Pipelines could build my project, I spent the last week after that trying to figure out why builds would fail on Azure with Python 3.7, not with Python 3.6 or on my local development machine.

One question I was asked: Why continuous integration if I’m just a hobbyist? I have two answers:

  1. Test-Driven Development with Python by Harry Percival recommends it:

    “Rather than let that happen, we can automate the running of functional tests by setting up a “Continuous Integration” or CI server. That way, in day-to-day development, we can just run the FT that we’re working on at that time, and rely on the CI server to run all the tests automatically and let us know if we’ve broken anything accidentally. The unit tests should stay fast enough that we can keep running them every few seconds.”

    Excerpt From: Harry Percival. “Test-Driven Development with Python.” Apple Books.

  2. It’s cool. And that’s the real reason. Having the little “Azure Pipelines Succeeded” badge on the Github repo page; hooking up the Slack integration to get a message when a build builds or fails; and knowing I’m doing things like a “real” developer might.

But I digress. I set up Azure Pipelines to run two builds - one in Python 3.6 and one in Python 3.7. After I make a commit to the SilverSaucer Github repository, Azure Pipelines automatically starts a job and builds the project.

Azure Builds

Two of the four tests passed.

The good news: The two tests using Python 3.6 pass and it builds!

The bad news: The exact same two tests fail on Python 3.7. (Log)

I made sure my development machine’s version of Python matched Azure’s and upgraded from Python 3.7.1 to 3.7.3 just to make sure - still failed.

I poked at it here and there for a few days and then asked for help in the Pyramid IRC channel. Right away, I received advice to add pysqlite3 and it worked! I used pip freeze to update my requirements.txt file and made sure pysqlite3 was in there, committed, and now I have a shiny badge on my repo.

I still don’t understand why it built on Python 3.6 but not Python 3.7. But it’s working and time to move on.

Coming in part 3: Hooking up Dependabot and the Python 3.7 builds fail again.

Learning pytest using continuous integration with Azure Pipelines (or SSH key hell) - Part 1

Introduction

I’m still on my quest to learn more Python and at the top of that list is learning pytest. I just can’t wrap my head around testing and I know my two Pyramid apps aren’t “complete” until there are tests. (I did write docs, so I have that going for me).

A couple months ago I (very easily) added continuous integration to NFLPool using Microsoft’s Azure Pipelines. I, like many other people, have been blown away by the right turn Microsoft made a few years back to embrace open source, and wanted to give Azure Pipelines a try. Every time I make a commit or Pyup.io submits a pull request to update a Python package, Azure Pipelines builds NFLPool. Of course it fails, because the tests I’ve written so far fail. To end this cycle, I really need to learn how to write tests!

I have a domain I don’t use, silversaucer.com. I have a few different ideas for some projects for the domain. I’m going to build another web app using Pyramid to start, with a goal of properly using classes in Pyramid (and not Pyramid handlers) to start. I’ve also been reading Test-Driven Development with Python, 2nd Edition, that I received from a recent Humble Bundle filled with Python books.

I was thinking that this was a perfect opportunity to learn pytest. I would create a new project in Pyramid and use TDD to write my tests as I write my code. If I’m going to do that, I might as well set up continuous integration right away and pretend I’m a real developer.

Setting up Pyramid

Since I’m going to be using a Test Driven Development philosophy, all I’m going to do is create a Pyramid project and not make any changes to it yet. I used Pyramid’s cookiecutter to create the project and then committed it to my Github repository. That’s it!

Hooking up Azure Pipelines

Here is where the fun starts. I’m not going to go through this process as Microsoft has great documentation for Azure and set up.

Walk through the setup and connect to your repository on Github. Azure Pipelines will create the needed YAML file, commit it and run your first build.

Here is where my build failed for Silver Saucer. I was getting the following error:

Obtaining silversaucer from git+git@github.com:prcutler/silversaucer.git@75932f389536b59993fa780b281170849ff92238#egg=silversaucer (from -r requirements.txt (line 38))
  Cloning git@github.com:prcutler/silversaucer.git (to revision 75932f389536b59993fa780b281170849ff92238) to ./src/silversaucer
  Running command git clone -q git@github.com:prcutler/silversaucer.git /home/vsts/work/1/s/src/silversaucer
  Host key verification failed.
  fatal: Could not read from remote repository.

Here is where I first lost hours over the course of a few days. Azure is pulling my repository using git, not https. I would compare this to my Azure Pipeline for NFLPool, which for some reason pulls my repository using https and works fine. I know a little bit about SSH keys. I’m no expert, but all of my Digital Ocean and servers at home use SSH key authentication to log in and not passwords (yay me for good opsec!) and I have my SSH key on multiple computers without any issues.

Lots of search queries later, I learned how you can change a git repository’s remote URL, but this appears just to be on your local machine. I’m sure I’m missing something simple to make it a global change, but I never figured it out.

Ok, let’s add my SSH key to Azure Pipelines. Again, Microsoft has good developer documentation on how to do this.

  • Step 1: Add your public key to your Azure profile.
  • Step 2: In your projects in Azure Pipelines, go to Pipelines -> Library and choose Secure files. Add your private key (usually id_rsa).
  • Step 3: Add the SSH Task to Azure Pipelines and make sure you authorize the private key - follow Microsoft’s developer documentation for the SSH Task. Update your YAML file:
    # Install SSH Key
        # Install an SSH key prior to a build or release
        - task: InstallSSHKey@0
          inputs:
            hostName: 
            sshPublicKey: 
            #sshPassphrase: # Optional
            sshKeySecureFile:

The hostname input confused me at first, but here you’re going to go into your ~/.ssh directory and copy and paste the Github entry in your known_hosts file. (This is a hidden directory in your home folder on macOS or Linux. I’m not sure where it is on Windows, sorry!) Paste your public key in sshPublicKey: and the name of your private key that you uploaded in Step 2 above. If your repository is public on Github, you are not going to want to add your sshPassphrase to your YAML file.

My SSH key fails

Again, I lost hours here. I have no idea why, but the SSH key I’ve been using for the last couple of years will not work. I had to create a second SSH key for Azure Pipelines and delete my known_hosts file, clone the repository again (which then updated known_hosts) and paste in the new fingerprint from known_hosts to the YAML file. The new key works fine, but I’ll be damned if I can figure out why my normal key doesn’t work. I’ve compared the fingerprint of my usual key, used it to clone other repositories, but Azure Pipelines refuses to work with it as I would just receive the above error over and over again.

I’m not thrilled with having a second SSH key, and now I have to go to the other two computers I work with and copy it over and add it to my keyring. But it works.

What’s next

Coming up in Part 2: Pytest works in Azure Pipelines in Python 3.6 (but not Python 3.7!)