Stable and Predictable Python Environments with Version Pinning

Managing Python Dependencies in Your Projects

Managing Python dependencies can be challenging, especially when transitioning projects between different environments or ensuring that all team members are using the same package versions. A common tool for managing these dependencies is a requirements.txt file, which lists the necessary packages for a project. However, often this file does not specify exact package versions, leading to inconsistencies and challenging errors across various setups.

Specifying package versions is crucial to prevent unexpected changes across different environments, whether among team members, testing setups, or production systems. In a recent Django project, my requirements.txt file listed all packages but omitted their versions. Each time I fixed bugs or added features, a new Docker image was built, always pulling the latest package versions. This approach was problematic as it could introduce untested changes to the environment.

A common method to pin packages involves installing the required packages via pip, followed by running pip freeze to list all installed packages and their versions. I found this method unwieldy because it mixed direct dependencies with transitive ones, complicating the removal process when updating the requirements.txt. To streamline this, I developed a Bash script that compares the output of pip freeze to the entries in requirements.txt, focusing only on the directly used packages and not including all dependencies.

To tackle this problem, we can deploy a Bash script that pins the versions of currently installed packages to match those specified in requirements.txt. This strategy guarantees that you and your team maintain a consistent environment, accurately reflecting the setup used during development. Below, I will provide a detailed Bash script and demonstrate how it can be seamlessly integrated into your project.

Why Pinning Versions Is Crucial?

Pinning versions in your requirements.txt file ensures that every environment the project is deployed in uses the exact same package versions. This practice minimizes “works on my machine” problems and improves the robustness of your deployment process. It’s particularly crucial in environments where stability and predictability are required, such as production or CI/CD pipelines.

The Bash Script

#!/bin/bash

# Define the path to the original requirements.txt file
requirements_file="requirements.txt"
# Define the path for the new requirements file with pinned versions
pinned_requirements_file="pinned_requirements.txt"

if [ ! -f "$requirements_file" ]; then
    echo "Error: requirements.txt not found."
    exit 1
fi

touch "$pinned_requirements_file"

while IFS= read -r package || [ -n "$package" ]; do
    # Clean up the package name to handle cases with extras and remove any versioning
    package_name=$(echo $package | sed 's/\[.*\]//;s/==.*//')

    # Get the version of the package installed, accounting for extras if present
    installed_info=$(pip freeze | grep -i "^$package_name")

    # Check if the exact requirement with extras exists in the pip freeze output
    if echo "$package" | grep -q '\['; then
        # If extras are specified, refine the search to include them
        extras=$(echo $package | grep -o '\[.*\]')
        exact_match=$(echo "$installed_info" | grep -i "$extras")
        if [ ! -z "$exact_match" ]; then
            echo "$exact_match" >> "$pinned_requirements_file"
        else
            echo "$package (with extras) not installed as specified" >> "$pinned_requirements_file"
        fi
    else
        if [ ! -z "$installed_info" ]; then
            echo "$installed_info" | head -n 1 >> "$pinned_requirements_file"
        else
            echo "$package not installed"
        fi
    fi
done < "$requirements_file"

echo "Pinned requirements have been written to $pinned_requirements_file. Please review to make sure all expected packages are included."

Conclusion

Using a Bash script to pin Python package versions is a simple yet effective way to enhance the reproducibility and consistency of your project environments. Including such scripts in your project makes the setup process easier for others.