Managing Credentials in Pipenv¶

This guide covers best practices for securely managing credentials and authentication in Pipenv, including environment variables, private repositories, and security considerations.

Credentials in Package Sources¶

When working with private package repositories or authenticated services, you need to securely manage credentials in your Pipenv workflow.

Environment Variable Expansion¶

Pipenv automatically expands environment variables in your Pipfile, providing a secure way to inject credentials without storing them in version control.

Basic Environment Variable Syntax¶

You can use environment variables in your Pipfile using the following syntax:

${VARIABLE_NAME} - Standard syntax
$VARIABLE_NAME - Alternative syntax
%VARIABLE_NAME% - Windows-specific syntax (also supported on all platforms)

Example: Private Repository Authentication¶

[[source]]
url = "https://${USERNAME}:${PASSWORD}@mypypi.example.com/simple"
verify_ssl = true
name = "private-pypi"

When Pipenv reads this Pipfile, it will replace ${USERNAME} and ${PASSWORD} with the values of the corresponding environment variables.

Setting Environment Variables¶

Before running Pipenv commands, set the required environment variables:

Linux/macOS:

$ export USERNAME="your-username"
$ export PASSWORD="your-password"
$ pipenv install

Windows (Command Prompt):

> set USERNAME=your-username
> set PASSWORD=your-password
> pipenv install

Windows (PowerShell):

> $env:USERNAME="your-username"
> $env:PASSWORD="your-password"
> pipenv install

Security Considerations¶

Pipenv hashes your Pipfile before expanding environment variables and substitutes them again when installing from the lock file. This means:

Your credentials are never stored in the lock file
You don’t need to commit any secrets to version control
Different developers can use different credentials with the same Pipfile

Warning

While environment variables provide better security than hardcoded credentials, they are still accessible to any process running with the same user permissions. For highly sensitive credentials, consider using a dedicated secrets management solution.

URL Encoding for Special Characters¶

If your credentials contain special characters, they must be URL-encoded according to RFC 3986.

For example, if your password is p@ssw0rd!, it should be encoded as p%40ssw0rd%21:

[[source]]
url = "https://${USERNAME}:${PASSWORD_ENCODED}@mypypi.example.com/simple"
verify_ssl = true
name = "private-pypi"

You can generate URL-encoded strings using Python:

import urllib.parse
print(urllib.parse.quote("p@ssw0rd!", safe=""))
# Output: p%40ssw0rd%21

Credentials in Package Requirements¶

Environment variables can also be used in package requirement specifiers, but with some limitations.

VCS Repository Authentication¶

For version control system (VCS) repositories that require authentication:

[packages]
requests = {git = "git://${USERNAME}:${PASSWORD}@private.git.com/psf/requests.git", ref = "2.22.0"}

Note

For VCS repositories, only the ${VAR_NAME} syntax is supported. Neither $VAR_NAME nor %VAR_NAME% will work in this context.

Runtime vs. Install-time Expansion¶

It’s important to understand that environment variables are expanded at runtime, not when the Pipfile or Pipfile.lock is created. This means:

The entries in Pipfile and Pipfile.lock remain untouched
You need to have the environment variables set every time you run Pipenv commands
Different environments (development, CI/CD, production) can use different credentials

Keychain Integration¶

For certain cloud providers, Pipenv supports keychain integration for authentication to private registries.

Google Cloud Artifact Registry¶

Google Cloud Artifact Registry supports authentication via keychain:

Install the required packages:

$ pipenv run pip install keyring keyrings.google-artifactregistry-auth

Configure your Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://europe-python.pkg.dev/my-project/python/simple"
verify_ssl = true
name = "private-gcp"

[packages]
flask = "*"
private-test-package = {version = "*", index = "private-gcp"}

If the keychain might ask for user input, you may need to disable input enforcement:
```
[pipenv]
disable_pip_input = false
```

Azure Artifact Registry¶

For Azure Artifact Registry:

Install the required packages:

$ pipenv run pip install keyring keyrings.azureartifacts

Configure your Pipfile similar to the Google Cloud example above.

AWS CodeArtifact¶

For AWS CodeArtifact:

Install the required packages:

$ pipenv run pip install keyring keyrings.aws-codeartifact

Configure your Pipfile with the appropriate AWS CodeArtifact URL.

Best Practices for Credential Management¶

Use Environment Variables¶

Always use environment variables instead of hardcoding credentials:

# Good
url = "https://${USERNAME}:${PASSWORD}@private-repo.example.com/simple"

# Bad
url = "https://actual-username:actual-password@private-repo.example.com/simple"

Store Credentials in .env Files¶

For local development, store credentials in .env files that are not committed to version control:

# .env
USERNAME=your-username
PASSWORD=your-password

Add .env to your .gitignore:

# .gitignore
.env

Pipenv will automatically load variables from .env files when you run commands.

Provide Templates¶

Include a template .env.example file in your repository:

# .env.example
USERNAME=
PASSWORD=

This helps other developers understand which environment variables they need to set.

Use Different Credentials for Different Environments¶

Maintain separate credential sets for different environments:

# .env.development
USERNAME=dev-username
PASSWORD=dev-password

# .env.production
USERNAME=prod-username
PASSWORD=prod-password

Use them with:

$ PIPENV_DOTENV_LOCATION=.env.development pipenv install

Rotate Credentials Regularly¶

Regularly rotate your credentials to minimize the impact of potential leaks:

Generate new credentials
Update your environment variables or .env files
Revoke the old credentials

Consider Credential Managers¶

For team environments, consider using dedicated credential management solutions:

These can be integrated into your workflow to provide secure, centralized credential management.

CI/CD Integration¶

GitHub Actions¶

For GitHub Actions, use secrets to store credentials:

name: Python Package

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.10'
    - name: Install dependencies
      env:
        USERNAME: ${{ secrets.PYPI_USERNAME }}
        PASSWORD: ${{ secrets.PYPI_PASSWORD }}
      run: |
        pip install pipenv
        pipenv install --dev

GitLab CI¶

For GitLab CI, use CI/CD variables:

stages:
  - test

test:
  stage: test
  image: python:3.10
  variables:
    USERNAME: ${PYPI_USERNAME}
    PASSWORD: ${PYPI_PASSWORD}
  script:
    - pip install pipenv
    - pipenv install --dev
    - pipenv run pytest

Jenkins¶

For Jenkins, use credentials binding:

pipeline {
    agent {
        docker {
            image 'python:3.10'
        }
    }
    stages {
        stage('Build') {
            steps {
                withCredentials([
                    string(credentialsId: 'pypi-username', variable: 'USERNAME'),
                    string(credentialsId: 'pypi-password', variable: 'PASSWORD')
                ]) {
                    sh 'pip install pipenv'
                    sh 'pipenv install --dev'
                    sh 'pipenv run pytest'
                }
            }
        }
    }
}

Troubleshooting¶

Common Issues¶

Environment Variables Not Being Expanded¶

If environment variables aren’t being expanded:

Verify the variables are set correctly:
```
$ echo $USERNAME
$ echo $PASSWORD
```
Check the syntax in your Pipfile:
- Use ${VARIABLE_NAME} for the most reliable expansion
- Ensure there are no typos in variable names

Try setting the variables directly in the command:

$ USERNAME=user PASSWORD=pass pipenv install

Authentication Failures¶

If you’re experiencing authentication failures:

Verify your credentials work outside of Pipenv:

$ curl -u "${USERNAME}:${PASSWORD}" https://private-repo.example.com/simple

Check for special characters that might need URL encoding
Ensure your credentials have the necessary permissions

Keychain Integration Issues¶

If keychain integration isn’t working:

Verify the keychain packages are installed:
```
$ pipenv run pip list | grep keyring
```
Check if the keychain is properly configured for your system
Try disabling pip input enforcement:
```
[pipenv]
disable_pip_input = false
```

Security Considerations¶

Credential Leakage Risks¶

Be aware of these common credential leakage risks:

Command history: Credentials passed directly in commands are stored in shell history
Process listing: Environment variables set on the command line may be visible in process listings
Log files: Debug logs might include expanded environment variables
Core dumps: Application crashes might include memory containing credentials

Mitigating Risks¶

To mitigate these risks:

Use .env files instead of setting variables on the command line
Limit access to environments where credentials are used
Use temporary credentials or tokens when possible
Implement the principle of least privilege for all credentials
Monitor for unauthorized access to your repositories

Conclusion¶

Properly managing credentials in Pipenv is essential for security and maintainability. By using environment variables, .env files, and following best practices, you can securely authenticate to private repositories and services without compromising sensitive information.

Remember that credential management is an ongoing process that requires regular review and updates to maintain security. Always follow the principle of least privilege and rotate credentials regularly to minimize potential security risks.