Over time, I’ve collected some notes and examples that I often reference in my day-to-day work. Whether you’re debugging a tricky issue or designing error-handling for a new feature, I hope this blog post becomes a handy reference for you - as a fellow engineer - as well.

📋 The basics

First, a refresher on the basics.

try:
    # Code that may raise an exception.
    value = int(input("Enter your age: "))
    assert value >= 0
except (ValueError, AssertionError):
    # Defines how to handle the error(s).
    print("Only positive integers please!")
else:
    # This runs if no exception is raised.
    print(f"You are {value} years old.")
finally:
    # Always runs, regardless of whether an exception was raised or not.
    print("Execution complete.")

🪵 Tips on logging exceptions

And before we dive into the patterns below, some tips on logging exceptions:

Log the printable representation of the exception

By default, log the repr() of the exception to provide a clear description of what went wrong.

import logging

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

def validate_age(age):
    if age < 0:
        raise ValueError

try:
    validate_age(-2)
except ValueError as e:
    logger.error("An error occurred: %r", e)

ERROR:root:An error occurred: ValueError()

Why not log the string representation?

Using str() may result in the loss of key information about our exceptions.

try:
    validate_age(-2)
except ValueError as e:
    logger.error("An error occurred: %s", e)  # <-- Notice the difference here

ERROR:root:An error occurred:                 # <-- Wait! What's the error?!

Include the traceback for context

To make debugging easier, we can include the traceback using exc_info=True. It also works when logging at other levels (DEBUG, INFO, WARNING, etc.).

try:
    validate_age(-2)
except ValueError as e:
    logger.error("An error occurred: %r", e, exc_info=True)

ERROR:__main__:An error occurred: ValueError()
Traceback (most recent call last):
  File "/Users/opto/code/keep_calm_eng_blog.py", line 21, in <module>
    validate_age(-2)
  File "/Users/opto/code/keep_calm_eng_blog.py", line 18, in validate_age
    raise ValueError
ValueError

Note: logger.error("%r", e, exc_info=True) is equivalent to logger.exception("%r", e).

🧑‍💻 Common patterns

We'll use some fictional application models and relationships (in an Opto Investments context) to exemplify the patterns below:

An Investor models an entity that can own assets
A Fund models an asset that can be owned by investors
A Position is a relationship between an Investor (owner) and a Fund (the asset owned) with some extra details such as amount

🔄 Transform the exception

Use case

Expose a domain-specific error to the caller by transforming a low-level exception.

Example

The positions table in our DB has a foreign key constraint on investor_id which references the investors table
If a user tries adding a new Position for an investor_id that doesn't exist, our ORM throws a DBIntegrityError

@dataclass
class Position:
    id: UUID
    investor_id: UUID  # Foreign key relationship to investors table in the DB.
    fund_id: UUID
    amount: Decimal


@internal_positions_router.post("")
def add_position(position_create: PositionCreate) -> Position:
    try:
        new_position = db.add_position(position_create)
        return new_position
    except DBIntegrityError as e:
        logger.warning("%r", e, exc_info=True)
        raise PositionCreationError("Unable to create Position. Does the Investor exist?") from e

Through exception chaining, we transform the DBIntegrityError into an error that better maps to the higher-level logic of our application. Caller also sees the original exception in the traceback.

WARNING:__main__:DBIntegrityError()
Traceback (most recent call last):
  File "/Users/opto/code/keep_calm_eng_blog.py", line 79, in add_position
    raise DBIntegrityError
DBIntegrityError
Traceback (most recent call last):
  File "/Users/opto/code/keep_calm_eng_blog.py", line 79, in add_position
    raise DBIntegrityError
DBIntegrityError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/opto/code/keep_calm_eng_blog.py", line 88, in <module>
    add_position()
  File "/Users/opto/code/keep_calm_eng_blog.py", line 84, in add_position
    raise PositionCreationError(
PositionCreationError: Unable to create Position. Does the Investor exist?

Remember to

Add from e (or from None) when you raise the new exception. Otherwise, you’ll technically be causing a new exception while attempting to handle the original exception:

WARNING:__main__:DBIntegrityError()
Traceback (most recent call last):
  File "/Users/opto/code/keep_calm_eng_blog.py", line 79, in add_position
    raise DBIntegrityError
DBIntegrityError
Traceback (most recent call last):
  File "/Users/opto/code/keep_calm_eng_blog.py", line 79, in add_position
    raise DBIntegrityError
DBIntegrityError

During handling of the above exception, another exception occurred:  # <-- NOTICE THE DIFFERENCE HERE!

Traceback (most recent call last):
  File "/Users/opto/code/keep_calm_eng_blog.py", line 86, in <module>
    add_position()
  File "/Users/opto/code/keep_calm_eng_blog.py", line 84, in add_position
    raise PositionCreationError("Unable to create Position. Does the Investor exist?")
PositionCreationError: Unable to create Position. Does the Investor exist?

🚫 Suppress the original exception

Use case

Hide implementation details from the caller by suppressing the original exception.

Example

The below is now an “external” endpoint, we may not want external callers to get much insight into the internals of our application
We suppress the exception that hints to the type of ORM and relationships we’ve built in the back-end and instead raise a standard HTTPError

@dataclass
class Position:
    id: UUID
    investor_id: UUID  # Foreign key relationship to investors table in the DB.
    fund_id: UUID
    amount: Decimal


@external_positions_router.post("")  # <-- External endpoint, hit by external users.
def add_position(position_create: PositionCreate) -> Position:
    try:
        new_position = db.add_position(position_create)
    except DBIntegrityError as e:
        logger.warning("%r", e, exc_info=True)
        # ^-- Notice we can still log helpful internal details.
        raise HTTPError(status=400, description="Invalid payload.") from None
        # ^-- But the end user only sees this.

    return new_position

Here, we prevent the original exception from being exposed by using from None. This approach is useful for external-facing APIs.

⚠️ Alert on exception

Use case

Trigger an action before the program terminates.

Example

We have a daily job that syncs investor data with an external third party.
We want to send a helpful summary message to a Slack channel when it finishes syncing or an alert if it fails.

def run_job():
    try:
        job_summary = sync_investors(investor_list)
    except Exception as e:  # <-- Broad except to catch all errors.
        send_message_to_slack(slack_webhook_url, f"Error running job: {e!r}")  # <-- Alert to Slack.
        raise e  # <-- Notice we don't swallow the exception, we still raise it.
    else:
        send_message_to_slack(slack_webhook_url, f"Job finished: {job_summary}")

🧹 Finally, don’t forget about cleaning up

Use case

Build test fixtures that clean up after themselves by leveraging finally.

Example

@pytest.fixture
async def create_test_investor(test_async_client, investors_payload):
    try:
        response = await test_async_client.post("/internal/investors", json=investors_payload)
        # The yield statement pauses the fixture execution and provides the 
        # test investor details to the calling test function.
        yield [Investor.model_validate(investor) for investor in response.json()]
    finally:
        # After the test completes, we ensure the investors are deleted from
        # our DB, even if the test raises an exception.
        db.delete_all_investors()