Accelerate Your Unit Tests with Parallelization

Why Parallel Testing?

Running unit tests sequentially can be a real test of patience. Imagine this: you have a suite of 100 unit tests, each taking approximately 50 seconds to complete. If you run them one after the other, you’re looking at a staggering 83 minutes of waiting time! To optimize this process, consider parallelizing your unit tests with the pytest-xdist library, which can significantly reduce the total execution time.

Example Code

Let’s illustrate this with a simple string manipulation utility in string_utils.py:

def reverse_string(s: str) -> str:
    """Returns the reversed version of the input string."""
    return s[::-1]

def to_uppercase(s: str) -> str:
    """Returns the input string in uppercase."""
    return s.upper()

def count_characters(s: str) -> int:
    """Returns the number of characters in the input string."""
    return len(s)

def is_palindrome(s: str) -> bool:
    """Checks if the input string is a palindrome."""
    return s == reverse_string(s)

Unit Tests:

In test_string_utils.py, we have:

from string_utils import reverse_string, to_uppercase, count_characters, is_palindrome
import time

def test_reverse_string():
    """Test the reverse_string function."""
    time.sleep(2)  # Simulate a delay
    assert reverse_string("hello") == "olleh"

def test_to_uppercase():
    """Test the to_uppercase function."""
    time.sleep(2)  # Simulate a delay
    assert to_uppercase("hello") == "HELLO"

def test_count_characters():
    """Test the count_characters function."""
    time.sleep(2)  # Simulate a delay
    assert count_characters("hello") == 5

def test_is_palindrome():
    """Test the is_palindrome function."""
    time.sleep(2)  # Simulate a delay
    assert is_palindrome("racecar") is True
    assert is_palindrome("hello") is False

Measure Execution Times

Run Tests Sequentially:
Execute the tests normally:
```
pytest ./tests/ -v -s
```
Time Taken: Approximately 8 seconds (2 seconds per test).
Run Tests in Parallel:
Ensure you have pytest-xdist installed:
```
pip install pytest-xdist
```
Now run the tests in parallel:
```
pytest ./tests/ -v -s -n auto
```
breakdown:
- -v: Enables verbose output, showing detailed information about each test and its result.
- -s: Allows print statements and logging output to be displayed in the terminal during test execution.
- -n auto: Runs tests in parallel using all available CPU cores, speeding up the test execution time.
Time Taken: Reduced to 6 seconds!

Conclusion

By using pytest-xdist, you can easily speed up your unit tests. This makes your development process more efficient and gives you faster feedback.

Thank you for reading, and stay tuned for more insights!

Cheers

Why Parallel Testing?

Example Code

Measure Execution Times

Conclusion

References and more reading