Jeff Wright Jeff Wright - 7 months ago 22
Python Question

How to run all PyTest tests even if some of them fail?

I am looking for a way to run all of my unit tests in PyTest, even if some of them fail. I know there must be a simple way to do this. I checked the CLi options and looked through this site for similar questions/answers but didn't see anything. Sorry if this has already been answered.

For example, consider the following code snippet, with PyTest code alongside it:

def parrot(i):
return i

def test_parrot():
assert parrot(0) == 0
assert parrot(1) == 1
assert parrot(2) == 1
assert parrot(2) == 2


By default, the execution stops at the first failure:

$ python -m pytest fail_me.py
=================== test session starts ===================
platform linux2 -- Python 2.7.10, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: /home/npsrt/Documents/repo/codewars, inifile:
collected 1 items

fail_me.py F

=================== FAILURES ===================
___________________ test_parrot ___________________

def test_parrot():
assert parrot(0) == 0
assert parrot(1) == 1
> assert parrot(2) == 1
E assert 2 == 1
E + where 2 = parrot(2)

fail_me.py:7: AssertionError
=================== 1 failed in 0.05 seconds ===================


What I'd like to do is to have the code continue to execute even after PyTest encounters the first failure.

Answer

It ran all of your tests. You only wrote one test, and that test ran!

If you want nonfatal assertions, where a test will keep going if an assertion fails (like Google Test's EXPECT macros), try pytest-expect, which provides that functionality. Here's the example their site gives:

def test_func(expect):
    expect('a' == 'b')
    expect(1 != 1)
    a = 1
    b = 2
    expect(a == b, 'a:%s b:%s' % (a,b))

You can see that expectation failures don't stop the test, and all failed expectations get reported:

$ python -m pytest test_expect.py
================ test session starts =================
platform darwin -- Python 2.7.9 -- py-1.4.26 -- pytest-2.7.0
rootdir: /Users/okken/example, inifile: 
plugins: expect
collected 1 items 

test_expect.py F

====================== FAILURES ======================
_____________________ test_func ______________________
>    expect('a' == 'b')
test_expect.py:2
--------
>    expect(1 != 1)
test_expect.py:3
--------
>    expect(a == b, 'a:%s b:%s' % (a,b))
a:1 b:2
test_expect.py:6
--------
Failed Expectations:3
============== 1 failed in 0.01 seconds ==============