Lorenzo Camaione Lorenzo Camaione - 4 months ago 17
Android Question

AWS Device Farm Espresso tests order

I'm starting using Device Farm. My problem is that it is ignoring the order that I used to write tests. In local all works fine because Espresso executes tests in a certain order. To declare that order I used alphabetical order. All my classes starts with a letter (E.g. "A_EspressoTest") so I can choose which class has to be ran first.

Into my classes I use

@FixMethodOrder(MethodSorters.NAME_ASCENDING)


to declare in which order my tests have to be ran.

It seems also like Device Farm ignores all my annotations (E.g. "@Test" ) because it is running also methods that doesn't have that annotation.

Answer

Lorenzo,

As of today, there is no way to specify the order of test execution with Espresso. Additionally, your observation about the @Test annotation is correct, we don't currently use that when discovering what test classes/test methods are selected.

AWS Device Farm currently discovers tests based on JUnit 3 style naming conventions (classes starting/ending with the word Test and methods within those classes starting with the word test.

For example:

// This class would be automatically discovered (by name).
public class LoginTests extends InstrumentationTestCase {
    // This method would be automatically discovered (by name).
    public void testLoginFormUsingInvalidCredentials() {
        // ...
    }

    // This method would **not** be automatically discovered.
    @Test
    public void loginWithValidCredentials() {
        // ...
    }
}

// This class would **not** be automatically discovered.
public class Login extends InstrumentationTestCase {
    // This method would **not** be automatically discovered since the class was not discovered.
    public void testLoginFormWithValidCredentials() {
        // ...
    }
}

With all that said, we've heard plenty of feedback and requests for supporting test discovery using all JUnit 4 annotations and it's an area of improvement we're definitely taking a look at.

Hope that helps!

Best,

Andrew @ AWS Device Farm

Comments