Xcode - Single XCTest suite gets run multiple times

[evliu]

Description

Our github action runs xcodebuild test -project Project.xcodeproj -scheme Project\ DEV -destination 'platform=iOS Simulator,name=iPhone 16,OS=18.2' -parallel-testing-enabled NO | xcpretty && exit ${PIPESTATUS[0]}

It starts the test correctly, but then midway through the current tests, it starts the same test again for no known reason and fails our tests intermittently but passes just fine after business hours.
Logs tend to look like this

All tests
Test Suite Project.xctest started
AppInitFlowControllerTests
    ✓ testCheckServerMaintenanceThrowsCorrectError (0.015 seconds)
...
Selected tests
Test Suite Project.xctest started
...
VersionHistoryModelTests
    ✓ testVersionHistoryModel (0.002 seconds)
    ✓ testVersionHistoryModelInvalidDate (0.083 seconds)
Executed 225 tests, with 0 failures (0 unexpected) in 4.165 (9.503) seconds

Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
2025-03-03 18:13:48.387 xcodebuild[[359](https://github.com/cg-tech/project-mobile/actions/runs/13599568322/job/38119598047#step:6:360)1:22114] [MT] IDETestOperationsObserverDebug: 216.435 elapsed -- Testing started completed.
2025-03-03 18:13:48.388 xcodebuild[3591:22114] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2025-03-03 18:13:48.388 xcodebuild[3591:22114] [MT] IDETestOperationsObserverDebug: 216.435 sec, +216.435 sec -- end
Failing tests:
	PushNotificationServiceTests.successfullySendsDeviceToken()
** TEST FAILED **

Our Project.yml for the testing looks like this

  Test:
    runs-on: macOS-15
    needs: Build
    steps:
      - uses: actions/checkout@v4

      - name: Select Xcode
        run: sudo xcode-select -switch /Applications/Xcode_16.2.app && /usr/bin/xcodebuild -version

      - name: Testing App iPhone 16 iOS 18.2
        run: xcodebuild test -project Project.xcodeproj -scheme Project\ DEV -destination 'platform=iOS Simulator,name=iPhone 16,OS=18.2' -parallel-testing-enabled NO | xcpretty && exit ${PIPESTATUS[0]}

Our test plan only has one set of XCTest, and locally, it always just runs that test suite and passes with no issues.

Any insights would be amazing, I've spent hours looking into this for many months now. Right now, we just re-run the action until it passes. It's also always a random test that seemingly fails even though it passes on the first run of the XCTest suite.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022
• Windows Server 2025

Image version and build link

all versions of the macos images for years now

Is it regression?

No

Expected behavior

The XCTest suite runs once and passes.

Actual behavior

The XCTtest suite runs more than once (up to three times) and one besides the first fails sometimes even though it says 0 tests executed

Repro steps

We run our test action, with the main part being xcodebuild test -project Project.xcodeproj -scheme Project\ DEV -destination 'platform=iOS Simulator,name=iPhone 16,OS=18.2' -parallel-testing-enabled NO | xcpretty && exit ${PIPESTATUS[0]}

[archita105]

Hi @evliu
We are looking into it. Will keep you posted.

[archita105]

Hi @evliu
I attempted to reproduce the issue by following the steps outlined in the issue description but did not encounter any XCTest-related or test-related problems while testing the Xcode application. Please refer to the attached screenshot of the workflow and the successful build. Could you please review your project configuration?
Let me know if you have any additional questions or encounter any issues related to the macOS images.

[evliu]

One other difference I do see from my local is this

[21:09:00]: Running Tests: ▸ Touching AuthFoundation_WebAuthenticationUI.bundle (in target 'AuthFoundation_WebAuthenticationUI' from project 'AuthFoundation')
[21:09:00]: Running Tests: ▸ Touching AuthFoundation_OktaOAuth2.bundle (in target 'AuthFoundation_OktaOAuth2' from project 'AuthFoundation')
[21:09:13]: Running Tests: ▸ Touching AuthFoundation_AuthFoundation.bundle (in target 'AuthFoundation_AuthFoundation' from project 'AuthFoundation')
[21:09:13]: Running Tests: ▸ Touching Apollo_ApolloAPI.bundle (in target 'Apollo_ApolloAPI' from project 'Apollo')
[21:09:13]: ▸ Linking AuthFoundation.o
[21:09:13]: Running Tests: ▸ Touching Apollo_Apollo.bundle (in target 'Apollo_Apollo' from project 'Apollo')

I only get Running Tests: with my local code locally, but the runner shows Running Tests: for 3rd party dependencies. Does that have anything to do with it?

Also, I see in your one, All Tests is printed twice but just once in mine. What happens if you use a small runner with a thousand tests?

I have no idea how the tests are running a second time, and executing 0 tests with a failure

[21:14:44]: ▸ All tests
[21:14:44]: ▸ Test Suite Project.xctest started
...
[21:14:59]: ▸ VersionHistoryModelTests
[21:14:59]: ▸     ✓ testVersionHistoryModel (0.006 seconds)
[21:14:59]: ▸     ✓ testVersionHistoryModelInvalidDate (0.157 seconds)
[21:14:59]: ▸ 	 Executed 225 tests, with 0 failures (0 unexpected) in 8.527 (15.150) seconds
[21:14:59]: ▸ 
[21:15:36]: ▸ Selected tests
[21:15:36]: ▸ Test Suite Project.xctest started
[21:15:36]: ▸ 	 Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
[21:15:36]: ▸ 
[21:16:52]: ▸ 2025-03-06 21:16:52.661 xcodebuild[5196:27553] [MT] IDETestOperationsObserverDebug: 335.529 elapsed -- Testing started completed.
[21:16:52]: ▸ 2025-03-06 21:16:52.661 xcodebuild[5196:27553] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[21:16:52]: ▸ 2025-03-06 21:16:52.661 xcodebuild[5196:27553] [MT] IDETestOperationsObserverDebug: 335.529 sec, +335.529 sec -- end
[21:17:01]: ▸ Failing tests:
[21:17:01]: ▸ 	ServiceTests.alwaysDifferentRandomTestFailshere()
[21:17:01]: ▸ ** TEST FAILED **
[21:17:02]: Exit status: 65
+--------------------------+
|       Test Results       |
+--------------------+-----+
| Number of tests    | 246 |
| Number of failures | 1   |
+--------------------+-----+
...
+---------------------------------------+
|           fastlane summary            |
+------+------------------+-------------+
| Step | Action           | Time (in s) |
+------+------------------+-------------+
| 1    | opt_out_usage    | 0           |
| 2    | default_platform | 0           |
| 3    | is_ci            | 0           |
| 💥   | run_tests        | 637         |
+------+------------------+-------------+