Skip to content

chore: increase parallelization in e2e tests #12276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 21, 2023
Merged

chore: increase parallelization in e2e tests #12276

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
83f2bbc
chore: increase parallelization of tests
sobolk Mar 21, 2023
0e75f93
chore: run searchable migration solo
sobolk Mar 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .circleci/config.base.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -515,7 +515,7 @@ jobs:
source .circleci/local_publish_helpers.sh
changeNpmGlobalPath
cd packages/amplify-migration-tests
retry yarn run migration_v5.2.0 --no-cache --maxWorkers=3 --forceExit $TEST_SUITE
retry yarn run migration_v5.2.0 --no-cache --maxWorkers=4 --forceExit $TEST_SUITE
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we can actually use all 4 cores since jest needs 1 cpu to manage the worker pool & OS; maybe its working now ? I noticed when I tried 4 , it just runs the 4th test at the end of the first 3

Copy link
Contributor Author

@sobolk sobolk Mar 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We added --forceExit since then. 4 core CPU can handle more than 4 workers as long as they interrupt and yield the cpu from time to time.
See https://app.circleci.com/pipelines/github/aws-amplify/amplify-cli/17621/workflows/123658d9-cf34-4eaf-b32e-def425cac409

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting maxWorkers beyond CPU count - 1 would be harmful if our tests were CPU bound. But they mostly spawn CLI and wait for it asynchronously.

no_output_timeout: 65m
- run: *scan_e2e_test_artifacts
- store_test_results:
Expand Down Expand Up @@ -545,7 +545,7 @@ jobs:
source .circleci/local_publish_helpers.sh
changeNpmGlobalPath
cd packages/amplify-migration-tests
retry yarn run migration_v6.1.0 --no-cache --maxWorkers=3 --forceExit $TEST_SUITE
retry yarn run migration_v6.1.0 --no-cache --maxWorkers=4 --forceExit $TEST_SUITE
no_output_timeout: 65m
- run: *scan_e2e_test_artifacts
- store_test_results:
Expand Down Expand Up @@ -577,7 +577,7 @@ jobs:
cd packages/amplify-migration-tests
unset IS_AMPLIFY_CI
echo $IS_AMPLIFY_CI
retry yarn run migration_v10.5.1 --no-cache --maxWorkers=3 --forceExit $TEST_SUITE
retry yarn run migration_v10.5.1 --no-cache --maxWorkers=4 --forceExit $TEST_SUITE
no_output_timeout: 65m
- run: *scan_e2e_test_artifacts
- store_test_results:
Expand Down Expand Up @@ -607,7 +607,7 @@ jobs:
source .circleci/local_publish_helpers.sh
changeNpmGlobalPath
cd packages/amplify-migration-tests
retry yarn run migration_v4.28.2_nonmultienv_layers --no-cache --maxWorkers=3 --forceExit $TEST_SUITE
retry yarn run migration_v4.28.2_nonmultienv_layers --no-cache --maxWorkers=4 --forceExit $TEST_SUITE
no_output_timeout: 90m
- run: *scan_e2e_test_artifacts
- store_test_results:
Expand Down Expand Up @@ -637,7 +637,7 @@ jobs:
source .circleci/local_publish_helpers.sh
changeNpmGlobalPath
cd packages/amplify-migration-tests
retry yarn run migration_v4.52.0_multienv_layers --no-cache --maxWorkers=3 --forceExit $TEST_SUITE
retry yarn run migration_v4.52.0_multienv_layers --no-cache --maxWorkers=4 --forceExit $TEST_SUITE
no_output_timeout: 90m
- run: *scan_e2e_test_artifacts
- store_test_results:
Expand Down Expand Up @@ -665,7 +665,7 @@ jobs:
source .circleci/local_publish_helpers.sh
amplify -v
cd packages/amplify-console-integration-tests
retry yarn run console-integration --no-cache --maxWorkers=3
retry yarn run console-integration --no-cache --maxWorkers=4
name: 'Run Amplify Console integration tests'
no_output_timeout: 90m
- run: *scan_e2e_test_artifacts
Expand Down
2 changes: 1 addition & 1 deletion .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ jobs:
source .circleci/local_publish_helpers.sh
amplify -v
cd packages/amplify-console-integration-tests
retry yarn run console-integration --no-cache --maxWorkers=3
retry yarn run console-integration --no-cache --maxWorkers=4
name: Run Amplify Console integration tests
no_output_timeout: 90m
- run:
Expand Down
4 changes: 2 additions & 2 deletions .circleci/local_publish_helpers.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,9 +273,9 @@ function runE2eTest {
if [ -f $FAILED_TEST_REGEX_FILE ]; then
# read the content of failed tests
failedTests=$(<$FAILED_TEST_REGEX_FILE)
yarn run e2e --forceExit --no-cache --maxWorkers=3 $TEST_SUITE -t "$failedTests"
yarn run e2e --forceExit --no-cache --maxWorkers=4 $TEST_SUITE -t "$failedTests"
else
yarn run e2e --forceExit --no-cache --maxWorkers=3 $TEST_SUITE
yarn run e2e --forceExit --no-cache --maxWorkers=4 $TEST_SUITE
fi
}

Expand Down
4 changes: 2 additions & 2 deletions scripts/split-e2e-tests-v2.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ const RUN_SOLO = [
'src/__tests__/schema-auth-11-a.test.ts',
'src/__tests__/schema-auth-15.test.ts',
'src/__tests__/schema-connection-1.test.ts',
'src/__tests__/studio-modelgen.test.ts',
'src/__tests__/transformer-migrations/searchable-migration.test.ts',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

turns out studio-model gen is happy with some company.
but searchable migration is not on windows if we crank up parallelization.

];
/**
* Most Windows tests only run on 'dev', except for this list of smoke tests.
Expand Down Expand Up @@ -155,7 +155,7 @@ const TEST_EXCLUSIONS: { l: string[]; w: string[] } = {
'src/__tests__/pinpoint/notifications-pinpoint-config-util.ts',
],
};
const MAX_WORKERS = 3;
const MAX_WORKERS = 4;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we do something like this for the base.yml and config.yml file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not worth the effort given timeline for this script.

type OS_TYPE = 'w' | 'l';
type CandidateJob = {
region: string;
Expand Down