Skip to content

Conversation

kmcginnes
Copy link
Collaborator

@kmcginnes kmcginnes commented Jul 13, 2024

Description

In a few cases we can not avoid sending many parallel requests to the server. Since some servers implement throttling, we want to batch up those requests in to smaller groups and execute them in serial.

  • Update openCypher keyword search query template
    • Fix limit and offset
    • Add ability to handle multiple vertex types in a single request
    • Sort by ID
    • Better formatting of strings
  • Consolidate multiple openCypher search requests in to a single request
  • Add batchPromisesSerially() function to manage the batching logic
  • Batch all the Promise.all() occurrences in the code
  • Default concurrent request batch size is set to 10

Validation

  • Tested openCypher search
  • Tested openCypher schema sync
  • Tested sparql schema sync

Related Issues

Check List

  • I confirm that my contribution is made under the terms of the Apache 2.0
    license.
  • I have run pnpm checks to ensure code compiles and meets standards.
  • I have run pnpm test to check if all tests are passing.
  • I have covered new added functionality with unit tests if necessary.
  • I have added an entry in the Changelog.md.

@kmcginnes kmcginnes force-pushed the batch-parallel-requests branch 5 times, most recently from 0a0e56f to 44f6fc6 Compare July 16, 2024 16:44
@kmcginnes kmcginnes requested a review from michaelnchin July 16, 2024 16:44
@kmcginnes kmcginnes marked this pull request as ready for review July 16, 2024 16:44
@kmcginnes kmcginnes mentioned this pull request Jul 16, 2024
5 tasks
@kmcginnes kmcginnes force-pushed the batch-parallel-requests branch 2 times, most recently from 05034bf to 09a360a Compare July 17, 2024 00:00
michaelnchin
michaelnchin previously approved these changes Jul 17, 2024
@kmcginnes kmcginnes force-pushed the batch-parallel-requests branch 2 times, most recently from 74a21da to bd0c407 Compare July 17, 2024 13:49
@kmcginnes kmcginnes force-pushed the batch-parallel-requests branch from bd0c407 to 798598e Compare July 17, 2024 13:59
@kmcginnes kmcginnes merged commit 1d757a4 into aws:main Jul 17, 2024
@kmcginnes kmcginnes deleted the batch-parallel-requests branch July 17, 2024 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Task] Batch parallel requests
2 participants