Change pip_index_url to pip_index_urls list in Katib SDK

[wassimbensalem]

Problem

Katib SDK currently supports only ONE pip index URL via pip_index_url parameter. Users cannot install packages from multiple sources simultaneously (e.g., public PyPI + private repos).

Use Case

ML teams need both:

• Public packages: scikit-learn, pandas from PyPI
• Private packages: Company ML libraries from private repos

Currently forced to choose one OR the other, not both.

Proposed Solution

Replace pip_index_url (string) with pip_index_urls (list).

# Current (limitation)
client.tune(
    name="experiment",
    pip_index_url="https://my-repo.com/simple/"  # Only one repo
)

# Proposed (multiple repos)
client.tune(
    name="experiment", 
    pip_packages=["scikit-learn", "my-private-lib"],
    pip_index_urls=[  # List of repos
        "https://pypi.org/simple/",
        "https://my-repo.com/simple/"
    ]
)

# Default behavior (if None provided)
client.tune(name="experiment")  # Uses PyPI only

Implementation

1. Change pip_index_url parameter to pip_index_urls (list) in tune() method
2. Create utility function in utils.py to format list as --index-url (first) and --extra-index-url (rest)
3. Modify get_script_for_python_packages() to handle the list
4. Default to PyPI when pip_index_urls=None

Files to Modify

• sdk/python/v1beta1/kubeflow/katib/api/katib_client.py
• sdk/python/v1beta1/kubeflow/katib/utils/utils.py

I am willing to implement and submit PR.

/area sdk
/good-first-issue

Why is this needed?

ML teams often need packages from multiple sources in the same experiment:

Public packages (scikit-learn, pandas) from PyPI
Private packages (company ML libs) from private repos

Currently Katib forces you to choose only ONE repo, blocking common enterprise ML workflows where you need both public and private dependencies.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[google-oss-prow]

@wassimbensalem:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

What you would like to be added?

Problem

Katib SDK currently supports only ONE pip index URL via pip_index_url parameter. Users cannot install packages from multiple sources simultaneously (e.g., public PyPI + private repos).

Use Case

ML teams need both:

• Public packages: scikit-learn, pandas from PyPI
• Private packages: Company ML libraries from private repos

Currently forced to choose one OR the other, not both.

Proposed Solution

Replace pip_index_url (string) with pip_index_urls (list).

# Current (limitation)
client.tune(
   name="experiment",
   pip_index_url="https://my-repo.com/simple/"  # Only one repo
)

# Proposed (multiple repos)
client.tune(
   name="experiment", 
   pip_packages=["scikit-learn", "my-private-lib"],
   pip_index_urls=[  # List of repos
       "https://pypi.org/simple/",
       "https://my-repo.com/simple/"
   ]
)

# Default behavior (if None provided)
client.tune(name="experiment")  # Uses PyPI only

Implementation

1. Change pip_index_url parameter to pip_index_urls (list) in tune() method
2. Create utility function in utils.py to format list as --index-url (first) and --extra-index-url (rest)
3. Modify get_script_for_python_packages() to handle the list
4. Default to PyPI when pip_index_urls=None

Files to Modify

• sdk/python/v1beta1/kubeflow/katib/api/katib_client.py
• sdk/python/v1beta1/kubeflow/katib/utils/utils.py

I am willing to implement and submit PR.

/area sdk
/good-first-issue

Why is this needed?

ML teams often need packages from multiple sources in the same experiment:

Public packages (scikit-learn, pandas) from PyPI
Private packages (company ML libs) from private repos

Currently Katib forces you to choose only ONE repo, blocking common enterprise ML workflows where you need both public and private dependencies.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wassimbensalem]

/assign

[andreyvelich]

Thank you for opening this @wassimbensalem!
Yeah, this is known issue, as a workaround you can do the following with your pip_index_url:

pip_index_url = "https://pypi.org/simple/ --extra-index-url https://private-repo.com/simple"

Since we just insert the string here.

Would it work for you ?

[andreyvelich]

/remove-label lifecycle/needs-triage

[andreyvelich]

cc @kramaranya @astefanutti @szaher for the same use-cases in Kubeflow SDK.

[wassimbensalem]

@andreyvelich thanks a lot for the reply. I think it should work, but is my implementation in the PR wrong or not needed?