Dft

[1485840691]

What does this PR do?

Support dynamic fine tuning

Fixes #3877 (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[kashif]

thanks @1485840691 can we just have the sft script example only in this PR?

[1485840691]

thanks @1485840691 can we just have the sft script example only in this PR?

@kashif . Thanks, aligned with main branch. The code is from a fork from the trl:main branch. But I created another PR and submit to the main branch of the fork. Seems my account could only create a single fork from trl. So this results in the un-related commits in the change history.

[qgallouedec]

Thanks a lot for this contribution!
I'd prefer having this directly in SFTTrainer.compute_loss. Let's say with a new arg loss_type="dft" (default to "cross_entropy"). What do you think?

[qgallouedec]

Closing via #4042