-
Notifications
You must be signed in to change notification settings - Fork 830
Description
DiskBasedResponseCacheProvider
supports an options file that can be used to tweak the default time-to-live for cache entries and a cache mode setting which can be used to completely turn off caching, or to completely block calls to the LLM when there is a cache miss.
AzureStorageResponseCacheProvider
does not support similar options file or cache mode functionality - instead it supports tweaking the default time-to-live in code via a constructor parameter.
The cache modes functionality as well as the options file above is pretty specialized and should not be required in normal usage scenarios. We ideally don't want to support this functionality out of the box in the eval library. For advanced usage scenarios that require the special mode above to block LLM calls on cache misses, it should be possible (and relatively easy) to implement a custom IResponseCacheProvider
s with that behavior.
Logging this issue to remove the options file and modes functionality, and to align the constructors for DiskBasedResponseCacheProvider
and AzureStorageResponseCacheProvider
such that both support tweaking the default time-to-live for cache entries via code.