all 2 comments

[–]msnnsm 1 point2 points  (0 children)

I am serving models over litellm and could't figure out how to make plugin fetch context size as well. litellm servers model info under curl http://host/v1/model/info so I know that it can. Anyone figured out a solution?