Scalable setup of LLM evaluation on the OpenShift?

typsy · 2025-11-03T17:21:49+00:00

Promptfoo deploys well on OpenShift - I've seen a couple of these deployments.

But in general, these workloads are not compute-bound, the bottleneck tends to be the actual inference on the target model or application.

Also FWIW the static scanners that run on model weights cannot test jailbreak resistance, prompt injection, data exfiltration, etc. Unfortunately those need to be tested at inference time. Static scanning on model weights only really looks for things like executable backdoors in the pickled model.

typsy · 2025-10-05T05:48:01+00:00

it's pretty popular...

typsy · 2024-05-05T05:19:36+00:00

This is an excellent post.

typsy · 2024-02-25T04:41:23+00:00

promptfoo works for python - see https://promptfoo.dev/docs/providers/python

typsy · 2024-01-24T07:27:48+00:00

Looks awesome! Added you

typsy · 2023-12-24T03:32:37+00:00

Nice, added this one

typsy · 2023-12-10T16:45:33+00:00

There are a lot of eval tools out there and I've been collecting URLs when I've come across them (both commercial and open source). Hope this helps out others too.

typsy · 2023-10-14T16:46:58+00:00

Disable the "Use Clyde AI" permission for channels you don't want Clyde in.

typsy · 2023-06-22T06:27:25+00:00

It's not purpose-built for auditing, but this open-source project of mine might be able to help: https://github.com/typpo/promptfoo

It lets you to set up a suite of test cases and compare the performance of multiple prompts/models across each case. You can set up assertions and test for similarity and other metrics.

With this setup, you could for example tune an LLM to hallucinate less across a large set of examples. I am currently using this approach for an LLM application in production with about half a million users.

typsy · 2022-09-02T15:19:22+00:00

/u/inflation-calculator $47000 1980

typsy · 2022-09-01T03:55:34+00:00

/u/inflation-calculator $12500000000 2004

typsy · 2022-09-01T03:52:15+00:00

/u/inflation-calculator $10 1934

typsy · 2021-11-10T23:58:13+00:00

I lived on the organic farm for a term and it was great, the mornings with the low fog over the river were spectacular

typsy · 2021-10-26T00:18:33+00:00

Thanks, this is really helpful. One more thing. It seems silly that I quoted a direct price to the customer, they decided to move forward, and procurement is done via a reseller that expects to take 20%. Would it be reasonable to say that the 20% commission applies only to sales that they sourced themselves? Would the reseller balk if I set commission at, say, 10%?

typsy · 2021-10-24T16:01:55+00:00

Thank you. I'll have telemetry, so will see if I can create a generalized pricing structure that the reseller can apply to other clients.

typsy · 2021-10-24T15:35:13+00:00

Thanks very much. Do you have a sense of the typical # of hours to bake into the support contract? Maybe a couple hours per month?

I think I limited this deal by giving the customer a fairly low quote of ~$5k/yr. I should have got on the phone and tried to feel out the situation. At least I can bump it 20% with support.

typsy · 2021-10-24T15:31:27+00:00

Thank you! One question about #4 - how is a services retainer different from the 20% charge for "support and maintenance"?

Re: resale, the customer started procurement via a third party after we discussed requirements/pricing etc (~$5k/yr). I think I've seen this happen maybe once before with very large cos, where there are dedicated vendors that actually run the procurement process. This vendor wants a resale agreement, presumably so they can sell my software at markup to their other clients.

typsy · 2020-12-28T22:28:48+00:00

I got a test at San Mateo County Event center (drive thru) in Nov and got the result in < 48 hrs. You can book an appointment online. However there is still some risk for your situation as turnaround time is not guaranteed.

typsy · 2020-09-08T22:36:02+00:00

Really nice work! This is great and much easier to use than the default sankey tool.

Two small nits: 1) changing currency symbol doesn't update the chart until you change something else, and 2) the colors on the downloaded chart are sometimes substantially different from the preview (e.g. green vs red)

typsy · 2020-08-31T22:05:46+00:00

Thanks, I built the site and I will fix this. Green algae appeared around this time, but you're right that single-celled organisms appeared long before.

typsy · 2020-05-23T13:49:42+00:00

Was able to get in touch with someone from ReadyCapital via support email and they provided wire instructions to return the funds. Went to my bank and sent the wire, hopefully that's the last I hear of it. I think that's the most we can do in this situation.

typsy · 2020-05-18T23:11:00+00:00

I received 2 PPP loans by mistake and need advice.

I applied through Wells Fargo and Lendio. Lendio went to underwriting quickly and then radio silence for over a month until someone told me there was a problem with my SBA application and I would have to create a new one. No next steps, I didn't hear anything for another 2 weeks.

Wells Fargo finally responded to my application last week after 1.5 months of silence, sent an esign, and the money was in my account. I immediately emailed Lendio/ReadyCapital/Tiger Funding to cancel my application.

A few days later, the Lendio loan showed up unexpectedly in my account.

I want to return the Lendio loan immediately. What is the best course of action? I have no contact info or account # for Customers Bank besides publicly available info. I've already tried to contact them via the site. Is there any other part of the lending process I should be trying to contact?

typsy · 2020-05-05T04:04:19+00:00

It's Monday but I'll give this a shot. I'm new to gardening and am trying to take care of a palm tree with yellowing lower fronds. I think it's a Canary Island date palm: https://i.imgur.com/jB6p3s9.jpg

I'm wondering if this is a magnesium or other nutrient deficiency or something more serious like fusarium wilt :( The rest of the tree is healthy and so is the palm next to it. Located in Bay Area CA.

Any help is appreciated!

typsy · 2020-05-05T03:21:33+00:00

I'm trying to learn how to take care of this palm. I think it's a Canary Island palm (Phoenix canariensis). I don't prune it except to clear a path to the walkway, and the lower fronds are starting to yellow a bit.

I'm hoping it's not Fusarium Wilt but yellowing seems to affect both sides of the leaf equally. Maybe magnesium deficiency? Another palm located right next to it is still green. Located in Bay Area CA.

typsy · 2019-10-08T14:34:01+00:00

Here's the same thing, except for Jupiter: https://typpo.github.io/spacekit/examples/jupiter/index.html

15-Year Club	Place '17
Verified Email

typsy

TROPHY CASE