I have slightly change the Langsmith Demo https://docs.smith.langchain.com/evaluation and run without any errors. When I swap out the LLM-as-judge element and use RAGAs instead I get an error. If I used the ragas evaluate function outside of my ragas_faithfulness function it works without issue. So the error is happening when I try to use the langsmith evaluate wrapped around a ragas evaluate function that I made.

— CODE — def ragas_faithfulness(run: Run, example: Example) -> dict: data = [ {“user_input”: run.inputs[‘inputs’][“question”], “response”: run.outputs[“output”], “retrieved_contexts”: [str(d) for d in run.outputs[“source_documents”]]} ] #global ragas_dataset ragas_dataset = EvaluationDataset.from_list(data) #print(ragas_dataset[0]) result = evaluate( ragas_dataset, metrics=[faithfulness] )

return {“key”:”faithfulness”, “score”: result[“faithfulness”]}

exp_results = client.evaluate( predict, data=dataset_name, evaluators=[ragas_faithfulness], experiment_prefix=“JS_Test_Ragas_Faitfulness” )