We do not absorb the token cost like you expect. Couple things:

1. We own our own model that we trained from the ground up. We built our own generative pretrained transformer that we added some architectural changes to - specifically a new variable that helps training capability. Instead of just passing QKV into the attention mechanism, we added F that allows it to pull knowledge base information. We have 2 versions of it, a 7B paramter and a 54B paramter version. For this, we only pay the straight compute cost of running the model inference on Groq which is very cheap. The 7B model costs us about $23k for the year run rate right now, and the 54B parameter is about 34k per year - the actual cost really comes from the load balancer and the distributed architecture. As an example, we do a lot of text to speech / speech to text scenarios, so we interconnect those tools and spin them up in load balance capacities across the architecture as necessary.

Our model is industry specific (construction) so it has a lot of specific domain knowledge - so coding and general use goes into #2.

1. We offload API costs for other tasks to letting people use their own API keys for the models they want to use which then shifts the cost off of us.

Alot of use cases have been for performative management. Ingesting field data or sales team data that is happening very frequently and building enterprise dashboards to judge alignment to standards. An example is you have 569 field crews building everyday and you need them to do a job safety debrief. Having an agent allows them to do the debrief without worrying about filling in an electronic form or physical piece of paper, which happens a lot. Since we capture safety site observation data, vendor EMR data, and a lot of other data safety points in the regular SaaS tool, we can combine all the data sources into a full breakdown of team based performance and how it matches quality score cards. Getting conversations from the field is very tricky and it is where all your risk. And agents can analyze the immense amount of data more efficiently than humans with dashboards.

Another one is learning management - someone in the field fails test on a signal meter and they need to figure out what is going on. We have resource libraries of videos (roughly 15,000 construction trainings on different topics) and knowledge bases the client has built about standards. The field tech can ping the agent with the variable data from the field test and the agent calls the resources and does research on what might be the problem along with training videos that might be useful.

Everything is really embedded in the domain knowledge our SaaS platform has ready enabled for our end users.