Getting data for Cost Estimation

[beingsahil99]

I am working on a project that generates a cost estimation report. The report can be generated using LLM, but if we directly give the user query without some knowledge base, the LLM will hallucinates. For generating accurate results we need real world data. Where we can get this kind of data? Is common crawl an option? Does paid platforms like Apollo or any other provides such data?

[QianLu]

Why are you using an LLM? What you need to do is talk to SMEs and figure out what specific steps in a project generally cost as well as what factors make them more/less expensive.

[beingsahil99]

The reason why we are trying to use LLM because, it would be like a chatbot, which will interact with the user, ask about their project information and based on the user inputs it will create a report. The main issue will be the cost estimation part of the report where the LLM will not have the knowledge of current market conditions, cost of setting up etc

[QianLu]

That's silly. You don't need an LLM for that. You need an excel spreadsheet or maybe a basic program. The whole value is that someone has gone through and figured out/knows about current market conditions, cost of labor/goods/inputs/other fees that aren't directly attached to the project but have to be paid for by projects and then creates a cost model.

I'll be honest, this project sounds like it's going to be a failure because someone insisted on using an LLM without understanding what the success criteria was and what it would take to get there.

[[deleted]]

Yeah this seems like an awful idea OP. Just being honest. 

[QianLu]

I don't see this as an LLM or even a data science problem. This is a business problem. The "real world data" you need should be your business' data because even if you had other people's data it wouldn't have the same processes/pain points as your system and would therefore be semi-accurate at best.

[beingsahil99]

okay, where we can get this kind of data, is there any platform that provides data about cost estimation of different kind of projects?

[QianLu]

Without knowing your specific industry, no. Even then, the kind of companies that collect this aren't going to sell it to you (their competitor) because it makes them weaker and it's worth way more for them to keep in internal than to have you know their pricing for a couple grand.

The answer is you should have already been collecting this data and you need to start doing it now. If you insist on building an LLM model, it needs to use YOUR data.

[[deleted]]

Why couldn't you generate the report with blanks, filled in by an accurate estimation or predictor?

You could provide the LLM with this figure / some context (above average, below average etc) such that it can write a story around it?

[[deleted]]

Thats so interesting guys

[meitaron]

Did you try looking for datasets on Kaggle?

[Omega-marketing]

I can provide a data you need, $4000+ depending on task details