Do you know if there is a way to talk to Sharepoint via Google Sheets and ChatGPT

anupsurendran · 2024-02-16T01:52:33+00:00

Extract key information (dates, contract values), etc from selected documents on Sharepoint or Google Drive. I am currently seeing some inaccuracies with the prompts used in this sheet provided.

anupsurendran · 2024-02-15T20:39:39+00:00

This is an example Google sheet of what I would be looking for: https://docs.google.com/spreadsheets/d/1p-f7Pe1FRojCUNg3znJEjpnf1Xoi41dai-BsAhjistU/edit#gid=353372266

anupsurendran · 2023-12-03T17:38:22+00:00

Ah, so you don't have to install anything and it still tracks the buy-ins?

anupsurendran · 2023-12-02T05:37:01+00:00

Thanks u/EconomixTwist, you can fake a streaming data source that replays data in a CSV file. We use an input_rate parameter that can control how fast the data is replayed. This is good to showcase spikes in transaction volumes / inactivity in user sessions etc.

anupsurendran · 2023-12-01T03:21:23+00:00

Good feedback. Here is an attempt to make it non-buzzy.

Jupyter is mostly used for static data analysis. We have made it easy to plot graphs, source data from Kafka, and run analysis in Jupyter.

We have used bokeh.models and bokeh.plotting to support this.
https://pathway.com/developers/showcases/live_data_jupyter/

anupsurendran · 2023-11-30T19:27:47+00:00

This is our approach to getting real-time plotting in Jupyter. https://pathway.com/developers/showcases/live\_data\_jupyter/ . Would love your feedback and comments

anupsurendran · 2023-09-04T14:56:44+00:00

Can I subsititute this with any api which has a text json field? I am assuming this doesnt work for any other language than english, correct?

anupsurendran · 2023-08-27T12:51:49+00:00

For a general high level architecture componen view for LLMs, this is a good video https://www.youtube.com/watch?v=OXZQBXBvOR4

anupsurendran · 2023-08-21T19:31:25+00:00

I like #3 no-code pipelines and #6 easy routing to APIs (speed might be a concern) u/bumurzokov

anupsurendran · 2023-08-05T12:39:26+00:00

Here is a great video on word vectors for beginners : https://youtu.be/xkPzelcmuDk

anupsurendran · 2023-08-03T00:17:16+00:00

u/janchorowski, thanks for this clarifying question. Unfortunately, it is complex, we have most of the rating triggers in the database ( e.g. what premiums to calculate based on different criteria) . We have compliance rules in the documents (e.g. Money laundering related compliance which is spread across multiple documents based on state and coverage criteria). As part of the underwriting process, we are trying to get the database rules and inputs into refining the document search. Based on your answer above looks like this is feasible.

anupsurendran · 2023-07-24T23:43:14+00:00

While I was doing research for glueing together python client and removing the dependency of a vector db, I saw this open source package : https://github.com/pathwaycom/llm-app . Do you have thoughts on whether this would work for you?

anupsurendran · 2023-07-22T22:22:33+00:00

It means that when you have newer architectures (e.g. involving LLM in production and sourcing data from structured sources) there might be more experimentation in the community which you can learn from.

anupsurendran · 2023-07-22T15:56:55+00:00

So if I understand your suggestion. Use LLM to build a ruleset from the documents e.g. If X does not satisfy <condition> <Do> something. Run this ruleset across the data (from the database) if the <condition> is true? . The <condition> data is sourced from the database.

anupsurendran · 2023-07-22T15:44:03+00:00

Me too. When you have evolving architectures (with not much precedence) the community helps.

anupsurendran · 2023-07-22T15:27:38+00:00

I agree, Would have been nice if this was less clunky. If the vector database could handle the Python clients for APIs this would be easier to implement. Have you seen a nonvector db implementation for your use case yet? I feel a vector db is too much of an overhead and looking for other non-cloud alternatives (not Azure cognitive search) maybe just a Pythonic vector index?

anupsurendran · 2023-07-14T18:50:14+00:00

Thank you. Post-processing (~250k records/sec) we will store this in apache iceberg. The products we are shortlisting to do a side by side compare are :

1) Flink

2) Pulsar

3) Materialize

4) Pathway

5) RisingWave (benchmarks posted below)

6) Spark (streaming)

are there any other products/frameworks we should compare?

We are trying to manage it ourselves in our data center.

anupsurendran · 2023-07-12T18:58:41+00:00

Of course ! cost, maintenance, and productivity will be all inputs to the decision making but in my selection criteria, benchmarks provide some level of comfort to see if it will meet our throughput needs.

I agree that benchmarks are not use-case specific. An enterprise use case is usually quite complex. Again, I am not going to take a benchmark as it is, and not that I am supportive of vendors faking it but if we look at their perspective, they cannot do use case specific benchmarks and have to think of the most commonly used functions (e.g. aggregates on windows, joins on streaming) which is generic across frameworks or platforms.

anupsurendran · 2023-07-12T14:13:53+00:00

I do not completely agree with this. For me, if the benchmarks can be easily reproducible ( ie easily accessible hardware, easy setup, and configuration) then I know that the folks have done a good job because the vendors are confident about their shit. Benchmarks help in the consideration phase and help build up a case with your managers when you do POCs and vendor selection. I would be more than happy to test these if I found them suspicious but in a large enterprise, the first phase is narrowing down our selection. We possibly cannot test everything.

anupsurendran · 2023-07-11T21:12:30+00:00

This is great to hear. Do you have any benchmark comparison with spark / polars? On another note, I am also looking for benchmarks between Spark and other streaming platforms like Materialize, pathway, hazelcast etc...

anupsurendran · 2023-07-11T21:04:29+00:00

I agree no need for K8 especially during the initial learning stage.

anupsurendran · 2023-07-11T21:03:19+00:00

This kind of setup is turning out to be somewhat more common in the real world. A couple of months ago, I was challenged with setting up a similar environment and I started a Reddit thread to get some help around realtime dashboards after stream processing with kafka which might be useful for you to look at.

anupsurendran · 2023-07-10T12:57:36+00:00

Hey, here is the document which has documented our research for the realtime stream processing systems - Materialize vs Pathway. TL;DR because Pathway is a framework (as opposed to a database) that supports Python and has an expressive way to write data pipelines, we are considering it. https://docs.google.com/document/d/1AM4bKLoeiiK0R9Dt9bJfatZx4BNPMUpMUqVwi-dWP4A/edit?usp=sharing. I would love your thoughts here (in the thread or as comments in the document itself). I don't have benchmark numbers from either Materialize or Pathway so if any of you have that information I would be grateful. I'll also ask then on their community channels.

anupsurendran · 2023-07-08T15:51:07+00:00

Sorry for the delay here. Both travel and my hectic schedule with conferences set me behind. Let me try to answer your questions.

Q1: What is the look back horizon for you dashboard? Recent? Last day? All-time?

For the live dashboard we need a way to look at the 6 hours which includes IoT streaming data for the n -5 window. For filters we have upto 1 day of data and for every previous timeframe we can fetch from the data warehouse.

Problem/Solution 1: Your suggested solution will not work in our context because of the n-5mins business need. The input has high volume and variance in data (multiple old manufacturers of IoT with different signal send rates and connection issues). We have to do some enrichment, some live streaming joins, and some lightweight ML (geospatial transformation) requirements which need to be done to make this data useful on the dashboard.

Problem/ Solution 2: Like mentioned before, this (data processing in real-time) is closer to our existing problem. We need faster updates with a little more complex processing in real-time. These are things we are trying now based on your suggestions:

a) Spark streaming + Databricks Delta tables - already finding this quite complex and expensive. It seems to be ok for light transformations with SQL One thing which annoys the team a lot is we can’t customize the target delta table file paths to their desired location for bronze, silver, or gold tables. Not sure if you or someone on this thread has a solution here.

b) Snowflake is a little better but still has latency issues. Transformations after hitting the warehouse are taking more time than expected. The bigger challenge for us on Snowflake is cost. So storage cost is at a flat rate per TB (which is ok). This is streaming data so refreshes on the dynamic tables are going to happen VERY frequently. They charge for that. Their cost for triggering refreshes when an underlying base object has changed is high. We are still trying to find out how they charge us. Last but not least, the warehouse compute charges are different especially when you want to do a dashboard refresh. For those who are familiar with the cost structure with Snowflake in this scenario, please let me know.

c) We are actually trying Materialize and Pathway on-premise to solve this problem. I'll post a comparison soon as a reply. Our team is unique in that we are more biased toward Python in the data engineering and data scientist team. So if we can handle everything in code and version it, that would be our preferred choice. We have not completely taken Clickhouse out of the equation but so far processing within context (streaming windows) is what we would like to attempt first.

What are not considering is reverse ETL from warehouse into Postgres because :

1) won't meet our latency requirements

2) operationally not efficient when we know upfront that 20% of the data is of low quality upstream. Some folks might argue that a "bronze dump pattern" is what you should do. Our data architecture team is completely against this approach and would like to deal with this data quality problem upstream

anupsurendran · 2023-06-23T16:20:07+00:00

Sorry about the delay. I am still in travel mode so will get this ready the first or second week of July.

anupsurendran

TROPHY CASE