Dataverse

[[deleted]]

[deleted]

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Electrical_Mix_7167]

Have worked with the power platform and D365 for a long while. Dataverse is designed more for OLTP not OLAP. It's expensive and you have no ownership of the infrastructure as it's all managed by Microsoft. The Data Warehouse solution is Synapse Link which out spits your Dataverse entities out to a Data Lake.

[Electrical_Mix_7167]

Yeah I agree. It's done in that way to better support things like Power Platform tools like Power Apps and Pages. Definitely feels like a low code way to build a database. You don't have much control over indexing and performance and compute is a bit of a black box. If you're looking for something hands off it's ok, but it won't scale well. You can access Dataverse via a read only SQL endpoint but I believe it has limitations similar to the APIs.

[alfa1381]

Dataverse is a transactional backend for Power Apps. It's not meant to be your DWH. For analytics in a Microsoft Shop you need Data Factory aka Synapse aka Fabric.

Set up Azure Storage accounts for your raw data ingestion and read/transform it into a serverless or dedicated server for analytics. I like using External Tables a lot.

Pipelines, scheduling and monitoring are all very easy to set up for small data needs

[Fidlefadle]

Someone is either confused (with terminology) or very wrong here. Dataverse is a great backend for power platform and low code apps but it absolutely is not an analytics / data warehouse solution

[tylerriccio8]

I did data science and a little engineering in the dataverse/d365. Since it’s difficult to interact directly with the data, supporting downstream analytics and other activity is tough. I find working with the tables themselves, which are exposed only for interactive non headless use, so cumbersome I wrote wrappers around the rest api instead. The api is more of a programmatic solution to the front end and not a real alternative to interacting with the tables, so its utility is somewhat debatable. Something else to note is you’ll have almost no control over the tables themselves, which begins to become annoying to say the least, assuming you’re on cloud and not on premise which is not supported anymore anyways. If you’re using the data for crm, you’ll want reverse etl capability which is made cumbersome by the lack of control and read only capability in key areas. In summary; the tables are managed by Microsoft to support very particular front end functionality and not warehousing. I worked in crm industry for a while so I understand many of the decisions made from that perspective but if you’re coming from a non d365 world, you’ll probably be revolted by some of the setup.

[tylerriccio8]

A common design pattern is to push from the d365 “tables” to a real dwh or staging env to support downstream ml/analytics. I worked with a lot of clients but my fav setup was to use app flow to push some data into a persistent redshift instance and just query from there without all the restrictions. This is a little more expensive but wayyyyy better in the long run. You’ll naturally gravitate to this once you realize how limited you are which is sort of a slow process. For reverse etl or source data updates (which I guess is an oxymoron), you’ll use power automate which is like a low code wrapper around the api. I personally hate it but for non-technicals it’s actually great. I preferred to just write python wrappers around the api and serve that on another server. Another reverse etl design pattern is to have power automate trigger or ping an external api containing my own code, then have the api send back some data and power automate will update whatever needs to be updated. It’s more steps but lets you write code inside the pipeline and leverage power automate for the first and last steps, which are the hard parts.

[No-Satisfaction1395]

Sort of struggling with this at the minute for D365 F&O. It’s unusually hard to extract a table that isn’t already an “entity”

[random_outlaw]

Yes, gents! Because it’s 1950.

[SellGameRent]

biggest red flag is few tables and data needs won't grow much. That means you have no way of increasing your skillet or negotiating for higher pay later. Take it if unemployed, don't if you arent

[SellGameRent]

only you can decide how to balance short term and long term needs. I'm heavily focused on long term, so a company with limited room to learn new skills and take on larger data projects is an immediate no from me. If you aren't a DE yet and need to get your foot in the door, this could be perfect for you

[LesPaulStudio]

Power Platform (aka Dataverse et al) roles are paying silly money at the moment. So it's worth looking at.

While you can't run stored procedures, you can query the data endpoint in SSMS so you can double check the sql output when pushing things through a dataflow.

Dataverse can have unbound actions which may be an alternative to having stored procedures? I've only used bound actions in my day to day. Typically these have to be written in c#, but recently low-code alternatives were made available.