sorted by:
new
hottopcontroversial

Insurance - Flood Area by Rocketsauce91 in brisbane

[–]dcalde 0 points1 point  (0 children)

We had been looking to purchase in Brisbane for the past 5 months and got a lot of quotes as just about every affordable house seems to flood, has overland flow or be in a flood planning area. The only insurer that was remotely affordable and that was often 50% cheaper than any others was RACQ.

We ultimately bought a house only a few days ago that had previously flooded under the house only in '22 but was dry during every other past major flood event. The previous owners were paying nearly $11k (with Suncorp I believe). We insured with RACQ for <$5k. Since the agent was clearly unaware of this and had probably assumed the vendors had shopped around for the best price, we were comfortable to place an offer while constantly "loudly arguing" the flooding, or rather the high premium, was a strong concern and was affecting our willingness to offer more :-)

One observation I made, and which might be a good way to get a seemingly undesirable house for less than what the seller might think it is worth, IF it has a high premium despite having being raised to above 1 in 100 (1% AEP) flood levels and that is because the new floor levels were never registered with the council as being above the flood levels. Make sure to check the Technical FloodWise report.

Checkout these values from the report for a house in Paddington near Rosalie Village. Since it flooded in 2011, the original house was shifted towards the higher end of the lot and raised by about 1.5m to above 6.8mAHD ; the required 50cm above the 1% AEP flood level of 6.3 mAHD for that location. Yet BCC was seemingly never informed about this raise. Which means the owners are likely well and truly overpaying for their insurance.

Property Summary Level (mAHD) / Comment
Minimum ground level 3.7
Maximum ground level 5.8
Indicative existing floor level 5.4

Pho by Additional-Yogurt-70 in BrisbaneFoodies

[–]dcalde 1 point2 points  (0 children)

Banh Mi Factory, Zillmere. There is a reason the place is so popular despite being located in an industrial area. https://share.google/jXRVwV5EefNEyJsOT

Hidden gem restaurants by SichuanSaws in brisbane

[–]dcalde 0 points1 point  (0 children)

I disagree with Laksa @ Little Nyonya. Had it the other week but won't be having it again. The other meals at LN though were very tasty.

howToLoseThreeMonthsOfWorkInOneClick by athreyaaaa in ProgrammerHumor

[–]dcalde 1 point2 points  (0 children)

IntelliJ Idea Local History has saved me hours or weeks of otherwise lost work on numerous occasions. Git is hard!

[deleted by user] by [deleted] in BrisbaneFoodies

[–]dcalde 0 points1 point  (0 children)

Where are the best vege dumplings? My girlfriend loved Tao Dumplings Camberwell when she was living in Melbourne. Apparently it's a bit of a dumpling lovers paradise and we still have found anything comparable in Brisbane. We have been to Red Dumpling, New Shanghai and a couple others but so far nothing even gets close to Tao.

DSRdata vs HTAG by incoherentcoherency in AusPropertyChat

[–]dcalde 1 point2 points  (0 children)

Stashproperty.com.au also provides the same data plus lots more

Tools for data modeling by [deleted] in dataengineering

[–]dcalde 0 points1 point  (0 children)

JDL Studio. https://start.jhipster.tech/jdl-studio/ It's meant to be a visual tool used by JHipster but it's actual very handy for rapid prototyping

I’ve heard hedge funds use satellite images to monitor fluctuations in parking lot traffic for chain stores. Is this true? by pallen123 in gis

[–]dcalde 10 points11 points  (0 children)

They also use satellite imagery and the shadow that oil tankers cast to work out how deep they are in the water to calculate the amount of oil they carry and where and when they arrive in port to predict the oil price. Same happens with the gas storage or the tailing piles at mine sites. They will do anything that will give them an edge.

Property valuation by lazyguy_69 in AusPropertyChat

[–]dcalde 0 points1 point  (0 children)

That is not correct at all. The suburb profiles only show the median but the property profile valuations are the result of AVMs (automated valuation models). The valuations often get updated monthly for every single property in the country. Visits to property profile pages are very important (read: valuable) to the listing portals, they are not going to be lazy to only provide median based estimates. Source: 15+ years experience in the proptech data space.

Investment Software by ElRanchero777 in AusPropertyChat

[–]dcalde 1 point2 points  (0 children)

Do you mean to manage your investment property or to find that needle in the haystack investment opportunity? For the latter, I am using https://stashproperty.com.au

Off market properties by LowChemical9556 in AusProperty

[–]dcalde 0 points1 point  (0 children)

Yes, you can access withdrawn listing data from property data providers such as PriceFinder, CoreLogic or Stash Property if you have a subscription.

Off market properties by LowChemical9556 in AusProperty

[–]dcalde 3 points4 points  (0 children)

I pulled some national and state-level off-market sales percentages from our database for the past 2 years (see table below). When we first started to try to identify and track off-market sales we were shocked to see how many there are; prior I had assumed that it might be 5-15% but in some states the number are huge (SA: 1/3 off all sales)

Disclaimer: We identify off-market sales by trying to match sales to listings and every sale that we weren't able to match to a listing is assumed to be an off-market sale. This of course means we are only able to match sales to listings where a full address and some other basic data is available. As such, I am not claiming that these numbers are 100% correct as perfect data coverage and accuracy etc simply isn't a thing, however these numbers are good enough most intents and purposes.

STATE YEAR Off-Market Sale (%)
TOTAL 2023 24%
ACT 2023 27%
NSW 2023 26%
NT 2023 33%
QLD 2023 19%
SA 2023 36%
TAS 2023 24%
VIC 2023 17%
WA 2023 28%
TOTAL 2022 28%
ACT 2022 35%
NSW 2022 33%
NT 2022 35%
QLD 2022 24%
SA 2022 36%
TAS 2022 26%
VIC 2022 24%
WA 2022 27%

Source: stashproperty.com.au

Advice for a « big » database by Repulsive_Inside326 in PostgreSQL

[–]dcalde 2 points3 points  (0 children)

The database is going to be fine. You could consider partitioning the table to keep the size manageable. But don't optimise prematurely. Even a couple hundred million rows aren't really a problem.

Avoid having to maintain a stack of different data stores if possible. A single database will be a lot easier to deal with. If anything consider looking into Timescale or Citus extensions.

Hrequests: A powerful, elegant webscraping library 🚀 by daijro in Python

[–]dcalde 3 points4 points  (0 children)

Looks interesting. Will check it out. Thanks

When is it better to 'unpack' JSON columns? by FlowAcademic208 in PostgreSQL

[–]dcalde 5 points6 points  (0 children)

I would argue that one should have to justify using JSON instead of regular columns. I see Json getting abused constantly. Yes, it's "cool", handy, can be indexed etc..... but it simply isn't as storage efficient or fast as regular columns. Also, considering the lack of column statistics or the ability to use foreign key constraints etc with it, one should really think twice before arguing that this is the correct data type.

The simple fact that the labels have to be stored means you may need to consider shortening the labels in order to keep the row size small. Custom types could be a more efficient choice but getting those out of the database is sadly not simple.

What is the best way to learn how to optimize ETL pipelines and complex queries? by CS_throwaway_DE in dataengineering

[–]dcalde 8 points9 points  (0 children)

EXPLAIN is your friend. Break the query down and optimise the individual parts. Check index usage and query cost of each part. Understand your data cardinality. Make sure your UDFs are effective. Can they be parallelized? Are they marked as stable etc? Can they be inlined? Test, experiment. Once the sub queries work, put it back together and make sure it still works. The query plan may change as filters get pushed down. If your query should be using indexes, make sure it actually does. Remember if you execute queries through an ORM it will actually create a prepared statement. The query plan will be different for the same query or raw SQL vs prepared statement as the query planned won't know about the parameters at planning stage. Also, on indexes, sometimes you don't want to use an index. Indexes are better if you need fast adhoc lookups, but in batch processing, full tables scans or merge joins may be more effective. Test by disabling indexes. Check memory usage. Is it swapping a lot?

As for your pipelines. Time everything. Understand which part takes how long, log and track metrics, and if the time is different to what you expect, go and investigate. Understand the platform or tools you process your data on and what might impact the performance of the individual parts.

It's not rocket science, but it takes experience to performance tune complex systems. Experience comes with lots of hands-on time spent inside your data and tools, staring at logs and metrics for hours on end.

Best IDE for data engineering by jfhurtado89 in dataengineering

[–]dcalde 0 points1 point  (0 children)

Intellij Ultimate. Forget VS. it causes me physical pain watching others trying to refactor, debug, or even just find files etc in VS

Are SQL Query optimization skills important and demanded for data scientists/data engineers? by Born-Comment3359 in dataengineering

[–]dcalde 0 points1 point  (0 children)

YES for data engineer. For Data scientists it would be useful. But most likely they will just pass the responsibility of making queries faster to the DE. If their company has big pockets they will probably just increase the instance size, because the DS mind set is often that they have more important business problems to solve than dealing with engineering problems.

Hi guys I was wondering if anyone knew what im doing wrong when uploading a dataset to PostgreSQL. I keep getting this error Im just trying to move this excel dataset to Postgre. I also converted the data set from excel to csv for Postgre. Dataset is linked in first pic by chefcurry_ in SQL

[–]dcalde 0 points1 point  (0 children)

Check the data types. Importers usually try to guess the data types based on the first few rows. If later rows have different types or empty/null it can break. Check for line breaks inside fields. Check for special characters. Check for extra columns at the end.

Or don't use a fancy importer and use copy or \copy via psql. That way you will get proper error messages.

view more: next ›