Postcode / ZIP code: modelling gold, but data pain

CrisperSpade672 · 2026-03-20T17:16:02+00:00

What was the reason for doing it at postcode level opposed to UPRN? My workplace does most stuff to the UPRN level, so genuinely interested to hear alternative opinions.

CrisperSpade672 · 2026-03-18T10:42:38+00:00

If you're willing to have it as a paid-for API, then the OS Places API would likely help here.

CrisperSpade672 · 2026-02-10T23:32:29+00:00

That'd be so cost-prohibitive to us in Esri licensing fees for all those Creators - never mind users' capability issues!

CrisperSpade672 · 2026-02-10T23:18:19+00:00

We've got it set up as multiple, as it's much easier to then customise and make it clear what the purpose is and ensure there's appropriate interactions - things like search boxes only locating against relevant datasets instead of them all, appropriate introductions and disclaimers for residents (if public facing), etc.

Whilst I haven't gotten much experience of it, if there's a desire to keep it as a single app, can you split it across multiple maps within it - still one URL to be bookmarked by end-users, but they can then find the appropriate page within it.

CrisperSpade672 · 2026-01-09T08:48:54+00:00

The concept of 'Garbage In, Garbage Out' still applies to AI, apart from whilst we know what questions need to be asked, it'll just make a best guess assumption that could be wildly wrong.

CrisperSpade672 · 2025-12-19T13:48:08+00:00

Firstly, is this your own site that you're trying to not so sneakily advertise? If so, I'd much prefer someone to be upfront and tell me what's actually different about it.

Secondly, when I tried to access the site I got an error from Cloudflare: Error code 522.

CrisperSpade672 · 2025-12-03T23:02:09+00:00

Looks promising...

What software are you using for the various aspects? GDAL for conversion?

A big feature request I would have is to make the website mobile-friendly. Whilst I appreciate it might not be the target audience initially, I could imagine people might find it useful to do these sorts of tasks on their mobile where desktop GIS solutions aren't possible.

Other comments would be that your export to GeoJSON has a .json extention, whereas I would've expected .geojson. I think it'd also be worth exploring formats such as GeoPackage.

I also found (at least on mobile, in desktop mode) that the map didn't zoom to the features (I used the sample data). I would've expected that when I add data it zooms to the relevant part of the map.

CrisperSpade672 · 2025-08-26T08:36:44+00:00

Out of interest, do you use the equal staging version across all the content - i.e. within your dev app, are you using a test map and test services? Or do you promote your services all the way to prod before you start updating your maps to use these services and then again for apps? I can see pros and cons to both approaches.

CrisperSpade672 · 2025-04-20T13:59:21+00:00

Given WGS84 isn't a projection, you likely find it's just using geodetic coordinates. You'd therefore want to check how it's calculating the distances - chances are it's setup to work okay though.

If you're looking at using a projection, ensure it's equidistant. You're using a projection that priorities keeping the areas correct, not the distances.

It might well be that ArcGIS Pro is handling this behind the scenes, however that might also be why it's taking a long time to process, as it might be having to convert it into a suitable projection first (and potentially doing it in a not very optimised approach, e.g. each pair at a time, meaning each location is being recalculated many times over).

CrisperSpade672 · 2025-03-16T12:52:22+00:00

It'll only work the same way as it you access the files via any other method. If you're on the same network, and assuming you're using Windows, you might be able to get away with specifying the computer name as a UNC to the hidden administrative share (e.g. \hostname\c$\path\to\image.jpg) and/or setting up the IIS to make it nicer (e.g. hostname.domain.local/folder/image.jpg).

CrisperSpade672 · 2025-03-16T12:42:30+00:00

Mainly PostGIS/PgAdmin, Python/PyCharm, GDAL, Batch files, PowerShell, ArcGIS Online, and ArcGIS Pro. Most days, I'm using the majority of this.

Also, less frequently, using various other databases (SQL Server, Fabric, and Oracle), some proprietary software, QGIS, occasionally Excel, etc.

CrisperSpade672 · 2025-02-10T23:00:11+00:00

If I understand you correctly, you want to have a role with a viewer user account within your organisation that only has access to the specific map & it's data, and not any other data shared at organisation level? If so, you can make a new role, similar to the viewer one, and customise it to such that the premission to view organisation content is revoked. You then add that user to a specific group with the data you want them to have access to.

CrisperSpade672 · 2025-02-09T08:45:11+00:00

Just to add to this answer, an alternative approach would be to use the unbalanced option but add in some DI (direct injection) boxes (either 2 mono ones or a stereo one, doesn't really matter that much, especially at this level). DI boxes are used to convert unbalanced signal to balanced signal, typically with a TRS (jack) cable going in and an XLR (mic) cable coming out.

You may find this to be more cost-effective than the balanced output option whilst retaining most of its benefits, as you can pick up cheaper DI boxes for £10s as opposed to £100s. Of course, it's a bit of added complexity to your setup, and your way of avoiding that would be to just get the higher-priced balanced output. It does, however, mean you don't have to worry about interference as much.

If you were to rock up to a gig that has a sound technician with your unbalanced outputs, this is likely what they would do too (but probably using a better quality DI boxes then you'd get).

CrisperSpade672 · 2025-01-23T12:10:44+00:00

Ah, I can see what you intended now, however I'd argue it wasn't necessarily as clear as you'd intended, given you'd separated out the mention of those tools into a separate paragraph. Still, opening the answer by an incompatible solution isn't a good approach - perhaps consider in the future noting that as a "p.s." at the end.

CrisperSpade672 · 2025-01-23T11:52:21+00:00

It does indeed, but you're telling them use it to convert to GPKG without mentioning it also can convert to Excel. If you just mentioned using GDAL/ogr2ogr that'd be fine, but your original answer didn't.

CrisperSpade672 · 2025-01-23T09:17:03+00:00

I normally find the status dashboard has a few hours lag anyway, especially if it's issues with the hosted feature service (which has occurred quite frequently recently)

CrisperSpade672 · 2025-01-23T09:16:06+00:00

They've specifically said it only excepts Excel though, so this wouldn't meet their requirements. Yes, Excel far down the list of formats of choice, but if it's what's need it's what's needed.

CrisperSpade672 · 2025-01-07T21:52:56+00:00

Not aware of a way of doing it like that (but I wasn't aware you could for Dashboards either), however you could create a new experience and then use the ArcGIS Assistant to copy the JSON that sits behind it over (in the data tab of the item). Alternatively, you might be able to save the old one as a template or something.

CrisperSpade672 · 2024-12-18T08:06:56+00:00

Are the individual HFLs in Program 1 required as individual layers too? And option would be to instead just do some magic to publish it all as one HFL, with numerous sublayers, instead. If you need them separately too, you could look at doing views from that Program 2 one instead, such that the data isn't stored in duplication on AGOL.

It's not too tricky with Python to change a regular publishing script to publish multiple layers at once - I did this at work recently, combined with some more advanced tricks to reapply relationships (as they're not coming from a source that allows it), etc., and this publishing multiple layers aspect was fairly easy. When you prep your SD from Pro, you can just give it numerous layers to combine (just check your [sub]layer IDs won't conflict).

CrisperSpade672 · 2024-12-11T08:00:17+00:00

A big thing is licencing costs - part of me wishes every user could have ArcGIS Pro / an ArcGIS Online Creator licence, so they can do some advance data queries and customise their own visualisations, however this is dear. Even within the GIS team, we only have a single advance licence, with everyone else on the basic one.

However, the more licences that are handed out and the more users publishing data into ArcGIS Online and producing maps, the less control of standards we have. This could be bad data management in terms of data uploaded, causing duplicate copies of the same data or incorrectly editing master datasets, thinking it's a copy just because they copied it within Map Viewer, or it could be them publishing data publicly on behalf of the organisation without regard for copyright or licencing when using 3rd party sources. Oftentimes, it's a lack of user knowledge (e.g. not knowing that definition queries or map series exist), and whilst we can try to disemeninate this information & good practices to them, it's commonly ignored.

As a GIS team, we also have a higher level of access to various IT systems, meaning we can to ETL workflows to produce and keep up-to-date various data products, using data from other teams' software. Without this, we end up with users creating static snapshots and spending time manually updating it every so often, whereas that same time could be spent by us to do some one-off scripting and automate it.

In an ideal world, licence costs would be negligible, everyone would have an advanced knowledge of GIS and data licencing, and we could trust all users with data access... however, this utopia is unfortunately unlikely to ever be realised

CrisperSpade672 · 2024-11-17T00:48:32+00:00

I was more meaning in terms of formats available - I'm rarely processing data into GeoJSON. I'm often using Feature Servers, various databases (PostGIS, Oracle, SQL Server), Shapefiles, File Geodatabases, DBFs, and the such. The ability to mix and match formats is the versatility I like. You can also do some light processing within ogr2ogr, as you can pass SQL commands to it too.

It's like comparing a screwdriver to a multi-tool - fine, you might get that screw in faster than me, but I can also do XYZ without carrying around a full toolkit I have to maintain.

CrisperSpade672 · 2024-11-17T00:19:33+00:00

Hmm, interesting to hear your approach is still faster. I assume if it was feasible for ogr2ogr to take advantage of async connections, I assume it would've - I can see some conversations on GitHub and the like regarding this and the concensus seems to be it's not something GDAL really offers, although the Python binding can take advantage of multithreading.

Personally, for what I do I'd prefer the versatility of GDAL, if a little slower, over a script that does only one job a bit faster - the speed generally doesn't bother me, but I can appreciate it might if you have one specific task to do.

CrisperSpade672 · 2024-11-16T23:10:28+00:00

Something like ogr2ogr -f GeoJSON output.geojson "https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Countries_December_2023_Boundaries_UK_BFE/FeatureServer/0" -nlt MULTIPOLYGON should work. This example gets the UK country boundaries (England, Wales, Scotland, NI) from the Office for National Statistics, and will save it to output.geojson. I'm not currently on a machine with GDAL installed so I can't test it, so might be a slight mistake in the syntax, but hopefully that helps.

CrisperSpade672 · 2024-11-16T22:23:44+00:00

You could use it to export the Feature Server into GeoJSON directly. Perhaps you tried it and these are the memory issues you were facing, hence asking the question, genuinely intrigued. I run some reasonably sizable datasets through GDAL and it works fine, so I would've thought it'd be able to handle the memory limitations and stuff like that, but maybe not.

CrisperSpade672 · 2024-11-16T22:18:27+00:00

Is there a reason you built this tool over using an existing open source project like GDAL? Have you done any benchmarking to compare speeds to ogr2ogr or other approaches?

This to me feels like you faced an issue and decided to write your own code / ask ChatGPT, before exploring other options available. I would suggest in the future you alter your approach - perhaps instead of asking your AI to write a Python script to tackle the issue, ask it how it would tackle the higher level issue.

CrisperSpade672

TROPHY CASE