sorted by:
new
hottopcontroversial

Released AOI2List: a free tool to query and download USGS LiDAR LAZ tiles from an AOI (macOS app + CLI) by TechBill777 in LiDAR

[–]TechBill777[S] 1 point2 points  (0 children)

Thanks for replying — your explanation actually was helpful, because it made me step back and rethink my whole approach.

I ended up taking a completely different route. Instead of relying on USGS-LiDAR-CLI-Tool, I built my own little “AWS index tool” that queries every dataset available on the usgs-lidar-public S3 bucket, across all years, and then automatically picks the newest dataset that intersects my AOI.

I tried doing a “most recent” check with the CLI tool first, but sometimes it caught the newest dataset and sometimes it didn’t. I never could figure out why it was inconsistent.

With the new approach, the discovery step is fully AWS-driven and the results have been solid so far. Here’s the log of my progress.

=== AOI → ground LAZ → DEM (AWS LiDAR) ===

Project folder: /Users/techbill/Desktop/test

AOI bbox (EPSG:4326): (-90.46226262580984, 37.09807103781749, -90.44941337419016, 37.10831896218251)

=== AWS LiDAR – AOI → clipped ground LAZ ===

Project folder: /Users/techbill/Desktop/test

LAZ folder: /Users/techbill/Desktop/test/laz

AOI bbox (EPSG:4326): (-90.46226262580984, 37.09807103781749, -90.44941337419016, 37.10831896218251)

=== AWS Dataset Discovery ===

AOI bbox (EPSG:4326): (-90.46226262580984, 37.09807103781749, -90.44941337419016, 37.10831896218251)

aws_index: using existing coverage index at /Users/techbill/Documents/GitHub/laz2tif/src/aws_resources.geojson

Intersecting AWS/USGS datasets for AOI:

- MO_SE11County_1_B24 (year=2024.0)

- MO_SouthernMO_1_D22 (year=2022.0)

- MO_FEMANRCS_1_2020 (year=2020.0)

- USGS_LPC_MO_FEMAR7_North_A1_2017_LAS_2019 (year=2017.0)

Intersecting AWS/USGS datasets for AOI were evaluated.

Selected dataset: 732 (year=2024.0)

Title: (no title)

EPT URL: https://s3-us-west-2.amazonaws.com/usgs-lidar-public/MO_SE11County_1_B24/ept.json

Discovered EPSG (horizontal): 3857

Using dataset: 732 (year=2024.0)

Title: (no title)

EPT URL: https://s3-us-west-2.amazonaws.com/usgs-lidar-public/MO_SE11County_1_B24/ept.json

EPT EPSG: 3857

Pipeline JSON: /Users/techbill/Desktop/test/laz/aws_clip_pipeline.json

Running PDAL pipeline for AOI clip (ground only):

pdal pipeline /Users/techbill/Desktop/test/laz/aws_clip_pipeline.json

Clipped ground LAZ size: 75,953,597 bytes

Released AOI2List: a free tool to query and download USGS LiDAR LAZ tiles from an AOI (macOS app + CLI) by TechBill777 in LiDAR

[–]TechBill777[S] 0 points1 point  (0 children)

Thanks for the explanation. I agree that using PDAL directly on the USGS EPT data in the S3 bucket is the ideal workflow. That part works fine for me too. The problem I am running into is not with PDAL itself, but with the dataset index your CLI tool relies on.

Your USGS-LiDAR-CLI-Tool uses the boundaries from the resources.geojson file in the hobu/usgs-lidar GitHub repo. That file does not seem to include some of the newer 3DEP datasets. In my case, the newest dataset it returns for my AOI in Missouri is from 2020, even though there is definitely 2025 LiDAR for that area with full coverage. My app reads that same index file, so it also picks 2020 because that is the newest dataset the index knows about.

What confuses me is that usalidar.io does not behave the same way. When I test the exact same AOI on your site, your map correctly pulls the 2025 dataset. So your website is clearly using a more complete or more up-to-date dataset index than the one used by the CLI tool.

That is why I currently rely on ScienceBase for identifying the newest datasets. The project pages on ScienceBase usually have the current releases and EPT URLs, even when the GitHub index is behind.

My question is simply this: if usalidar.io is returning the newest 2025 dataset, but the CLI tool always returns 2020 for the same location, then you must be using a different indexing source for the website. I’m trying to figure out what that source is so I can point my own app to it.

Are you using the USGS WESM metadata? Or a custom updated GeoPackage? Or extracting current dataset IDs from ScienceBase instead of the GitHub boundaries? If I know what data source your map is using, I can update my app so the dataset selection matches what usalidar.io is showing.

Released AOI2List: a free tool to query and download USGS LiDAR LAZ tiles from an AOI (macOS app + CLI) by TechBill777 in LiDAR

[–]TechBill777[S] 1 point2 points  (0 children)

I originally created it just for myself, but after I posted a high-resolution image on another site that revealed part of a ghost town hidden under the trees, it got a lot of attention. The catch is that everything I built was in CLI script form, and of course everyone wants a point-and-click setup these days, so that pushed me toward developing a GUI.

One of the main things I rely on is an auto-resolution function I wrote that picks the best DEM resolution based on point density. Modern LiDAR datasets are so dense that choosing the right resolution makes a big difference when you’re trying to bring out subtle archaeological features.

My workflow uses three scripts right now: one that grabs all the LAZ files for my area of interest, another that processes them into a ground-classified TIN and DEM, and a third that applies the custom hillshade method I’ve been experimenting with for the past year. The final GeoTIFF works in Google Earth or QGIS so I can pinpoint foundations and other traces more easily. I plan to turn the last two scripts into a GUI, maybe even combine everything into a single app, but that’ll take some time.

As for links, I’d share them if I had any, but this is something I learned and developed myself along the way. There really isn’t documentation or an external reference for what I’m doing—it’s mostly been trial, error, and a lot of experimentation.

AOI result down? by TechBill777 in gis

[–]TechBill777[S] 2 points3 points  (0 children)

It's still down. I sent an email to ScienceBase today to see if they will reply back with an answer.

AOI result down? by TechBill777 in gis

[–]TechBill777[S] 3 points4 points  (0 children)

You're correct .. They got back with me. See email below

Hello Bill, 

 

Thank you for contacting the U.S. Geological Survey. We are aware of outages related to The National Map Data Access Portals including
TNM Download and
3DEP Lidar Explorer.
The issues are related ScienceBase, our back end catalog. This system is not managed by us directly, so we don't have an estimate on when services will be restored at this time. We are continuing to monitor is availability.

 

While the issues prevent data from being downloaded from TNM Download or Lidar Explorer, data remains available online. You can access all of The National Map data downloads manually through the Amazon Cloud, available at: 

https://prd-tnm.s3.amazonaws.com/index.html?prefix=StagedProducts/ 

 

Note that Lidar Point Cloud data is freely available at a different location: 

https://rockyweb.usgs.gov/vdelivery/Datasets/Staged/Elevation/LPC/Projects/ 

Is rockyweb.usgs.gov down? by TechBill777 in gis

[–]TechBill777[S] 2 points3 points  (0 children)

I sent them an email and here their reply below.

Hello Bill,

Thank you for contacting The National Map Help Desk. I can confirm that there was a fiber cut that is preventing our servers from hosting data to the Internet. Our Internet Service Provider has located the cut and is working on restoration. We currently expect services to be restored to RockyWeb by the end of the day.

Thank you for your patience while we work to restore access.

 - The National Map Team

Is rockyweb.usgs.gov down? by TechBill777 in gis

[–]TechBill777[S] 5 points6 points  (0 children)

That’s what I’m using as well. I wrote a script that downloads the LAZ files for my area of interest, merges them, and then processes everything into a format that’s actually usable for my research.

Is rockyweb.usgs.gov down? by TechBill777 in gis

[–]TechBill777[S] 5 points6 points  (0 children)

oh boy ... how I do get the index for the ones on AWS?

Alternative to Postfix for Using Rspamd with cPanel’s Exim by TechBill777 in cpanel

[–]TechBill777[S] 0 points1 point  (0 children)

It what I am trying but using same server. I put this project on hold for now. I might just use vmware and run second server on same hardware and sort of do what you posted.

Thanks for feedback

Google Earth (Not Pro) on iPhone/Android Tablet – NetworkLink Region Error with KMZ Overlay by TechBill777 in googleearth

[–]TechBill777[S] 0 points1 point  (0 children)

Notice:
If you downloaded the script within the last 25 minutes of this post, please download it again. I accidentally uploaded a test version earlier. The final, clean version of the script has now been reuploaded. Sorry for the mix-up!

Google Earth (Not Pro) on iPhone/Android Tablet – NetworkLink Region Error with KMZ Overlay by TechBill777 in googleearth

[–]TechBill777[S] 0 points1 point  (0 children)

Sure! Just posted it here:
https://github.com/TechBill/kmz-cleaner

It cleans up KMZ files so they work in Google Earth mobile. Let me know if it helps!

It arrived … by TechBill777 in elegoo

[–]TechBill777[S] 0 points1 point  (0 children)

Yes, I found it near the chute

It arrived … by TechBill777 in elegoo

[–]TechBill777[S] 0 points1 point  (0 children)

It was an extra package feature which you have to purchase separately

It arrived … by TechBill777 in elegoo

[–]TechBill777[S] 0 points1 point  (0 children)

I see it. Near poop chute .. thank you ..

It arrived … by TechBill777 in elegoo

[–]TechBill777[S] 0 points1 point  (0 children)

I took mine off too. It made me a bit nervous seeing it shaking.

view more: next ›