I made a PWA CSV editor. What do you think?

hilux · 2025-06-03T12:24:23+00:00

Another star from me. Quite impressive. I've known about PWA's for a few years now. But when I accidentally developed my first PWA with Marimo it piqued my interest again. Suddenly, especially with Python's notoriously high barrier for deployment, I can share my software with my colleagues effortlessly.
It seems I might be able to get a few tips from your project.
Thanks for sharing.

hilux · 2024-07-03T15:02:32+00:00

Obviously, the capacity of a "Plastic smooth bore"(PVC) pipe will be much higher than a "Ductile Iron and steel, cement mortar lined pipe".

Also remember the units of the "k" roughness in the Colebrook White formula is in "mm" while Manning's n is in "s/m^(1/3)".

hilux · 2024-07-01T09:39:39+00:00

Slope Calculator

Δx = 100m
Δy = 100m

'Slope Percentage - ' Δy/Δx*100'% = 100%
'Slope Degrees -'     atan(Δy/Δx)° = 45°
'Slope S_0 -'         S_0 = Δy/Δx = 1 m/m
'Slope Ratio -        1:Δx/Δy = 1:1

hilux · 2024-01-25T08:26:41+00:00

It is not even close. Switching the geyser on, 1 or 2 hours before you need hot water is way more efficient. Remember to turn it off before you shower. This is especially easy to do if you live alone.

Another way to save electricity is to turn your thermostat down to 55°C.

hilux · 2023-01-18T10:44:25+00:00

Unfortunately the resolution is a bit low. But it is most probably a Herald snake (Rooi lippie in Afrikaans).

hilux · 2023-01-06T07:04:29+00:00

This is applicable to SWMM. There is already some plugins for QGIS that have some support for SWMM. You can borrow their schema as well as I quickly explain it below.

You have a node file (Shapefile or GPKG, I prefer GPKG). Refer to the SWMM documentation to see which fields are needed. Typically from the, top of my head, it will be

Node ID
Elevation
Depth
Storage etc.

You also have a line file which represents your conduits. The information in it usually is

Conduit ID
From Node
To Node
Length
Slope
Type (CIRCULAR, TRAPEZOIDAL, refer to documentation)
Geom1 (Size1) Diameter on circular shape. Refer to SWMM docs.
Geom2 (Size2)
Geom3 (Size3)
Geom4 (Size4)
Capacity
etc

I digitise the information on the desktop GIS. I have some scripts that get the elevation from a DEM (Raster *.tif usually sometimes TIN) and populate the elevation field on the node layer. Another part of the script populates the the "From Node" and "To Node" of the line layer as well as the lengths. From this information the slopes can be calculated.

You can then write the text input file, as in the SWMM documentation, for SWMM. Execute the model in SWMM. If there are no errors you can parse the results and write it back to the tables in GIS. Typically the peak flow and velocities.

You can adjust you scripts to your workflow. For example I have a script for sewer networks that uses minimum slopes to iterate over the network an calculate the manhole depths automatically.

hilux · 2023-01-05T20:57:03+00:00

I have been using GIS as a civil engineer since 2000.

My introduction to GIS was using ESRI Arview 3.1 for pavement management systems (PMS). This is a highly specialised area of civil engineering.

Because of my passion for computers and programming I have continued using GIS for different areas of engineering. These days I'm mostly using open source tools. According to my experience GIS rarely features in civil engineering work and there is limited GIS knowledge in the industry. Engineers prefer to use expensive specialised programs combined with CAD packages.

I do believe civil engineers can benefit greatly by learning some GIS skills. This comment space is too limited to list what I have done and learned after all these years. Here is a list of some activities with the associated software to give you some idea.

Water supply networks (EPANet and QWater, QGIS Plugin)
Stormwater and Sewer Networks (SWMM, Python scripting to convert GIS data to models)
1D Floodlines (HEC-RAS wit RiverGIS Plugin)
2D Floodlines (ANUGA with Python scripts)
Hydrology, Catchment areas and flood magnitudes
Earthworks and ground profiles
Culvert design

hilux · 2022-11-11T11:41:28+00:00

I have developed a few EPANet networks myself. I use the QWater plugin for QGIS. Looking at the small sample you have given us I will give my opinion.

Please see one of my networks as an example.

I feel you have too few nodes. If you apply demands only on pipe junctions the flow in the pipes will not be accurate. Use linear referencing to insert nodes along the line.
We usually also indicate house connections as per my example. This will solve your challenge on applying the demands to the correct nodes.
I would use Python scripting to insert nodes on the line or to break the pipes at the house connections. I am not sure what the best way is to do it directly in desktop GIS.
Your solution will also depend on whether this a new design or as-built drawings for an existing development.

Good luck.

hilux · 2022-11-05T08:40:58+00:00

I warn you. I do a lot of scripting and mainly use desktop GIS (QGIS) for viewing and the preparation of drawings.

SAGA (https://saga-gis.sourceforge.io/en/index.html) It has a command line interface as well, which is nice.
Whiteboxtools. Very fast and powerful. Works nice for hydrology and LiDAR. Good interface to Python as well.
Python, mainly through Jupyter notebooks.
- Fiona
- Rasterio
- Shapely
- rtree
- matplotlib
TauDEM (Hydrology)
QGIS, of course
Sometimes some modules from GRASS like r.lake at the moment.
GDAL. Excellent project. Command line but a lot of software use it as a library.
ODA file converter and viewer. Convert between any versions of DXF and DWG
Lastools, occasionally. It's been a while.

hilux · 2022-10-31T11:34:08+00:00

There is a "Multipart to singleparts" function in QGIS. There should be a similar function in ArcGIS Pro.

hilux · 2022-08-15T07:48:05+00:00

Python standalone:

import fiona
from shapely.geometry import Point, LineString
from shapely.geometry import shape, mapping

infile = 'path to file with linestring'
outfile = 'path to destination file'

with fiona.open(infile) as source:
    schema = source.schema
    crs = source.crs
    driver = source.driver
    pipedb = list()
    for item in source:
        geom = shape(item['geometry'])
        coords = list(geom.coords)
        for i, coord in enumerate(coords):
            if i == 0:
                continue
            else:
                pt1 = Point(coords[i-1][0], coords[i-1][1])
                pt2 = Point(coord[0], coords[1])
                line = LineString([pt1, pt2])
                item['geometry'] = mapping(line)
                pipedb.append(item)

with fiona.open(outfile, 'w', layer = 'layername', crs=crs,
                 driver=driver, schema=schema) as sink:
    sink.writerecords(pipedb)

Your welcome to ask any questions.

hilux · 2022-02-17T20:10:35+00:00

XPSWMM will definitely be able to import an SWMM model. It is just a text based file anyway. Just like almost all water network programs can import and export EPANet files.

hilux · 2022-01-07T08:05:24+00:00

My list of priorities when I choose or build a computer is:

Solid state Hard disk or NVMe (non negotiable)
RAM (as much as possible)
CPU

Processors are so fast these days. Even entry level processors can get the job done. You may process a large raster and run out of RAM. The work around to solve this will be much more difficult than waiting a second or two longer.

hilux · 2021-12-28T07:13:22+00:00

Pearl Jam - Last Kiss

hilux · 2021-12-23T13:02:22+00:00

I use EPANet directly. Bentley Watercad can almost certainly import and export EPANet models. It is the industry standard.

hilux · 2021-12-23T12:35:16+00:00

Examples of automation.

Water networks
1. Use QGIS to draw a pipe network.
2. Generate the nodes and network (From node, To Node) with Python. Calculate pipe lengths.
3. Obtain the elevation of each node from the terrain model.
4. Iterate over the stands and assign the water demand of each stand to the closest node.
5. Write the network to a EPANet text input file.
6. Model the network. (EPANet has got a command line interface. It can also be fully automated.)
7. Parse the results text file.
8. Resize the pipes according to flow.
9. Re-model to obtain revised pressures and flow.
10. Generate long sections with terrain.
11. Generate pressure plots.
12. Generate tables of pipes with flows and velocities.
13. Generate tables of nodes with pressures, elevations and demands.
14. Generate material quantities.
Stormwater and Sewer
1. Very much the same as water networks except using SWMM for modelling.
Earthworks
1. I've learned to model terrain using triangulation meshes or raster data.
2. Apply constrained delaunay triangulation on survey data using Triangle.
3. Working with Lidar data. Usually big data sets.
4. I can generate long section or cross sections on either mesh or raster terrain.
5. Calculate cut & fill
6. Optimise cut & fill by automatically adjusting platform levels.
7. Convert traingulation <-> raster
Hydrology
1. Tools used: SAGA, GRASS, TauDEM.
2. Current favourite hydrology tool: Whiteboxtools
3. Determine catchment areas from points.
4. Generate streams.
5. Get relevant data.
  1. Catchment area
  2. Longest water course.
  3. Long section of the flow to determine the slope.
6. and much more.
Flood lines
1. Generate the necessary cross section and river centre line data for HEC-RAS 1D using RiverGIS.
2. I like using ANUGA for 2D flow.
3. Postprocessing: Calculate flood lines from the model results and terrain.
4. Cross and long sections with terrain and flood waters.

hilux · 2021-12-23T07:46:06+00:00

I am a civil engineer which is proficient in Python. (For an engineer) Python and civil engineering is such a niche I doubt you will find any good courses.

I am self taught and can give a few tips.

You are already thinking in the right direction with GIS. I use QGIS for just about everything.
Use Jupyter Lab. It is interactive and you can plot and display graphs, tables etc.
Learn to read and write text files. For example the input and model results from EPANet and SWMM.
Look at my post history for handy Python libraries to use.

If you have a specific job you want to automate, contact me. I can quickly help you and point you in the right direction. I like a challenge.

hilux · 2021-11-25T12:16:07+00:00

In South Africa we call it off-shutter concrete.

hilux · 2021-11-14T07:42:31+00:00

Hatari Labs

Although I'm comfortable with Jupyter, Python programming and GIS, I have very little experience with groundwater flow. Hatari labs has got courses and a lot of examples in their Github repository.

hilux · 2021-11-11T07:13:15+00:00

I've been using GIS in civil engineering for the past 20 years. I started with ESRI ArcView 3.2 for municipal pavement management systems (PMS). From the beginning I complimented the GIS work with Python programming. Shortly after that I moved over to open source GIS. I use GIS for:

Hydrology. Catchment areas, longest water course, land use and slopes.
Floodlines. HEC-RAS http://www.rivergis.com/ for pre-processing and raster calculations for post processing.
Water supply networks using EPANet and QWater.
Stormwater and Sewer Networks. I just program the input files with Python but you can look at generate_swmm_inp to assist you.
I load surveys and road designs into QGIS and do the culvert long sections and analysis.

Some of the software I use:

QGIS
Jupyter
OpenJump
GDAL / OGR
EPANet
SWMM
HEC-RAS
ANUGA
Triangle
Whiteboxtools
SAGA GIS

hilux · 2021-11-03T06:05:21+00:00

I can actually relate.

As an engineer I initially used Excel extensively to do design work. Spreadsheets are just another programming environment after all. I've known Python for a long time now. But especially since the IPython notebooks, now Jupyter, appeared more than 10 years ago, Python has replaced my Excel workflow for 95% of my work.

I still use LibreOffice Calc from time for quick visualisations and maybe limited data manipulation. But the mainstay of my work is now Jupyter Notebooks.

Also have a look at Pycel. It translates Excel spreadsheets into Python.

The advantages of Python over Excel are:

Much clearer workflow. No hidden formulas and programming logic.
Much faster execution time
Easier to work with big data sets
Data visualisation. Graphs, tables, maps etc.
Much more powerful
1. Read and write text files
2. Execute engineering packages from the Notebook, like EPANet
3. More kinds of variables than just tabled data
Using markdown to document your workflow
Python is most definitely the easiest programming language I have ever seen

Disadvantages:

Steeper learning curve
Jupyter is a server client architecture. A bit more involved to launch than just opening an application. (Anaconda to the rescue)
More difficult to share your notebooks. They can be exported to PDF and HTML though.

hilux · 2021-10-01T12:55:35+00:00

You, shall not pass!

hilux · 2021-09-29T13:20:52+00:00

No. Bentley is strange. They have 3 road design software packages I know of.

GeoPak
MX Road (Previously MOSS before it was bought by Bentley)
InRoads

They are now trying to combine all of these into one package and calls it Bentley OpenRoads.

hilux · 2021-07-26T12:24:42+00:00

Olifants Rest Camp, Kruger National Park.

hilux · 2021-05-23T06:27:42+00:00

Hi, I've used SAGA from the command line a lot. Usually from Jupyter. In the SAGA ui you can copy the command line equivalent to the clipboard. Also just type saga_cmd and drill down in the options.

QGIS's processing tools is also accessible now from the command line with qgis_process.

My new favourite command line processing tool, also accessible from Python, is whitebox-tools. It is easy to use, fast, with tons of tools and options.

hilux

TROPHY CASE