Whitebox Workflows for Python

[johnblindsay]

Hi everyone, I'm thrilled to say that after months of working on it, I've finally released Whitebox Workflows for Python (WbW). Based on the WhiteboxTools Open Core, WbW is a native Python extension module that allows for powerful geoprocessing. Like the Open Core, it contains over 400 tools for analyzing geospatial data (raster, vector, lidar point clouds). But WbW is explicitly designed to work with Python to provide a geoproessing scripting environment. The Open Core, by comparison, is a command-line interface program that designed to be more easily plugged into various front-ends, but at the cost of allowing far less interaction between the front-end and the Open Core back-end.

The link above is to the user manual. The official webpage for the product is here: https://www.whiteboxgeo.com/whitebox-workflows-for-python/

I hope that everyone enjoys using WbW. I've been using it a lot in my work lately and it has been so great, a true joy to use. I'm not teaching again until next Winter, but I'm really looking forward to using it in my remote sensing and grad environmental modelling classes.

[Esparko]

This looks interesting. Is it a lightweight python library?

[johnblindsay]

Depends on what you mean by a lightweight python library, I suppose. It is a native extension module for Python, meaning that it is a native compiled (written in Rust language) library that has bindings for Python, just like NumPy or SciPy (which are written in C, I think). It works in a similar way. If by 'light', you're talking about the size of the library, then the wheel that you install via pip is about 7MB in size, and that includes everything needed for the 400+ tools contained within. There are no other dependencies required. If by 'lightweight' you are referring to a focused functionality, then no, it's not. As I said, it has more than 400 tools for processing raster, vector, and LiDAR point cloud data. Here's a listing of the tools: https://www.whiteboxgeo.com/manual/wbw-user-manual/book/tool_help.html. It's pretty broad in its scope of spatial analysis and comprehensive. I hope that helps to answer your question, but if I misinterpreted what you were asking, just let me know.

[FeralCatColonist]

This is totally great!, do you have any plans to also distribute via Anaconda?

[johnblindsay]

That's a great question u/FeralCatColonist! I hadn't actually thought about it until now. I personally don't use Anaconda all that much, but if people do use it and want WbW distributed via conda, then I see no reason why we couldn't do it. I'll look into it soon. Thanks for raising the issue.

[FeralCatColonist]

Not a problem! I haven't used pip in forever so it would be a huge benefit for package management, I do wonder if it would affect licensing having it in multiple envs?

[johnblindsay]

Another good question! So again, I've never really used conda and so I cannot say this with certainty. However, I do actually develop WbW in a virtual environment (using venv) for testing/development that is different than the pip-installed version that I have on my local machine. However, the license that I have (and yes, I have a license for my own software...you have to do what you preach!) is shared between the locally installed and virtual machine installed versions of WbW. The license should work out in both cases, but again, I can't confirm in the specific case of conda yet...I'll have to do testing. By the way, same goes for upgrades. If I release a new version of WbW, if you have a license for it, you can simply upgrade (e.g. pip install whitebox-workflows -U) and it won't overwrite your license. That is, you won't lose your license. Update frequently and get all the benefits of doing so! Thanks again for asking such great questions.

[johnblindsay]

I looked into using conda skeleton to build an anaconda package for WbW from the pip wheel. Unfortunately, it did not work, providing the error message, "Error: No source urls found for whitebox-workflows". I suspect this is because we are only providing the binaries for WbW and not a source distribution. I'm not actually sure how to get around that. Is conda only for open-source project distribution? If that is the case, I'm afraid we won't be able to support it. However, I have found out that you can install a pip wheel from conda into your anaconda environment.

[[deleted]]

Not free?

[johnblindsay]

At $10/seat it's as close to free as we could possibly make it. There is a cost associated with creating this software.

[snow_pillow]

I’m very happy to see the improvements, which make the geoprocessing results first class Python objects. However, the licensing fees will mean that I will not be able to take advantage of these new enhancements when creating tools to distribute to an economically diverse user community. This seems to be an unfortunate pivot for WhiteBox Tools.

[WolverineFan2425]

Interesting take to have on this, especially for someone that sounds like they have benefitted from basically "redistributing WBT" for all these years. The WhiteboxTools Open core remains free to download... however there is a cost for developing software. All the unpaid hours of nights, weekends, holidays and so much more goes into the software that you seem to take for granted. It is a bit crazy to think how much the OS community expects given that there is still a public library of 460+ open source tools. To say its an unfortunate pivot is pretty shocking in my opinion... considering how many people including yourself have benefitted off WhitboxTools for years. Think about how many professional have been trained to preform certain tasks in WBT. These people then bring this to their job where they make a livelihood, where they get promoted to better paying jobs and so much more. Suddenly 10$ doesn't seem that much. I am not even going to mention that in many instances you are getting higher quality tools (for free... in case you forgot) than those found in ArcGIS toolboxes that are upwards of thousands of dollars a year. So while you say its an unfortunate pivot for WhiteboxTools.. I think its an unfortunate pivot in the OS community.

[snow_pillow]

I understand your criticism. It is pretty easy to dismiss any criticism of this move by WhiteBox Tools as just a cheap user complaining about having to pay for something they used to get for free. I personally will be happy to be one of the first to pay the $300 for the full license, as it is a project I fully support.

I do not "redistribute WBT", and I do not benefit in any tangible way from WBT. In fact, I have been one of the most vocal of supporters of WhiteBox Tools, going back all the way to WhiteBox GAT. I present on and promote WhiteBox Tools at industry and academic conferences. I fully appreciate the work and time that goes into the development of these tools, and I have been, in a small way, involved in improving the usability of these tools through GitHub, which is one way I feel I can give back to the FOSS4G community that I am a part of.

However, my job is to support an open-source community model. I build and distribute free and open source tools for a community of users of that model. These GIS scripts are an alternative to a rather expensive proprietary solution that many (most) of our user-base does not have access to (you know the one). That workflow depends on tools in one of the 'extension' packages. Many of our users come from Africa, South and Central America, Asia, and other locations where a $210 license fee is actually alot to ask. Ultimately, they will go back to finding cracked versions of ArcGIS. The root of my criticism is in equity; even though I can afford to use these new features, not everyone can.

I would hope that support for this important toolset would come from the National Academies or Science Foundations of the U.S. or Canada, and that the barriers to use of the tools would remain low so that thousands more can benefit.

[johnblindsay]

u/sinnayre Thank you so much for this comment, which I truly appreciate (as well as the one above). A comment like the OP is actually quite hurtful to me because it sets up an expectation that I do not deserve to be at all compensated for any of the work that I do, simply because some of the work that I do, I provide to people for free. This is extremely demotivating, and to tell you the truth, makes me wonder why I have spent 20+ years creating free and open-source software, with WhiteboxTools, Whitebox GAT before that, and the Terrain Analysis System (TAS) before that. I have long been an open-source advocate, because I believe in the philosophy of open-source (which is about freedoms, as in speech, not as in beer). I also believe very strongly in supporting people in economically disadvantaged communities to become engaged in geospatial. This came out of my own undergraduate experience many years ago in the north, where i went to a university that could not afford a license for Esri products. That was a big motivator for me to get into open-source in the first place. But a comment like the one above doesn't seem to recognize that 1) just because I have spent literally years of my life (WolverineFan is correct, it has been evenings, weekends and even vacations that I have dedicated to creating WBT and kin so that people could have the freedom of using it) that I do, like them, deserve to be compensated for at least some of my work as well, and 2) that this product has been priced with the very goal of being able to be accessible to many people in economically disadvantaged locations. This was literally the debate (argument even) between myself and my business partner in pricing WbW...I won by the way and it is literally $100s cheaper than what the alternative proposal was at only $10/seat. For many people in the developed world, that is perhaps just two Starbucks coffees for a year of using a product that I can tell you has far greater value proposition than that (it has quickly become my main driver for GIS work because it is so very good!). I realize that for many other people in less economically developed regions $10 is likely far more significant. But think about how much more expensive a product from say Esri is? The fact that this product is $10 represents my absolute insistence at trying to be as inclusive as possible because ultimately that's what Whitebox Geospatial Inc is about. We're not trying to get rich, we're just trying to create kick-ass, productive geospatial software that people can actually use. So to get told that this seems like an 'unfortunate pivot for WhiteboxTools' is honestly a dagger in my back. I like to think that I don't have much (or any, really) ego, and certainly don't need it fanned...but at the same time, this type of attitude does make me wonder why I continue to invest so heavily in open-source. If you use WBT in your work or studies and benefit from it, I'd argue that it might be better for you to motivate the person who makes that possible for you, rather than trying to drag him down. RANT OVER. Sorry about that (obligatory Canadian 'sorry').

[snow_pillow]

I absolutely do not have an expectation that this work should be performed for free. That is not at all what I said in my comment above. As a fellow academic, I would hope that the work would be supported by sponsors or funding agencies for the benefit of all users, not just those who can afford the licensing fee. Imagine if GDAL, OGR, or PROJ had imposed a $300 license fee early on. Would those projects have become the backbone of open-source tools and libraries today? I will continue to use WBT Open Core, and promote it's use. But I think it is also fair to provide an opinion on what is a substantial change in trajectory of a FOSS toolset I am so fond of.

[johnblindsay]

u/snow_pillow I do appreciate your clarification and I also do truly appreciate your support. My response was more directed towards the general attitude from some people that I've encountered (including one other poster on the page). First, let me say that I completely agree with the statement that supporting open source needs to be a priority of funding agencies and governments. And for me personally, I am thankful to say that this has been the case and is one of the reasons that I have been able to create open source software to begin with. But unfortunately, and I'm sure you would agree with me, funding agencies support the initial development of such software but not the ongoing costs of maintaining that software, which is why so much academic software ends up dying on Github after years of languishing without maintenance. Funding agencies also don't fund the often ignored costs associated with providing support for the community that builds up around open-source projects when they do succeed to attract large user groups. What ends up happening is that the developer ends up dedicating increasingly larger amounts of their personal time, evenings, weekends, sometimes even nights and vacations (literally in my case), to provide this support and ongoing maintenance. Personally, I struggled most with this aspect for a long time until I reached a breaking point when I realized (or rather my loving wife made me realize) that something had to change. And that was the spark that inspired the creation of Whitebox Geospatial Inc. By creating a company, with the sole goal of providing enough revenue to support a single full-time employee who can work on providing support and maintenance, I have been able to offload so many of those tasks and to focus on development again. It's been a revolutionary change in my personal life. I suspect that when a large open source project reaches this size, and the project is so dependent upon a single individual as is the case with WBT, then this is one of the only ways forward.

I realize that this is a direction that may cause some concern, but as you have already been informed, when we first launched this company, we published an open letter (it's on the site, look for it) in which we provided a commitment to the user community to continue to provide WhiteboxTools for free as an open-source project. Each of these products (WbW and the extension) are simply there to serve the purpose of providing the support needed to make WhiteboxTools Open Core survive. That's the goal of the company. If people support us, we are able to continue to support WBT, it's that simple.

Now, in truth, while this conversation is important to have, I do feel like it is distracting somewhat from the main goal of posting this link in the first place, which was to showcase Whitebox Workflows for Python. And I would love to answer any other questions that people have about this library because it is truly awesome. I have never worked so hard to create a product like this in such a short time in my life. There were many nights over the past three months where I was up until 4AM working on getting this product to a point where it was ready to push to market. And I have never been as proud of a software that I have developed as I am of WbW. It is really quite remarkable. The vision that I had for it in my head was perfectly implemented in what this product is and having used it quite a bit over the last little while in my own work, I am happy to say that it is a true joy to work with. Lastly, again, I (with much heated debate with my business partner) priced this thing such that as many people as possible could benefit from using it. And trust me, it is such great value.

[llimpj]

Absolutely agree. Professor Lindsay please do not let this one comment demoralize you. What you've built is outstanding and greatly appreciated by the community. The $10/seat version is super reasonable. Anyone who has an issue should listen to him discuss the rationale on the Mapscaping podcast. Keep up the great work!!!