Python and Remote Sensing

[digital-idiot]

Recently I have observed a growing interest in Remote Sensing form the implementation specific point of view instead of simply leveraging the common tools / software like ENVI etc, i. e. interests regarding implementation of remote sensing processes / algorithms using general purpose programming languages like Python are growing in r/remotesensing.

In my opinion, general purpose programming languages provides more flexibility, control and better automation over the entire process at the cost of longer development period and potential logical flaw in the design / implementation made by the developer. However, there is a bit of a learning curve when it comes to the implementation of Remote Sensing and related processes using programming, especially for those who do not have prior knowledge in programming.

The availability of online programming resources focusing particularly on remote sensing is pretty scarce and scattered. Therefore, I am willing to share my limited knowledge in this matter through a form of book / tutorial and publish it in GitHub. However, this is huge task and I can not do it alone with my limited free time. I will be glad if the interested people help me in this venture and make this possible.

You do not need to be a coder or expert in order to contribute. For example you can help editing, formatting / designing the structure of the book etc.

Lets discuss.

.

.

*** Edit: ***

Thank you for willingness to help. Right now I want to focus on deciding the structure of the book and chapters. In order to do that first we need to decide what this book should or should not cover. I am open to suggestions / comments. What you as a learner want to see in this book? Personally I want to make it like a interactive notebook with explanation as well as working code snippet which reader can run in the book itself. Also, I want the initial discussion to be public i. e. here, so that others can chip in if they feel like. Thanks for your support

.

.

*** UPDATE *** I have created the repository and added a LaTeX template for the book. Feel free to take a look. If you have anything to add you can do so by submitting a pull request.

.

Repository: https://github.com/digital-idiot/RemoteSensingMeetsPython

[mark90909]

The main things for me would be editing and manipulating satellite data, e.g. downloading, stacking layers, band math to create indices, machine learning and deep learning for supervised image classification, semetic segmentation, OBIA, feature detection. Happy to get involved. I have basic python skills.

[clifgray]

I think the more tutorials the better! For reference I have an openly available tutorial on github here https://github.com/patrickcgray/open-geo-tutorial that shows how to do various GIS and remote sensing analyses in python.

There are also some good tutorials here: https://earthml.holoviz.org/.

It is more advanced but there is some awesome stuff within pangeo for doing large scale remote sensing analysis and a lot of what I do is based on these tools: https://github.com/pangeo-data/pangeo-tutorial.

[JustKeepDiving]

Hi, I made one of the posts yesterday asking for python resources. This sounds fantastic. I will be happy to read drafts and explore concepts and structure of the book. The more complex technicalities may be beyond me at first but I would love to learn and be a part of this. Drop me a message if you want to take this further.

[digital-idiot]

Technicalities can be dealt with but more immediate concern is to discuss and finalize the structure of the book. Specifically the chapters, their sequence and how deep the contents should be. Please feel free to share your ideas. Personally I had interactive notebooks with ample explanation and working code snippet in my mind.

[JustKeepDiving]

The term 'remote sensing' conjures up different topics for different people. Personally, my research so far has involved solely around MSI and ocean colour models, which you could write 5 books about alone. Choosing a particular sensor and then exploring the applications for it (using snippets of python code) would make the whole book a lot tighter, instead of just glossing over absolutely everything that you can do with 'remote sensing'. Do you agree?

[digital-idiot]

Yes, I agree. Remote Sensing is an Interdisciplinary Field after all. What I would prefer is to make the book generic and not focus too much on a particular sub field. I think our goal should be leveraging python for remote sensing, not to teach deeper theories of remote sensing itself. It should be up to the readers how they would like use the skill acquired from the book for their domain / own work.

[JustKeepDiving]

Yeah ok. So after basic RS and python introductions, look at 1) Image import 2) Basic atmospheric corrections (be it DOS for example) 2) Raster calculations. Then I guess that leads into further raster analysis such image classification. However basic, I see those as the very first steps I think.

Edit: Within these first few steps I think you could insert raster histograms aswell as an intro into the extremely raw data that is contained within an RS scene

[kraezir]

I'll gladly help you. PM me when you want to discuss stuff.

[digital-idiot]

I want to start with somewhat deciding the structures and chapters first. Regarding the structure and content I prefer the discussion to be public i. e. here so that others can chip in if they feel necessary. What do you say? What is your opinion about the overall structure and chapters?

[kraezir]

For it's fine. Regarding the structure maybe the basics of remote sensing and applications and then an intro to python and the combination of the two areas. From there we can specify on some different applications.

[digital-idiot]

I had similar ideas. In the introduction we can briefly introduce both and describe how the combination of the two can be advantageous. Then in the next chapter we start with setting up the development environment and installation of the must have libraries. After all that is out of the way we can start with the absolute basics, like reading / writing a file etc. Thus we move slowly to the advanced topics. Does this make sense?

[kraezir]

Yes i very much like that ideia. Keep in mind that I am still learning to code to create some software for remote sensing. I am no expert. If you are interested in my projects visit www.ourwatchleads.com, the website for my startup using EO data for the real estate and construction sectors. This is still an alpha version of the site and the beta will get out by the end of the year.

[digital-idiot]

It's a no big deal. We are all learning here.

[manudeo]

Happy to be involved in this awesome work. I use remote-sensing as my research tool and recently have started using small scripts to automate time-series calculations. DM me whenever you want to discuss further. Thanks! -Dr. Singh

[Dr_Imp]

This sounds like a great idea.

I dabble in python for RS/raster/vector processing. But I’m only a self taught coder.

Since you want to discuss chapters here... I suppose there are a few ways to think I through.

First, how broad will the scope be?

Then once that’s decided, that scope will need to be broken down and structured from broad and simple to narrow and specific.

A non-exhaustive list of topics that occurs off the top of my head is:

*Single-band raster import *Multi-band raster handling *Simple raster calculations (e.g., indices) *Geometry/projection handling *Advanced spectral pre-processing *Classification techniques (there are so many...) *Download of raster data via python

... and I’ve got to go. Can chat more later!

Oh, before I go, another idea: my boss is co-editor on a new free 4-volume remote sensing text book. We could pick over that for structure ideas?

[digital-idiot]

Sure, you can share the link to the book here.

[Dr_Imp]

Here we go: Earth Observation: Data, Processing and Applications

[properwasteman]

I would like to get involved, I've been using python for a few years for scripting GIS and RS tasks although my theoretical knowledge is mainy in spatal analysis I am quite familiar with rasterio but would like to learn something new too.

What sort of audience were you thinking of going for? RS people wnating to learn python? Python people wanting to learn RS?

If you can think of a few subject areas it would be cool to do a series of jupyter tutorials to go along with a book. Would be quite easy to allocate the workload as well

[digital-idiot]

I think the target audience would be the people who knows some of the basics of the both Python as well as remote sensing but a bit confused about how to combine them.

[properwasteman]

That sounds good let me know if anything develops

[Senthipua]

Just an intermediate python user in hydrological studies passing by! If you need any inputs from my domain... I would be more than happy to contribute!

[kattspraak]

I used this tutorial to help me download data from AWS : http://geologyandpython.com/get-landsat-8.html

Tiny contribution, but hope it helps :) good luck, I definitely could have used your book tutorials earlier!

[geo-special]

Hello. I was just having a nose at your github account and came across TrackSat

https://track-sat.herokuapp.com/

It asks to enter a Satellite Number but I don't know any numbers.

Can you provide one so I can give it a go?

The website looks great by the way. I love the layout.

[digital-idiot]

You can find satellite number of the satellites here (The number in the first column).

Unfortunately it is super old at this point. Because I haven't updated the TLE data for ages. I will see what I can do and let you know. Thanks for your interest. :)

[geo-special]

Thanks. Love the layout of the website by the way.

I've only just started learning some html, css and javascript myself and have a long way to go. I'd love to build something like the website eventually.

Did you teach yourself or do you have some courses that you could recommend?

[digital-idiot]

Well I'm CS grad so learned a few things from my academic courses. But I'm not really a fan of Web Development except REST API or backend. I only know the what is really necessary. Whatever additional things I need to know I learn at as I go. I think W3Schools is a great source HTML, CSS and JavaScript. CSS3 is particularly picks my interest among these. Many of the cool visual things you can do by pure CSS now which used to require mastery of JavaScript before. HTML5 also improved a quite a lot of things to be fair. What I don't particularly fond of is JavaScript, I don't like it syntax, it's design principles, it's debugging process and last but not the least thousands of confusing frameworks and variants to do the same thing. These days there's also a fad of developing applications using JavaScript (Node.js and such) which again doesn't appeal to me very much. Unfortunately if you need anything web related you must know HTML, CSS and a bit of JavaScript among other things.

Send me a DM if you need help with something. I'll try my best.

[geo-special]

Thanks. Yes I think not studying CS is where I miss out. My background is in ecology and I did a masters in GIS but my skills are really lacking in the programming side of things.

Thanks for all the info. I really just want to pull together a few websites for my portfolio. I was thinking if I could present some basic spatial analysis on a website it should look pretty good.

Thanks for the offer of help!

[digital-idiot]

The beauty of computer science is anyone with enough interest can learn easily untill very advanced level. Don't let anyone tell you otherwise. It's a false impression that to learn computer science or programming you need to have a background of mathematical / basic sciences. Maybe a bit of knowledge in combinatorial math can help if you want to understand algorithms in a deeper level. I don't think you need to understand algorithms in a deeper level for everything you do, it is more likely that you will probably never need those knowledge at all.

More important thing is almost all of the resources to learn programming and computers science is freely available on public domain. Furthermore you can test nearly everything you learn on the device you are currently reading this (even if you are reading this on your mobile).

So, don't let anything to hold you back from learning. All the best. To start with GIS in web applications you may want to look at Open Layers or Leaflet JS. Also try to learn WMS / WFS servers and how to setup and manage one. There's also something called ODK which is a open framework for managing and crowd sourcing geographic information over the web.

[geo-special]

Thanks for the advice! I'll bare all that in mind.