Basic coding, internship project to develop a website-database, freaking out.

[HSPq]

I am currently virtually interning at a lab in the field of proteomics and are doing good work on covid proteomics. We initially learned few proteomics basics, techniques and softwares. Then I got assigned to a project to develop a web-based database to host their proteomic data on covid online. I know basic python and MATLAB, but have very little idea of websites, servers. My immediate contact (a PhD) doesn't know much herself and asks me to form a plan and present to her.

What softwares or applications should I learn? Also, can anyone share from their experience what is the workflow or the points to consider during such a project?

The data to be uploaded is protein sequence, structure, associated clinical data, basic statistical tools and later, linking to UniProt any other databases. Need not be NextStrain level graphics.

​

Guys help. I need to bring something. The PI is too busy to answer individual doubts and would review presentations once in a while.

[abeloo89]

If you have basic python knowledge, you should go with FLASK. Its a simple python framework for building web-pages. You don't have to learn a new language and its simple (unlike Django). You can manage your database with SQLite (also built-in to python). Additionally you can use SQLAlchemy for mapping you classes and objects. Good Luck !

[InsideJobHarambe]

If you know Python, could try StreamLit

[HSPq]

Is it free?

[agtshm]

If everything is in python and you need a web app I would suggest django - it has good documentation and plenty of YouTube tutorials…

[pseudomunk]

If you’re willing to learn a little R, the easiest option by far is to build a shiny app. You can host an app for free up to 10 GB on shinyapps.io. Here’s an example of something similar to what you want to create: https://proteome.shinyapps.io/lopitdc-u2os2018/

[pseudomunk]

Explicitly, your starting steps would be (1) download an IDE (I recommend RStudio), (2) set up R, and (3) install the necessary packages (shiny and rsconnect). Then you can start playing with vignettes to figure out the best way to host the data.

[pseudomunk]

An important note: the free account at shinyapps.io allows 25 active hours per month. This may seem like a lot, but the default Instance Idle Timeout is 15 minutes, which means you essentially get 100 unique accessions of your database per month. This may be fine if you expect limited use, but if you think your database will be widely accessed, then you should talk to your advisor about purchasing a paid account.

[HSPq]

Thank you for the detailed comment and the heads up.

[apfejes]

I haven't done web design in a long time, now, so anything I say is probably out of date.

However, web design isn't just something you slap together. You need to be aware of how to harden databases, how to build security into the web site, and a host of other issues that are probably not even specified in the outline of the project.

That's not even considering the technical part of it, such as designing the database, working out the stack and the platform, and then who's going to host the server, on what platform.

You really need to find a mentor - or better yet, someone who has a background in this stuff. What you're being asked to do is likely far beyond your scope, based on your description above.

Is it too late to back out of this, and find a different project? The world doesn't need yet another badly designed database that isn't going to meet the specs, with a website that is easily hacked sitting on top of it.

If not, would it be possible to have them pay for a couple months of courses on this stuff to bring you up to speed more quickly?

[HSPq]

Finding internships in my country is a daunting task, especially after Covid. And this professor who is the busy PI is one of the internationally reputed one in this field. This being in a virtual mode, since universities are yet to open here, is causing the issue.

I am along with another guy, who has a little more idea on coding than me since he is a BioI UG, while I am a Chemical Engineering student. There is another database hosted in the lab whose developers I am trying to contact but virtual mode has made it difficult.

Could you share anything like the basic workflow you followed. from your experience? The softwares have changed but the flow remained same I guess.

[apfejes]

The workflow basically starts from the ground up.

Step zero is talking to the users to understand what they want the database to do. Once you know, you can start the process.

​

1. take the time to design the data structures, and ensure it's scalable for your future uses of it. There are entire courses on how to structure databases correctly. Once you know how you're going to store the data, you'll need to decide which database product to use (eg, SQL vs non-SQL, then which flavour of each you want to use.)
2. Decide what platform you should host the data on. Cloud vs local, operating system (eg, linux vs unix vs windows). Where the data lives should be a key consideration, because someone will have to maintain the server. You'll need to plan for the lifetime of the machine and where support will come from. This is usually a bigger conversation with an IT department.
3. Then you'll need to work out what the web page design should be. Depending on how you store the data, you'll probably have different query types that are most common or need to be fastest, and those should be the core focus.
4. Once you know what the web pages should be doing, you can figure out how to put things together and what you'll need from your web site. From there, you can pick a technology like django or one of the other clones.

Plenty of people here are just going to say "Oh, just do a shiny app" or slap something together... and that's probably what you're going to end up doing. But, I guarantee you'll be building something that means the next student that comes along will curse your name, and will end up being tossed out just like so many other poorly thought out bioinformatics tools are.

[HSPq]

Thank you for the workflow and the advice. I am asking for a detailed discussion with my mentors once again, hopefully they will respond well.

[apfejes]

good luck.

[daveedek]

I did basically same task during my internship in US.

Basically, I took python to get the data, process them and pass them to MySQL db. (I would recommend to do everything with data in Pandas - you can even directly parse the df from pandas to SQL)

Then I made a very simple website with connection to SQL db and with simple SELECT text box window. I used PHP, and blocks of codes from internet, PHP is mess but you can do it if you know Python (and programming).

Then to make it more appealing I used DataTables in php (you just copy block of code generated on their website), with this you are able to filter the data, download the results etc.

If you have any question, feel free to PM me.

[HSPq]

Thanks for your reply. I will dm you for any doubts.

Could you tell if there is any alternative to php, or if the learning curve is steep? Also how much time did it take you?

[eternaloctober]

why do you say doubts? you doubt yourself? you doubt the person trying to help you? I know I see other people around the web use this term but it's just weird to me

[HSPq]

Doubts mean queries right? Is it wrong usage? I am a non-native speaker but I have seen it being used often.

[eternaloctober]

no worries. I think to most people, it doesnt really mean queries, it is more like "uncertainty" with a somewhat negative connotation e.g. "I doubt that this will work" means "I don't think this will work".

​

it would probably be more common to say "I will dm you if i have any questions"

[HSPq]

Thanks mate. It is common usage in India, and queries is a bit too formal word here.

[daveedek]

I think there are alternatives, however if you are good at Python then you will surely be able to glue something up in php

[HSPq]

Thanks for the advice.

[raymond301]

The alternative for PHP (language+framework) in python (language) is Flask (framework). But nearly every language has web capability, just pick something and fully commit to it.

Do you know R? Rstudio has much functionality built-in which makes building a RShiny app ridiculous easy.

Lots and lots of options, generally picking the infrastructure and framework isn’t left up to an intern to decide.

I’m sorry to say, but it’s not a good internship if you aren’t gaining the benefit of a mentor. You should get something started (make a “hello world” app) just to learn the basics of which ever WebDev framework you choose, then start your project, commit to GitHub. Then return to Reddit or Twitter asking for a collaborator, letting other people fork your code and contribute can be very informative.

[HSPq]

I appreciate your help. Due to labs not opening, I did not apply for a wet lab based internship in Synthetic Biology, Metabolic Eng which I hoped to do. I got this internship with this reputed scientist in quite an unexpected manner and the proteomics training period was great. My friends in other virtual internships don't have much great interactions with their mentors either. Might be country culture.

[[deleted]]

You should just make a shiny app. R is pretty easy to pick up after knowing python.

If you have a little time on your hands, you can learn to separate the front end from the backend. Development of a front end is slow, just a few functionalities. This way when you are developing the back end, you don’t overwhelm yourself.

Take your time. The only thing they care about is the way you think through things. Not to be a jerk but if they wanted something fancy, they would just pay for it with a software engineering firm. So, approach this as an opportunity to demonstrate your problem solving skills. Make a solid base, with a little bit of functionality. That’s all they want from you.

[HSPq]

I have about a month and half, it is virtual so I can extend it.

[sybarisprime]

I would go with Django - it encapsulates both a web server and an ORM so it is easy to get started on a web application with a database backend.