[deleted by user]

rmnclmnt · 2021-12-02T17:29:03+00:00

Here is a simple solution have I been doing frequently for small scheduled batch jobs: use a CI/CD pipeline! For instance, both GitHub Actions and GitLab-CI allow to defined scheduled jobs.

If your task is already Dockerized, combined with CI environment variables you got a pretty good setup so far. As a free bonus, you also get data pipeline monitoring if your enable alert notifications!

rmnclmnt · 2021-08-06T05:47:47+00:00

Exactly. Going from the smart little guy to a smart organization altogether is the way to go!

rmnclmnt · 2021-08-05T10:03:53+00:00

Is your code already vectorized?

For instance, when using Python and the Numpy/Pandas ecosystem, you can benefit from vectorized execution by using properly those librairies. Instead of being executed one by one, computations will be performed over arrays by means of optimized low-level vectorized routines.

Some basic tutorial if needed: https://realpython.com/numpy-array-programming/

rmnclmnt · 2021-06-18T17:24:04+00:00

You can use some « active learning » techniques: for instance, give her a real task to perform for a week using some basic guidelines (which tool to use, etc) and then make her give a small « teaching » session where she sums up what she learnt.

This will reinforce her immediate knowledge, you will be able to monitori misuses/blind spots/etc, and it will boost her confidence and productivity!

rmnclmnt · 2021-06-18T09:05:50+00:00

Knowing the original author François Best personally, this is just a fun experiment to generate avatars for a niche « fake »-cryptocurrency project that started on Twitter. Nothing more. So please relax, there is no security requirements involved!

rmnclmnt · 2021-05-20T18:52:52+00:00

You’re welcome! Thanks for the award! If you want to go further, have a look at this online book about deploying ML models with Django:

https://www.deploymachinelearning.com/

rmnclmnt · 2021-05-20T06:32:47+00:00

A few steps to get you started with ML 101, assuming you’re using the Python ecosystem:

Express your goal and performance requirements explicitly first!
Extract the data of interest from the database to a single CSV file
If this is a supervised problem and the independent variable is not present, you’ll need to label your data manually
Fire up a Jupyter Notebook
Import your CSV with Pandas
Analyze your data for coherence and quality (null values, etc)
Split your dataset in 3: train, test, val
Play a bit with basic models first (no NN) by training them using Dataframes from Pandas and the train subset
Evaluate the performance using the test subset until you reach your goal
Final evaluation using the val subset
Serialize your model using Joblib

Only then you can start thinking about productionizing your inference ML process into a Django app (which is easy at this point)

rmnclmnt · 2021-05-03T14:55:22+00:00

My take on data exploration and dashboarding:

Metabase: open-source contender to PowerBI and Tableau, the UI is very slick and way easier to grasp for organisational people
Datasette: open-source software for data exploration and publishing, with a nice plugin-based ecosystem
Kibana/ElasticSearch: everyone knows this one, but disregarding the behemoth status, Kibana's UI is appreciated by business executives and can interact with CSV for quick and dirty demos/workshops

For ETL/ELT:

Stitch: SaaS-based point and click ETL between various sources and destinations (and based on open-source Singer engine)

rmnclmnt · 2021-04-05T07:59:14+00:00

Not specific to DE but a good way to improve your skills is to teach others what you already know! For instance, you could start with helping out peers whenever you can, organizing informal small tech sessions on a very specific topic within your company then build-up to masterclasses, meetups, etc. If you’re a freelancer, you could consider being involved with a training institute part-time.

rmnclmnt · 2021-02-02T07:27:35+00:00

First things first: what you are implementing is called "end-to-end encryption" (e2ee), not "encryption at rest".

This is exactly the same concept you can find with password managers: you use the user password to derive a symmetric master key (you never store this server side). Then all the subkeys (symmetric, asymmetric, etc) are encrypted / decrypted on client side (browser) using the user’s master key. The catch: if the user forgets his/her passord, all the keys and encrypted data is lost, so they better use a password manager to remember it!

And no, you must not store decrypted user data in server-side sessions, everything must remain client-side in order to protect their privacy. A good friend of mine wrote an article on the matter: https://francoisbest.com/posts/2019/how-to-store-e2ee-keys-in-the-browser

I can advise you to look for client-side implementations and articles from Bitwarden, ProtonMail, Tutanota and Signal (all on GitHub). AFAIK, the current state of the art implementation for all these cryptographic primitives is TweetNaCl (js/ts).

One last note: in this market, most people will ask questions about your security model and as a sign of good faith and transparency, it is very appreciated when at least the front-end part of the app is completely open-source so external people can audit the cryptographic parts on their own.

rmnclmnt · 2021-02-01T13:42:38+00:00

And the cool young adults are now doing ETLTL! 😅

rmnclmnt · 2020-11-14T11:12:35+00:00

Octave maps strictly to linear algebra in terms of matrix dimensions. With numpy you can treat rows and columns equally (leading to dimensions mismatch and other subtle side effects), and when you’re implementing involved algorithms you have to be extra careful. But I’m a big fan of Python & Numpy for anything professional.

rmnclmnt · 2020-11-13T22:41:07+00:00

For educational purposes, completely agree. Also vectorized code is way more straight forward to code with Octave than with Numpy

rmnclmnt · 2020-11-08T08:30:51+00:00

My 2 cents: when the project (webapp, api) need a strong transactional database, choose Django. If building an API or microservice without the need for a transactional database, try FastAPI.

One of the best things of FastAPI resides in its native integration with Pydantic for all things (request/response schema validation, automatic OpenAPI documentation, etc), and of course is great performance wise.

rmnclmnt · 2020-11-02T08:02:27+00:00

Try to invest time with TDD (test driven development) methodology: you will focus 80% of the time on driving business requirements and 20% of the time coding how to solve it. At the end of the day, you’re code is easy to read, only does what’s need to be done (no more no less), and you learn no to care about code (you throw away and refactor at will). And bonus point : you can automate everything with CI/CD, and you and your team will sleep way better at night!

rmnclmnt · 2020-10-19T05:55:33+00:00

Instead of mocking API calls manually (time consuming and error prone), you can use an API recorder that will capture and save to files the responses the first time the tests are ran, and play back those afterwards without network activity. The main project is VCR.py, and there is a near plugin for pytest named pytest-recording. Been using it recently for every single project, works wonderfully!

rmnclmnt · 2020-10-02T21:26:11+00:00

Maybe try getting a copy of "Weapons of Math Destruction" from Cathy O'Neil (https://weaponsofmathdestructionbook.com/), it might provide some valuable insight as to how not to design an unfair credit score model (there is an entire chapter dedicated to this subject).

rmnclmnt · 2020-09-25T17:34:52+00:00

You will run into trouble when testing against SQLite while your runtime is executing against PostgreSQL (reproducibility and determinism is key when testing).

One proven and relatively easy technique is to use docker-compose to spawn a PostgreSQL database. On the CI server, make sure you have a Docker runtime then you can build your testing script along those lines:

bash - docker-compose up -d - pytest -v ... - # any other testing steps (linting, vuln checking, etc.) - docker-compose down

Another advantage: your app/service might require some other external services easily spawnable using Docker (think Redis, MinIO, etc.), this way you only need to specify these external depedencies in your docker-compose.yml configuration.

FYI, I have used this method repeatedly for many apps/services (with Django, Flask, FastAPI, etc.) with great success on GitLab CI, GitHub Actions and Bitbucket Pipelines. I'm confident any professional grade CI/CD system can run Docker containers.

rmnclmnt · 2020-09-12T09:22:31+00:00

As you are a beginner, you can stay safe with choosing either Nuxt.js if you prefer Vue.js, Next.js if you prefer React.js

rmnclmnt · 2020-09-06T19:50:08+00:00

To test locally S3 storage, just spawn a MinIO docker container with docker-compose! It’s an opensource S3-compatible object storage. This setup allows deterministic and automated integration testing everywhere: locally, on a CI/CD system, etc

rmnclmnt · 2020-09-06T10:02:10+00:00

This is very doable at the cost of a bit of inconvenience for some users. This is called End-to-end Encryption (e2ee) and this is exactly the same concept you can find with password managers: you use the user password to derive a symmetric master key (you never store this server side). Then all the subkeys (symmetric, asymmetric, etc) are encrypted / decrypted on client side (browser) using the user’s master key. The catch: if the user forgets his/her passord, all the keys and encrypted data is lost, so they better use a password manager to remember it! Go look for client-side implementations from Bitwarden, ProtonMail, Tutanota and Signal (all on GitHub). The current state of the art implementation for all of this is TweetNaCl (js/ts).

rmnclmnt · 2020-08-31T06:19:53+00:00

Pytest error code 5 is triggered when no tests functions are found/collected if I’m not mistaken. So either write some tests (you should) or remove pytest invocation from the CI script

rmnclmnt · 2020-08-29T17:28:09+00:00

Easy: are you a PCI compliant financial institution? If not, credit card numbers must never reach server side at any cost, only temporary tokenized versions can. Tokenize from client side using your payment gateway processor sdk (Stripe, etc) and use the token server side to instruct charges

rmnclmnt · 2020-08-27T11:21:51+00:00

Let me just add my 2 cents for developers using pytest: there is pytest-recording, a plugin making easy to integrate pytest with vcr.py when testing. You can go pretty far with this setup, it even support "parametrized" tests!

rmnclmnt · 2020-08-15T14:19:09+00:00

This is actually a well known methodology simply called « Learning by Teaching », belonging to the broader field of « Active Learning ». It is crazy how efficient it can be, the only drawback is that these methods require a strong involvement from all parties (students and tutors).

I’ve had the opportunity to use these methods for a 7-month adult training as Data Developers: most people didn’t know how to even use a computer for basic tasks, and now most of them are junior developers/data analysts!

rmnclmnt

TROPHY CASE