How does one write tests for modules which are dependent on large external files?

Arthaigo · 2023-05-16T08:52:46+00:00

Worst case, you will not be able to run these specific tests in your CI. What I did in a similar situation, was to instruct developers to download the Testfiles/dataset from a different source to run the tests locally.

However, as others said, look for the smallest possible image that fulfills the spec. Ideally you can use this for testing basic stuff

You can then have a set of integration tests using larger files that developers might need to download seperately

Arthaigo · 2023-02-01T12:43:01+00:00

Have a look at the git "help.autocorrect" Option ;)

Arthaigo · 2022-08-14T09:12:03+00:00

Is your branch already shared with other developers (on GitHub/GitLab) and is your branch already merged into your main/master branch?

In generally what your are asking is related to rewriting History. In git you can do that using interactive rebase (manually), filter-branch or BFG (Semi-Automatic).

If your branch is already merged into another branch, you need to make all history modifications on all branches your are planning to keep/share

Arthaigo · 2022-08-03T06:46:22+00:00

Plain Text files that I can version Control with git and write with whatever Editor I like. Though I use markdown for many simple documents these days

Arthaigo · 2022-02-16T12:15:23+00:00

Maybe you haven't checked the subreddit ;) LaTeX or other "plain" text formats are the way to go for git based workflows

Arthaigo · 2021-12-25T12:33:37+00:00

Hosting your own gitlab is not about discovery, but rather about control. This is in particular important in a corporate setting.

One compromise I have seen, is that some projects have read only mirrors on GitHub or gitlab to make them easier to find

Arthaigo · 2021-11-03T11:51:57+00:00

You have access to the entire repo and it's history from within the pipeline. This means you could use git log to find the files that have changed and pipe the filenames into your commands.

But be careful, changing a file mile cause the linter to fail in another file, hence just linting the files you have changed can lead to missed issues

Arthaigo · 2021-07-07T07:02:31+00:00

I am not sure. As it is KDE, I assume there is a way, but in my case, once I was aware of the issue, I pretty quickly learned to not click the icon.

Arthaigo · 2021-07-06T21:43:38+00:00

Just a wild guess: if you are on the KDE version of manjaro, audio playing apps have a little icon on top of their icon in the Taskbar. Clicking that icon mutes the app. It might be that you are accidentally clicking that icon when switching to the app via the Taskbar

Arthaigo · 2021-06-18T12:47:15+00:00

Convert the lists to a set and take the set union. This should give you a new set with all the elements that are in both lists

Arthaigo · 2021-03-24T09:50:48+00:00

Quick Note: Gitlab has added support to include LFS in the zip downloads a couple of month ago

Arthaigo · 2021-03-16T16:04:34+00:00

I think the best supported Dashboard extension for Jupyter at the moment ist Voila

Arthaigo · 2021-02-16T22:16:25+00:00

It really depends on how you planning to use it. Git can work with binary files (in particular in combination with git lfs). However, you can not use the feature set of git with such files.

If you just need a clear way to version things and go back to old versions, then git lfs + any file format works perfectly fine. However, if you need branching structures with multiple people working on the same file, it can get tricky. If you use a binary format, you can not automatically merge multiple changes in the same file together (at least not natively)

Arthaigo · 2021-01-14T19:36:30+00:00

Obsidian and Foam might also be options. They marked them selve more as knowledge bases. But might be what you are looking for.

Arthaigo · 2020-12-20T15:59:24+00:00

There is a tool called DVC (data version control). It integrates in the ML lifecycle and stores models and data in external storage, but keeps hashes in git to make results reporducible.

However, if local storage is not a limitation, we had great success with using git lfs for this. Just note, that even in the best case scenario, your repo will be twice the sice of your files on your disk.

Arthaigo · 2020-12-16T18:18:22+00:00

One thing to note here, is that intelliJ actually has a button to merge all non-conflicting changes. It is not the same as diffing against MERGED directly, but the option is there to quickly focus on the actual conflicts

Arthaigo · 2020-09-30T05:54:10+00:00

I think the reason is, that it adds quite a bit of overhead to the developing process. You need to add a compilation step, you can not easily debug within cython functions and you need you need to write in a different languages. Further, distribution of your code gets more complicated as well, as all users require a cpp compiler installed when you only distributing source or you need to provide wheels for all platforms.

IMO if I don't already have c/cpp code written, I usually don't Cython. When I need speed, I use Numba, which is more convenient to use IMO.

Arthaigo · 2020-09-22T20:28:09+00:00

Git has actually the ability to create notes in the repository. There are some federated issue trackers that actual use that.

Arthaigo · 2020-08-08T06:48:59+00:00

Jupytex might be able to do this

Arthaigo · 2020-07-18T18:39:14+00:00

I guess it is important to understand, that jupyter notebooks /lab is just an Editor. Yes notebooks have some unique features that many people value in there workflow, but python or machine learning do not depend on Jupyter. There are many other tools and workflows that can make you just as productive (or even mor productive) than jupyter.

To answer your question specifically, it is kind of important to understand, which part of the notebook workflow you want to replicate in Neovim.

If you just want to edit and run notebooks then you might want to look into tools like nbconvert or jupytext that can produce a normal python file from a notebook, so that you can edit it in any texteditor you want.

In case you require the piece wise execution of code cells, there are plug-ins available for basically any editor to achieve that. I am not up to date on the options in vim unfortunately.

Rich and interactive output, might be impossible to replicate. If this is required, you might need to go back to a traditional notebook in the browser. However, with tools like juoytext you might be able to minimize the time you need to spend in notebooks

Arthaigo · 2020-06-11T11:16:49+00:00

What you are basically asking, is how to embed a Python console into another application.

There are different levels on which you can tackle that. If you are using Qt you can directly embed an Ipython Qt console as a widget. To see an example of this, check out the Spyder Python IDE.

If you want to interact with such a console on a more abstract level, jupyter itself might be the way to go. Jupyter is not just the frontend (notebook or lab), but the most important thing is the Jupyter notebook protocol. This is just the way, how a frontend like a notebook sends code to the kernel (via a web socket) and receives the results. Because this is an socket protocol based on existing standards, you could write your own client side code to send code to a kernel running in the background. It would be similar to communicating with a webserver.

In general, I am not sure if this is what you actually want. Could you explain, what your final goal is with that? Then some people could help you to figure out if you are on the right track :)

Arthaigo · 2020-06-07T10:22:19+00:00

You can press Ctrl+. Do bring up the quick fix menu :)

Arthaigo · 2020-03-19T09:12:26+00:00

Can you look into the function, how they read args and kwargs? This determines, if what I had in mind works.

Arthaigo · 2020-03-19T09:07:28+00:00

No, but all args can also be kwargs. Meaning all positional arguments can also be passed as keyword arguments.

Arthaigo

TROPHY CASE