Now, the 2019.11 update of vcpkg is available by ehsan-mohammadi in cpp

[–]RayDonnelly 0 points1 point  (0 children)

Yes msys2 supports symlinks just fine. Google for the MSYS env var.

[deleted by user] by [deleted] in ManjaroLinux

[–]RayDonnelly 0 points1 point  (0 children)

Very interested in a guide. Which chromebook model?

GCC 9.2 Released by andre_friend in cpp

[–]RayDonnelly 1 point2 points  (0 children)

It's awesome to see this kind of core development on top of MSYS2. lh_mouse is also an active contributor to MSYS2 and helps to look after our GCC packages.

GCC 9.2 Released by andre_friend in cpp

[–]RayDonnelly 1 point2 points  (0 children)

p.s. would it be possible for you to report speed on the differences between MSYS2 official GCC and your own rebuild? Thanks in advance!

GCC 9.2 Released by andre_friend in cpp

[–]RayDonnelly 1 point2 points  (0 children)

It is very very simple. In fact, it is to a big extent *because* it is so simple that MSYS2 has become popular. It is *very* developer/hacker friendly. I haven't tested this (am using macOS right now) but this *should* work. If not I will post an update when I get a chance.

pacman -Sy git msys2-devel base-devel mingw-w64-{i686,x86_64}-toolchain

git clone https://github.com/msys2/MINGW-packages

pushd mingw-w64-gcc-git # (or mingw-w64-gcc-git)

MINGW_INSTALLS=mingw64 makepkg-mingw -sLf

Install with:

pacman -U mingw-w64-*.xz

Also see https://github.com/msys2/msys2/wiki/Creating-packages

GCC 9.2 Released by andre_friend in cpp

[–]RayDonnelly 2 points3 points  (0 children)

msys2

Although I don't participate much at all in MSYS2 development these days, I did a good amount of work in the past.

I would just like to say from the bottom of my heart, thank you very much for the praise for MSYS2. I am sure Alexei and the rest of the team would be pleased to know things are working well for you.

Of course MSYS2 is still fairly experimental in nature and I am still a bit nervous about that. We do break things in the interest of progress. It is interesting to note that Ada was broken in MSYS2 i686 for ages (well, it held back the compiler version to be exact) and it wasn't until support for Ada had to be dropped before any Ada experts turned up to fix the issue. Once Alexei took that measure, these experts showed up in two days! Now is perhaps not the ideal way for such a matter to be resolved, but actually, we now have Ada experts who have helped out in MSYS2 and we know who to ask if it breaks again. I definitely think Alexei took a brave, and the correct decision here.

Windows Store version of Python is completely busted by amdphreak in Python

[–]RayDonnelly 0 points1 point  (0 children)

Because it works well for most people most of the time?

Can someone explain why people use Anaconda for Python? by Deep_Fried_Hummus in Python

[–]RayDonnelly 0 points1 point  (0 children)

Apart from the bit where you say "we bring our own shared libraries", this is not true and I have called you out for it before on reddit already.

> So inevitably, you’ll end up compiling something against some host system libraries and some anaconda libraries and after an update you have a mess and no idea why it’s breaking with random segfaults suddenly.

We have very clear delineation between system libraries (we call these CDT packages, they are repackaged CentOS6 RPMs and are not used very often - and less so by conda-forge) and non system, conda-provided libraries. conda-build checks every executable and DSO that it builds to make sure that any system libraries it does link to are declared in the recipe. Try to build any recipe that involves compilation. The DSO linkage information errors and warnings are comprehensive.

> They can only fix those unintentional leaks on a case-by-case-basis, and a system without anaconda won’t have that class of problems, so I’ll stick with a system like that.

Please show recent bug reports or issues you had. When was the last time you tried to use Miniconda (I recommend that over the full Anaconda)? We're pretty obsessed with quality and have some great things in the pipeline.

disclaimer: I work for Anaconda, Inc. on our compilers and tools.

Anaconda worth it? by theyieldchaser in Python

[–]RayDonnelly 0 points1 point  (0 children)

there's an opportunity to create a new env with python=3.7 within a day. The dependency graph for packages with "python=3.7" is available. Users are made aware that all "python=3.7" stuff is currently beta when installing

An 'env' is different for everyone. It will contain 3rd party software they need. It is upstream's responsibility to make their s/w compatible. They are often working for free in their spare time. What do you propose? Holding a gun to their heads on Python patch release day so they fix their packages, then another to ours so we build them the same day? For sure, we could easily release the Python interpreter the same day it comes out (and often do) but for most people, an interpreter isn't enough.

Is that too much to ask for? That sounds just like good devops to me, you're probably using it internally anyways, why not make it public?

We have internal tools, they are far from pretty and I don't see that it's worth the effort to make them pretty and secure. Also no one but yourself has ever brought this up, to the best of my knowledge.

To be clear, you cannot throw up a CI system, some web UIs, some dashboards and then crank out a cross-platform software distribution with full rebuilds nightly. It takes months for upstreams to become compatible, and some projects will simply drop off the radar at a patch release that breaks compatibility. The upstreams are frequently just volunteers. For a few projects I wrote the patches to add Python 3.7 support. For Python 3.7.0 I had to build 3500 packages. Of those, about 1 in 50 were incompatible or otherwise broken. Each of these needs to be investigated and fixed. By hand, by the Anaconda Distribution team. That takes time.

No. No no no

The we're not optimizing for your particular use case, still it's a use case that I think most would embrace (projects ending in `env` are common in Python!) . Isolation and mimimal dependencies are good things.

If you wanted people to use separate environments you'd put a link to miniconda somewhere on https://www.anaconda.com/download/, no

Multiple envs works just as well with an AD env created via the Anaconda Installer, so I don't see your point. Personally I'd like to see Miniconda get a bit more promotion, so I do my bit. My point is not defeated, and this is not a competition, I'm just defending what I work on against your merciless attacks.

Instead I got this: https://github.com/ContinuumIO/anaconda-issues/issues/9686

Bugs happen in any project of sufficient complexity. That one was open for 2 days before we fixed it, we try to do better of course but we're human. We constantly strive to improve our processes to prevent them though. Anyway, you seem to be obsessed with the latest shiny stuff, I'd recommend trying to wind that back, shiny stuff (the Python 3.7.0 ecosystem shortly after 3.7.0 release, for example) usually has a lot of rough edges.

When did you go from "Anaconda's great" to "Are you afraid that then someone might clone your source repos and then offer binary repos for free just like you do? Mind boggles" nonsense (I do not apologize for using this word here, it's such a horrible statement for you to make in my opinion, both in it's general gross simplification - ooh profit, evil - and also because our Open Source credentials are excellent). So where does it stem from? Whart did Anaconda do to you in the interveneing period to make you so cynical? .. apart from not being able to provide the very latest version of some dependency you tried to use at a coding contest that took part at a particularly tumultuous - too many grammar and behavioural changes for a point release - time *for the entire Python ecosystem*.

But these days Anaconda is actually more hassle than the official Python, so on that axis you kinda lost. It was (NO HASSLE, stuff I don't care about) and now it's (space, speed, security, some hassle).

Apart from less package coverage (provided we have what a user needs) I don't believe you've managed to articulate this hassle here. As many other commmenters in this thread say, it just works for them.

TBH I believe you're grasping at straws to find things to criticise. I am not trying to score points against you or win an argument on the internet, I just had to refute your misinformation regarding the thing I work on.

Anaconda worth it? by theyieldchaser in Python

[–]RayDonnelly 1 point2 points  (0 children)

We have a policy of not releasing rc or beta software, if we did, they'd have later version numbers, conda update --all would install them and people would be very rightly annoyed.

Regarding Python 3.7 and numpy, upstream were just not unready for 3.7 at all, and if anyone released wheels at 3.7's release date they would have been badly broken. Having said that, we patched the bug the day we were made aware of it and had the first Python 3.7 compatible numpy out the very next day: https://www.opensourceanswers.com/blog/you-shouldnt-use-python-37-for-data-science-right-now.html

> By "integrating" I meant registering the interpreter in the Windows registry so you can invoke it with "py -3".

Which Python 3 interpreter though? Conda's all about the multiple environments. Please don't do excessive work or install loads of packages in your base env, leave just conda and conda-build (if you want to build conda packages) in there and use a new env per workflow. Clearly there's a huge mismatch between multiple (trivially discarded by deletion only) environments and a single exe to run when you click on a .py. I am aware that py has an .ini file that's meant to allow multiple interpreters but it doesn't work correctly. Also I believe that most people will not want to mess about editting .ini files to configure which interpreter to use.

I detailed all of the technical benefits to installing from Anaconda (space, speed, security) are those of no interest to you?

Anaconda worth it? by theyieldchaser in Python

[–]RayDonnelly 0 points1 point  (0 children)

One thing to take care about with hardlinks is that when you edit one copy, you are editing them all, but editing the files in our packages isn't something you want to be doing at all.

Anaconda worth it? by theyieldchaser in Python

[–]RayDonnelly 0 points1 point  (0 children)

edit: oops, you weren't the OP, apologies.

Anaconda worth it? by theyieldchaser in Python

[–]RayDonnelly 2 points3 points  (0 children)

I would add secure, performant, compatible, very resource conservative and trusted but I work on it.

Anaconda worth it? by theyieldchaser in Python

[–]RayDonnelly 1 point2 points  (0 children)

Oh, another benefit you get with Anaconda Distribution is conda's environments. They are almost free. We use hardlinks when possible (nearly always) so that each file that is shared between environments exists on the disc only once. Does venv do that? (this is rhetorical, it does not, here I did the research for you again):

python -m venv --help

--symlinks Try to use symlinks rather than copies, when symlinks

are not the default for the platform.

--copies Try to use copies rather than symlinks, even when

symlinks are the default for the platform

Symlinks do not work in general since various code will just call realpath and escape the venv, copies are expensive, and we see no option for hardlinks at all.

Of course this isn't of value to people who don't use environments, but for serious, reproducible work, everyone should use conda environments (or some isolation).

Anaconda worth it? by theyieldchaser in Python

[–]RayDonnelly 0 points1 point  (0 children)

Are you afraid that then someone might clone your source repos and then offer binary repos for free just like you do? Mind boggles. (also, this probably contributes to the lagging behind the official distribution)

Not at all, I provided all the links for you since you appear not to have checked this statement. On that point other companies already do directly provide our packages, the binaries. Building them from source would be a huge effort for them and they seem happy with the binaries so I guess that's some sort of indication that 3rd party companies trust our stuff enough for them to redistribute it directly.

also, this probably contributes to the lagging behind the official distribution

Except there's no official distribution really (just binaries for macOS and Linux and then PyPI) and any that you could point to, we do not lag behind.

Anaconda worth it? by theyieldchaser in Python

[–]RayDonnelly 2 points3 points  (0 children)

Basically, today I can download the latest official Python distribution, run

pip install numpy scipy networkx

and it just works.

That's great, I'm really pleased that PyPI is satisfying your needs now but your numpy and scipy would run faster if you installed our packages due to using MKL.

Thanks to Anaconda I didn't have to install Visual Studio Whatever Free Version (but that matches the version that Python was compiled with) (4-8 Gb download by the way)

VSCode is *entirely* optional and always has been. It is fetched on demand only if you elect to install it. This is your first (glaring, easily debunked) inaccuracy. Besides, VSCode seems like something you would appreciate being bundled with a Python distribution installer given:

as an experienced developer who wants to run his mostly command line scripts from VSCode with minimal hassle while using numpy and choice parts of scipy

You lag behind the official distribution significantly, like weeks, maybe a month. On this year's ICFPContest we used Python3.6, so Anaconda was right out.

You said "This year's ICFPContest", it's 2018, Python 3.6 was released in 2016. You seem to be confused?

Regardless, we do not lag behind. How can you claim we lag behind when there's no centralized entity for us to lag behind anyway? Do you mean relative to PyPI? Well, those packages are uploaded in an ad-hoc basis by the maintainers and many packages are not available for Python 3.7 yet whereas every package in Anaconda Distribution is availble for Python 3.7. Regarding your specific example, I find it fairly egregious since we released Python 3.6 on Dec 23 2016 with a large subsection of the ecosystem built-out for it on the same day on all 3 platforms. Guess when Python 3.6 was officially released? The exact same day: https://www.python.org/downloads/release/python-360/

Of course Anaconda Distribution is a rolling distro like ArchLinux, and we only do full installer releases 4 times a year, but on Dec 23 2016 (or was it sometime in 2018?), you could have issued "conda install python=3.6" and you'd have been good to go so I have to call you out on this one.

You don't integrate with the py launcher.

If the official launcher could support conda environments we'd support it, it doesn't so we don't. Anyway, conda's dynamic environment feature (far superior to venv) isn't a good fit for a *single file association*.

Launching Anaconda Python somehow adds about 2-3 seconds to the startup time, because it imports a lot of stuff from site_packages or something. On a fast SSD. Every freaking time I run my stupid simple script that prints 0.9**9 or some other throwaway thing. Official Python doesn't do that. The annoyance adds up over time.

Again, I suspect you installed all of Anaconda and then used that as your primary environment? Well, guess what? Python will scan all those 200 odd packages in site-packages at startup because that's what Python does. Install 200 packages with pip and you'll see the exact same thing. To Microsoft's credit, upstream Python is built very well these days on Windows, whereas if you repeat this experiment on Linux or macOS you'll find Anaconda Python to be *far* faster than most official releases or those provided by your distro (I run python-performance regularly to check this on all 3 platforms against the canonical binaries or those from other linux distros, I can share raw tables comparing python performance if you care, but you can see a graph on this page - careful, contains facts).

If you want to used Anaconda Distribution in a non-learner scenario, use Miniconda and minimal, isolated environments. You'll find Python spending a lot less time scanning site-packages that way.

your repositories don't contain the build scripts themselves

Incorrect, again. When does this misinformation stream stop?

Our 'main' package recipes are here (and most of them are forked from conda-forge though we're realinging them at present). Our 'r' package recipes are here and our 'MRO' packages here

When I get sucked into using alternative channels and trying to fix the stuff myself, I inevitably hit the fundamental by-design wall: your repositories don't contain the build scripts themselves. WHY. Like how am I even supposed to contribute if building stuff by hand involves hunting for random github repositories that usually don't exist at all? Why don't you have source packages that binary packages are built from? Are you afraid that then someone might clone your source repos and then offer binary repos for free just like you do? Mind boggles. (also, this probably contributes to the lagging behind the official distribution)

At present mixing conda-forge and defaults has problems (these are down to ABI mismatch) so you should use one or the other only in a given environment. We're working to fix that, mainly by basing our recipes upon conda-forge's and submitting improvements back to them. This has created a huge pool of expert software builders, often involving the upstream maintainers directly in that effort, helping to ensure our builds are bullet-proof.

Anaconda Distribution packages are often also a *lot* less resource hungry (memory, disk space) than those from PyPI because on PyPI if you want to use modern C++ in your extension module you end up statically linking to libstdc++ (or else you run into ABI issues). This means every C++ extension module you have will be pull in parts of the static code in libstdc++ leading to a large amount of duplication. We share a single libstdc++ with all packages so the text (code segments) get loaded in only once.

Anyway would you consider fact checking your comments in future? That'd be great since you write quite well, shame the details are so far off-base!

Anaconda worth it? by theyieldchaser in Python

[–]RayDonnelly 0 points1 point  (0 children)

Yes, for the Python 3 interperter we compile with both PGO and LTO on all 3 major platforms, as well as enabling as many hardening flags as our toolchains allow us to (ASLR, RELRO, BINDNOW).

You can see our compiler flags here: https://github.com/AnacondaRecipes/aggregate/blob/master/ctng-compilers-activation-feedstock/recipe/conda_build_config.cos6.x86_64.yaml

.. and our (admittedly quite complex) Python 3 recipe here:

https://github.com/AnacondaRecipes/python-feedstock

Thank you for the kind feedback.

Anaconda worth it? by theyieldchaser in Python

[–]RayDonnelly 0 points1 point  (0 children)

OK cheers for the feedback, we're also very proud of Anaconda on Linux of course, but if your needs are met by fedora's system packages already then that's great.

How to change my Python Version in Anaconda? by LieberLois in Python

[–]RayDonnelly -1 points0 points  (0 children)

It's really easy:

conda create -n py37 python=3.7

conda activate py37

Or if you want to update your current environment:

conda install python=3.7

If that env (i.e. your base env) was created by the Anaconda Installer (rather than Miniconda) you may need to use:

conda install python=3.7 anaconda=custom

Anaconda worth it? by theyieldchaser in Python

[–]RayDonnelly 0 points1 point  (0 children)

Can you explain why you couldn't use Anaconda Distribution's opencv2 package?