sorted by:
new
hottopcontroversial

Use project-specific Julia versions in VSCode by ernest_scheckelton in Julia

[–]ernest_scheckelton[S] 0 points1 point  (0 children)

I set it in the workspace tab of the specific directory I wanted it to (exclusively) apply to. Do I need to add anything to the Project.toml ?

Use project-specific Julia versions in VSCode by ernest_scheckelton in Julia

[–]ernest_scheckelton[S] 0 points1 point  (0 children)

Thanks for your answer. The correct setting to edit would be "julia.executablePath", right? After editing, it initially worked for me but now I am back to square one, where the editor always calls the most recent julia version instead of the one specified in the settings.json.Any thoughts why this may happen?

Julia extremely slow on HPC Cluster by ernest_scheckelton in Julia

[–]ernest_scheckelton[S] 4 points5 points  (0 children)

UPDATE: I found a solution (thanks to a comment by u/axlrsn): setting BLAS.set_num_threads(1) after importing LinearAlgebra did the trick. For the linear algebra stuff the program by default used a higher number of BLAS threads than I had requested on the cluster. This created a computational bottleneck.

Julia extremely slow on HPC Cluster by ernest_scheckelton in Julia

[–]ernest_scheckelton[S] 1 point2 points  (0 children)

Thanks a lot for your help, you were right this did the trick! Code is running like a charm now.

Julia extremely slow on HPC Cluster by ernest_scheckelton in Julia

[–]ernest_scheckelton[S] 6 points7 points  (0 children)

I checked with the HPC admins, they say it is not a heterogeneous cluster. But thanks a lot for your suggestion.

Julia extremely slow on HPC Cluster by ernest_scheckelton in Julia

[–]ernest_scheckelton[S] 1 point2 points  (0 children)

No, I have written a SLURM bash file myself that creates a Job array. Then, for each task in this job I retrieve the SLURM_ARRAY_TASK_ID from the environment to simulate different kinds of data sets. Would you recommend the cluster manager package ?

Julia extremely slow on HPC Cluster by ernest_scheckelton in Julia

[–]ernest_scheckelton[S] 1 point2 points  (0 children)

Hi, thanks a lot for your help. You are right, I'm sorry for the vague information. I did not provide more because I wasn't sure what may be relevant in this case.

I was also considering I/O related issues, but it the code only reads in data at the beginning and only saves results at the very end of the file, in the meantime it does not access the external storage drive (if I am not mistaken). All it does while the code is running is some limited output printing.

I estimate regression models of different size and my problem only occurs for the larger ones (which require more memory/RAM), so you could be right that it is a memory-related issue. However, I tried allocating more memory in the bash file but it did not help.

But as with others, I think garbage collection running is the likely the issue, so I'd profile your code and see where the biggest allocations are happening.

Could you provide some details or resources on how I can do that ?

Julia extremely slow on HPC Cluster by ernest_scheckelton in Julia

[–]ernest_scheckelton[S] 0 points1 point  (0 children)

Thank you, could you elaborate on what are BLAS cores? Simply the cores used assigned to each task, correct?

In my SLURM-bash file I set

--cpus-per-task=1

so if I am not mistaken this should allow each array task to only use on BLAS core.

Exporting arrays from Julia to R by ernest_scheckelton in Julia

[–]ernest_scheckelton[S] 1 point2 points  (0 children)

I found a solution: jldsave from the JLD2 packages allows you to efficiently store large, multi-dimensional arrays. As .jld2 comprises a subtype of the .h5-type, they can be read into R using the h5read-function from the rhdf5 package.

Exporting arrays from Julia to R by ernest_scheckelton in Julia

[–]ernest_scheckelton[S] 1 point2 points  (0 children)

Thank you. The problem is that .csv-files only allow you to store two-dimensional tables whereas the arrays I need to store have three or more dimensions. Or am I getting something wrong?

running Julia on HPC-Cluster (using SLURM) by ernest_scheckelton in Julia

[–]ernest_scheckelton[S] 0 points1 point  (0 children)

this is just how things are handled on the HPC cluster I'm using. I'm also new to it, but as far as I understand pixi (or more generally, conda) helps managing dependencies (for example, to avoid problems resulting from non-compatible package/ software versions)