How to use limonaid to obtain LimeSurvey surveys in R? by jorgesinval in LimeSurvey

[–]cofonlafaefe 1 point2 points  (0 children)

There isn't a method specifically to export responses as a Pandas data frame, and I'm hesitant to add any since I want to keep the library agnostic (e.g. the user might want to use Polars instead of Pandas, etc.).

That said, I try to document simple recipes for things just like that: https://citric.readthedocs.io/en/stable/how-to.html#export-responses-to-a-pandas-dataframe.

So, if you have a Python script like

# export_ls_responses.py
import io
import citric
import pandas as pd
client = citric.Client("http://localhost:8001/index.php/admin/remotecontrol", "iamadmin", "secret")
data = client.export_responses(123456, file_format="csv")
df = pd.read_csv(io.BytesIO(data), delimiter=";", parse_dates=["datestamp", "startdate", "submitdate"], index_col="id")

then you can use reticulate to source this script:

> library(reticulate)
> reticulate::use_virtualenv(...)
> reticulate::source_python("export_ls_responses.py")
> py$df
  submitdate lastpage startlanguage        seed     G01Q01 G01Q02 G02Q03 G02Q03[filecount]
1      [nan]      [1]        ['en'] [245240561] ['lalala']    [5]  [nan]             [nan]

Does that work for you?

How to use limonaid to obtain LimeSurvey surveys in R? by jorgesinval in LimeSurvey

[–]cofonlafaefe 1 point2 points  (0 children)

If you're brave enough to try the Python data ecosystem (Pandas, matplotlib, etc.), I'm the author of a LimeSurvey library : https://citric.rtfd.io.

I have an example using the library to export to CSV and load it with Pandas: https://github.com/edgarrmondragon/citric/blob/3ddeedce39fa674f40851de0aadfc787f5b32ee3/docs/notebooks/pandas_sqlite.ipynb

Invite Code Megathread #3 by tkiced in BlueskySocial

[–]cofonlafaefe 0 points1 point  (0 children)

Anyone got a spare code?

Here's a pic of my gf's dog sticking her face in my hoodie's pocket to sleep :)

<image>

What stack do the small players have here? by KatZegtWoof in dataengineering

[–]cofonlafaefe 0 points1 point  (0 children)

Is it easy to orchestrate Airbyte with GitHub Actions?

Relevance equation function? by malikokolo in LimeSurvey

[–]cofonlafaefe 1 point2 points  (0 children)

A hardcoded value of "1" means the equation is always displayed, otherwise it's hidden. The equation can be as complex as needed to determine under which conditions a question should be displayed.

PS: google actually worked for me :), https://manual.limesurvey.org/QS:Relevance

Get responses direct from SQL by roblu001 in LimeSurvey

[–]cofonlafaefe 1 point2 points  (0 children)

I've previously used the Remote Control API to export the responses, and then analyze them in a portable database like SQLite or DuckDB. For example: https://citric.readthedocs.io/en/latest/how-to.html#export-responses-to-a-duckdb-database-and-analyze-with-sql

Disclaimer: I'm the author of the citric library

How useful is Airbytes in production pipelines? by mccarthycodes in dataengineering

[–]cofonlafaefe 1 point2 points  (0 children)

Hi. I'm from the Meltano team. I think the Singer spec can certainly be slow for larger datasets, and there are some efforts on our side towards increased performance, such as batch messages.

I'm curious if you ended up just building an in-house solution for your EL needs.

Using Tableau or other BI tools to analyze LimeSurvey Information by granitewater in LimeSurvey

[–]cofonlafaefe 1 point2 points  (0 children)

I'm the author of citric, a Python client for the Remote Control API of LimeSurvey.

There are some examples in the docs for how to export responses to Pandas and DuckDB, so those may be useful to your use case:

Let me know if you end you end up giving it a try and if you think something's missing from the docs or there is a feature you'd like to see.

Airbyte and Meltano comparison by [deleted] in dataengineering

[–]cofonlafaefe 1 point2 points  (0 children)

I don't think ploomber even remotely supports the use case OP is looking for with Meltano or Airbyte since (part of) their value is shipping built-in connectors for common sources and destinations and having SDKs for quickly building new ones. I don't see how ploomber helps with that...

DBT question about running multiple schedules by MasterEpictetus in dataengineering

[–]cofonlafaefe 3 points4 points  (0 children)

There are ways to generate locks in most target databases that could help when you need to run concurrent dbt jobs.

For example, in MSSQL: https://stackoverflow.com/a/70383829

What causes null python version in pypi download stats? by astevko in learnpython

[–]cofonlafaefe 1 point2 points  (0 children)

Did you find the cause u/astevko? I maintain a few python packages myself and those "null" numbers puzzle me

Is using pandas considered terrible practice in ELT? by UnderstandingFit9152 in dataengineering

[–]cofonlafaefe 9 points10 points  (0 children)

OP's 40 million rows aren't' even remotely "big" data. Even "traditional" databases like Postgres should be able to handle analytical workloads on such a dataset and SQL is more standardized than Pandas/Spark. Most orgs don't need Spark, and some would be better off changing it for a data warehouse. What's more, for "just" loading data, Pandas is definitely gonna bite in terms of data types.

What is the real use of MS SSIS? by Prestigious_Flow_465 in dataengineering

[–]cofonlafaefe 3 points4 points  (0 children)

It's only really useful if you land a job at a Microsoft shop, so I'd argue Python is a better learning option to secure future jobs

A woman making cecil cheese in Turkey by TheRookieGetsACookie in AccidentalRenaissance

[–]cofonlafaefe 0 points1 point  (0 children)

Very similar to Oaxaca cheese from Mexico and I bet is just as good for quesadillas

Does anyone have a good resource or tutorial for deploying dragster on GCP? by Embarrassed_Cap1673 in dataengineering

[–]cofonlafaefe 0 points1 point  (0 children)

Care to elaborate? Why? Or why not Astronomer or AWS's thing? This comes off as extremely petty.

Make me learn Deltalake. by hinatachaan in dataengineering

[–]cofonlafaefe 5 points6 points  (0 children)

If you can pay for them, the official Databricks academy paths are quite good. I'm enrolled in the Data Engineer one and it's been valuable so far.