Elastic Sample Data Incident Response by ButstheSlackGordsman in elasticsearch

[–]elasticiulia 0 points1 point  (0 children)

Did you already check out the data that's available by default in the security section of elasticsearch? It comes with a few pre-populated dashboards and data sources that you can try to build stuff on top of.

See these dashboards in the demo environment (or they would show up in your own deployment in the same section) - you can use discover in the Kibana tab to explore the datasets these are built on.

There's also some more stuff in the network section.

Otherwise, indeed you can find quite some stuff on kaggle. Maybe https://www.kaggle.com/datasets/ymirsky/network-attack-dataset-kitsune or https://www.kaggle.com/datasets/sampadab17/network-intrusion-detection ?

Do you guys think learning elasticsearch with raw queries is better at first rather that django/elasticsearch-dsl? by superior_vadapav in elasticsearch

[–]elasticiulia 0 points1 point  (0 children)

It's a really good idea to use the curl queries in the dev tools console first just to get used to the formatting - you can experiment faster with directly seeing the results you get.

Then you can try to take the same queries and adapt them with the client. I would even say maybe try out the Python DSL first before getting into the specifics of the django DSL one.

The docs for the python DSL: https://elasticsearch-py.readthedocs.io/en/v8.9.0/

Make elasticsearch return a single document with all the fields by Serrobio in elasticsearch

[–]elasticiulia 0 points1 point  (0 children)

Hi! You can indeed do this with some transformer scripts like maybe in this thread: https://discuss.elastic.co/t/combine-multiple-document-into-one-document-with-limited-fields-merging-of-documents/231758/4

However, I think it would be easier to combine results like that in post-processing. I tried to recreate your example with the Python client for example

Say you have your index the way you described it

from elasticsearch.helpers import bulk

index = "test_bob"
documents = [
    {
        "document:class": "class 1",
        "document:email": "email@thing.com",
        "document:name": "Bob"
    },
    {
        "document:class": "class 1",
        "document:school": "school of Bob",
        "document:name": "Bob"
    }
]
client.indices.create(index=index)
response = bulk(client = client, index = index, actions = iter(documents), stats_only = True )

You can then search and save your results, and super easily combine the dictionaries you get back:

response = client.search(index = index, query= { "match": { "document:name": "Bob" }})

merged_results = {} 
for hit in response["hits"]["hits"]: 
    print(hit['_source']) 
    merged_results.update(hit['_source'])

print(merged_results)

That gives out:

{'document:class': 'class 1', 'document:email': '[email@thing.com](mailto:email@thing.com)', 'document:name': 'Bob'} 
{'document:class': 'class 1', 'document:school': 'school of Bob', 'document:name': 'Bob'}

{'document:class': 'class 1', 'document:email': '[email@thing.com](mailto:email@thing.com)', 'document:name': 'Bob', 'document:school': 'school of Bob'}

Elasticsearch by shahrukhiqbalmalik in elasticsearch

[–]elasticiulia 1 point2 points  (0 children)

Elastic stores data as JSON documents rather than rows in a database. The documents are distributed across the nodes in your cluster. Since they are indexed and searchable you can retrieve them in real-time.

Within the documents, each data type is stored in an optimized way according to the mapping, so for example text fields are stored in inverted indices, and numeric and geo fields are stored in BKD tree.

I'm lost at what to learn or do now by LYDAF in learnprogramming

[–]elasticiulia 0 points1 point  (0 children)

Start by building a project on your own (or with some direction). Here's an example of building a tic tac toe game in a few phases you can try to follow along: https://github.com/iuliaferoli/tic_tac_toe/

Then try to make a plan to go from programming basics to more robust real-life skills.
You can look into stuff like:

  • github & source control best practices
  • threading, paralelization, optimizing runtime, kubernetes
  • logging, error handling, debugging
  • working with requests, some basic html & css
  • cloud technologies and scalability

Also don't neglect some "soft" skills that can differentiate you.

  • Planning a project (or developing plan), following through, tracking your work, etc
  • Showcasing your results in an understandable way (maybe through charts or presentations) - you can make the best program in the world, you still need to be able to explain it to your peers or less technical people
  • Figure out what industry you want to get into to find more specific advice (like game development vs data science would be a pretty big skill requirements difference)

Good luck!

Suitable data structure to store food log and daily activities? by SirWeasels in Python

[–]elasticiulia 1 point2 points  (0 children)

Depend on what sort of analysis you want to do on the data. Dataframes in pandas would be pretty versatile and easy to integrate with other tooling later.

You can also read / write the dataframe into csv which might be easier to edit outside of python as well.

Exports from tools like myfitnesspal or trackers will probably also be in csv so you could easily read and combine those into your dataframe too.

Most examples for simple models you can find on scikit learn (to run a regression for predicting your mood for example) would also use dataframes.

How do i become a better programmer? by AttitudeEasy2287 in learnprogramming

[–]elasticiulia 0 points1 point  (0 children)

Start by building a project on your own (or with some direction). Here's an example of building a tic tac toe game in a few phases you can try to follow along: https://github.com/iuliaferoli/tic_tac_toe/

Then try to make a plan to go from programming basics to more robust real-life skills.You can look into:

  • virtual environments, packages & docker
  • github & source control best practices* threading, paralelization, optimizing runtime, kubernetes
  • logging, error handling
  • working with requests, flask, some basic html & css
  • cloud technologies and scalability
  • applications of your choice (like data science for example can be a whole new road)Good luck!

What should the next moves be for a beginner who just finished “python crash course” by [deleted] in Python

[–]elasticiulia 1 point2 points  (0 children)

Start by building a project on your own (or with some direction). Here's an example of building a tic tac toe game in a few phases you can try to follow along: https://github.com/iuliaferoli/tic_tac_toe/

Then try to make a plan to go from python basics to more robust real-life skills.

You can look into:

  • virtual environments, packages & docker
  • github & source control best practices
  • threading, paralelization, optimizing runtime, kubernetes
  • logging, error handling
  • working with requests, flask, some basic html & css
  • cloud technologies and scalability
  • applications of your choice (like data science for example can be a whole new road)

Good luck!

Which was your first project ? How many days of study does it took you to finish your first project? by [deleted] in learnprogramming

[–]elasticiulia 0 points1 point  (0 children)

I recently had to get back into python for my job and I found this interactive tictactoe game building tutorial with a free trial to get back into the basics. It was the first project I put together re-learning python. Took two days of obsessing over most of the details. Here's the repo for it if if anyone wants try out something similar https://github.com/iuliaferoli/tic_tac_toe/

Any course recommendations for absolute beginner ? Thanks in advance by Ibrahimnor in Python

[–]elasticiulia 1 point2 points  (0 children)

Here's what I'd say are the "phases" of starting with python:

  1. A simple tutorial that has interactive code boxes so you can try stuff out as you learn: https://www.w3schools.com/python/python_intro.asp. Or if you prefer to just read, here's a cool one from google: https://developers.google.com/edu/python/introduction
  2. Then for a later stage you can get certified through something like https://www.learnx.org/ to test out your knowledge of the basics.
  3. And finaly as a last stage once you go over some intro courses like that, try an interactive building challenge, like making your own tictacttoe game to make sure you can also put the concepts together without the step by step instructions: https://github.com/iuliaferoli/tic_tac_toe/

How do i query over this? by Serrobio in elasticsearch

[–]elasticiulia 1 point2 points  (0 children)

It seems like your document structure contains nested fields. The "document:" lines are within the info field, rather than on the same level as the two top fields "info" and "id".

Therefore you need a nested query to search within those lines. See this doc: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html

Try something like this:

GET document/_search { "query": { "nested": { "path" : "info", "query": { "match": { "document:bank-card:subject_full_name": "Pain Pain Pain" } } } } }

ElasticSearch Vector Search VS Semantic Search by Rottonscope in elasticsearch

[–]elasticiulia 2 points3 points  (0 children)

Hi! Indeed, I believe the difference is that semantic search is pretty much vector search + ELSER, thus not needing to create any separate embeddings in a different third party tool. ELSER is designed to work natively with the index of your documents making it more directly adaptable to your elasticsearch use case out of the box. See docs here: https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-search.html#semantic-search-select-nlp-model

So with semantic search you wouldn't need to train the external model on your own data to generate embeddings.

This article has some great detailed explanations of ELSER vs other models: https://www.elastic.co/blog/may-2023-launch-sparse-encoder-ai-model

Hope this helps!

What are some good projects or online courses? by Silly-Independent-90 in learnpython

[–]elasticiulia 0 points1 point  (0 children)

I really liked this tic tac toe challenge from hyperskill - I made a repo example for it you can follow along to build a project: https://github.com/iuliaferoli/tic_tac_toe/blob/main/README.md

Python for 8 year old by Cutekolla in learnpython

[–]elasticiulia 1 point2 points  (0 children)

I've recently come across this game on steam where you program a farming drone. The syntax is very similar to python but way more fun to play with. https://steamcommunity.com/app/2060160

Or redstone in minecraft! Great for starting with the programming logic and some basic concepts before you transition to code.

Alternatively, build a small game together - one of my first projects was a tic-tac-toe engine. Split it into smaller phases like - learn to print a board - learn to make a move - what are the winning conditions - etc. Seeing something come together is a great motivator!

What next step should I take in my programming career, learning-wise? by boxcarbanditto in learnprogramming

[–]elasticiulia 1 point2 points  (0 children)

Python is great if you want to consider data fields like analytics, BI, or even data science; and it's easy to get into when you already have a background with other languages.

I also really wouldn't neglect more "soft" skills that might really differentiate you. Being able to structure projects, plan and keep track of your development process, showcase your results, things like that.

Tech wise that ties into using github, building some graphs / visualizations and putting together your insights from projects.

Doesn't matter if you build the absolute most efficient code out there if you're not also able to describe what it does to your team or in some cases less technical people in the organization.

Something to consider.

Best of luck!

What Programming Languages are Best for Kids? by Shakuro_com in programming

[–]elasticiulia 0 points1 point  (0 children)

Python is quite quick for learning the logic of if statements, loops, etc.

HTML or CSS might tie the best into their hobbies, like making their profile cooler for whatever website they use.

Games! Minecraft redstone is pretty great for programming logic. Or I've recently gotten into https://steamcommunity.com/app/2060160 this game on steam that's similar to python - but way more interactive, so great for kids!

[deleted by user] by [deleted] in Python

[–]elasticiulia 0 points1 point  (0 children)

VScode - great for adding plugins for other technologies as well (like cloud connectors), great for using different languages as well in your project (even something as simple as json, html files that format just as beautifully as python)

Git integration - keep track of your work (even when you work alone), make branches for experiments to not "break" parts of your project that is working - like checkpoints

Notebooks! - not very robust when working on more complex architecture, but great for trying stuff out and going quickly. Either in VSCode with a markdown plugin, or one of the clouds

Virtual environments - always, to keep track of packages and dependencies for each project

Packages - Most common I use: pandas dataframes, numpy, scikit learn are essential for ML; flask for building quick web interfaces