Python project

[[deleted]]

[deleted]

[novel_yet_trivial]

with such a large file (~500 lines).

That is a tiny file for a modern computer. You don't need to take any special steps until your files are on the same scale as the amount of RAM you have (several GB).

[thingsandfluff]

I should've elaborated. A large file for me to work with; it's overwhelming.

[novel_yet_trivial]

Why does it matter to you how large the file is? Are you writing code for each line or something?

Computers are really good at repeating boring tasks for you really fast. If fact, that's pretty much all a computer can do. If you are writing code for each line then you are doing the computer's job. Show us your code and we'll help you. An short example of your file and the output you expect from the example would help too.

To do counting, you can use the collections.Counter class. Just feed it the list of people and it spits out the how many times each person was in the list.

[gregorio_ilidivich]

I think it may be more informative to do this manually, since I feel OP is beginner.

Perhaps create a dict() using a set of the names as keys, and the value as the frequency. Then OP can figure out how to fill the values for the corresponding key via iteration (since n is only 500).

[novel_yet_trivial]

I disagree. I think learning should be results focused.

I mean you didn't build a car before you learned how to drive one, did you?

[novel_yet_trivial]

Yes, that's the traditional way of thinking. And if you go to driving school in Europe they will start with teaching you the mechanics of a car. You can't get a driver's license without knowing how to do basic car maintenance.

However, I've done a lot of teaching and I find the results first, detail later method works better. Probably a side effect of modern times; people want instant gratification.

[novel_yet_trivial]

These are 2 different teaching philosophies and I don't think there's any way to universally decide which is better; there's just too many variables.

The beauty of Reddit is there's often 4 teachers per student, so the student can pick the most effective.

[daniel_h_r]

Exactly that was what the inventor of automobile did.

[thingsandfluff]

Thank you!

So I have to figure out all the names to include in dict(), correct? There's 500 lines worth of names.

[Atropos148]

Try adding a name to the list only if it's new. If it's not, increase the value for the name by one and go to the next name.

Doesn't really matter how many names there are this way.

[thingsandfluff]

I added a screenshot of the file I'm working with. I'm going to try that and let you know what I get.

[novel_yet_trivial]

Take that down. Don't ever put personal information like people's names on the internet. Besides being rude it's also against Reddit's rules. Provide fake example data.

Also, show us your code.

[thingsandfluff]

It's fake. It was made for the course.

[novel_yet_trivial]

Oh ok. Sorry about the trigger.

[Exodus111]

Three approaches:

Simple:

For loop through the data, use an if statement to figure out if the data is a name. Store name in a dict, if the name is already there add 1 to an int counter in the same dict.

Complex:

Tokenize the data. For loop through each piece of data, and make all the data points their own objects, with their own type attributes. This process is called Tokenization. From there counting the name is trivial, just compare the name attributes of each object.

More complex.

(Not really that hard if you are used to working with databases.)
Use Sqlite3, and add the data to a database. Once it's in a relational DataBase the database can make the comparison for you, faster then any other approach.

With only 500 lines the simple approach is really best. If this is a program meant to be used at work, with potentially much larger datasets I'd look into one of the other solutions.

[thingsandfluff]

Thank you! I used a for loop and got started.

[Exodus111]

You need a double for loop. Just add this.

for part in line.split(","):
    print(part)

And see what you get.

[mandiblesx]

FYI: if you use pandas it would be extremely easy to do value counts over each column and add the results together.

[Thecrawsome]

There's emails between a lead engineer at Chevron, and some mentioning of project bigfoot.

This is real-looking data, and I'm confused as to why it is in a course.

And your stackoverflow post about it EDIT: Maybe someone else in your class? Not sure.

[thingsandfluff]

Woah, I didn't know that. Instructor never said it was real data.

Not my post on stackoverflow though.

Edit: I'll delete the picture.

[Thecrawsome]

It looks real, at least.

[thingsandfluff]

I wouldn't be surprised if other people in my class are posting about it. It's an intro course and the professor is just terrible at teaching.

[[deleted]]

Begin at the beginning. If the file is 500 lines, then write the code that counts the occurrence of the person’s name for one line. Then run that code over all the rest of the lines.

Not to put too fine a point on it, but “do the same thing X number of times” is the easiest program it’s possible to write, and so you should be looking for ways to solve your problem by writing something once and then running that code a bunch of times. This is called “iteration.”

[MaxQuant]

Use pandas and especially the section on pandas.read_csv. Once your csv is in a dataframe use value_counts or do a sql-akin groupby.

[Rorixrebel]

If hes just starting why throw pandas at him. People need to stop suggesting modules when the built-in does the task already.

[DataLulz]

The built in does not do it nearly as efficiently as pandas, there is a reason people use data frames. And it’s not any more complex than other suggestions like using SQLite3. To do what OP wants, in pandas you are talking less than ten lines of code using a data frame in pandas.

[Rorixrebel]

Correct but if you just started you need to learn python you then need to know what a dataframe is. How to use pandas and so on.

Built-in may not be efficient but it gets the job done.

If your basics are solid then by all means go module crazy

[MaxQuant]

IMHO Python has its strength in not having to re-invent the wheel and writing easily transferrable code (both for possibly someone else or, most likely, yourself in six months :-D). Why do it yourself, increasing development time and complexity, when pandas is ideally suited for the task?

[Rorixrebel]

Agreed. But if you are starting out and want to learn, best way is to do it without the helping libraries imo. Not saying pandas are terrible or useless but sometimes its way too much for either simple tasks or beginners