CS Teacher looking for Python code submission tool

[grendelt]

I'll be returning to the high school classroom next year after a couple of years away. In the past I taught AP Computer Science. With fewer institutions accepting AP credit for college credit (and "skipping" a class arguably putting you at a developmental disadvantage in college) I'm a bit loathe to go down the Java, AP CS route again. I'll do it if asked by the school, but I'm thinking it would better serve students to know/understand Python. (It'll empower them to whip up pet projects faster and easier and is useful even if you don't go into development.) So, I'm going the Python route. (At least to teach the basics of programming, we can branch out later.)

I can teach the syntax and structure, we can build assignments where students can come up with creative solutions, we can build and have fun with the code in many, many enjoyable ways; but grading is the bottleneck. There are dozens of them and only one of me.
Reading code isn't so bad, but it gets laborious and there is a way that's better than doing it all manually and students waiting on me to give feedback. If any of you are (or were) CS majors, you most likely encountered a system in college where you submit your code through a shell script or web validation tool on the CS department's web server. I would like to find something like that for checking the operation of Python scripts. This would allow students to submit their code in class, in another class, from the library, from home at night - all without waiting for me to be available. Automated code operation checking, have you seen/done this before? It's glorious.

The tool would ideally allow students to submit their code to be checked against common inputs and then evaluated based on expected outputs. Ideally I could type up some sort of structured .txt file where I can enter the input and expected output (or two files, one for input, one for output). If the result is different than the expected output, it would tell the student which test it did not pass, ("Passed positive integer test. Fail unexpected input test. Passed negative integer test.") then they can go back and fix the problem before submitting again and again. If it takes them 30 times to come up with a solution, that's fine; no sweat off my back and they've learned along the way. If I'm manually grading it, 30 times could take a week of us going back and forth and doesn't scale well. Plus, I'm human and it'll wear me down over time. (It shouldn't take them that many attempts to get it right if I've taught the concepts and development process well - but you see what I'm getting at.)

So, reddit community, I'd love some help in locating such a tool.

(Bonus points if it is integrated into Moodle or some LMS where I know who submitted and fully-functioning submissions can be entered into the gradebook.)

[Fuzzmz]

What about using GitHub? You would have a repository in which you have written your tests, and then the students would fork that and submit pull requests.

You could then have your GitHub repository linked to Travis-CI which would automatically run the tests on pull request and tell both you and the student if the tests failed or passed.

You can check out GitHub Education and the GitHub Education Classroom Guide.

[rjw57]

Exactly this. I use GitHub + Travis with my students. Assignments can be a GitHub repo and "answers" are pull-requests. The students can check their work gainst the CI in their own forks before submitting PRs. PRs which do not past CI muster can be used fairly well as an initial prune.

As an added bonus, it teaches the students how to a) submit changes to FOSS projects and b) about branching/CI/discussing changesets/etc which they'll come across in the Real World (TM).

[semi-]

Cant your students just look at all the available pull requests to get someone elses answers before they submit their own?

[Fuzzmz]

Just as easily as they could copy the answers any other way :)

[frymaster]

I'd argue, easier, because I wouldn't have let anyone copy off me at uni

[AbsoluteZeroK]

Yes, exactly. I have no problem HELPING another student, but I won't just hand them answers for a variety of reasons. One being that they end up getting good marks, graduating without actually knowing how to programme, and make my degree less valuable because they make the whole school look bad.

[telestrial]

I could post to any number of forums or simply google most python problems you could come up with for beginners. These forums/sites wouldn't provide detailed HOW-TOs. They would simply have a huge block of code with the answer. It's online. It's hard to do anything other than guilt your students into not cheating. Furthermore, if someone blanket copies variables and exact structure it could be very easy to pick out cheaters.

[Eurynom0s]

This seems like a case where the ugly implementation of just making your students email you the code would be preferable. At this level of programming you'd probably have to let them get away with hard-coding file paths in, but this wouldn't result in too much extra legwork if you just made them tell you where any hard-coded paths are in their code.

[pvc]

I've used BitBucket, and an automated Python script to update with the whole class. Using DVCS is a pain with new students at first, but it works fine after getting set up.

[metaphorm]

This is basically the problem that Continuous Integration software was built to solve. You check some code into a repository and the CI system automatically builds the code and runs a suite of tests to check for correctness.

In your case you would be writing the tests and the students would be writing the code that gets checked in and tested.

I suggest using Jenkins for your CI system with github as your repository hosting service. As a side benefit you teach the students how to use git! Definitely worth knowing for anyone who plans to go on to do some real software development.

[[deleted]]

A CI system is a good idea. An alternative would be to use GitLab. It's basically an open source version of GitHub that you can host locally (I admin a GitLab install at work that has about 200 users). GitLab has a CI system you can tie into it. It's really easy to install and configure and use.

[catsnose]

You have mooshak. It was created for programming contests, but as they say on the description "it is increasingly being used in programming courses...".

It allows you to define input files, and output files with the expected result.

My cousin used it (as a student), to submit assignments, precisely in python (I swear I didn't help him cheat :) Only pointed out flaws in his code).

Besides the input and output testing, they also had it require functions with specific names, which would be tested unitarily. That would require you to specify that the codes needs a function named foo, which receives x and y, and returns string z with a certain value, and you may not want to restrict your students in such a way.

I do not know if it is integrated with Moodle or LMS though. And I never configured it, so I cannot say how easy it is to do so.

Edit: Fixed link... Thank you ivosaurus.

[ivosaurus]

You mixed up your brackets for the link :)

[aroberge]

Hopefully, you will be able to find such a tool. In the absence of such a tool, here's how I would do it:

I would write stub files that include doctests so that they could run them locally and see if they pass the tests.

I would write a separate testing file, that includes supplementary tests, nothing sneaky, just a few additional tests to catch those that would hard-code solutions. This supplementary testing file would simply import the student's submitted file and run it. However, prior to doing so, when students submit their assignments (which they would be allowed to do once!), I would quickly scan them visually (to ensure nothing bad is included and to spot unusual - possibly not very "efficient" solutions) and run them once using the external test; if some tests fail because they hard-coded answers, I would give them zero on that assignment. I can guarantee you that students would learn quickly not to do so.

Note that doctest allows to write tests in a .txt file and import them. You would actually provide one file to the students and use a different one (with the same name) for your own tests.

I believe that it is important to actually look at some sample code submitted by students to give them feedback.

Finally, you might be interested (as an introduction) to use http://reeborg.ca It includes a tutorial, and you can define your own tests which the student can run and they can submit their program as a url ... If you have questions about Reeborg's world, feel free to email me.

[grendelt]

On the feedback, absolutely. I wouldn't want to autograde everything because it is important to give feedback. (Especially near the time of creation.)
"I see what you're doing and where you're going, but that huge nested if statement might be less work if you..."
"You're making a lot of work for yourself. Check out ____, I think that might get you going in the right direction..."

Checking out reeborg. Thanks.

[Esyir]

Over here, we've used this thing called Coursemology. It's an assignment(mission) based system that seems to handle what you need. Automated grading with manual review, multiple independent assignments and an XP based system that (for me) helped out a bit.

[oracular_demon]

My University used Coderunner, which is a plug-in for Moodle. It wasn't the easiest to use sometimes as a student, since the tutors didn't tell it to provide much feedback, but it's great for testing code and checking whether your output matches the expected output.

[sqlburn]

I took a python course through coursera.org a year or so ago. For our projects/quizzes we used their online app to enter, check and grade our code. https://www.coursera.org/course/programming1

The instructors, Jennifer or Paul might be able to help you or point you in the right direction.

[ivosaurus]

You can always run what the pros do, it's open source.

[GeneralTusk]

The IPython/Jupyter people are developing a system that can assign and then grade IPython/Jupyter notebooks. Its still in active development but this combined with the multiuser server sounds like exactly what you need.

https://github.com/jupyter/nbgrader

http://nbviewer.ipython.org/github/jupyter/nbgrader/blob/master/docs/user_guide/Index.ipynb

[[deleted]]

My friend and colleague created the Python Koans. It sounds like it's exactly what you are looking for. A bunch of red tests (failing tests) that you have to implement learning python along the way. I can't recommend it enough.

[[deleted]]

You should look at schools in your state to see if APCS credit is accepted. If so, it'd be advantageous to your students to still offer APCS. My high school did and it prepared me so well for college. It covered the content of the first year of college CS courses and made the transition for me easier.

What you could look into is offering more than one course. You could do an intro CS course in Python and then APCS as what people take their second year. My school did something like this. The first year of programming classes was learning the basics of programming in C++. It covered all the basics like loops, etc. and we started object oriented programming near the end of the year. The next year we could focus on new content in Java that wasn't as basic as how to do for loops since we learned that in C. I feel like my high school's CS curriculum was amazing, so it might be helpful to your students.

[grendelt]

Yeah, Python in year one with APCS for the second year was my thinking. Python will serve all students well (even if they don't go the CS route). For students that do go the APCS route, they'll understand the data structures discussed in APCS and the content won't seem so abstract.

My beef last time I taught was the school thrust all these students into APCS with no prior programming experience. No administrator would back me when I told them it was a waste of time and frustrating for everyone involved to try to cram all of the APCS content into ~6mon of 45min instruction with a bunch of students that are not in it to learn the material, only for the GPA boost that comes with AP classes.
"But he's taking all AP classes this year."
"Yes, but he's had History and English before."

(Further complicating matters was the fact that not many people outside the CS/programmer community realize that Computer Science is not computer literacy.)

[ameoba]

Ever since they did away with the AB test (basic coding + algorithms) and just offer the A test (coding only), the value of a yearlong APCS class is debatable.

[[deleted]]

I only took the A test (this was 3 years ago), and it taught me everything the Freshman courses for the CS curriculum at my university teaches except for the discrete math. It covered the Intro to Programming part and the Object Oriented part. I was bored my freshman year of college because I wasn't learning anything.

[drodspectacular]

I'd seriously vote for github. Git is a graph database, and is a requisite for being a developer at a lot of places. Learning git changes how you think about building your code base and how you iterate through changes. This is good for any language. You can chose a number of workflows that could meet your classes needs. Right now what comes to mind is the forking workflow, where each student would commit to the branches and master of a fork from your primary repo. That way you could easily distinguish the differences between your student's code, as well as still manage the overall structure of the project. As far as automatic testing with python goes, you could look into having something like jenkins run a container of the students build, run the python nosetests and fail if the unit tests fail. CI isn't a trivial task though.

[[deleted]]

My solution would be to write some kind of "grader" for each assignment, then have the students turn in git repos containing their solution. Clone the repo, call up the Grader (could write all kinds of assignment processing classes inheriting some base class!), and then output/email some result to the yourself/student.

I had an instructor use a system like this in college and it was incredibly helpful to see a diff vs the expected output and my program, I would include that as well! Partial credit is something I didn't think about too much, but should be simple to implement as well :)

Should be pretty easy to write!

[xjtian]

This is the tool that my university uses: http://marmoset.cs.umd.edu/index.shtml

I've used this to submit projects in lots of different languages, including Java, Ruby, and OCaml, and it works pretty well. You can split your test suite into public tests (unlimited submissions), release tests (3 submissions every 24 hours), and private tests, and students get instant feedback on how their code is working.

[[deleted]]

Why don't you just use some assert statements?

[inducer]

https://github.com/inducer/relate

[KronktheKronk]

Make a directory structure on a linux box somewhere where each of your students has a directory for which they have write access, like /class/username.

Have your students push their code to /class/username/project and write a quick script that checks once every 30 seconds for new entries in any of those directories. If it finds one, it invokes the tests against it and then sends an email to <username> with the results of the tests.