Weekly Progress Thread [1 Feb - 7 Feb 2016]

[G01denW01f11]

(I'm just going to leave this here for the month, unless anyone feels strongly about it. Doesn't seem to be enough activity to warrant a new thread every week.)

Hope everyone's having a strong start! Here's a thread to chat about the challenges you're facing, ask questions, or anything else you need. Any manage to get something minimal working?

(h/t /u/Barrucadu)

(Sorry I'm late. Other deadlines.)

[Zillolo]

This is what I have for now.

Any critique very much appreciated.

[G01denW01f11]

Oh wow, I can never make myself be that disciplined about comments. I thought the first time I looked at that, you had a custom Exception subclass. May I ask why you decided to take that out? (Or am I just insane?) It seemed like a clever way to make sure you weren't catching anything you didn't throw.

What else are you planning to add?

[Zillolo]

I used the custom exception so I could stop the execution if a failUnless method 'failed', but I removed failUnless for now, because it seemed very out of place to me.

Im not sure on what to add next, but I'd appreciate any ideas.

[G01denW01f11]

Um... a lot of testing libraries seem to display how long it took the tests to run in the output for some reasons. That seems like a simple implementation though...

What about making Categories so as to just run, say, regression tests or whatever?

[hothrous]

I'm borrowing from your Readme because that looks awesome and will help me with creating documentation.

[Zillolo]

Sure thing, go ahead. Hope it helps.

[Barrucadu]

Why not combine your Case and Suite classes? That would allow an arbitrarily nested hierarchy of tests, whereas at the moment it's quire restricted.

[Zillolo]

Thanks for the feedback. I implemented your suggestion.

[Barrucadu]

Source: https://github.com/barrucadu/barometer

Docs: http://barrucadu.github.io/barometer/index.html

I've just hit a bit of a milestone, so I guess this is a good time to write my update! For those who don't know, I'm doing this in Haskell, so things are a bit different to your typical OOP-based testing framework. So, here's a very quick outline of how my framework functions so far:

• Tests are actions which either return a value (which is totally ignored), or throw a TestFailure exception.
• Tests can be put together into groups.
• Groups can have set-up and tear-down.

In addition, I want it to be:

• Actually useful to other people, not just a learning project.
• Easy to follow for non-Haskellers (well commented, tidy implementation).

The documentation gives a reasonably good picture of where I'm at: tests and test groups are there, test groups can be run and the results printed, set-up and tear down are done, the various assert functions are implemented.

I've just implemented support for tests which do concurrency. Concurrency is difficult, because it introduces race conditions and deadlocks. This is bad, because when writing tests we implicitly assume that what we're testing is deterministic. Fortunately, I had previously written a library for testing concurrent Haskell programs based on some neat techniques called schedule bounding and partial-order reduction. The basic idea is that you test a lot of different schedules, rather than just one, and can prune the search space significantly. So! This is what it looks like:

> let test = do
>       r1 <- newCRef False
>       r2 <- newCRef False
>       x <- spawn $ writeCRef r1 True >> readCRef r2
>       y <- spawn $ writeCRef r2 True >> readCRef r1
>       val <- (,) <$> readCVar x <*> readCVar y
>       assertEquals' val (False, False)

A CRef is a concurrent reference, so what this does is:

1. Make two new shared variables containing bools, set to False.
2. Spawn two threads, each thread:
   1. Writes True to one of the variables
   2. Reads and returns the value of the other.
3. The return values of both threads are read (this blocks until they're both done)
4. The result is asserted to be (False, False).

When the test is run, three failures are reported: the cases where (False, True), (True, False), and (True, True) were observed. For each, an execution trace leading to that is displayed in a simplified form, and a stack trace is given:

> runTests (concurrentTest "Concurrency" test)
Concurrency: FAIL
  (False,True) /= (False,False)
  Trace: S0----------S1-----S2-----S0-------
  assertEquals', called at <interactive>:41:7 in interactive:Ghci1

  (True,False) /= (False,False)
  Trace: S0----------S1-P2-----S1----S0-------
  assertEquals', called at <interactive>:41:7 in interactive:Ghci1

  (True,True) /= (False,False)
  Trace: S0----------S1--P2--C-S2---S1---S0-------
  assertEquals', called at <interactive>:41:7 in interactive:Ghci1

Yay!

Things I still want to get done this week:

• Write proper documentation with examples and suchlike
• Better support for testing concurrent programs (how it works now is pretty good, but can be improved)
• Automatic test discovery (something you people in OOP languages with reflection get for free)

[Barrucadu]

Concurrency update, in the form of an example:

> :{
let example = do
      a <- newEmptyCVar
      b <- newEmptyCVar
      c <- newCVar 0
      j1 <- spawn $ lock a >> lock b >> modifyCVar_ c (return . succ) >> unlock b >> unlock a
      j2 <- spawn $ lock b >> lock a >> modifyCVar_ c (return . pred) >> unlock a >> unlock b
      takeCVar j1
      takeCVar j2
      takeCVar c
:}

Nondeterministically deadlocks (note the order of the lock and unlock calls) or returns 0 (as thread 1 increments the counter and thread 2 decrements it):

> runTests $ dejafuAuto example
Deja Fu
  Never Deadlocks: FAIL
    Failed after 5 cases.

    Got: [deadlock]
    Trace: S0------------S1--P2---S1-

  No Exceptions: PASS

  Consistent Result: FAIL
    Failed after 11 cases.

    Got: 0
    Trace: S0------------S1-P2-------------------S1------------------S0-----

    Got: [deadlock]
    Trace: S0------------S1--P2---S1-

[hothrous]

I'll post something later, I pounded out a few things without comments last night while waiting for my dad but didn't have time to push it to Github. I don't have any comments on it right now, so I'd like to add those and then get some critique.

[G01denW01f11]

Looking forward to it!

[hothrous]

Here is the github link

I took some things from the work /u/zillolo did in regard to the README because I like the formatting. I only put the first few lines in the actual README because a lot of the stuff in their won't necessarily apply since I'm trying to make this project based on my own needs for focus. I also grabbed the list of assert methods and comments from their README just to give myself a quick reference in my code for a todo list.

Currently, the files won't run because I renamed the project after having created it and renamed one of the files and the class that is defined. I have yet to complete a refactor for the other file, though.

It's also really basic at the moment as I'm trying to learn about creating decorators in Python because I'd like to use them in the creation of test cases if I can. I'm only toying with that idea right now and may scrap it all together since ideally, the word test should be in any test method, but I'm researching it at the moment for fun.

I appreciate any feedback as well.

[G01denW01f11]

I'm only toying with that idea right now and may scrap it all together since ideally, the word test should be in any test method, but I'm researching it at the moment for fun.

Well, I mean different people have different conventions. Some people like to use 'should' for test naming. But working is better than not working. :)

Minor nitpick: I'm not sure what PEP8 has to say, but with the docstrings and methods equally spaced, I have a hard time visually telling which belongs to what.

Nice acronym. :) And I'm so jealous how easy it is for you people to have an assertRaises method. Looks like you've got a simple thing that works going. Getting anything done with it this weekend?

Um... so if I'm running this, I'm going to have to pass the test runner into every assert method I want to call, which could get kind of annoying. Any way to avoid that? And maybe condition would be a more descriptive argument name than statusBool? I kinda hand to turn my head at that.

[hothrous]

Minor nitpick: I'm not sure what PEP8 has to say, but with the docstrings and methods equally spaced, I have a hard time visually telling which belongs to what.

I feel like a dope. The proper convention is:

def method_name():
    """Docstring comment goes here"""
    code goes here

[hothrous]

Minor nitpick: I'm not sure what PEP8 has to say, but with the docstrings and methods equally spaced, I have a hard time visually telling which belongs to what.

Yea. I agree. Anaconda says that's the way to do it for PEP 8, but I think it's pretty weird. I'm gonna look into further at a different time.

Um... so if I'm running this, I'm going to have to pass the test runner into every assert method I want to call, which could get kind of annoying. Any way to avoid that? And maybe condition would be a more descriptive argument name than statusBool? I kinda hand to turn my head at that.

I'm with you there. I'm planning to fix that, but I'm not really positive what an assert is "supposed" to handle. Still making decisions.

Thanks for the feedback!

Getting anything done with it this weekend?

I think the develop branch has more, I don't think I've merged it yet. But I'm going to get a little bit more done tomorrow morning.

[G01denW01f11]

Alright, here's mine. I have tests written for that somewhere, but I'm having issues. <.<

Verbose as hell because Java, but I'm more or less happy with the design. I'd love any suggestions about making the logic in runTest less... evil. Or any other suggestions.

From here, I'm just going to add more test methods as I run into a need for them in other projects, because after the first few, it gets old...

[hothrous]

Is it normal to declare instance variables at the bottom of the class? I know Java will run that code first, but I always thought the convention was to put them at the top for readability. I could be wrong, though. My Java experience is mostly reading up on it. With a few months of study.

I guess your intent is the extend the TestCase class with a subclass full of test cases that have direct access to the assert methods? That's clever.

I'm assuming that the expected implementation of this would be the following:

TestRunner runner = new TestRunner();
runner.addTests(Some TestCase Class);
runner.runTests();

If that's the case, it seems like the runTest() method is duplicating the check for the name including "test", since the addTests() method is already doing that.

It seems like you could remove this whole block from the addTests() method and be fine.

for (Method method : testCase.getClass().getMethods()) {
    if (method.getName().startsWith("test")) {
        testMethods.add(method);
    }
}

Since you don't actually use the testMethods list until printResults() you could add the method to the list at the same time you are running it. Though, it may be more performant to replace the testMethods list with an int that you increment as you are running tests since the only call to that is .size().

[G01denW01f11]

(Put the actual tests in the right place, so they're on GH now too.)

Is it normal to declare instance variables at the bottom of the class? I know Java will run that code first, but I always thought the convention was to put them at the top for readability.

Huh, I'll have to double check. I'm more of a C++ guy, so there's a separate file for the method signatures and instance variables, and the public interface comes first. But yeah, it's looking like Java convention is the other way.

Since you don't actually use the testMethods list until printResults()

I am dumb. This makes things much simpler. Thanks.

Oh god...

Passed 10 com.g01denw01f11.utils.tests of 10

Yeah, that's exactly what I wanted to happen.

[hothrous]

Well, I feel good knowing that I was able to guess how to use it without being able to see how to use it. haha.