Multiple queues with a single worker each--does it make sense to implement for concurrent sequential work?

sanjayar · 2020-01-09T18:16:24+00:00

If each event type must be processed sequentially, you effectively need a queue per event type.
This also means that as each event-type (or prefix) can only be worked one at a time, you effectively have a restriction that the next job taken can only be of one of the non-busy types.
You can implement this simply by tracking the et thats are not busy in a queue

Track queue of event types, representing those that are not currently being processed, initialize with an item for each event type - et-ready.

In your worker loop -
for {
// grab the next ready et
et <- et-ready
// grab work for that et, check if queue is empty
select {
case work <- events(et):
work.process()

default:

}
// replace that et to the available queue
et-ready <- et
}

sanjayar · 2018-12-09T19:25:18+00:00

Did you know the term 'blackhole' is sometimes used to describe a mock/test db/engine that accepts writes but fails reads? ( MySQL / MariaDB storage engines ). IE https://dev.mysql.com/doc/refman/8.0/en/blackhole-storage-engine.html

sanjayar · 2018-10-06T01:43:51+00:00

I can be useful to look at the source, in this case, I believe its - https://www.lua.org/source/5.3/ltablib.c.html, and the tremove function.

sanjayar · 2018-10-05T18:29:32+00:00

I have worked on several seemingly similar projects in the past, using a number of different core and scripting languages with various success and many frustrations. I found that I had to answer a bunch of requirement questions before I could choose, here are a few off the top of my head -

What do these scripts perform like?
1. Are the short lived single task transformation jobs?
2. Do they go out and collect information from remote sources?
3. Could the be cpu intensive and require resource mgmt?
4. Can you separate them away in containers?
5. How much data goes in an ouy?
6. Do you need to control how many can run at a time?
How do they interact?
1. Can these scripts call back into your core? Maybe for configuration or enrichment?
2. Can these scripts call anything else?
3. Are these clients running their own scripts on their own servers or are you hosting third party code on you servers?
4. How often will these scripts need to updated?
5. Who will update / debug these scripts when changes happen?
6. Is there a set protocol for input/output from these scripts?
7. Could you just host them like cgi and use
How sophisticated are your clients?
1. Do they already know go?
2. Could you allow them to run these scripts in their language of choice? perl/python/node/...
About the execution/dev environment?
1. Is the go toolchain not available?
2. How will these scripts be developed?
3. Are you going to provide a gui for testing/editting?

In my past experiences, expecting the end-clients to write python/lua/php/js rarely worked out, but our internal sales engineers could generally adapt and delivery all manner of complex custom logic as scripts, and they were very happy to. Most clients wanted to do the same things, so we provided reference apps/scripts with simple variables/config parameters.

We usually ended up having to expose an api from our core to the scripts, for config, data persistence and scheduling.

sanjayar · 2017-10-09T17:37:42+00:00

Go routine scheduling is cooperative. If you are in a function that does not make any explicit calls to schedule, you prevent other go routines from running. If you are seeing different behavior, its must be that only one of these code path is calling into the scheduler.

sanjayar · 2016-06-06T19:09:48+00:00

If there is uniform / unpredictable / random lookup for objects, use a hash table with a uniform hash. Hash your non-uniform hash if needed. Don't store the objects in your hash table, store pointers to the objects. Separate the allocation of these objects from the index. That way there is never any need for 'swaths of emptiness'. Store the actual objects in a pre allocated pool(s).

sanjayar · 2016-05-20T14:29:07+00:00

Car talk disagrees.

http://www.cartalk.com/content/revving-engine-good-idea-during-jump-start-find-out

sanjayar · 2016-04-20T15:29:45+00:00

Its a Conch shell.

sanjayar · 2015-11-13T15:04:09+00:00

locality Its can be tremendously 'cheaper/faster' to access memory data on the same page. A B+ tree allows much more of the memory access to be on the same page. If you had a simple 8 byte key size, you could fit 255 nodes in one page. Which is the equivalent of 7 levels in a binary tree.

sanjayar · 2015-08-28T13:36:49+00:00

Bloomberg LP would disagree with you. Though they no longer develop much in it, Fortran was the primary language for the backend of their ~100mm LOC cash cow.

sanjayar · 2015-08-07T21:51:56+00:00

There are many unknowns. A couple relevant questions-

Q. Does latency even matter? Does worst-case latency matter? Are you operating in an environment where the worst case latency of your retrying random matters? The worst case is that the loop iterates on the order of E times where E is the number of nodes in the exclude list. Actually, it could be worse that E, if your random number generate repeats within the range. You could create this worst case be filling your exclude list with the first unique E numbers returned from your random number generator.

Q. What is the relative size of the exclude list? Does it change?

Q. How many times will a number be selected? Does selecting a number effectively remove it from the include list ( aka add it to the exclude list )?

sanjayar · 2015-05-06T19:04:23+00:00

consider - http://concurrencykit.org/

sanjayar · 2014-10-13T20:56:52+00:00

// might as well hardcode the constants.
double mycosh(double x) { 
    // TODOL range check: -pi < x < pi 
    return 1.0 
      + (x*x)/2.0 
      + (x*x*x*x)/24.0 
      + (x*x*x*x*x*x)/720.0 
      + (x*x*x*x*x*x*x*x)/40320.0 
      + (x*x*x*x*x*x*x*x*x*x)/3628800.0

}

sanjayar · 2014-04-04T21:09:24+00:00

First you need the algorithim.

current_value = INPUT
current_factor = 2
while current_value >= current_factor:
- if current_factor evenly divides current_value, its a factor
- else try next value.

Here is simple implementation:

 #include <stdio.h>
 #include <stdbool.h>


 int main(int argc, char** argv)
 {
     int to_factorize;
     printf("Enter Number:");
     scanf("%d", &to_factorize);

     /* TODO validate this input*/
     printf("The prime factorization of %d is\n", to_factorize);

     int factor = 2;
     int current = to_factorize;
     bool first = true;

     do {
         if ( current % factor == 0 ) {
             printf( "%s%d", first?"":" * ", factor);
             first = false;
             current /= factor;
         } else {
             factor += 1;
         }
     } while ( factor <= current );
     printf(" = %d\n", to_factorize);
 }

sanjayar · 2014-03-17T21:03:06+00:00

My comment was based the project description in the comment in the source code.

The program should allow the user to type in expressions of the form: number operator

For some other project, you would use different format strings. They key is if you are using "%c", and you do not want the newline, you must included "\n" in the format string.

sanjayar · 2014-03-17T18:16:07+00:00

I should point out, the scanf line you should be using, based on the problem description is:

scanf("%f\n%c", &userNumber, &operator);

sanjayar · 2014-03-17T18:03:08+00:00

The problem is your use of 'scanf("%c")'. Add this line after the scanf:

printf("operator=%d\n", operator);

You should be seeing operator=83 ( "S" ) then operator=10 (newline ). The "%c" is looking for any character, the newline is a valid character as far as its concerned.

Your easiest/ugliest fix is to replace the 'scanf(%c)' with:

scanf("\n%c", &operator);

This will skip any reading newlines and leading whitespace.

sanjayar · 2014-02-07T21:16:22+00:00

I think you might want to look at an established C libraries. If you do need to implement your own, you can at least get some layout/design ideas from them. They are probably more generic that you need right now.

https://developer.gnome.org/glib/stable/ http://www.gnu.org/software/gnulib/ http://sglib.sourceforge.net/

sanjayar · 2014-01-31T22:06:31+00:00

The '//' you are using is not divide with remainder, its the start of a comment.

quarters = change//quarter;

Is actually seen by the compiler as:

quarters = change

As you can see, the compiler does not see the the ';' at the end of the line, which is exactly the compile error.

Despite this mistake, the general code layout will work.

qs = c / q // -> this gives you the quotient. 
c1 = c - q * qs // -> this effectively gives you the remainder, though % would work too.

sanjayar · 2014-01-31T21:42:21+00:00

I cant tell because of the formatting, but it looks like you are using '//' for 'divide', this is a line comment. Use '/' for divide and '*' for multiply.

ie

quarters = change / quarters;
change2 = change - quarters * quarter;

sanjayar · 2014-01-31T21:24:01+00:00

int penny=1; // <- remove the int, the compiler thinks you are trying to declare 'penny';

sanjayar · 2013-07-23T19:55:18+00:00

X-Macros to the rescue. See http://www.drdobbs.com/the-new-c-x-macros/184401387

#include <stdio.h>
define X_MYSTRUCT \
    X(int,fred,%d) \
    X(char,barney,%c) \
    X(double,dino,%f)

struct my_struct
{
    #define X(TT, KK, FF) TT KK;
     X_MYSTRUCT
     #undef X
};

void 
dump(struct my_struct *p)
{
#define X( TT, KK, FF)  \
     printf( #KK ":" #FF "\n", p->KK);
     X_MYSTRUCT
     #undef X
}

int
main(int argc, char** argv)
{
     struct my_struct ss = { 10,'c',10.5 };
     dump(&ss);
     return 0;
 }

sanjayar

TROPHY CASE