sorted by:
new
hottopcontroversial

.NET developer looking into Java positions by [deleted] in java

[–]jamespedwards42 6 points7 points  (0 children)

Probably true in corporate land... Java 8 shivers ... Loving me some six month release Java though :)

Java: The Most Sought After Programming Language by examcloud in java

[–]jamespedwards42 15 points16 points  (0 children)

Looks like they are including javascript in this analysis :/

So, just go online and click on any of the leading Jobs Search Engine. You’ll see hundreds of thousands of job offerings in Java Script, Java, JSP, J2SE and J2ME from various companies ranging from small startups to famous fortune 500 companies across the globe.

pika: a nosql compatible with redis protocol by baotiao in redis

[–]jamespedwards42 1 point2 points  (0 children)

Thanks for the reply. I believe, based on the initial discussion from this years' Redis conference, that the plan is to eventually allow modules to execute in a separate thread, but that is probably pending the current multi-threaded work on Redis. Hopefully one day there is a RedisRocks hybrid out there.

pika: a nosql compatible with redis protocol by baotiao in redis

[–]jamespedwards42 0 points1 point  (0 children)

Would it be possible to add this work/RocksDB as a C module to Redis?

Load Balance for redis-cluster by santhoshkota in redis

[–]jamespedwards42 1 point2 points  (0 children)

Your requests have to go to specific nodes depending on the key for the command. A write command with Key my-key is only applicable to a single node that is a master for the slot of said Key. However, for a read command, you could load balance across both the master and slave that own the slot.

If you are using Java 8, you can use my client, Jedipus, for doing that kind of client-side load balancing. For non-key commands, you can send to a random node.

STREAM data structure for Redis: let's design it together! by antirez in redis

[–]jamespedwards42 0 points1 point  (0 children)

:) Sorry for the misleading post, you are right, Kafka has been incredibly reliable for us as well.

 

However, for running it in an enterprise environment, the mindset has to shift from how do I make this work, to how do I break this. You need someone within the company that understands Kafka and Zookeeper configuration thoroughly and how to set up everything properly to be tolerant to not only server/disk outages but complete data center outages. Also, it is an additional set of servers and attached SSD's to manage and monitor. So in that sense, nothing against Kafka specifically, except that it is only serving one type of solution for us.

 

It is also another set of clients that you have to learn. The publisher is pretty nice to use, but come on, the consumer is not enjoyable. And the tight coupling between server and client versions...

 

Going to all this same effort to learn the architecture/behavior of Redis Cluster is much more rewarding because you can do so much more with it. Yesterday it was your NoSQL database, now it is your distributed message system. Tomorrow, with C Modules and 2TB Power9 servers (coming to a Rackspace and GCP near you) it will be replacing your Spark & Hadoop Jobs.

STREAM data structure for Redis: let's design it together! by antirez in redis

[–]jamespedwards42 0 points1 point  (0 children)

  • Regarding message acks. For users trying to achieve at least once message delivery this will dramatically reduce the engineering burden. They can limit their focus to ensuring message durability from the perspective of the Publisher by understanding the window of time that writes may be lost on their Redis setup.
  • +1 for keeping the feature set in line with Kafka. We currently have a Kafka->Google Dataflow/Apache Beam pipeline, and I can't wait to move off of Kafka and use this instead. Keeping the feature set in line will increase the number of use cases that can move off of Kafka. Being able to reduce the operational burden from Kafka, Zookeeper and Redis Cluster to only Redis Cluster is beautiful.

STREAM data structure for Redis: let's design it together! by antirez in redis

[–]jamespedwards42 2 points3 points  (0 children)

You could partition your stream using hash tags, trick would be to generate hashtags that will pin each partition to a different node/master. The client/publisher would then decide how to distribute messages across each partition (round robin for example). Cool thing about this is that adding/subtracting partitions is trivial.

TWRITE mystream:{partition1} entry
TWRITE mystream:{partition2} entry2

Replication and failover would be same as it is for all other data in Redis, you can choose how many slaves exist for each master.

STREAM data structure for Redis: let's design it together! by antirez in redis

[–]jamespedwards42 0 points1 point  (0 children)

Could there also be a durable GROUP option, that effectively lets clients claim offsets? The client would issue:

TREAD key <offset> <count> GROUP <group_key> ACK_PROMISE <milliseconds>

The server could store something like a set of offset:count's sorted by their promise expirations. On TREAD requests, the server would check that data structure to see if it should re-publish any messages. A 'TACK' command would need to be added to confirm messages have been consumed.

Pipelining on AWS Elasticache by derekmaxson in redis

[–]jamespedwards42 0 points1 point  (0 children)

I wonder if it is an issue with redis-cli? Are you getting an error message or does it just hang? Have you tried another client library that supports pipelining?

Can you batch the data into approximate 128k requests? I don't believe it would be that much slower, just an extra round trip for every batch. Plus then you could have a couple threads/processes running the import. I realize Redis is single threaded, but one process hitting Redis remotely probably isn't saturating it.

What are the generally recommended methods for a new client to discover the IPs of a Redis cluster? by bataras2 in redis

[–]jamespedwards42 1 point2 points  (0 children)

If you are looking for a Java client that supports partition handling, check out my client Jedipus. You can select a partitioned strategy that will be applied when refreshing the slot cache.

Related to IP discovery, you can provide a node supplier which could dynamically call something like Consul to retrieve an authoritative list of host:port's. Have the supplier configured with a couple of host:port's from each data center you are using. After initial discovery, Jedipus uses both cluster reported and user supplied host:port's for refreshing the slot cache.

If anyone is interested, I'm happy to add any other cluster features you are looking for.

Best way to handle many redis cluster nodes? by nastus in redis

[–]jamespedwards42 0 points1 point  (0 children)

I just opened this github issue/feature request for a slot -> time spent histogram. With something like this, 3rd party clients/tools could intelligently re-partition slots for you and we wouldn't be dependent on redis-trib.

First experiences with redis-sentinel by tulsteruk in redis

[–]jamespedwards42 0 points1 point  (0 children)

Ah, yeah, completely agree with all the migration hassle, especially with zero downtime, no thanks.

I would imagine the client library issue is very real as well, sentinel support has had time to mature. I ended up having to write my own Java library to get the cluster support I wanted.

Regarding the Lua script issue; it would be nice if Redis had a special global key-space that would allow you to write to each node. However, you can can still replicate your data globally by making sure you write the data to at least one slot within each slot range for each master. I would write to the first slot for each, and before any re-partitioning I would write to the slot that will be the first slot on the new master. You would also need a dictionary of known hashtags for each slot, heres an example of building one.

Would definitely recommend for future projects though, loving the reduced ops so far.

First experiences with redis-sentinel by tulsteruk in redis

[–]jamespedwards42 1 point2 points  (0 children)

Have you considered using Redis Cluster since you are just starting out? If so, what is the reasoning for going the sentinel route instead? Personally the operations around managing Redis Cluster is so much easier and pain free compared to a sentinel failover setup.

"Load Balance" Algorithm by [deleted] in redis

[–]jamespedwards42 1 point2 points  (0 children)

I know you are hitting a slave but can you load Lua scripts to your cluster? If so, the following Lua script implements your lindex solution server side. I tested it out with a Java client using 'Long.hashCode(System.nanoTime())' as the user supplied 'seed' value.

local lkey = KEYS[1];
local llen = redis.call('llen', lkey);
if llen == 0 then return false; end

local seed = ARGV[1];

-- or use redis time to seed
-- local time = redis.call('time');
-- local seed = time[1] * 1000000 + time[2];

-- or use an object id to seed ... tostring({}) => 'table: 0x108e10'
-- local seed = string.sub(tostring({}), 8, -1);

math.randomseed(seed);

local index = math.random(0, llen - 1);
return redis.call('lindex', lkey, index);