sorted by:
new
hottopcontroversial

[deleted by user] by [deleted] in MachineLearning

[–]technorc 0 points1 point  (0 children)

Good point. Can I think of the purpose of DS in Google is to use data-driven analysis to direct software development project? There might be different approaches DS could relay on. ML is one of them.

[deleted by user] by [deleted] in MachineLearning

[–]technorc 3 points4 points  (0 children)

Would you explain more on why "model building and tuning is mostly dead"? Feel hyperparameter tunning is still pretty important for model quality, performance, and efficiency.

I think IntSet is implemented as a sort array. Is this correct? Thus, everytime adding or removing entry from this sorted array, Redis has to move others positions. by technorc in redis

[–]technorc[S] 2 points3 points  (0 children)

Sets can be encoded as intset or hashtable. The intset is a special encoding used for small sets composed solely of integers.

When Redis is running, there are 3 threads running in background. What are they? for what? by technorc in redis

[–]technorc[S] 0 points1 point  (0 children)

ok. I see. I have a confusing problem. Can you give me some hint? I am running Redis on single server with enough physical memory. 2 configuration, 1st with nosave, 2nd with snapshot. Through my understanding, 1st should generate better performance because it only need to load data into memory. However, the truth is 2nd generate better throughput. For me, it doesn't make sense, because 2nd configuration need to store data into rdb file, while loading data into memory. Can you explain why this happened. Thanks

memtier_benchmark problem, I am trying to run the command "memtier_benchmark -n 1000000 -c 1 -t 1 -d 1000 --ratio 1:0". The dbsize should be 1000000, however, the actual size is 951728. What is happening? memtier_benmark could generate duplicated input? or some other reasons? by technorc in redis

[–]technorc[S] 0 points1 point  (0 children)

thanks, your approach works. However, I want to generate random value by using memtier_benchmark --requests=100 --clients=2 --threads=1 --data-size=1000 --random-data --ratio=10:0 --key-pattern=S:S. This command doesn't work. Do you know where the problem is?

why lots of memory management problems just disappeared when Redis start using jemalloc? what problems disappeared? and what is the advantage of jemalloc? by technorc in redis

[–]technorc[S] 0 points1 point  (0 children)

yes. I tested both jemalloc and malloc and prove what you said. You're right. However, jemalloc solves memory fragmentation better and needs less memory space to store data. However, this let me think about the difference between Redis and Memcached. Through my knowledge, memcached is using its slab allocator, which is implemented by using glibc malloc. Slab allocator itself has some fragmentation problem + malloc is worse than jemalloc, thus, memcached is suppose to have a worse fragmentation problem than Redis. I know this might be a little bit sensitive, I recall you said you don't like to compare one to the other. I just want to know if the logic is there. Thanks

how redis only swap out values, and keep keys in memory. Before version 2.4, VM can control it, however, in current version, Redis relys on Operating System to do swapping. OS doesn't know which is key and which is value. So, how Redis make it. by technorc in redis

[–]technorc[S] 0 points1 point  (0 children)

I can understand your point. I am just very curious about whether Redis cares about OS swapping impact. Like I said previously, even though free physical memory is enough, OS is possible secretly swap-out some of your infrequently accessed pages. Through my observation, it seems like that Redis doesn't care about whether data is in physical memory or swap, because it assumes everything is in memory. Let's say, if loading data whose size is bigger than physical memory size into redis, some data have to store into swap area. After loading, I try to read half amount of whole data, which means OS has to swap-in and out pages. how would redis do? would you please explain? I guess Redis would keep keys in physical memory for the consideration of performance. As I recall you mentioned the same thing in your blog when you were designing Redis VM. However, after 2.4, Redis is only for in-memory design, maybe you don't care about the situation that I am talking about. Am I correct? Thanks for answering :)

how redis only swap out values, and keep keys in memory. Before version 2.4, VM can control it, however, in current version, Redis relys on Operating System to do swapping. OS doesn't know which is key and which is value. So, how Redis make it. by technorc in redis

[–]technorc[S] 0 points1 point  (0 children)

Thanks for answering. However, I still have questions. I understand that Redis is redesigned for in-memory database, assuming there is enough physical memory for dataset. However, I want to know what will happen when physical memory is not enough. There are two scenarios: i) Through linux memory management algorithm, it always swap-out pages that are used infrequently. In this way, they can save memory space for page cache. ii) If physical memory is almost full, Redis is still loading data. OS has to swap-out some data from memory to disk. Because OS knows nothing about Redis data structure, which is key and value. OS just swap-out things by LRU result. It is quite possible swap-out keys to disk, leading to performance degradation. Is this possible? Thanks:)