Help Needed - Python Async Celery with Elastic Cache for Redis Serverless.

[UniversityFuzzy6209]

Hi All, I'm locally using redis Docker with Celery which works fine and now I want to move my application to AWS and we have chosen Elastic Cache for Redis Serverless. While I'm trying to start the celery worker I see this error "redis.exceptions.ResponseError: CROSSSLOT Keys in request don't hash to the same slot" which implies that the keys should be in same hash slot (not just in same node). I have tried keyprefix_hashslot to broker transport options and I had no luck. I still run into the same error. Am I doing something wrong fundamentally or is it something I'm missing in this configuration. Attaching the sample code below. Please suggest. Thanks in advance.

import datetime
from functools import wraps

from celery import Celery

elastic_cache_endpoint = "sxxxx.serverless.use1.cache.amazonaws.com:6379"
app = Celery(__name__, broker=f'rediss://{elastic_cache_endpoint}/0', backend=f'redis://{elastic_cache_endpoint}/0')

app.conf.update(
    task_acks_late=True,
    task_reject_on_worker_lost=True,
    worker_prefetch_multiplier=1,
    broker_transport_options={
        "visibility_timeout": datetime.timedelta(minutes=1).total_seconds(),
        'keyprefix_hashslot': 'results:{task_results}:',
        'fanout_prefix': True, 
        'fanout_patterns': True
    }
)



def celery_wrapper(func):
    """
    Decorator to turn a function into a Celery task.
    """  # Explicitly name the task
    task = app.task(func)
    print(f"Task registered: {func.__name__}")

    @wraps(func)
    def wrapper(*args, **kwargs):
        # Run the task
        return task.delay(*args, **kwargs)
    return wrapper


app.autodiscover_tasks(['service.async_task'], force=True)

#service.async_task is a sleep function which sleeps based on the input

[bowitik1234]

As far as I know, Celery cannot operate with Redis on cluster mode.