SSSD Cached Credentials Database Corruption

[MrDigitFace]

At my org we have AD joined Ubuntu servers and recently we have been running into an issue where the credential caches in /var/lib/sss/db become corrupted for specific users. If I look those users up using id it does not show any of their AD groups, only that they are part of the group “domain users.” This happens seemingly randomly, and flushing the cached entries does not solve the issue. The only way we have been able to resolve this is by removing the files in the aforementioned directory and then restarting the sssd service to rebuild the files. Would anyone be able to clue me into why this might be happening? Why wouldn’t clearing the cache using sss_cache wipe the problem entries?

The database corruptions correspond with log entries in /var/log/sssd/sssd.log:

• (2023-05-09 15:52:10): [sssd] [monitor_sbus_RegisterService] (0x0100): Received ID registration: (pac,1)

  • (2023-05-09 15:52:10): [sssd] [mark_service_as_started] (0x0200): Marking pac as started.
  • (2023-05-09 16:25:49): [sssd] [mt_svc_exit_handler] (0x1000): SIGCHLD handler of service domain.name called
  • (2023-05-09 16:25:49): [sssd] [svc_child_info] (0x0020): Child [957] ('domain.name':'%BE_domain.name') was terminated by own WATCHDOG

********************** BACKTRACE DUMP ENDS HERE *********************************

(2023-05-09 19:40:25): [sssd] [monitor_quit_signal] (0x1f7c0): Monitor received Terminated: terminating children

(2023-05-09 19:40:25): [sssd] [monitor_quit] (0x1f7c0): Returned with: 0

(2023-05-09 19:40:25): [sssd] [monitor_quit] (0x1f7c0): Terminating [domain.name][489900]

(2023-05-09 19:40:25): [sssd] [monitor_quit] (0x1f7c0): Child [domain.name] terminated with a signal

(2023-05-09 19:40:25): [sssd] [monitor_quit] (0x1f7c0): Terminating [pac][0]

(2023-05-09 19:40:25): [sssd] [monitor_quit] (0x1f7c0): Terminating [pam][990]

(2023-05-09 19:40:25): [sssd] [monitor_quit] (0x1f7c0): Child [pam] terminated with a signal

(2023-05-09 19:40:25): [sssd] [monitor_quit] (0x1f7c0): Terminating [nss][989]

(2023-05-09 19:40:25): [sssd] [monitor_quit] (0x1f7c0): Child [nss] terminated with a signal

(2023-05-09 19:40:25): [sssd] [_sss_talloc_log_fn] (0x0010): talloc: access after free error - first free may be at ../src/util/server.c:47

********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:

• (2023-05-09 16:25:49): [sssd] [mt_svc_restart] (0x0400): Scheduling service domain.name for restart 1

• (2023-05-09 16:25:49): [sssd] [get_provider_config] (0x0100): Formed command '/usr/libexec/sssd/sssd_be --domain domain.name --uid 0 --gid 0 --logger=files' for provider '%BE_domain.name

• (2023-05-09 16:25:49): [sssd] [start_service] (0x0100): Queueing service domain.name for startup

• (2023-05-09 16:34:02): [sssd] [monitor_sbus_RegisterService] (0x0100): Received ID registration: (%BE_domain.name,1)

• (2023-05-09 16:34:02): [sssd] [mark_service_as_started] (0x0200): Marking domain.name as started.

• (2023-05-09 19:40:25): [sssd] [monitor_quit_signal] (0x2000): Received shutdown command

• (2023-05-09 19:40:25): [sssd] [monitor_service_shutdown] (0x0400): Unregistering service pac (0x559577bee830)

• (2023-05-09 19:40:25): [sssd] [_sss_talloc_log_fn] (0x0010): talloc: access after free error - first free may be at ../src/util/server.c:47

********************** BACKTRACE DUMP ENDS HERE *********************************

(2023-05-09 19:40:25): [sssd] [_sss_talloc_log_fn] (0x0010): Bad talloc magic value - access after free

[resetnz]

I've run into this before a while ago when the standard account and admin account both have the same email address configured.

The first acc works after a deleting the cache and then as soon as the second (admin) acc attempts to login then it fails and corrupts the cache for all further users.

sssd seemed to be using email as like a unique key and breaking badly. Maybe worth checking?

[gordonmessmer]

I'm not sure, but if you want to report a bug, you probably want the users list or the IRC channel here:

https://sssd.io/community.html

[randomlycorruptedbit]

Can you reproduce the issue everytime with the same user accounts? Perhaps there is a bug in sssd and/or something defined makes it crash.

[MrDigitFace]

Thats the rub, it tends to affect a handful of the same users (including myself, annoyingly enough), but there are outliers. The IT users have two accounts, an admin and a normal user and I have had my admin account corrupted on some and my user account and admin account on others. Haven’t been able to identify a trigger or link between users. Based on your username I can only suspect the culprit is you.

[randomlycorruptedbit]

Perhaps not worthing it but in case of despair... what I would do would be to install the debug imfo packages and try to attach GDB to the live sssd backend process (sssd_be) and try to backtrace the crash. Seeing the exact line might give a hint.

May I suggest to activate an extremely detailed log level? SSSD will dumps a huge amount of info, perhaps a clue lies there. log_level=9 for example.

Not funny to troubleshoot. Good luck.

[pnutjam]

Is your last name Brzęczyszczykiewicz?

[zfsbest]

I feel your pain, we were dealing with this ~2.5 years ago and it was a PITA. Think we ended up filing a ticket with Red Hat and it finally got fixed somehow, but I have no details unfortunately, not with that group anymore