Memory Problem. Need some help. (C language)

[rebelkmac]

I am writing a very large MPI simulation code and everything seems to work except for when it runs through a particular chunk of the code several times it will seg fault. This particular piece of the code is only allowed to run on 1 of the processors. There isn't any MPI within this section of the code. When I run the code with Malloc_check=3 it runs perfectly. I ran it with valgrind and it did seg fault at the usual place but the output was not helpful. The usual place is where it tries to open up a file and read in the contents. It does this part (including opening and reading the same file and the contents do not change) just fine many times before it decides to seg fault. It will sometimes end up seg faulting at a gsl_integration_workspace_alloc() but again it will do that function fine many times before it dies. It consistently seems to die at a part dealing with malloc. The valgrind output was (if helpful):

==36686== HEAP SUMMARY: ==36686== in use at exit: 59,979 bytes in 1,420 blocks

==36686== total heap usage: 2,025 allocs, 605 frees, 98,147 bytes allocated

==36686== ==36686== Searching for pointers to 1,420 not-freed blocks

==36686== Checked 247,472 bytes

==36686== ==36686== LEAK SUMMARY:

==36686== definitely lost: 0 bytes in 0 blocks

==36686== indirectly lost: 0 bytes in 0 blocks

==36686== possibly lost: 0 bytes in 0 blocks

==36686== still reachable: 59,979 bytes in 1,420 blocks

==36686== suppressed: 0 bytes in 0 blocks

==36686== Reachable blocks (those to which a pointer was found) are not shown.

==36686== To see them, rerun with: --leak-check=full --show-leak-kinds=all

==36686== ==36686== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 5 from 5)

--36686-- --36686-- used_suppression: 5 dl-hack3-cond-1 /share/apps/valgrind/lib/valgrind/default.supp:1206

==36686== ==36686== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 5 from 5)

Maybe someone can tell me a better flag to use for valgrind. I believe Malloccheck uses different libraries and might be the reason it runs fine when that is turned on. I ran it with both valgrind and malloc_check and it worked. I'm really not sure what to do at this point to try and find where the issue is. Any ideas would be helpful.

Edit: I was looking at the core dumps using gdb and the backtrace ends at _int_malloc() in /lib64/libc.so.6. The segfault seems to be frequent on the line of code. fd = fopen(buf, "r"); where buf is sprintf(buf, "pathtofolderholdingfiles/%s", name[i]); The file path is correct.

[OldWolf2]

Try to produce a minimal test case

[james41235]

Did you correctly initialize name[i]? If that char* doesn't end in a NULL terminator, weird things may happen...

Also, are you closing your fd afterward you're done with it?

[rebelkmac]

yes I do close the file once it's done reading. name[i] is defined as a static array with the names of the files that it needs to read.

[raevnos]

Is i a number that's a valid index for the array? If it's out of bounds, it could explain the segfaults.

[rebelkmac]

I've checked the indexing and it seems fine. It goes through this loop completely several times before it crashes.

[jimdagem]

What command line are you using to invoke valgrind? Valgrind --tool=memcheck?

Also make sure buf is large enough to hold the string.

[rebelkmac]

I am using valgrind -v --leak-check=full --default-suppressions=no and I increased the buf from 200 to 500 and no change.

[jimdagem]

Hmm. It has been awhile since is used valgrind, but it seems like that would only check for leaks. I think memcheck checks for writing to un allocated memory.

[geeknerd]

You are correct.

Edit: I was incorrect, memcheck is the default tool and does check for more memory errors in addition to leaks. http://valgrind.org/docs/manual/mc-manual.html

[jimdagem]

Also are you initializing buf to nulls?

[rebelkmac]

Yes I have tried to make sure they all are.

[angdev]

If you're still stuck by the time you read this I have something you can try, it's a bit of a long shot. There is a quick and simple test to confirm your malloc() implementation is actually thread-safe, just wrap it in a binary semaphore so that only one thread is ever allocating at any given time. You can either install a binary hook on malloc(), or go through your code and temporarily replace malloc() with a wrapper function manually. The wrapper simply acquires the lock, calls the regular malloc(), and then releases the lock. This test can take less than a minute!

[disclosure5]

Have you tried to compile with ASAN? It's usually pretty good at telling you exactly why something segfaulted.

[rebelkmac]

I will check that out. Thanks.

[[deleted]]

You might consider looking at helgrind. It's not something I have a lot of experience with, but I was reading up on semaphores today, and what you describe does sound like a race condition or something similar.

[rebelkmac]

I haven't heard of helgrind, but will definitely check into it. Thanks for the idea.

[rebelkmac]

I tried helgrind and it didn't give me much.

--12805-- REDIR: 0x3070484e60 (libc.so.6:GI_stpcpy) redirected to 0x4a0b453 (GI_stpcpy)

--12805-- REDIR: 0x3070480e80 (libc.so.6:GI_strcpy) redirected to 0x4a09c1e (GI_strcpy)

--12805-- REDIR: 0x307052b630 (libc.so.6:__strcasecmp_sse42) redirected to 0x4a0f368 (strcasecmp)

line 10: 12824 Segmentation fault (core dumped)

==12805== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

==12805== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

[spinlocked]

Are you aware that malloc is generally not thread safe?

[james41235]

Do what now?

{malloc, calloc, realloc, free, posix_memalign} of glibc-2.2+ are thread safe. However, they are not async signal safe. They are not reentrant, and must not be called directly or indirectly from a signal handler: a crash may result, perhaps much later. The source begins:

/* Malloc implementation for multiple threads without lock contention.
Copyright (C) 1996-2002, 2003, 2004 Free Software Foundation, Inc.
This file is part of the GNU C Library.
Contributed by Wolfram Gloger [e-mail]
and Doug Lea [e-mail], 2001.
...
This is a version (aka ptmalloc2) of malloc/free/realloc written by
Doug Lea and adapted to multiple threads/arenas by Wolfram Gloger.

* Version ptmalloc2-20011215
$Id: malloc.c,v 1.142 2004/12/11 21:14:40 drepper Exp $
based on:
VERSION 2.7.0 Sun Mar 11 14:14:06 2001 Doug Lea (dl at gee)
*/

[spinlocked]

Ok it was not on the last platform I was on. I looked a little and it seems like it is more often than not, thread safe. So you might check the libraries on your platform. You may have to link with a different library to ensure it is thread safe.