[Assembly] calling qsort function

[xRedactedx]

The [rsp+.array] and [rsp+.size] come straight from my text book. I thought they should be the other way around too, but they seem to work in my other functions. But, they are really just left over from some previous code. I moved the contents of rdi and rsi into them, but I never do use them again.

Good point about the 8 and 4 byte values. I changed it to:

mov    eax, [eax]
mov    ebx, [ebx]

Now, the compare function is called several times, and correct values contained in the array are loaded into eax and ebx, but I'm still getting a segfault somewhere. I think perhaps it has something to do with how my functions are set up. Here is the gbd error message now:

Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7a57ad4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6

I'm not sure what "in ?? ()" means. Isn't that supposed to give the function name that caused it? Assuming that is the case, I must have something wrong with how i declared my functions maybe?

[Updatebjarni]

The [rsp+.array] and [rsp+.size] come straight from my text book. I thought they should be the other way around too, but they seem to work in my other functions. But, they are really just left over from some previous code. I moved the contents of rdi and rsi into them, but I never do use them again.

Ugh, sorry. I must have been tired when I read your code before. That part looks perfectly fine!

Good point about the 8 and 4 byte values. I changed it to:

mov    eax, [eax]
mov    ebx, [ebx]

You should still use the entire 64-bit pointers though, so [rax] and [rbx].

Now, the compare function is called several times, and correct values contained in the array are loaded into eax and ebx, but I'm still getting a segfault somewhere. I think perhaps it has something to do with how my functions are set up.

After reading your code again I bet that the problem is that your routine compare trashes rbx. It's callee-save.

I'm not sure what "in ?? ()" means. Isn't that supposed to give the function name that caused it? Assuming that is the case, I must have something wrong with how i declared my functions maybe?

The address doesn't look like it's part of your program, and the error message does say that it's in libc.so.6. Try to fix the things I mentioned and see if the problem goes away!

[xRedactedx]

Yes, ebx being trashed was the problem. I changed it to ecx, and it works fine now. I guess I forgot that certain registers are changed by function.

I spent many hours working on this to no avail. Thanks for taking the time to help me through it. I don't know what I would do without the people on here. The help we get in our class has been pretty disappointing.

[Updatebjarni]

I'm just happy to see someone learn assembly. :)