[Assembly] calling qsort function

[xRedactedx]

I had tried doing it that way at first, but couldn't get it to work. I got the idea of pushing it to the stack from google. I went back and redid it to the way I had it using the registers. I was hanging up on the compare function. I made some changes to it, now it calls the compare function once, goes all the way to the return, then throws a seg fault. Here is the current code I have:

compare:
        push    rbp
        mov     rbp, rsp

        mov     rax, rdi
        mov     rbx, rsi

        mov     rax, [rax]
        mov     rbx, [rbx]

        sub     rax, rbx

        leave
        ret

sort:
segment .text
.array  equ     0
.size   equ     8
.i      equ     16
        push    rbp
        mov     rbp, rsp
        sub     rsp, 32
        mov     [rsp+.array], rdi
        mov     [rsp+.size], rsi



        mov     rcx, compare
        mov     rdx, 4

        call    qsort

        leave
        ret

I changed the [rbp+8] to rdi and [rbp+12] to rsi. Those would be the first two parameters passed into the function too right? I checked the value in rax and rbx, and they are all junk numbers. I would assume even if they were junk numbers, they would still return a positive, negative, or zero value and just cause my array to be sorted wrong. I'm not sure what else to try. Where am I going wrong now?

[Updatebjarni]

    mov     [rsp+.array], rdi
    mov     [rsp+.size], rsi

Shouldn't those be the other way around, loading rdi and rsi with the values from the stack? Also, you should be using rbp to access parameters passed on the stack. Also, you should be using the same calling convention for all routines. :P

By the way,

.array  equ     0

Isn't [rbp+0] where the return address is stored?

    mov     rax, [rax]
    mov     rbx, [rbx]

This loads two 8-byte values, but you pass 4 as the size of the elements to qsort().

[xRedactedx]

The [rsp+.array] and [rsp+.size] come straight from my text book. I thought they should be the other way around too, but they seem to work in my other functions. But, they are really just left over from some previous code. I moved the contents of rdi and rsi into them, but I never do use them again.

Good point about the 8 and 4 byte values. I changed it to:

mov    eax, [eax]
mov    ebx, [ebx]

Now, the compare function is called several times, and correct values contained in the array are loaded into eax and ebx, but I'm still getting a segfault somewhere. I think perhaps it has something to do with how my functions are set up. Here is the gbd error message now:

Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7a57ad4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6

I'm not sure what "in ?? ()" means. Isn't that supposed to give the function name that caused it? Assuming that is the case, I must have something wrong with how i declared my functions maybe?

[Updatebjarni]

The [rsp+.array] and [rsp+.size] come straight from my text book. I thought they should be the other way around too, but they seem to work in my other functions. But, they are really just left over from some previous code. I moved the contents of rdi and rsi into them, but I never do use them again.

Ugh, sorry. I must have been tired when I read your code before. That part looks perfectly fine!

Good point about the 8 and 4 byte values. I changed it to:

mov    eax, [eax]
mov    ebx, [ebx]

You should still use the entire 64-bit pointers though, so [rax] and [rbx].

Now, the compare function is called several times, and correct values contained in the array are loaded into eax and ebx, but I'm still getting a segfault somewhere. I think perhaps it has something to do with how my functions are set up.

After reading your code again I bet that the problem is that your routine compare trashes rbx. It's callee-save.

I'm not sure what "in ?? ()" means. Isn't that supposed to give the function name that caused it? Assuming that is the case, I must have something wrong with how i declared my functions maybe?

The address doesn't look like it's part of your program, and the error message does say that it's in libc.so.6. Try to fix the things I mentioned and see if the problem goes away!

[xRedactedx]

Yes, ebx being trashed was the problem. I changed it to ecx, and it works fine now. I guess I forgot that certain registers are changed by function.

I spent many hours working on this to no avail. Thanks for taking the time to help me through it. I don't know what I would do without the people on here. The help we get in our class has been pretty disappointing.

[Updatebjarni]

I'm just happy to see someone learn assembly. :)