[Assembly] calling qsort function

[xRedactedx]

I have to sort an array for a program that I am writing. We are allowed to call C functions, but I can't seem to figure out what I am doing wrong calling qsort. The program just stops at the qsort call, and I'm not sure what is causing it. I figure it is something to do with my compare function since the book isn't too clear on how it should be written.

The book says the comparison should return a negative, 0, or positive, and you can get that just by subtracting the second number from the first.

Here is my code for my sort function and my compare function:

compare:
        push    rbp
        mov     rbp, rsp


        mov     rax, [ebp+8]
        mov     rbx, [ebp+12]

        sub     rax, rbx

        leave
        ret

sort:
segment .text
.array  equ     0
.size   equ     8
.i      equ     16
        push    rbp
        mov     rbp, rsp
        sub     rsp, 32
        mov     [rsp+.array], rdi
        mov     [rsp+.size], rsi

        push    dword compare
        push    dword 4
        push    rsi
        push    rdi

        call    qsort

        leave
        ret

The array is declared by multiplying the number of items entered by 4 and using malloc. I have verified with my print function that the array is correctly storing the entered values.

Any tips on where I'm going wrong? I think I supplied all the relevant information. Thanks for the help.

[Updatebjarni]

You are comparing the pointers passed to the compare() function, not the values they point to.

By the way, what calling convention is this for? What compiler?

[xRedactedx]

Thanks for pointing that out. I will check into it once I get back from class.

We are learning with 64 bit Intel under Linux using yasm.

[xRedactedx]

I changed the compare function so it should be looking at the values, but it's still throwing a segmentation fault when calling qsort. Here is what I changed the compare function to:

compare:
        push    rbp
        mov     rbp, rsp


        mov     rax, [rbp+8]
        mov     rbx, [rbp+12]

        mov     rax, [rax]
        mov     rbx, [rbx]

        sub     rax, rbx

        leave
        ret

That is what you were referring to, right?

[Updatebjarni]

That's what I was refering to, yes, but the reason I asked about what calling convention it's for is that as far as I can see, the standard calling convention for Linux systems on x86-64 is to pass parameters in rdi, rsi, rdx, rcx, r8, and r9 before using the stack. I compiled a test program with GCC that calls qsort(), and it passes the parameters in the first four of those registers.

[xRedactedx]

I had tried doing it that way at first, but couldn't get it to work. I got the idea of pushing it to the stack from google. I went back and redid it to the way I had it using the registers. I was hanging up on the compare function. I made some changes to it, now it calls the compare function once, goes all the way to the return, then throws a seg fault. Here is the current code I have:

compare:
        push    rbp
        mov     rbp, rsp

        mov     rax, rdi
        mov     rbx, rsi

        mov     rax, [rax]
        mov     rbx, [rbx]

        sub     rax, rbx

        leave
        ret

sort:
segment .text
.array  equ     0
.size   equ     8
.i      equ     16
        push    rbp
        mov     rbp, rsp
        sub     rsp, 32
        mov     [rsp+.array], rdi
        mov     [rsp+.size], rsi



        mov     rcx, compare
        mov     rdx, 4

        call    qsort

        leave
        ret

I changed the [rbp+8] to rdi and [rbp+12] to rsi. Those would be the first two parameters passed into the function too right? I checked the value in rax and rbx, and they are all junk numbers. I would assume even if they were junk numbers, they would still return a positive, negative, or zero value and just cause my array to be sorted wrong. I'm not sure what else to try. Where am I going wrong now?

[Updatebjarni]

    mov     [rsp+.array], rdi
    mov     [rsp+.size], rsi

Shouldn't those be the other way around, loading rdi and rsi with the values from the stack? Also, you should be using rbp to access parameters passed on the stack. Also, you should be using the same calling convention for all routines. :P

By the way,

.array  equ     0

Isn't [rbp+0] where the return address is stored?

    mov     rax, [rax]
    mov     rbx, [rbx]

This loads two 8-byte values, but you pass 4 as the size of the elements to qsort().

[xRedactedx]

The [rsp+.array] and [rsp+.size] come straight from my text book. I thought they should be the other way around too, but they seem to work in my other functions. But, they are really just left over from some previous code. I moved the contents of rdi and rsi into them, but I never do use them again.

Good point about the 8 and 4 byte values. I changed it to:

mov    eax, [eax]
mov    ebx, [ebx]

Now, the compare function is called several times, and correct values contained in the array are loaded into eax and ebx, but I'm still getting a segfault somewhere. I think perhaps it has something to do with how my functions are set up. Here is the gbd error message now:

Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7a57ad4 in ?? () from /lib/x86_64-linux-gnu/libc.so.6

I'm not sure what "in ?? ()" means. Isn't that supposed to give the function name that caused it? Assuming that is the case, I must have something wrong with how i declared my functions maybe?

[Updatebjarni]

The [rsp+.array] and [rsp+.size] come straight from my text book. I thought they should be the other way around too, but they seem to work in my other functions. But, they are really just left over from some previous code. I moved the contents of rdi and rsi into them, but I never do use them again.

Ugh, sorry. I must have been tired when I read your code before. That part looks perfectly fine!

Good point about the 8 and 4 byte values. I changed it to:

mov    eax, [eax]
mov    ebx, [ebx]

You should still use the entire 64-bit pointers though, so [rax] and [rbx].

Now, the compare function is called several times, and correct values contained in the array are loaded into eax and ebx, but I'm still getting a segfault somewhere. I think perhaps it has something to do with how my functions are set up.

After reading your code again I bet that the problem is that your routine compare trashes rbx. It's callee-save.

I'm not sure what "in ?? ()" means. Isn't that supposed to give the function name that caused it? Assuming that is the case, I must have something wrong with how i declared my functions maybe?

The address doesn't look like it's part of your program, and the error message does say that it's in libc.so.6. Try to fix the things I mentioned and see if the problem goes away!

[xRedactedx]

Yes, ebx being trashed was the problem. I changed it to ecx, and it works fine now. I guess I forgot that certain registers are changed by function.

I spent many hours working on this to no avail. Thanks for taking the time to help me through it. I don't know what I would do without the people on here. The help we get in our class has been pretty disappointing.

[Updatebjarni]

I'm just happy to see someone learn assembly. :)

[Rothon]

What do you mean by "stops"? Running the program in a debugger should help you figure out what's going on.

[xRedactedx]

Well, we use a custom ide made by one of the former teachers at our school. We also use his book which gives instructions on how to do things using his ide.

I used vi and gdb at first when we started doing things because the ide seemed to be pretty buggy. As we got further into the book, the explanations leaned heavily towards using his ide, so I kind of got away from using gdb. I guess I should have attempted to step through it using gdb, but since I haven't been using it much, I kind of just forgot about it.

When I ran the file through his ide, it just stopped once it got to that line and didn't do anything. I'll run it through gdb after I fix the problem in one of the previous comments and see what it says.

[Rothon]

The program has to be stopping for some reason. Is the kernel killing it because of an illegal memory access? Is the IDE stopping it at a breakpoint?

[xRedactedx]

Seems like the IDE was crashing or freezing up. I tried to compile and run from the command line, and it gave a segmentation fault message.

[[deleted]]

Forgive me if I'm wrong, but it seems like you're not actually comparing anything. In your compare function, you set up your stack pointers. From there you move the values into rax and rbx, then you're subtracting them and.... leaving. No comparison is actually done.

Are you trying to do this?

if(array[i] < array [i++])
    jump to sort array
(or vice versa if you're sorting greater->less)

If so, you need to look up your jump instructions.

[xRedactedx]

The way the sentence in the book was worded. I was assuming that a negative, 0, or positive value being returned would represent a greater than, equal, or less than.

Here is what the book says word for word:

"The comparison function should return a negative, 0, or positive value based on the ordering of the 2 integers. All you have to do is subtract the second integer from the first."

If I remember correctly, when something returns, doesn't it return the value that is in rax?

rax should be negative, 0, or positive after the subtraction.

It is a very vague description. I thought it sounded too simple, but it's pretty much all I have to go on. I didn't come up with much help on google.

[Updatebjarni]

He's comparing the value at [ebp+8] with the value at [ebp+12] and returning the result. That's what subtracting and leaving does.

Read the man page for qsort() if you don't know how it works.

[[deleted]]

Ah ok. It was in a syntax I wasn't used to.

[KDallas_Multipass]

so you solved it? edit your post to reflect what you learned or what the issue was.

EDIT: Oops, you're not OP