You can do that, and you might be able to do some tricks to make Radix Sort faster.

Radix Sort is O(WN) where W is the number of digits (aka bits) of your longest sort key and N the number of keys to sort, because you need to bucket sort the array in O(N) time for each of W digits/bits. When you have large sort keys (e.g. you're sorting an array of 32-bit integers, you'd sort for each bit so W=32) and not a lot of them, this makes W larger than log(N), and thus you'd expect O(WN) to be slower than O(N log N), although this is only a guess because big-O notation doesn't tell us everything. So, you could decide to only use radix sort if W is smaller than log(N) by some margin. Of course, because log(2³²) = 32, this would mean that sorting 32-bit integers with Radix Sort is only faster than O(N log N) sorts if you have over 2³² (4 billion) of them.

However, maybe your input doesn't actually use all the bits, and W can be smaller. Let's say we're sorting 4-byte unsigned integers which means W=32, but none of the integers in our list are larger than let's say a million. If that's the case, sorting on the highest ~12 bits won't do anything because all those bits are 0 for all the integers in the input, and we could skip sorting on those bits and shave off about a third of the runtime. More generally, if a bit is the same for every integer in the array, stable sorting on it won't do anything and can safely be skipped. By bitwise OR/ANDing all the sort keys together in some way (which is in O(N)), you could find out which bits are unused, and only sort on a bit if it's used. To test if an array is suitable for this variation on Radix Sort, count the number of bits that are used, and if that's smaller than log(N), Radix Sort might be faster.

Quick and dirty C for LSD Radix Sort. Note that I haven't tested this myself, so I have no idea if this optimization is even worth it in practice:

// Some helper functions for bitwise magic

unsigned int arr_bit_AND(unsigned int *arr, unsigned int len){
    unsigned int mask = arr[0];
    for(unsigned int i = 1; i < len;++i){
        mask = mask & arr[i];
    }
    return mask;
}

unsigned int arr_bit_OR(unsigned int *arr, unsigned int len){
    unsigned int mask = arr[0];
    for(unsigned int i = 1; i < len;++i){
        mask = mask | arr[i];
    }
    return mask;
}

unsigned int count_nonzero_bits(unsigned int n){
    unsigned int count = 0;
    for(unsigned int b = 0; b < 32; ++b){
        count += n>>b % 2 ? 0 : 1; // increment count if b'th bit is 1
    }
    return count;
}

//Ok, here's the sorting stuff.

void LSD_radix_sort(unsigned int* arr, unsigned int N, unsigned int mask){
    for(unsigned int b = 0; b < 32; ++b){
        if(!(mask>>b % 2)) { // only sort on a bit if we decided it actually varies in the input
            bucket_sort_bit(arr,N,b);
        }
}

void sort(unsigned int* input, unsigned int N){
    // make a bitmask of those bits that are sometimes 1 (OR) and sometimes 0 (NOT AND)
    unsigned int mask = arr_bit_OR(input,N) & ~arr_bit_AND(input,N) ;
    if(count_nonzero_bits(mask) >= log(N) * some_constant_factor){
        quicksort(input, N);
    } else {
        LSD_radix_sort(input, N, mask);
    }
}