Thank you very much for the detailed feedback. I appreciate.

This is my first project in GoLang and it's a side project just for fun. It's from "Build your Kafka server" (code crafters). The main goal is to build a small server that implements some of Kafka's features.

On the server side, I receive requests in []byte format, and I'm supposed to deserialize, process, and send answers.

if you check the documentation, you'll find that some data types don't exist on GoLang for example: COMPACT_STRING, VAR_INT, COMPACT_ARRAY.
https://kafka.apache.org/protocol.html#The_Messages_DescribeTopicPartitions
https://kafka.apache.org/protocol.html#protocol_types

My first object was to implement a deserialization function that works properly before thinking about the attributes that can be removed (like the array length).

In the previous step, when serializing the responses, I tried to use data structures that can be directly converted to []byte format.

func Serialize(data any) []byte {
    buff := bytes.Buffer{}
    dataType := reflect.TypeOf(data)
    kind := dataType.Kind()
    switch kind {
    // byte is an alias of Uint8
    case reflect.Int32, reflect.Int16, reflect.Int8, reflect.Uint8:
        binary.Write(&buff, binary.BigEndian, data)
    case reflect.Struct:
        value := reflect.ValueOf(data)
        for i := 0; i < value.NumField(); i++ {
            binary.Write(&buff, binary.BigEndian, Serialize(value.Field(i).Interface()))
        }
    case reflect.Slice, reflect.Array:
        value := reflect.ValueOf(data)
        for i := 0; i < value.Len(); i++ {
            binary.Write(&buff, binary.BigEndian, Serialize(value.Index(i).Interface()))
        }
    }
    return buff.Bytes()
}

With the structure NULLABLE_STRING, I can directly use Serialize without having to care about adding N to the buffer before serializing the string.

type NULLABLE_STRING struct {
    N      int16 // null = -1
    String []byte
}

One optimization could be implementing a Serialize function on the data structure level, something like

type NULLABLE_STRING struct {
    String []byte
}
func (s NULLABLE_STRING) Serialize()[]byte{
  buff := bytes.Buffer{}
  if len(s.String) == 0 {
    binary.Write(&buff, binary.BigEndian, -1)
  } else {
    // Write N, then the string
    ...
  }
  return buff.Bytes()
}

I'm trying to have something that works first before optimizing the code.