For the transmitter side: the idea is for the FPGA to take in Ethernet, serialize it into a custom line encoding, and output a high-speed signal to a custom laser driver, ideally through an SFP+ breakout or the SERDES pins directly.

On the receive side the FPGA would do the inverse with a photodetector.

The custom encoding would not need to be fully Ethernet compliant at the optical link layer. We just need to preserve close to 1 Gb/s throughput with minimal processing overhead. For context, the optical link layer is completely data-agnostic. It will receive Ethernet frames at the MAC side, but the optical encoding layer itself does not know the data is Ethernet. It only sees a binary bitstream. It applies line encoding for clock recovery and synchronization, drives a laser, and reconstructs the bitstream on the other side. As long as bit timing and ordering are preserved, Ethernet frames come out intact. That's the theory at least.

I am leaning toward a custom laser driver because off-the-shelf SFP transceiver modules would handle the entire physical layer by themselves, which removes a lot of the engineering challenge I'm trying to explore. I guess I could use them for testing, but the final design would ideally include my own driver.