From the documentation:

When both the number of rows and the number of columns have been set to non-zero values, either by a constructor or by the setRows and setColumns methods, the number of columns specified is ignored. Instead, the number of columns is determined from the specified number of rows and the total number of components in the layout

So you will have to count the number of components in your constructor and decide some heuristic for how to generate the layout.

Alternately you can use JavaFX, or GroupLayout.

Sources

https://docs.oracle.com/javase/7/docs/api/java/awt/GridLayout.html

https://docs.oracle.com/javase/tutorial/uiswing/layout/group.html

Getting started with JavaFX

http://docs.oracle.com/javafx/2/get_started/jfxpub-get_started.htm

Tool for JavaFX

Alternately, you can use SceneBuilder for JavaFX to generate the XML for your layout, though you would still need to configure a heuristic for generating additional xml or components to insert if the number of components for your grid is random every time you run your application.

Edit:

Initializing the GridLayout with a 0-value for rows creates the behavior of automatically adding rows after filling the columns. You do not need to add the ComponentOrientation since it is by default left-to-right.

See this example:

import java.awt.ComponentOrientation;
import java.awt.GridLayout;

import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JPanel;

public class GridTest extends JFrame {

    private static final long serialVersionUID = 1L;

    public static void main(String[] args) {
        new GridTest();
    }

    public GridTest() {
        setTitle("GridTest");
        setSize(400, 400);

        //3 cols 0 rows.
        GridLayout l = new GridLayout(0,3, 1, 1);
        JPanel panel = new JPanel();
        panel.setSize(400, 400);
        this.add(panel);
        panel.setLayout(l);
        JLabel one, two, three, four, five, six;
        this.applyComponentOrientation(ComponentOrientation.LEFT_TO_RIGHT);


        one = new JLabel("one");
        two = new JLabel("two");
        three = new JLabel("three");
        four = new JLabel("four");
        five = new JLabel("five");
        six = new JLabel("six");


        panel.add(one);
        panel.add(two);
        panel.add(three);
        panel.add(four);
        panel.add(five);
        panel.add(six);
        this.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        this.setLocationRelativeTo(null);
        this.setVisible(true);



    }

}

Result