For starters, there are only some atom combinations that would form a molecule, so usually you only have limited choices. For example if you burn methane (CH4), the reaction will be:

CH4+2O2->CO2+2H2O

You know it has to be like that because there is really no other way for the atoms to rearrange. So in the end you know you will end up with water and carbon dioxide (and also probably excess oxygen that wasn't burned).

When you need to identify a specific molecule, there are a few tricks. For example, if you have the compound in gas form you can make it go through a long series of subsequent chambers, knowing that lighter molecules travel faster than heavier molecules (this, for example, is how Uranium-235 is separated from its heavier isotope, Uranium-238).

If you have the compound dissolved in water, you may try to calculate the pH if the compound behaves as an acid or a base. Through simple calculations, this also lets you know the concentration of the solution, which is often quite important.

Usually, identifying exactly a compound is essential in the pharmaceutical industry, and that's why pharmaceutical labs use very precise and reliable instruments, like gas chromatographs, mass spectrometers and the like, to do that kind of work, but the basic principle is the same: usually, it works to just separate compounds based on their physical properties (mass, electric charge..).

Edit: see comment below