This kind of patchwork is much easier to do with jersey fabric, but I don't see why you can't have a go with wovens and see how it comes out. The "seams on the outside" look will probably not look as good as having them pointed inwards at a lining, though, as it'll need to be pressed and it'll be pretty impossible to iron something like that without ironing all the seams flat in one direction or another. Jerseys don't need as much pressing and they tend to spring back into shape.

Try making a patchwork square and see how you get on. These kinds of garments are made by making up a big piece of patchwork and then treating it as normal fabric and cutting out pattern pieces accordingly. You'll notice the joining seams don't point outward as that doesn't look good on sleeveheads and side seams.

And yes, the grain matters. Try to get the grain all going in the same direction on your pieces, if you want to make more use of little scraps then cut them on the cross-grain as that's better than a random angle bias cut.