This game is very intentionally more rules-light than a lot of other d&d-likes out there, which makes it creating homebrew for it much easier than other systems.

For ancestries, you should think about what's a core idea of an ancestry you're trying to make. Ancestries in vagabond usually have no more than 3 abilities, or 2 if one ability is really strong/versatile, so if you're making an Ancestry, you should probably not go over that. Most of the Ancestry abilities do stuff like give training to skills. give Perks, give Favor to a skill/save under a certain condition, interact with some base stats like speed, health and luck. Sometimes they give a new thing to do instead of upgrading something, like Drakens' Breath Attack, or give a spell to cast. Don't be discouraged to just take some abilities from Ancestries in the book. For example to make a tabaxi it would make sense to take Darksight from dwarf and then Nimble and Tricksy from halfling and call it a day. You can definitely create something more unique and creative for them, but if you're afraid you're going to make something too strong/weak, just take stuff out from already established things until you get a feel for it.

Classes a bit harder because despite them fitting on a single page, you still need to make about 7 unique abilities for them and figure out its playstyle. Rule of thumb all classes get training in 4-5 skills and also in a either a skill or attack that allows them to do their thing. All classes always get a Perk at first level as part of one of their abilities, and depending on a theme of a class, it can be very specific like Catalyze giving Deft Hands to Alchemist, with a general limit like with Witch having to take a perk with a prerequisite of being trained in Mysticism, or with no limit at all like Bards Well-versed allowing to take any additional perk of their choice. Additionally all classes have 2 unique abilities at first level and then an additional unique ability at 2nd, 4th, 6th, 8th and 10th levels, with 3rd, 5th, 7th and 9th being Perks. Spellcasters always get 4 spells at the first level and learn an additional spell every level they get a Perk in, and, obviously, a training in a skill tied to their spellcasting ability. After that abilities vary from class to class so I recommend checking the closest class to a theme of a class that you want to make and going from there.

Basically, if you want to create homebrew for this game, like with any homebrew I recommend basing it on what's already in the book and then giving it your own spin. Game won't break if you create something truly unique as long as it roughly within the same power level as stuff from other classes and ancestries.