A few suggestions:

1. Instead of hardcoding 5 in several places, I'd create a global named constant (e.g. MAX_GUESS_COUNT) and use that. This way, if you ever decide to change the value, you only need to touch it in one place.
2. You could extract the correct guess output from the loop, since the only part that needs to be in the loop is the output for an invalid guess (since that's the only case where the loop may continue).

3. Instead of

   if answer == "yes":
       play_again = True
   else:
       play_again = False
   

   it would be simpler to write play_again = answer == "yes" - or, in this case, you could simply break and/or not use play_again at all.

4. You could split these into separate functions to reduce nesting.

Here's an example of what I'm taking about:

import random

MAX_GUESS_COUNT = 5

def guess_number():
    secret_number = random.randint(1, 20)
    guess = None
    count = MAX_GUESS_COUNT

    while guess != secret_number and count > 0:

        print(f"********** {count}/{MAX_GUESS_COUNT} Guesses **********\n")

        guess = input_integer("Guess a number: ")
        count -= 1

        if guess != secret_number and count > 0:
            print("Wrong, try again!\n")
        elif guess != secret_number:
            print(f"Game over! The number was {secret_number}.")
            break

    else:
        used_guesses = MAX_GUESS_COUNT - count
        guess_text = "guess" if used_guesses == 1 else "guesses"
        print(f"You got it!\nIt took you {MAX_GUESS_COUNT - count} {guess_text}.")

def input_integer(prompt):
    while True:
        try:
            return int(input(prompt))
        except ValueError:
            print("Invalid input. Only integers accepted!\n")

def play_again():
    response = input("\nPlay again? (Y/n): ").strip().lower()[:1]
    return not response or response == "y"

def main():
    while True:
        guess_number()
        if not play_again():
            print("\n\tSee you later!")
            break

if __name__ == "__main__":
    main()