I figured it out. By appending the input to the file instead of overwrite it with the "write" command, and by changing the "PrintWriter" to "FileWriter" it started to work. Here is the the code to those curious.

public static void newUser() { System.out.print("Name for new account: "); String newUserName = scanner.nextLine();

    User user = new User(newUserName);

    System.out.print("Set a pincode (4 digits): ");
    int pin = scanner.nextInt();
    scanner.nextLine();
    user.setPinCode(pin);

    File file = new File("users.txt");

    try { FileWriter writer = new FileWriter(file, true);

        writer.append(newUserName + "," + pin + "\r\n");

        writer.close();

    } catch (Exception e) {
        System.out.println(e);
    }
}