How to stop the first initial of the currency code from appearing when currency symbol is turned on?

A$ is the currency symbol for AUD (Australian Dollar) in the "en" locale, according to CLDR. It's not a random letter being added.

To show AUD as "$" you need to ensure your locale is set to "en-AU". That will also show USD as "US$".

Locales are set via LocaleResolvers and the default one uses "$currentLanguage-$currentCountry", where $currentCountry defaults to the country from your store address. So make sure your store has an australian address set.

\Drupal\commerce_price\CurrencyFormatter passes the locale to the Commerce Guys INTL currency formatter from the constructor:

  /**
   * Constructs a new CurrencyFormatter object.
   *
   * @param \CommerceGuys\Intl\NumberFormat\NumberFormatRepositoryInterface $number_format_repository
   *   The number format repository.
   * @param \Drupal\commerce_price\Repository\CurrencyRepositoryInterface $currency_repository
   *   The currency repository.
   * @param \Drupal\commerce\CurrentLocaleInterface $current_locale
   *   The current locale.
   */
  public function __construct(NumberFormatRepositoryInterface $number_format_repository, CurrencyRepositoryInterface $currency_repository, CurrentLocaleInterface $current_locale) {
    $default_options = [
      'locale' => $current_locale->getLocale()->getLocaleCode(),
      // Show prices as-is. All digits (storage max is 6), non-rounded.
      'maximum_fraction_digits' => 6,
      'rounding_mode' => 'none',
    ];

    parent::__construct($number_format_repository, $currency_repository, $default_options);
  }

You should double check what is your current locale.
Also, you could potentially swap the CurrencyFormatter with your own and hardcode the passed locale to the desired one.