server IP blacklisted

Sorry for not responding to this earlier.

Is there an option to just check the DNS of the host and turn off the attempted sending of the check email?

There is not, but it should be simple enough to add it.

What are the implication of sending the test email?

There shouldn't be any, since many other systems and websites make similar checks.

Could it potentially get the server blacklisted?

Yes, I suppose it could, though it's probably not likely, but it really depends on how careful/paranoid/ignorant/incompetent/choose-your-adjective the administrator of the mail server is.

Please note that the referenced error message:

Mail rejected by Outlook.com for policy reasons.

is rather vague and will be challenging to pin down, especially when followed with

Reasons for rejection may be related to content with spam-like characteristics or IP/domain reputation.

which says "may," and implies "may not" and "may be related to something else entirely."

So, addressing those two specific things mentioned, first

content with spam-like characteristics

should not be related to this module, since there is no actual content included with the test (unless that is considered "content with spam-like characteristics"):

  // Conduct the test.
  fputs($connect, "HELO $localhost\r\n");
  $out = fgets($connect, 1024);
  fputs($connect, "MAIL FROM: <$from>\r\n");
  $from = fgets($connect, 1024);
  fputs($connect, "RCPT TO: <{$mail}>\r\n");
  $to = fgets($connect, 1024);
  fputs($connect, "QUIT\r\n");

The second one:

IP/domain reputation

would not be related at all to this module, but to the IP address of the server the site is hosted on and the IP address and domain name for the From address, which is defined as:

$from = variable_get('site_mail', ini_get('sendmail_from'));

and located on the Site information configuration page at /admin/config/system/site-information.
You will need to research this yourself, and that may be best started on Microsft's policy page, here: https://mail.live.com/mail/policies.aspx, especially at the Reputation Management section, although there are certainly numerous resources on the Internet to help.

I can confirm, from having run my own public "email address validator" (now taken down to restore my server's reputation), that Microsoft servers will block any server that performs multiple "will this address accept mail?" tests using SMTP without actually sending a message.

This makes sense, as spammers would find a list of valid email addresses at a given domain extremely valuable.

Testing for an existing MX record is perfectly safe, but talking to the MX server to just test whether an email address accepts mail can indeed result in a block on your server's IP.

I created #2827654: Provide a way for an admin to specify which tests to run to deal with the idea of selecting which tests to run when verifying.