Plaintext field formatter doesn't take "fromto" setting into account

I have a Date field that contains a from and to date (e.g. 2012-01-01 to 2012-12-31).

When I want to display this particular field through Views, I have set the formatter settings to "Plain" and the "Display" (key = fromto) to "Start date only".
However, this still shows me "2012-01-01 to 2012-12-31" instead of just "2012-01-01".

I traced this back to date_field_formatter_view() in date.field.inc.
There is a switch on $display['type']
In case this is "date_plain", my date gets run through the following code:
$element[$delta] = array('#markup' => t('!start-date to !end-date', array('!start-date' => $item['value'], '!end-date' => $item['value2'])));

Which is not what I want...

If I select the formatter "Date and time" however, this works correctly, because in that case, theme_date_display_combination() gets called, which has a code block that performs a switch on $options['fromto'], making sure only the starting date is shown.

I'll probably use the "Date and time" formatter for now, and try to strip out the HTML that is added to it (because that interfers with my use case). However this seems like an inconsistency that should be fixed. Either the "Display" select list should be removed in case "Plain" formatter is chosen, or (better yet) the Plain formatter should take this setting actually into account.