How or Where to Access Custom Field Handler Data in Processing

I've successfully created a couple of custom handlers extending from views_handler_field, and they display exactly what I want.
For my next trick I'd like to perform some processing actions based on the data in those fields. Problem is I can't find find them anywhere in the view object during processing. They just seem to magically appear after rendering.

I was expecting to find them somewhere in the result set and act on the rows, probably in hook_views_pre_render

Can someone provide guidance on how to access the data during views processing?