VBO Batch Delete PHP memory Issues

I've hit a problem with Drupal 7 using Views Bulk Operation to perform a simple task like batch deleting nodes.

Versions I am using:
Drupal 7.34
VBO 7.x-3.2

My view contains a few fields, exposed filters, and a VBO action for deleting nodes.

When the action is performed by admin user it works fine, but when performed by any other authenticated user, it hits the php memory limit.

I have investigated this problem and so far no luck. Hoping someone from the community might have a solution.

Here is what I think it is,

The Views is getting cached and when it's being unserialised, it fails with the php memory limit.

The unserialise error happens here : $cache->data = unserialize($cache->data); line 438 in cache.inc

I have used Views_cache_bully module to exclude the Views from being cached, turned off all views caching, but something is still wrong.

Looking forward to get some feedback and help from the community.

Much appreciated.

Jay