"I don't do that s--t because the man card means everything," the "Empire" alum said in a new podcast interview.