No, you don't need to remove or delete contacts that have unsubscribed. Unsubscribed users will remain in your list, but will be coded as Unsubscribed, meaning they no longer receive emails. 

If you desire to remove unsubscribed users, you can, but we do not recommend it.