No.  

You do not need to remove or delete contacts that have unsubscribed.  They will remain in your list, but will be coded as ‘Unsubscribed’, and thus, will no longer receive emails.

If you desire to remove them, you can, but we do NOT recommend doing so.