After you have removed a field from the view, you can find it in the “Unused Fields” tab at the left.
You can choose to then delete (only custom fields) them but both they and their data really will be gone forever.
