Hi,
When you delete it from PageLayout.aspx it only removes it from the page because it may be in use on other pages.
To delete it completely you can do it from the delete button in te feature instance settings or from the Content Manager Publish/Delete.. which has a button on the bottom of the page to delete it completely.
Hope it helps,
Joe