If you want to permanently delete an assignment, you can do so by going into the assignment in question via the Assignments tab.

Before you delete an Assignment you will need to make sure it is no longer assigned to any classes OR students by clicking into the assignment. Under 'Assigned to', it will show you the classes or students who have this assignment set to them. Delete the assignment for any classes or students. 

Once the assignment has been unassigned from all classes and students, the delete option will appear when you return to my assignments.

NB. If it was a Task assignment, the students submissions will also be deleted.