If you want to permanently delete an assignment, you can do so by going into the assignment in question via the Assignments tab.

Make sure that the Assignment is no longer assigned to any classes OR students by clicking into the assignment. Under 'Assigned to', it will show you the classes or students who have this assignment set to them. Delete the for any classes or students. 

Once the assignment has been unassigned from both classes and students, the delete option should appear when you return to my assignments.

NB. If it was a Task assignment, the students submissions will also be deleted.