You're not really adding and deleting sets, you're adding and deleting set breaks. The first row in the show is not a set break, it's the start of your show and you can't delete it.
If you now have the start of your show, followed immediately by a break, with no songs in between, you can just delete the break. In other words, delete the second row instead of the first row.