The idea is not to bother with restore operations if they don't change
the set of tags. This is actually a relatively common case.
In order to avoid fancy datastructures, this method is quadratic in
the number of tags; at least on my mail database this doesn't seem to
be a big problem.