Do make your ports clean up after themselves when they are deinstalled. This is usually accomplished by adding @dirrm lines for all directories that are specifically created by the port. You need to delete subdirectories before you can delete parent directories.
: lib/X11/oneko/pixmaps/cat.xpm lib/X11/oneko/sounds/cat.au : @dirrm lib/X11/oneko/pixmaps @dirrm lib/X11/oneko/sounds @dirrm lib/X11/oneko
However, sometimes @dirrm will give you errors because other ports also share the same subdirectory. You can call rmdir from @unexec to remove only empty directories without warning.
@unexec rmdir %D/share/doc/gimp 2>/dev/null || true
This will neither print any error messages nor cause pkg_delete to exit abnormally even if PREFIX/share/doc/gimp is not empty due to other ports installing some files in there.