So I'd be inclined to stick with the current M.N format for now, and keep incrementing 3.x unless and until we hit a significant enough change to bump it to 4.0.
+1 -- semver is not really useful here, and 3.20 looks perfectly fine in the eyes of developers (and probably beyond.)