SSH login delay

Some of my virtual machines running on Xen take a few seconds to login via SSH. This might be since that disabling management eth0's ipv6 disabled. There is no problem after logged in. Only login, it is a little bit slow, it takes more than 10 seconds maybe.
This problem is resolved by changing SSH configuration, should add following configurations to /etc/ssh/sshd_config, not ssh_config.

UseDNS no
AddressFamily inet