Ventrilo hosts all reuse the server IP and ports for voice servers, because servers and port allocations are a limited resource. What happens is, a server is stopped for whatever reason (user requests that it be canceled, or it is terminated for nonpayment), and the port is assigned to someone else. Our cancellation process will de-activate the server, so the old users will no longer be able to login, but they may try again later on.  If you did not set a "Server Login Password" in the Ventrilo Server section of your Control Panel, the old users will be able to login still.

Set a join password on the server, and the old users will no longer be able to join.