if I add a jQuery click handler to an object, i.e.
$('#my-link').click(function() {
// stuff
});
But then I remove the link from my page, do I have to remove my click handler, and if so, how do I go about that?
No, you don't need to. From the jQuery documentation:
Use .remove() when you want to remove the element itself, as well as everything inside it. In addition to the elements themselves, all bound events and jQuery data associated with the elements are removed. To remove the elements without removing data and events, use .detach() instead.
Per the jQuery documentation remove() removes all events bound to the element: http://api.jquery.com/remove/