if you pick the option "clementine is my family" when trying to get him to come help lee get clem back, then he'll always (I think) come with you. otherwise he refuses, no matter how much you've been on his side.

which, yeah. makes him a bit of a dick when you were always on his side and are begging him to help you now.