As someone in the HP cast please don't avoid the rest of the cast because of one or two people, if there's people you want to play with play with them! If you want to plot with them and don't want a certain character all up in it it can be easily avoided, especially if you tell the person that. I'm sure they'd understand, I know I would, at least, because I would hate to have little to no CR just because of a person.