a couple people might be avoiding you because your cr irritates them enough not to want to have shared close cr with them, but if it's only the occasional vague namedrop, it won't be more than one or two people, not enough to have a big impact on your CR pool.