In human lupus nephritis, tubulointerstitial inflammation (TII) is associated with expansion of B cells expressing anti-vimentin antibodies (AVAs). The mechanism by which AVAs are selected is unclear. Herein, we demonstrate that AVA somatic hypermutation (SHM) and selection increase affinity for vimentin. Indeed, germline reversion of several antibodies demonstrated that higher affinity AVAs can be selected from both low affinity B cell germline clones and even those that are strongly reactive with other autoantigens. While we demonstrated affinity maturation, enzyme-linked immunosorbent assays (ELISAs) suggested that affinity maturation might be a consequence of increasing polyreactivity or even non-specific binding. Therefore, it was unclear if there was also selection for increased specificity. Subsequent multi-color confocal microscopy studies indicated that while TII AVAs often appeared polyreactive by ELISA, they bound selectively to vimentin fibrils in whole cells or inflamed renal tissue. Using a novel machine learning pipeline (CytoSkaler) to quantify the cellular distribution of antibody staining, we demonstrated that TII AVAs were selected for both enhanced binding and specificity . Furthermore, reversion of single predicted amino acids in antibody variable regions indicated that we could use CytoSkaler to capture both negative and positive selection events. More broadly, our data suggest a new approach to assess and define antibody polyreactivity based on quantifying the distribution of binding to native and contextually relevant antigens.