Research shows that listeners’ abilities to categorize speakers based on their accents—whether regional, first-language, or foreign-language accents—are primarily based on social knowledge and familiarity with phonetic features of those varieties. Here, we explore whether English listeners’ categorization decisions are also influenced by the degree of perceptual similarity of those accents to their own, and whether a quantitative pronunciation distance metric can predict perceptual accent categorization. The current pronunciation distance metrics are restricted to segments (vowels and consonants), and they have not yet successfully integrated suprasegmental information (prosody and rhythm). To determine the independent contribution of suprasegmentals to accent categorization, we used low-pass and highpass filtering to compare their perceptual salience in low- and high-frequency spectral regions when segmental information was experimentally removed. Listeners responded to unmodified and filtered phrases produced by 24 speakers representing four distinct accent varieties. We found that accuracy of accent categorization was inconsistent with predictions based on the pronunciation distance, but the metric predicted the pattern of confusions: increasingly greater distance from listeners’ own accent decreased confusions with that accent. For filtered speech, categorization accuracy was comparatively lower but still above chance in either filtering condition. Together, across the four accent varieties, the contribution of suprasegmental information did not mirror the contribution of segmental information to accent categorization, considering both accuracy and confusions. Overall, however, the study provided evidence that listeners’ decisions are not solely based on social knowledge and accent familiarity, but phonetic similarity of accents also plays a role in categorization processes.