Irrelevant background speech is known to disrupt cognitive performance either by interfering with specific processes (e.g., serial rehearsal) or through attentional capture. Here, two experiments are presented to test the disruptive effect of whispered background speech in a serial recall task. According to an interference-by-process account, whispered speech should be less disruptive due to its reduced amplitude modulations and temporal fine structure compared to loud speech, thus providing weaker order cues to the auditory system and inducing less interference with seriation processes. However, due to the enhanced listening effort to process whispered speech (when presented in a comprehensible language), it could be argued that additional attentional resources will be demanded thus producing more disruption of serial recall. In Experiment 1, to-be-remembered letters were presented visually on the screen while either voiced or whispered to-be-ignored speech was presented via headphones. Half of the speech trials contained only a single German word that was played repeatedly (steady-state), whereas the other trials contained a full German sentence. Memory accuracy was significantly lower in the presence of sentential speech compared to steady-state words, indicating interference-by-process. More importantly, whispered speech produced more disruption compared to loud speech, indicating an additive effect of attentional capture potentially due to enhanced listening effort required to process semantic information. To test specifically whether the disruptive effect of whispered speech was due to enhanced listening effort, the same German speech materials were presented to a sample of English participants who did not speak or understand German in Experiment 2. It was found that in this case whispered speech was less disruptive than loud speech, indicating that psychoacoustic speech properties may dominate when no listening effort is demanded as in case of an incomprehensible language.