XAudio2 delayed sound - when playing multiple voices

XAudio2 delayed sound - when playing multiple voices

I am using XAudio2 to play sounds in my game. I have about 16 source
voices out of which 3 are looping effects which need to be played just
like a background music. I have another background music which starts
playing at regular intervals(once completed it waits for 1 minute and
restarts). Whenever another effect has to be played like button click, gun
shot etc, I search for an available source and attach the buffer to it and
starts playing.
My problem is once the background music has started playing, just the
looping effects will get played and other effects which need to be played
on game actions(such as button click) do not get played. And all of these
delayed effects get played once the background music is stopped. Also if I
use one looping effect(instead of 3) all these get played in the right
order.
I am using a submix voice for effects and background music is submitted
directly to the mastering voice.
Can someone please help me as I am stuck here