Effects of space charge on the mass purification in Penning traps