We experimentally investigated the role of background signal in time resolved photoluminescence experiments with single quantum dots on substrates. We show that the background fluorescence signal from thin gold films fabricated by electron-beam evaporation and from Al2O3 layers fabricated by atomic layer deposition have to be taken into consideration in experiments on the single photon level. Though all investigated components can be distinguished by their photoluminescence decay rates, the presence of the background signal prevents the observation of photon anti-bunching from single quantum dots. Moreover, a single quantum dot acts as a hot-spot enabling the plasmon supported fluorescence enhancement of gold.