Dispersion in accumulation is a widely observed phenomenon in metal-oxide-semiconductor gate stacks based on III-V compound semiconductors. The physical origin of this phenomenon is attributed to border traps located in the dielectric material adjacent to the semiconductor. Here, we study the role of the semiconductor substrate on the electrical quality of the first layers at atomic layer deposited (ALD) dielectrics. For this purpose, either Al2O3 or HfO2 dielectrics with variable thicknesses were deposited simultaneously on two technology important semiconductors - InGaAs and InP. Significantly larger dispersion was observed in InP based gate stacks compared to those based on InGaAs. The observed difference is attributed to a higher border trap density in dielectrics deposited on InP compared to those deposited on InGaAs. We therefore conclude that the substrate plays an important role in the determination of the electrical quality of the first dielectric monolayers deposited by ALD. An additional observation is that larger dispersion was obtained in HfO2 based capacitors compared to Al2O3 based capacitors, deposited on the same semiconductor. This phenomenon is attributed to the lower conduction band offset rather than to a higher border trap density.