Ultrashort laser pulses with femtosecond to attosecond pulse duration are the shortest systematic events humans can currently create. Characterization (amplitude and phase) of these pulses is a crucial ingredient in ultrafast science, e.g., exploring chemical reactions and electronic phase transitions. Here, we propose and demonstrate, numerically and experimentally, what is to the best of our knowledge, the first deep neural network technique to reconstruct ultrashort optical pulses. Employing deep neural networks for reconstruction of ultrashort pulses enables diagnostics of very weak pulses and offers new possibilities, e.g., reconstruction of pulses using measurement devices without knowing in advance the relations between the pulses and the measured signals. Finally, we demonstrate the ability to reconstruct ultrashort pulses from their experimentally measured frequency-resolved optical gating traces via deep networks that have been trained on simulated data.