Small angle neutron scattering (SANS), resonant inelastic X-ray scattering (RIXS) and X-ray absorption spectroscopy have been carried out to study the correlations of magnetic properties and electronic/atomic structures in an XY-like spin-glass (SG) state of Ni0.4Mn0.6TiO3 (NMTO) single crystal. SANS presents the texture of toroidal glass and H (magentic)-T (temperature) phase diagram in NMTO. The results of RIXS show the evidence of crystal field (d-d) excitations at Ni and Mn L3-edge. Importantly, the spin-flip excitation is observable which is prominent at Ni L3-edge RIXS spectra in NMTO mainly. Using temperature-dependent X-ray linear dichroism (XLD) and extended X-ray absorption fine structure (EXAFS) measurements, it has revealed that phonon softening/symmetry breaking occurred at low temperature near SG phase transition due to the local static disorder in Ni-O bond length, and further induced the Ni 3d electron preference occupation charge from in-plane (3dx2-y2) to out-of-plane (3dz2-r2) states, demonstrating the strong correlations of spin, orbital and lattice in an XY-like SG state of NMTO.