Condensed matter physics is an attractive field of complex phenomena.One of the major challenges in the field is to explore the exotic quantum states emergent from the interaction of many particles.Among these challenges,the interplay between mag-netism,superconductivity and topological phases has drawn intensive attention in past few decades.In experiments,the coupling between different phases can be realized in multiband superconductors,magnetic doped topological insulators and heterojunctions.In topological superconductors,the Berry phase of Dirac surface states can provide an extraphase for vortex bound states,which plays an essential role in the realization of Majorana zero mode(MZM).As a non-abelian anyon,MZM can create a global quantum state by braiding in space,which is topologically stable under local perturba-tion.Therefore,MZM is a promising platform to realize quantum computing by avoid-ing decoherence effect from the hardware aspect.In magnetic topological insulators,magnetism can break time reversal symmetry and induce quantum anomalous Hall ef-fect.Moreover,the abundant magnetic order can extend the categories of topological phases and create new types of quantum matters,like axion insulators and magnetic Weyl semimetals.In magnetic doped superconductors,the exchange scattering poten-tial of magnetic impurities can break Cooper-pairs and induce Yu-Shiba-Rusinov bound states.Furthermore,when magnetic impurities on a s-wave superconductor arranged in a 2D lattice,the hybridized YSR states can yield an effective topological chiral p-wave superconductor,which provide an ideal platform to realize MZM and topological quan-tum computer.As can be seen,multiphase-coupled materials are fertile soil for the emergence of novel quantum states.In this thesis,we will focus on three materials:Fe Te0.55Se0.45,Mn Bi2Te4and VTe2-superconductor heterojunction to explore the interplay between magnetism,superconductivity and topological phases by using scanning tunneling mi-croscope(STM).(1)By depositing Fe adatoms on Fe Te0.55Se0.45surface at low temperature,we realized quantum anomalous vortex and MZM in experiments and observed a strong zero-bias conductance peak at Fe adatoms.Fe Te0.55Se0.45single crystal possesses both Dirac topological surface states and high temperature superconductivity,which is an ideal platform to realize MZM.The superconductivity of bulk Fe Te0.55Se0.45can be induced in topological surface states by proximity effect.And the vortex induced by magnetic field can generate a MZM on the surface of Fe Te0.55Se0.45.Unlike magnetic flux induced MZM,MZM induced by Fe adatoms can be realized under zero field.The exchange interaction of Fe adatoms on the Fe Te0.55Se0.45can induce a quantum anoma-lous vortex due to spin-orbit coupling,which can provide an extraphase for Cd GM bound states in vortex to create MZM.In our experiments,we checked the origin of zero-bias conductance peak from three aspects:the magnetic response,the coupling dependence and the energy levels distribution.Furthermore,we realized the transition from YSR bound states to MZM with the increasement of exchange coupling by ap-proaching tip.(2)Based on a combined study on Mn Bi2Te4,we report the evidence of surface spin-flop transition induced by magnetic field using STM measurement and Monte Carlo simulation.Intrinsic antiferromagnetic topological insulator Mn Bi2Te4not only pos-sess a topological non-trivial band structure,but also holds rich and tunable topological phases.Bulk Mn Bi2Te4single crystal is a-symmetry protected?2topological insu-lator.Therefore,magnetic order plays a key role in the band structure,symmetry and topological properties of Mn Bi2Te4.By using STM/S at 400 m K,we studied the sur-face states of Mn Bi2Te4single crystal,grown out of solid-state reaction method.Under a magnetic field along c-axis,the interesting surface spin flop transition will tune the A-type antiferromagnetic order of Mn Bi2Te4into canted antiferromagnetic order,which lead to a d I/d V jump in STS experiments.Different from the bulk spin-flop transition,the critical field of surface spin-flop transition is asymmetric,which is consistent with the calculated result of Monte Carlo simulation according to Mills model.Therefore,the evolution of d I/d V spectra with magnetic field can be well explained by surface spin-flop transition.(3)By growing a monolayer graphene on Re(0001),we realized a more general superconducting substrate.Using this substrate,we fabricated a new heterostructure of VTe2and superconductor in experment.In general,the introduction of magnetism is much easier,which can be achieved through proximity effect or magnetic doping effect,while the introduction of superconductivity is more difficult,which mainly relies on the proximity effect of superconducting substrates.However,the strong interaction of the metallic substrates and bad thermal stability of van de Waals substrates restrict the fab-rication of 2D materials on superconductors.Therefore,we developed a more general superconducting substrates by growing a monolayer graphene on Re crystal,which have good thermal stability and weak surface interaction.Based on the Gr/Re substrate,we fabricated a VTe2-Gr-Re heterostructure and observed YSR bound states induced by lo-cal magnetic moments.This general superconducting substrate Gr/Re provides a solid foundation for the fabrication of exotic heterostructures in the future. |