Senior Data Engineer - Microsoft

Empower every person and organization on the planet to achieve more. That’s what inspires us, drives our work, and pushes us to build things that make a difference. 

Our culture is one of ambition and science. We aim to dream boldly, ignite imaginations, and push boundaries. This is not a team that is content with the status quo; we are driven by the thrill of pursuing high-risk, high-reward opportunities. We believe in a future that is brighter and better than the present, and we want to be the ones who make it happen, first.  

Our belief is that customer obsession requires investment in science and research explorations, and that the continual pursuit of the state of the art can yield ongoing benefits to our customers both today and on the ever-moving horizon.  

We engage in long-term sustainable research across Microsoft’s Security Division to develop truly differentiating security solutions that can be leveraged across our Defender and Sentinel suite of products and services. Some examples of our desired impact include formulating scientific methods to significantly boost the productivity of security professionals, leveraging AI breakthroughs to provide non-linear impact in prevention and detection effectiveness, effective threat intelligence to stay on top of the current threat landscape and effectively defend against novel cyberattacks. 

We oversee and coordinate AI investments for Microsoft Security as well as work with engineering partners to accelerate and scale progress by making AI tools and data more accessible to security researchers. And we serve as Microsoft Security’s central hub for industry and academic partnerships in security science and AI as well as lead mindshare efforts that elevate the perception of Microsoft as trailblazers of security science and AI.  

Learn more: 

Microsoft’s AI@Scale investments

Microsoft’s Chief Scientific Officer, Eric Horvitz’s testimony on applications of AI in cyber missions before the U.S. Senate Armed Services Committee 

Microsoft Security blog and Defender Security Research Group

#MSecR 

#MSFTSecurity

#appliedresearch 

#aiambition

#artificialintelligence

• Responsible for the technical leadership and end-to-end development of data solutions including creating data pipelines, data quality/ validation checks leveraging large, complex data sources across Microsoft Security. 
• Collaborate with researchers on the design, development, execution, and implementation of research projects that serve as a catalyst for technology transfer into our security products and services. 
• Research and develop an understanding of tools, technologies, and methods being used in the community that can be utilized to improve product quality, performance, and efficiency. 
• Incorporate state-of-the-art research or previously tested solutions occurring at Microsoft and academia and tune it to solve complex security challenges.

Required:   

• 3+ years of experience working and building distributed cloud services using Azure or related technology stacks.  
• 5+ years professional development experience with SQL, Python, Spark, R, or other programming languages 
• 5+ years professional experience in data analytics, data science, data engineering, or software engineering  

Preferred:

• Experience with Azure Data Factory, Cosmos, Kusto, Spark, Kafka and other big-data related technology stacks. 
• Prior knowledge of data modeling, and building scalable machine learning pipelines 
• Experience handling large scale datasets (TB+) 
• Team player, confident and enthusiastic; Proven partner engagement and cross-team collaboration skills   
• Excellent written and verbal communication skills, ability to simplify and explain complex ideas. 
• Expert knowledge in security, threat intelligence and hunting, anti-fraud systems, and social engineering research 

Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable laws, regulations and ordinances.  We also consider qualified applicants regardless of criminal histories, consistent with legal requirements. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request via the Accommodation request form.

Benefits/perks listed below may vary depending on the nature of your employment with Microsoft and the country where you work.