Lead Data Software Engineer Portugal or Remote

Responsibilities

• Lead the design, development, maintenance, and optimization of efficient and scalable data pipelines using Python and Apache Spark
• Architect streaming and batch solutions leveraging Delta Lake tables and Spark Structured Streaming, ensuring high system performance and dependability
• Collaborate closely with business partners, IT experts, architects, and engineers to gather requirements, define project goals, and design strategies
• Mentor and support team members through code reviews, technical training, and collaborative problem-solving
• Utilize relational data models and SQL to manage, query, and finesse large-scale data workflows
• Develop and enforce automated testing strategies to ensure high-quality, reliable, and maintainable solutions
• Monitor, troubleshoot, and anticipate complex data processing issues while implementing proactive measures to maintain operational stability across systems
• Document technical designs, processes, and architecture best practices with precision and forward-thinking clarity
• Identify and drive initiatives for continuous improvement, from operational efficiency to technical scalability, within data pipelines and team processes
• Engage with product owners, senior leadership, and team members to prioritize and deliver impactful tools and solutions aligned with business goals
• Take ownership of technical challenges and act as a primary point of contact for critical decisions and escalations within data engineering initiatives

Requirements

• 5+ years of experience in a software engineering or data engineering role working with large-scale distributed data systems
• At least 1 year of relevant leadership experience
• Advanced expertise in Python, Apache Spark, and Delta Lake
• Deep familiarity with relational data models, SQL, and complex data transformations
• Proven mastery of cloud-native technologies within the Microsoft Azure ecosystem, including Databricks and Azure DevOps
• Strong experience in designing and maintaining reliable automated testing frameworks for data solutions
• Exceptional analytical skills to break down complex problems, design actionable strategies, and deliver elegant solutions
• Demonstrated ability to mentor and coach team members to elevate overall technical and professional outcomes
• Fluency in English and excellent communication skills to lead and collaborate effectively with distributed cross-functional teams

Nice to have

• Deep understanding of Spark Structured Streaming for real-time and near real-time data processing use cases
• Significant experience working with financial data or similarly complex data domains requiring high accuracy and efficiency
• Proven track record in identifying and resolving performance bottlenecks in distributed systems, with a focus on both technical and organizational solutions
• Strong proficiency in documenting long-term data pipeline architecture and implementation approaches, specifically tailored for scalability and best practice alignment
• Prior experience in leading technical decision-making and steering large-scale data engineering projects to successful completion

We offer

• Competitive compensation depending on experience and skills
• Variety of projects within one company
• Being a part of a project following engineering excellence standards
• Individual career path and professional growth opportunities
• Internal events and communities
• Flexible work hours