Senior Data Engineer
Horizontal Talent

Job Info

---

Overview:

• Architect and refine our Databricks-centric data platform, emphasizing scalable integrations and advanced entity resolution strategies.
• Lead the charge towards a data mesh infrastructure, promoting domain-centric design and decentralized data management.
• Enhance our DataOps capabilities, emphasizing data observability, discovery, and lineage to maintain data integrity and accuracy.
• Pioneer the adoption of Databricks data lakehouse architectures, focused on simplifying data management and enhancing data processing capabilities.
• Serve as a technical liaison among analytics engineers, BI developers, and analysts within the Databricks environment, integrating solutions like Segment.io.
• Develop and optimize SQL and Python/Scala scripts within Databricks for complex data transformation and integration tasks.
• Work closely with cross-functional teams to align data strategies with operational needs and objectives, incorporating advanced data resolution methodologies.
• Stay at the forefront of industry trends and technological advancements to ensure our Databricks platform remains innovative and effective.

Must Have:

• Bachelor's
• Strong leadership skills and the ability to advocate for best practices in data management and processing.
• 5+ years of experience in a data engineering role, with history of implementing data platform modernization
• 2+ years of hands on databricks experience.
• In-depth understanding of Databricks, along with experience in other cloud data warehouses like Snowflake, Redshift, Big Query.
• Building data integration and orchestration within the databricks environment, with hands-on experience in technologies such as Delta Live Tables, CDC, dbt, airflow, Segment.io.
• Advanced skills in SQL, Python, and Scala, with an emphasis on their application within Databricks for complex data tasks.
• Knowledge of best practices in version control and CI/CD with Git and GitHub Actions; Agile project management with Jira and Confluence.
• Proficiency with the Azure data suite (Azure SQL, Data Factory, Synapse Analytics, Power BI).
• Solid understanding of data observability, discovery, and lineage, and their application in maintaining high data quality standards.
• Additional experience with AI and ML capabilities in the context of Databricks is a plus.

---