Data Engineer / DataOps / Data Architect

1/2023 – offen

Spearheaded the development of data infrastructure, focusing on renewable energy, and led the transition to data-driven decision-making by defining KPIs.
Evaluated and selected optimal data technologies like Hadoop and Spark, establishing a robust infrastructure on Microsoft Azure for storage, processing, and analytics.
Designed and implemented efficient ETL processes to maintain data quality and integrity, leveraging technologies such as Azure Synapse Analytics, Snowflake, and Databricks.
Developed data governance frameworks to comply with regulations like GDPR, and introduced data monitoring and alerting systems for high data accuracy and reliability.
Fostered professional growth within the team by mentoring junior data engineers and ensuring high standards across projects.
Achieved significant improvements in data accessibility, processing efficiency, and security, directly supporting enhanced organizational decision-making and operational scalability.

Apache Spark, Azure Databricks, Pandas DataFrame

7/2019 – 12/2022

Led data engineering projects in the Hadoop ecosystem, including Hive and Hue.
Managed the migration of Hadoop pipelines to Azure, enhancing data flow and performance.
Worked with teams to design data models and pipelines, supporting business initiatives.
Implemented Delta Lake and Unity Catalogue for improved data management in Databricks.
Optimized Hive query performance, boosting system efficiency.
Developed ETL processes for diverse data sources into Hadoop, ensuring data integrity.
Implemented security measures like encryption and access control in data handling.
Supported the integration of machine learning models within Hadoop, promoting smooth workflow.
Achieved significant infrastructure cost reduction, data processing time reduction, and enhanced query response through strategic migrations and optimizations.

Data Science, Azure Databricks, Azure Synapse Analytics, Big Data, Datawarehouse / DWH