Senior Data Engineer
- Verfügbarkeit einsehen
- 1 Referenz
- 105€/Stunde
- 40549 Düsseldorf
- auf Anfrage
- id | de | en
- 30.09.2024
Kurzvorstellung
Auszug Referenzen (1)
"Working with B.-W. has always been a pleasure. He is super friendly, helpful and always professional. We will miss him in our team."
1/2022 – 12/2023
Tätigkeitsbeschreibung
• Build and implement ETL-Pipelines using Python/Pandas
• Develop and deploy data-driven product & services used by customers
• Solve various data integration challenges
• Bring capabilities to deliver fast reporting and analytics solutions
Datenbankentwicklung, Pandas, Python
Qualifikationen
Projekt‐ & Berufserfahrung
4/2023 – 3/2024
Tätigkeitsbeschreibung
• Build a new data platform on Snowflake Data Cloud (Azure Cloud Platform)
• Data integration with Azure Databricks, Azure Data Factory and Snowpark for Python
• Data model with Data Vault 2.0 methodology and implementation the data model with Data Build Tool (DBT)
• Develop Proof of Concept for generative AI use case in Snowflake
• Build CI / CD Data pipelines on Gitlab for full automation of testing and deployment
• Provision and manage infrastructure in Azure Cloud and Snowflake with Terraform
Data Vault, Data Warehousing, Microsoft Azure, Snowflake
1/2022 – 12/2023
Tätigkeitsbeschreibung
• Build and implement ETL-Pipelines using Python/Pandas
• Develop and deploy data-driven product & services used by customers
• Solve various data integration challenges
• Bring capabilities to deliver fast reporting and analytics solutions
Datenbankentwicklung, Pandas, Python
11/2021 – 12/2021
Tätigkeitsbeschreibung
• Design and implement cloud data platform on Snowflake AWS Cloud:
- Migration of on-premise SQL-Server database to Snowflake on AWS
- Integration of Okta as Identity Provider, Tableau as BI Tool
• Design and implement the Snowflake Layered Security:
- Single Sign-On and SCIM Integration with Okta as Identity Provider
- Network Security with AWS Private Link
- Design Role Based Access Control (RBAC)
- Design Security Monitoring scripts
Cloud Computing, Snowflake
5/2021 – 10/2021
Tätigkeitsbeschreibung
• Guide customers through the process of migrating to Snowflake and develop methodologies to improve the migration process
• Deploy Snowflake following best practices, including ensuring knowledge transfer so that customers are properly enabled and are able to extend the capabilities of Snowflake on their own
• Work hands-on with customers to demonstrate and communicate implementation best practices on Snowflake technology
• Maintain deep understanding of competitive and complementary technologies and vendors and how to position Snowflake in relation to them
• Work with System Integrator consultants at a deep technical level to successfully position and deploy Snowflake in customer environments
• Provide guidance on how to resolve customer-specific technical challenges
• Support other members of the Professional Services team develop their expertise
• Collaborate with Product Management, Engineering, and Marketing to continuously improve Snowflake’s products and marketing.
Snowflake, Amazon Web Services (AWS), Cloud Computing, Data Warehousing, ETL, Google Cloud, Microsoft Azure, Solution Architektur
1/2019 – 4/2021
Tätigkeitsbeschreibung
Design and building a data lake on Microsoft Azure Cloud:
• Build ETL pipelines with Azure Databricks, Azure Data Factory, Azure Functions and Talend Data Management
• Data integration from variety of data sources (e.g. relational data from databases such as IBM DB2, MS SQL Server, SAP BW and semi-structured data from web services such as weather data, check24 data)
• Deployment of Azure Database for PostgreSQL for relational data within the data lake
• Deployment of Azure Data Lake Storage gen. 2 for unstructured / semi-structured data within the data lake
Apache Spark, Big Data, Cloud Computing, Data Warehousing, ETL, Java (allg.), Microsoft Azure, Python, Snowflake
12/2017 – 12/2018
Tätigkeitsbeschreibung
Building a scalable big data pipeline moving streaming data and batched data to a data lake on AWS Cloud:
• Data integration from variety of data sources IBM Mainframe, Oracle, MS SQL Server and SAP BW with Informatica Data Integration Tool and AWS Glue
• Develop python application for automatic generation of Informatica mappings, sessions and workflows
• Deployment of Streaming Platform Confluent Kafka, which is one of the main building blocks for data ingestion process
• Deployment of a Snowflake Data Cloud in AWS Cloud Platform for structured and semi-structured data within the data lake
• Deployment of AWS S3 storage for unstructured and semi-structured data within the data lake
Amazon Web Services (AWS), Apache Spark, Big Data, Cloud Computing, Data Warehousing, ETL, Python, Snowflake
3/2016 – 11/2017
Tätigkeitsbeschreibung
Building a big data platform on Azure HDInsight platform:
• Extract, transform and load data from variety of sources (Oracle, MS SQL Server, SAP BW) to Azure blob storage
• Use HDInsight tools Spark und Hive for transforming and storing data from the Azure blob storage to Hadoop-based storage (HDFS)
• Support data scientist team building a Hadoop-based analytical platform leveraging the analytic tools such as Spark Machine Learning, Apache Mahout
Support Proof of Concept activity designing a machine learning based predictive model for credit card fraud detection leveraging Azure HDInsight platform:
• Design a predictive model using HDInsight tools such as HDFS, HBase, Kafka, Spark
• Build a data pipeline, which extract the master data from HBase noSQL database through Kafka streaming platform to Spark for batch processing
• Build a data pipeline for the transaction data, which is streamed through Kafka streaming platform to Spark for real-time Processing
• Use Spark Machine Learning library to model a prediction for credit card fraud detection
Apache Hadoop, Apache Spark, Big Data, Cloud Computing, Data Warehousing, ETL, Python, SQL
6/2008 – 2/2016
Tätigkeitsbeschreibung
• Provide technical support to the customers (Vodafone Group, Deutsche Telekom, Telefonica O2) for deploying and testing of ZTE products
• Organize Proof of Concept with the customers to demonstrate ZTE product capabilities and to specify their potential use cases
• Team management in the ZTE data-based product development
• Planning and organization of pre-sales activities and meetings with suppliers (Qualcomm, Intel, Microsoft and Google)
Technischer Vertrieb
Zertifikate
Ausbildung
Leibniz Universität Hannover
Weitere Kenntnisse
- Snowflake Data Cloud
- Databricks Data Lakehouse
- Azure Synapse
- Google BigQuery
- PostgreSQL
- Azure SQL Server
- DB2
- Oracle
ETL / ELT Tool:
- Python (Jupyter Notebook)
- Snowpark for Python
- Azure Databricks (Pyspark)
- Azure Data Factory
- DBT
- Apache Kafka
- Talend Data Management
- Informatica Data Integration
Data Warehouse Modeling:
- Dimension Modeling, Data Vault Modeling
Cloud technology:
- Microsoft Azure
- Amazon Web Services
- Google Cloud
Programming skills:
- Python
- SQL
- Scala
- Java
Advanced Analytic / Machine Learning:
- Spark MLlib
- Scikit-learn
Operating System:
- Red Hat Linux
- Ubuntu Linux
- Windows
- macOS
Persönliche Daten
- Indonesisch (Muttersprache)
- Deutsch (Fließend)
- Englisch (Gut)
- Europäische Union
Kontaktdaten
Nur registrierte PREMIUM-Mitglieder von freelance.de können Kontaktdaten einsehen.
Jetzt Mitglied werden