Senior Software Engineer - Scala, Spark, Akka, Cats, Web APIs, microservices

10/2018 – 6/2020

Building and maintaining data analytics pipelines for our flagship banking product Q-Refinery.

Q-Refinery has various features such as customer bank transactions enrichment (brand, industry and geolocation) and analysis of customer spend behaviour, brand preference, income and expense classification, etc. The product consists of a few data pipelines built with Scala and Spark, processing between 30 and 100Tb of data, depending on the bank. We used Oozie as a scheduler to orchestrate the pipelines and to run in a MapR / yarn managed on-prem cluster. We put a lot of thinking into the data model definition and also the way we read and write the data, how it is partitioned and how large the part files, so we wrote our own [typed] Parquet/CSV/TSV reader/writer framework.

Product page: -Hyperlink entfernt-

Technologies used:
Big Data, Spark, Scala, ScalaZ, Cats, Cats Effects, FP, ScalaTest, Oozie

Big Data, Apache Spark, Scala

1/2018 – 9/2018

Part of the FX-Options team, building an application to be used mainly by traders, allowing them to better model the FX options risk and pricing.

Morgan Stanley has the largest Scala code repository in the UK with around 3 milion lines of code. They have an in house developed framework which abstracts futures, database calls and UI. Even though there was a pretty steep learning curve to learn the framework, there were a few opportunities to use more mainstream libraries such as Cats and Shapeless. I have written a small but very flexible form fields validation framework with Shapeless. The project highlight was the extensive amount of unit tests we wrote to cover complex FX options bookings for different scenarios.

Technologies used:
Scala, Shapeless, ScalaTest, TDD, Continuous Integration

Scala

5/2016 – 12/2017

Building automation tools for the SEM team managing the AdWords/Bing marketing campaigns for all the properties on the Hotels.com website, localized in 72 languages. The main set of tools are organised in a microservices architecture with REST API interfaces and interchanging events via Kafka or a proprietary message bus, presenting reports to the business user in a web interface. We’ve migrated some of the data processing jobs to Spark on AWS EMR allowing to scale better and I had the main role in drawing the architecture and getting the team up to speed with Spark.

Technologies used:
Scala, Akka, Akka-HTTP/Spray, React JS, Spark, ScalaTest, REST, JSON, TDD, Docker, AWS, Continuous Integration

Apache Spark, Docker, Scala, Amazon Web Services (AWS), Representational State Transfer (REST)