Data Engineer

An exciting opportunity has arisen for a motivated professional. About the Project: - You will work on a system that processes sensitive partner data locally on partner infrastructure (not in the cloud) before any transfer occurs. Security-first design is mandatory. Key Responsibilities: - Build a Python-based application that reads structured exports (CSV, Excel) from partner databases - Generates consistent pseudonymous identifiers using HMAC-SHA256 - Removes direct identifiers and generalizes quasi-identifiers - Detects residual sensitive information in free-text fields using NLP (SpaCy NER) - Handles real-world data quality issues safely and gracefully - Ensures no sensitive data leaks via logs, temp files, or memory - Test pseudonymous identifier consistency across multiple runs - Measure record linkage success and failure rates - Evaluate re-identification risk using statistical methods - Produce quantitative evidence suitable for research publication - Write clear technical specifications for anonymization methods - Deliver readable, well-commented code for external review - Create a methodology document enabling independent replication Required Skills & Experience: - Python: Strong pandas skills; experience with large, messy datasets; performance optimization - Cryptography: Hashing algorithms, HMAC, secure key management - Databases: PostgreSQL or MongoDB - Cloud & Big Data: AWS, Azure, or GCP; Spark or Hive - NLP: SpaCy or similar for Named Entity Recognition (using existing models) - Security Mindset (Critical): Defensive coding practices, No logging of sensitive values and Awareness of data leakage risks (logs, memory, temp files) - Data Analysis & Communication - Statistical analysis and metrics - Jupyter notebooks and visualizations - Ability to explain technical concepts to non-technical stakeholders - Comfortable coordinating remotely with partner IT teams