Data Engineer

Project description We are seeking a skilled, hands-on Data Engineer with demonstrated experience in Databricks, Apache Spark, Python, and real-time data streaming solutions. Responsibilities - Implement resource composition logic provided by the Product team into scalable and efficient PySpark code. - Develop, optimize, and maintain PySpark applications within the Databricks environment. - Utilize and maintain existing CI/CD pipelines in Databricks for automated build, testing, and deployment. - Write and maintain unit tests to ensure high code quality and reliability. - Perform data validation and quality checks to ensure compliance with defined business rules - Support and manage production deployments, ensuring system stability and performance. - Troubleshoot and resolve issues across development, testing, and production environments. - Collaborate with Product, QA, DevOps, and healthcare domain teams to ensure accurate implementation. - Maintain technical documentation for code, workflows, and deployment processes. - Pull from and push code to GitHub repositories, following version control and branching best practices. SKILLS Must have - 5+ years of experience in data engineering roles - Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience). - 4+ years of experience with Databricks data pipelines - Strong hands-on experience with PySpark and/or Apache Spark. - Experience working with CI/CD pipelines and DevOps best practices. - Proficiency with CI/CD, GitHub (pull requests, branching, code reviews). - Experience writing unit tests and performing data validation - Experience supporting production systems and deployments - Strong analytical and troubleshooting skills. Nice to have - Knowledge of FHIR (Fast Healthcare Interoperability Resources) standards. - Experience with healthcare data integration projects. - Familiarity with REST APIs and FHIR server integrations. - Experience in Agile/Scrum development environments.