Role: Data Engineer
Location: Halifax, Canada (ONSITE)
Duration: Long Term
Key Responsibilities:
- Design and implement complex ETL/ELT workflows for diverse data sources both on-prem and Cloud platform (AWS)
- Build and optimize data lake architectures on Cloudera Data Platform (CDP)
- Develop data models in Snowflake cloud warehouse and CDP environments
- Implement CI/CD pipelines using Helios for data infrastructure automation
- Collaborate with analysts to enable advanced analytics on data lake assets
- Ensure security/governance compliance for sensitive data at scale
- Technical Expertise Required:
- Python: Advanced scripting for data processing, API integration, and automation
- AWS : Amazon S3,AWS Glue
- Containerization: Docker image management; orchestration via Kubernetes/OCP4
- Data Platforms:
- Cloudera Data Platform (CDP) for Hadoop/Spark ecosystems
- Snowflake cloud data warehouse architecture
- Storage Solutions: Data lake design (S3/ADLS), structured/unstructured data optimization
- Pipeline Tools: Airflow, Kafka, Helios CI/CD
- Version Control: GitHub collaboration
- Cloud Services: AWS/Azure storage integration with CDP/Snowflake
Soft Skills:
- Problem-solving for hybrid cloud data challenges
- Cross-functional collaboration with data scientists and analysts
- Clear communication of complex technical solutions
- Nice-to-Have Qualifications:
- Certifications: Cloudera CCP, SnowPro, Red Hat OpenShift
- Experience with streaming platforms (Flink, Kafka Streams)
- MLOps patterns for model deployment
