Data Engineer (Cloud and DevOps)
Job Description
Description
• Build, design, and implement scalable data lake architecture using AWS S3 and Lake Formation.
• Build and optimize ETL pipelines using AWS Glue, EMR, and Spark.
• Implement event-driven workflows using EventBridge, SNS, and SQS.
• Design and query datasets using Athena.
• Manage metadata and data lineage using Amazon Neptune DB.
• Expose APIs for data subscription and sharing using API Gateway and AWS Lambda.
• Automate infrastructure provisioning using Terraform and CloudFormation.
• Ensure data security and compliance by implementing robust IAM policies and access controls.
• Develop and maintain a self-serve data portal using Angular and integrate it with backend services.
Must Have Technical/Functional Skills
- Strong experience in automating infrastructure provisioning using Terraform or CloudFormation (preferably with Terraform).
- Strong experience with AWS services: S3, Lake Formation, Glue, EMR, Athena, Lambda, EventBridge, SNS, SQS.
- Proficiency in Apache Spark for distributed data processing.
- Familiarity with Amazon Neptune DB for graph-based metadata management.
- Strong understanding of data lake architecture, data governance, and security best practices.
- Proficiency in SQL, Python, and Spark for data engineering tasks.
Nice to Have
- Experience with RESTful APIs.
- Knowledge of Angular is a plus.
Seniority Level
- Director
Employment Type
- Contract
Job Function
- Information Technology
Industries
- IT Services and IT Consulting
This job posting appears to be active and relevant.
#J-18808-Ljbffr
Company
Atlantis IT Group
Location
Toronto
Country
Canada
Salary
125.000
URL