Junior Data Engineer
Lahore, Punjab, Pakistan
Full Time
BIU Department
Mid Level
Job Title: Junior Data Engineer
Location: Lahore
Position Type: Full-Time
Job Overview:
We are seeking a Junior Data Engineer to join our data team as we build an enterprise data platform from the ground up. This role is crucial to our mission of transforming raw data into valuable insights that drive decision-making across the organization. You will be responsible for developing, managing, and optimizing data pipelines, data models, and ensuring data quality. The ideal candidate will have a solid background in SQL, Python, and experience with cloud services. Knowledge of both open source and cloud-based environments is a requirement.
Key Responsibilities:
•Develop Data Pipelines: Design, develop, and maintain scalable ETL processes to support data ingestion and transformation. Utilize Apache Kafka and AWS DMS for real-time Change Data Capture (CDC), ensuring continuous and reliable data streams into the data lake.
•Data Transformation: Leverage Apache Spark for large-scale data processing and transformation tasks within the data lake. Optimize transformation jobs for performance and cost-efficiency to ensure scalability and reliability.
•Data Modeling: Use DBT to create robust and reusable data models aligned with the business needs of various departments. Implement data marts and dimensional models to support analytical queries and reporting.
•Data Warehousing: Develop and maintain scalable data warehouses using PostgreSQL, Snowflake. Ensure the warehouse architecture supports efficient querying, reporting, and analysis.
•Change Data Capture (CDC): Develop and manage CDC processes using Kafka, ensuring that the data lake and warehouse reflect up-to-date data. Collaborate with database administrators and stakeholders to ensure reliable and performant CDC implementations.
• Metadata Management: Design and maintain a comprehensive metadata layer that facilitates dynamic pipeline creation, improving data discoverability and governance. Implement data cataloging solutions to ensure data assets are easily searchable and well-documented.
•Collaboration & Communication: Collaborate with business and data analysts to gather data requirements and translate them into technical specifications.
•Governance & Quality: Develop and maintain data quality checks to ensure data in the lake and warehouse remains accurate, consistent, and reliable.
•Research & Development (R&D): Continuously explore and evaluate new technologies, tools, and methodologies to enhance the efficiency, scalability, and reliability of the data platform. Stay updated with industry trends and best practices to apply innovative solutions to improve the data architecture.
Skills and Qualifications:
•2-3 years of proven experience as a Data Engineer or in a similar role
•Proficient in SQL and Python, with strong skills in data manipulation and processing.
•Hands-on experience with cloud services such as AWS, Azure, or GCP, especially tools like EMR, ADF, Glue, S3, CloudWatch, and DMS.
•Familiarity with data warehousing solutions
•Experience with both relational and non-relational databases and demonstrated ability to optimize query performance.
•Strong understanding of ETL processes and data pipeline development, with the ability to design and optimize workflows.
•Knowledge or Experience with Spark and DBT is a plus.
•Experience working in distributed environments, including clustering and sharding.
•Experience with streaming data processing using tools like Kafka is a strong plus.
•Experience or knowledge of building scalable applications from ground up, including server configurations.
•Knowledge of Change Data Capture (CDC) processes and data organization within a data lake.
•Understanding data cataloging and metadata management to ensure data quality and traceability.
•Excellent communication skills, with the ability to collaborate effectively across teams and explain complex concepts clearly.
•Self-motivated with strong problem-solving abilities, and a passion for continuous learning and professional growth.
How to Apply:
If you are a dedicated data warehouse infrastructure professional with a passion for protecting digital assets and a desire to work in a dynamic and collaborative environment, we encourage you to apply. Please submit your resume and a cover letter detailing your relevant experience to https://acemoneytransfer.applytojob.com/apply
ACE Money Transfer is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.
Location: Lahore
Position Type: Full-Time
Job Overview:
We are seeking a Junior Data Engineer to join our data team as we build an enterprise data platform from the ground up. This role is crucial to our mission of transforming raw data into valuable insights that drive decision-making across the organization. You will be responsible for developing, managing, and optimizing data pipelines, data models, and ensuring data quality. The ideal candidate will have a solid background in SQL, Python, and experience with cloud services. Knowledge of both open source and cloud-based environments is a requirement.
Key Responsibilities:
•Develop Data Pipelines: Design, develop, and maintain scalable ETL processes to support data ingestion and transformation. Utilize Apache Kafka and AWS DMS for real-time Change Data Capture (CDC), ensuring continuous and reliable data streams into the data lake.
•Data Transformation: Leverage Apache Spark for large-scale data processing and transformation tasks within the data lake. Optimize transformation jobs for performance and cost-efficiency to ensure scalability and reliability.
•Data Modeling: Use DBT to create robust and reusable data models aligned with the business needs of various departments. Implement data marts and dimensional models to support analytical queries and reporting.
•Data Warehousing: Develop and maintain scalable data warehouses using PostgreSQL, Snowflake. Ensure the warehouse architecture supports efficient querying, reporting, and analysis.
•Change Data Capture (CDC): Develop and manage CDC processes using Kafka, ensuring that the data lake and warehouse reflect up-to-date data. Collaborate with database administrators and stakeholders to ensure reliable and performant CDC implementations.
• Metadata Management: Design and maintain a comprehensive metadata layer that facilitates dynamic pipeline creation, improving data discoverability and governance. Implement data cataloging solutions to ensure data assets are easily searchable and well-documented.
•Collaboration & Communication: Collaborate with business and data analysts to gather data requirements and translate them into technical specifications.
•Governance & Quality: Develop and maintain data quality checks to ensure data in the lake and warehouse remains accurate, consistent, and reliable.
•Research & Development (R&D): Continuously explore and evaluate new technologies, tools, and methodologies to enhance the efficiency, scalability, and reliability of the data platform. Stay updated with industry trends and best practices to apply innovative solutions to improve the data architecture.
Skills and Qualifications:
•2-3 years of proven experience as a Data Engineer or in a similar role
•Proficient in SQL and Python, with strong skills in data manipulation and processing.
•Hands-on experience with cloud services such as AWS, Azure, or GCP, especially tools like EMR, ADF, Glue, S3, CloudWatch, and DMS.
•Familiarity with data warehousing solutions
•Experience with both relational and non-relational databases and demonstrated ability to optimize query performance.
•Strong understanding of ETL processes and data pipeline development, with the ability to design and optimize workflows.
•Knowledge or Experience with Spark and DBT is a plus.
•Experience working in distributed environments, including clustering and sharding.
•Experience with streaming data processing using tools like Kafka is a strong plus.
•Experience or knowledge of building scalable applications from ground up, including server configurations.
•Knowledge of Change Data Capture (CDC) processes and data organization within a data lake.
•Understanding data cataloging and metadata management to ensure data quality and traceability.
•Excellent communication skills, with the ability to collaborate effectively across teams and explain complex concepts clearly.
•Self-motivated with strong problem-solving abilities, and a passion for continuous learning and professional growth.
How to Apply:
If you are a dedicated data warehouse infrastructure professional with a passion for protecting digital assets and a desire to work in a dynamic and collaborative environment, we encourage you to apply. Please submit your resume and a cover letter detailing your relevant experience to https://acemoneytransfer.applytojob.com/apply
ACE Money Transfer is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.
Apply for this position
Required*