Role Overview
We are looking for an ETL QA Engineer with strong experience in Databricks and PySpark, combined with hands-on exposure to AI-assisted development and prompt engineering. The ideal candidate will be responsible for validating large-scale data pipelines, ensuring data quality, and leveraging tools like GitHub Copilot, GitHub Actions/Runners, and AI agents to improve testing efficiency and coverage. You will also contribute to building and validating AI-powered knowledge bases.
Required Skills & Experience
- Core
Technical Skills
- Strong hands-on experience in Databricks (notebooks, jobs, clusters, Delta Lake, SQL).
- Expert-level PySpark skills, including transformations, actions, window functions, and
optimization techniques.
- Solid understanding of ETL/ELT
concepts, data warehousing, and data modeling (star/snowflake
schemas, dimension/fact tables).
- Proven experience in data
validation, data quality checks, and reconciliation techniques.
- Practical experience with GitHub, GitHub Copilot, and GitHub Actions/Runners for CI/CD
pipelines.
- AI &
Prompt Engineering
- Experience crafting effective
prompts for large language models (LLMs) for coding, testing, and
documentation.
- Exposure to agent-based
architectures or tools (e.g., workflow/agent frameworks that
orchestrate multi-step AI tasks).
- Familiarity with AI-based
tools for code analysis, test generation, or knowledge management.
- Knowledge
Base & Documentation
- Experience building AI-driven
knowledge bases or documentation systems (e.g., using vector search,
embeddings, or LLM-based retrieval).
- Strong documentation skills,
with the ability to translate complex data/QA concepts into clear
knowledge articles.
- Testing & QA Practices
- Strong understanding of QA
methodologies: test planning, test design, defect management, and
regression testing.
- Experience with automated
testing frameworks (Python-based testing libraries such as pytest, unittest, or similar).
- Familiarity with data quality
tools/practices (e.g., validation rules, thresholds, anomaly detection).
Key Attributes
- Strong analytical and
problem-solving skills with a detail-oriented mindset.
- Passion for data quality,
automation, and continuous improvement.
- Ability to work in an agile,
fast-paced environment and collaborate across multiple teams like Business
and operations team and AD team.
- Curiosity and openness to adopting new AI tools and
practices for QA.
