We are seeking a skilled Mid-Level Databricks Engineer to join our data engineering team. The ideal candidate will play a key role in the migration of Hadoop workloads to the Databricks Lakehouse platform, contributing to the development, optimization, and monitoring of scalable and repeatable data solutions. You will work closely with cross-functional teams to ensure successful migration, enablement of advanced Databricks features, and ongoing performance improvements.
Key Responsibilities
Identify and categorize existing Hadoop jobs (ETL, batch, streaming) and data sources to support migration planning.
Participate in selecting scalable and repeatable migration use cases for Minimum Viable Product (MVP) initiatives.
Provision and configure Databricks workspaces, ensuring Lakehouse architecture and federation capabilities are enabled.
Execute pilot migrations of representative Hadoop workloads using tools such as Databricks Migration Accelerator or partner solutions.
Validate and monitor post-migration performance, cost efficiency, and data integrity.
Enable and test advanced Databricks features (e.g., Liquid Clustering, Lakehouse AI monitoring, Serverless warehouse, Unity Catalog) to optimize data layout and pipeline health.
Engage with data engineering, analytics, and governance teams to gather feedback and document learnings, blockers, and feature gaps.
Track and report on key success metrics such as migration time, query latency, cost savings, and feature adoption.
Contribute to the development of a phased roadmap for full-scale migration and advanced feature rollout.
Qualifications
Bachelor's degree in Computer Science, Information Technology, or a related field, or equivalent professional experience.
5-8 years of experience in data engineering, with hands-on experience in Databricks and Hadoop ecosystems.
Proficiency in Spark (Scala or PySpark), ETL pipeline development, and data migration practices.
Experience with Lakehouse architecture, Delta Lake, and Databricks advanced features (e.g., Lakehouse Federation, Liquid Clustering, Unity Catalog) is a strong plus.
Solid understanding of cloud platforms (Azure) and data governance concepts.
Strong analytical and problem-solving skills with attention to detail.
Ability to work collaboratively in cross-functional teams and communicate effectively with both technical and non-technical stakeholders.
Preferred Skills
Experience with Databricks Migration Accelerator or similar migration tools.
Familiarity with synthetic data generation and pipeline health monitoring.
Understanding of performance tuning, query optimization, and cost management on cloud data platforms.
Ability to document migration patterns and contribute to best practices for broader adoption.
What We Offer
Opportunity to work with cutting-edge data technologies and drive impactful migration projects.
...Nightingale Nurses is seeking a travel nurse RN Med Surg / Telemetry for a travel nursing job in Winchester, Virginia. Job Description... ...travel nursing assignments all over the US, as well as the highest pay rates and most extensive benefits in the industry....
...tree trimming, and pruning, to ensure clear sightlines and safe road conditions. Clean and grade ditches, stormwater drainage... ....E.D. Previous experience in road maintenance, landscaping, construction, or a related field is a plus, but no prior training will be provided...
...screen Must be able to read, write and speak English Are felony and conviction-free We offer Warehouse Associates: ~ Full-time... ...Pay increase upon hire ~$100 referral bonus when you refer a friend to work with WSI ~ Weekly pay and medical benefits ~ A...
...schedule ~ Free uniforms ~ Health insurance ~ Opportunity for advancement ~ Training & development Fish Window Cleaning in Warminster, PA is currently hiring for window cleaners in the BucksCo/MontCo/NE Philly area. We are looking for courteous, self-...
We are currently seeking home providers in: Iola / Allen County, KS Are you looking for a meaningful work opportunity in your home? Become a Host Home / Shared Living provider with Mosaic at Home! Average annual income of $47,000 (federally tax-free) plus room & board...