Overall Experience level:
7 years of recent GCP experience
Apache Hudi for big data processing and storage
7 years of handson experience Hadoop Hive or Spark Airflow or a workflow orchestration solution
Experience with programming languages: Python Java Scala etc.
Experience with scripting languages: Perl Shell etc.
is looking for a highly energetic and collaborative Data Engineer for a 12month engagement.
Responsibilities:
Responsibilities
Design develop and maintain robust and scalable ETL workflows and data pipelines using tools like Hive Spark and Airflow.
Implement and manage data storage and processing solutions using Apache Hudi and BigQuery.
Develop and optimize data pipelines for structured and unstructured data in GCP environments leveraging GCS for data storage.
Write clean maintainable and efficient code in Scala and Python to process and transform data.
Ensure data quality integrity and consistency by implementing appropriate data validation and monitoring techniques.
Work with crossfunctional teams to understand business requirements and deliver data solutions that drive insights and decisionmaking.
Troubleshoot and resolve performance and scalability issues in data processing and pipelines.
Stay updated with the latest developments in big data technologies and tools and incorporate them into the workflow as appropriate.
Required Skills and Qualifications
Proven experience as a Data Engineer preferably in a big data environment.
Expertise in Hive Spark and Apache Hudi for big data processing and storage.
Handson experience with BigQuery and Google Cloud Platform (GCP) services such as GCS Dataflow and Pub/Sub.
Strong programming skills in Scala and Python with experience in building data pipelines and ETL processes.
Proficiency with workflow orchestration tools like Apache Airflow.
Solid understanding of data warehousing concepts data modelling and schema design.
Knowledge of distributed systems and parallel processing.
Strong problemsolving skills and ability to work with large datasets in a fastpaced environment.
| Job Description: Expectations from this role: Act creatively to develop applications and select appropriate technical options optimizing application development maintenance and performance by employing design patterns and reusing proven solutions account for others developmental activities 1. Interpret the application/feature/component design to develop the same in accordance with specifications. 2. Code debug test document and communicate product/component/feature development stages. 3. Validate results with user representatives; integrates and commissions the overall solution 4. Select appropriate technical options for development such as reusing improving or reconfiguration of existing components or creating own solutions 5. Optimises efficiency cost and quality. 6. Influence and improve customer satisfaction 7. Set FAST goals for self/teamTypical performance measures: 1. Adherence to engineering process and standards (coding standards) 2. Adherence to project schedule / timelines 3. Number of technical issues uncovered during the execution of the project 4. Number of defects in the code 5. Number of defects post delivery 6. Number of non compliance issues 7. On time completion of mandatory compliance trainingsPerformance Areas: Code as per design Follow coding standards templates and checklists Review code for team and peers |
...you! Serenity is seeking a Receptionist for our Lake Mary, FL office; to join our team and drive success within our clinic. The position... ...people. Benefits~ Career Advancement Opportunity ~ Medical, Vision, and Dental Insurance(Serenity Covers 90% of your insurance...
...Hi Greetings from Conch Technologies Inc Position: Salesforce.com / Apex (SFDC) Location: New York City NY ( Local ) Duration : 12 Months Contract or Direct Fulltime Interview: In Person is must after 1st round Apex: 40% Lightning...
...is $12.00 cash or $14.00 check. There are lots of tips. This is an easy job for anyone.Saladworks(located at the corner of River Road & Edgewater in the bagel store to the right of the Animal Hospital)725 River RoadEdgewater, New Jersey 07020(***) ***-****(7...
...cardiology* Join large 39 provider practice that consists of 25 physicians and 14 APPs with 10 office locations throughout North Central... ...:4/5 weeks. STEMI call 1:4/5 weeks. * deal candidate would be Nuclear and Echo certified.* The incoming provider will be housed in...
...Language Trainers is a successful language training company working with freelance teachers of 70 different languages in hundreds of... ...Language, Anytime, Anywhere! We might have a job for you as a German language teacher. ID Reference1020498 Some details about...