Data Engineer

仕事内容

If you have passionate in big data technologies and love to organize data beautifully, this job is a perfect match for you.

This job position is responsible for establishing data platform and building data pipelines from different source systems. The objective is to make data easily accessible in our big data ecosystem. The candidates need to be familiar with Hadoop ecosystem, such as HDFS, HBase, Impala, Cassandra, and Kudu. SQL and NoSQL databases such as Oracle, ElasticSearch are also mandatory to this job.

You will work closely with data science teams and CIM DevOps teams to improve data transparency, data consistency and enable intelligent manufacturing.

Responsibilities

1. Design and optimize data structure, schema, and data cluster to improve data query performance and efficiency.
2. Deal with structured and unstructured data processing to establish data platform by ETL and/or data streaming.
3. Promote data platform to data scientists and domain users. Enrich data for data analytics, machine learning and AI applications.
4. Development and maintenance on data pipeline applications such as Kafka, ELK.
5. Study new technologies and conduct prove of concept process to solve new challenges.

Require knowledge and experience in:

• At least 2-years’ experience in Hadoop ecosystem, relational database, and data engineering
• At least 2-years’ experience in NoSQL and SQL performance tuning.
• Knowledge and experience in performance tuning for Big Data cluster.
• Knowledge and experience with the following skills is a plus:
• Distributed data processing in Spark
• Agile development, CI/CD
• Kafka
• Kubernetes
• BI and data analytics tools such as Tableau, Power BI, JMP, MATLAB
• Familiar with Git
• Good communication skills