<\> Casa do Dev
Notícias abrir
Ferramentas abrir
Vagas abrir
Aprender abrir
Comunidade abrir
Guia de Compras abrir
Vaga aberta publicada em 26/09/2026

Senior Data Engineer

Aptonet · Brasil

PJ Senior Remoto
Compartilhar WhatsApp LinkedIn Telegram X

Descrição da vaga

Job Summary

As a Senior Data Engineer, you will be responsible for designing, building, and optimizing big data pipelines. You will work closely with data scientists and other stakeholders to support the data needs of the organization.

Responsibilities

  • Design and implement high-velocity, high-volume data streaming solutions using Apache Kafka and Spark Streaming.
  • Develop real-time data processing and streaming techniques using Spark Structured Streaming and Kafka.
  • Troubleshoot and optimize Spark applications for peak performance.
  • Work closely with Python and/or Scala for data processing tasks (PySpark/Scala-Spark).
  • Utilize Databricks for cloud-based big data solutions.
  • Build, test, and optimize big data ingestion pipelines, architectures, and datasets.
  • Deploy data platforms on Azure or AWS and manage serverless technologies like S3, Kinesis/MSK, Lambda, and Glue.
  • Operate messaging platforms like Kafka, Amazon MSK, TIBCO EMS, or IBM MQ Series for asynchronous data communication.
  • Manage Databricks Notebooks, work with Delta Lake using both Python and Spark SQL, and manage Delta Live Tables and the Unity Catalog.
  • Ingest data from various formats including JSON, XML, and CSV.
  • Work with NoSQL databases like HBASE and/or Cassandra.
  • Perform shell scripting and other tasks on Unix/Linux platforms.
  • Work with other database solutions like Kudu/Impala or Delta Lake.

Qualifications - must have

  • Must have hands-on experience with high-velocity, high-volume stream processing: Apache Kafka and Spark Streaming.
  • Experience with real-time data processing and streaming techniques using Spark structured streaming and Kafka.
  • Deep knowledge of troubleshooting and tuning Spark applications.
  • Must have hands-on experience with Python and/or Scala (PySpark/Scala-Spark).
  • Must have experience with Databricks.
  • Must have hands-on experience building, testing, and optimizing big data ingestion pipelines, architectures, and datasets.
  • Experience in successfully building and deploying a new data platform on Azure/AWS.
  • Experience in Azure/AWS Serverless technologies, like S3, Kinesis/MSK, Lambda, and Glue.
  • Strong knowledge of Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series.
  • Experience with Databricks UI, Managing Databricks Notebooks, Delta Lake with Python, Delta Lake with Spark SQL, Delta Live Tables, Unity Catalog.
  • Experience with data ingestion of different file formats like JSON, XML, CSV.
  • Experience with NoSQL databases, including HBASE and/or Cassandra.
  • Knowledge of Unix/Linux platform and shell scripting is a must.
  • Experience with database solutions like Kudu/Impala, or Delta Lake.

Deixe seu preparo a postos

Enquanto voce se prepara para vagas como esta, um setup confortavel e material de estudo ajudam no dia a dia. Selecionamos itens que combinam com o momento:

Contem links de afiliados. Comprando por eles voce apoia o portal sem pagar nada a mais.

Responsabilidades

  • Design and implement high-velocity, high-volume data streaming solutions using Apache Kafka and Spark Streaming.
  • Develop real-time data processing and streaming techniques using Spark Structured Streaming and Kafka.
  • Troubleshoot and optimize Spark applications for peak performance.
  • Work closely with Python and/or Scala for data processing tasks (PySpark/Scala-Spark).
  • Utilize Databricks for cloud-based big data solutions.
  • Build, test, and optimize big data ingestion pipelines, architectures, and datasets.
  • Deploy data platforms on Azure or AWS and manage serverless technologies like S3, Kinesis/MSK, Lambda, and Glue.
  • Operate messaging platforms like Kafka, Amazon MSK, TIBCO EMS, or IBM MQ Series for asynchronous data communication.
  • Manage Databricks Notebooks, work with Delta Lake using both Python and Spark SQL, and manage Delta Live Tables and the Unity Catalog.
  • Ingest data from various formats including JSON, XML, and CSV.
  • Work with NoSQL databases like HBASE and/or Cassandra.
  • Perform shell scripting and other tasks on Unix/Linux platforms.
  • Work with other database solutions like Kudu/Impala or Delta Lake.

Requisitos

Must have hands-on experience with high-velocity, high-volume stream processing: Apache Kafka and Spark Streaming. Experience with real-time data processing and streaming techniques using Spark structured streaming and Kafka. Deep knowledge of troubleshooting and tuning Spark applications. Must have hands-on experience with Python and/or Scala (PySpark/Scala-Spark). Must have experience with Databricks. Must have hands-on experience building, testing, and optimizing big data ingestion pipelines, architectures, and datasets. Experience in successfully building and deploying a new data platform on Azure/AWS. Experience in Azure/AWS Serverless technologies, like S3, Kinesis/MSK, Lambda, and Glue. Strong knowledge of Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series. Experience with Databricks UI, Managing Databricks Notebooks, Delta Lake with Python, Delta Lake with Spark SQL, Delta Live Tables, Unity Catalog. Experience with data ingestion of different file formats like JSON, XML, CSV. Experience with NoSQL databases, including HBASE and/or Cassandra. Knowledge of Unix/Linux platform and shell scripting is a must. Experience with database solutions like Kudu/Impala, or Delta Lake.

Dados completos da vaga

Empresa
Aptonet
Modelo de trabalho
Remoto
Nível
Senior
Tipo de contratação
PJ
Localização
Brasil
Método de candidatura
Candidatura por link externo

Como se candidatar

Acesse a página de candidatura para ver os canais disponíveis.

Ir para página de candidatura