Spark on Kubernetes [SPOK] with Hudson Buzby, Solutions Architect at Spot.io

No episódio de hoje, Luan Moreno, Mateus Oliveira & Tiago Xavier entrevistaram Hudson Buzby, atualmente como Arquiteto de Soluções na Spot by NetApp.

SPOK ou Spark Operator on Kubernetes, é o deployment para Apache Spark que utiliza um Operator de Kubernetes para melhor gerenciar os drivers e executors usando Kubernetes como infraestrutra escalável.

 Com SPOK, você possui os seguintes benefícios:

• Melhor utilização de recursos escaláveis.
• Infraestrutura mais leve.
• Criação de serviços Serverless {Ocean} 

 
Falamos também nesse bate-papo sobre os seguintes temas:

• História
• Spark Operator on Kubernetes
• Ocean for Apache Spark
• Dicas das trincheiras

Nesta sessão você  aprenderá a utilizar melhor o Apache Spark no Kubernetes, e entender um pouco mais  porque os clientes da Spot embarcaram nesta jornada.

Spot
Hudson Buzby
Data Mechanics
Ocean for Apache Spark

Luan Moreno =
https://www.linkedin.com/in/luanmoreno/

Chapters

• 1:35 Guest Introduction - Tiago Xavier
• 2:06 Guest Introduction - Hudson Buzby
• 3:00 Since 2018 working with Data Engineering, what is your take about Apache Spark
• 6:25 Can you tell us which project with Spark you participate in that always have a special place in your memories?
• 10:34 Tell us about the history behind Data Mechanics
• 20:56 Why offer SPOK managed as a solution for Data teams running their workloads?
• 23:58 Difficulties throughout the implementation process? Tiago Xavier External shuffle (Spot Instance)
• 32:05 What about,Streaming applications on kubernets, like atructured streaming?
• 35:36 Today at Spot.io, what is the average number of Pods scaling daily?
• 38:38 What are the significant differences between having a traditional Apache Spark deployment and going to Spark on Kubernetes?
• 45:37 YARN vs. Kubernetes
• 47:57 What are the Advantages and disadvantages of having your entire data pipeline using Spark on Kubernetes?
• 52:34 Databricks vs. AWS EMR vs. GCP Dataproc vs. Spark on Kubernetes
• 1:00:25 What is Ocean for Apache Spark?.
• 1:00:30 What are the main features of Ocean for Apache Spark
• 1:15:48 What do you use for unit testing and tell us about apache Livy
• 1:18:36 Trench Experience Apache Zeppelin - Jupyter Gateway - Jupyter Labs (Kubernetes) Autotune & Deep Learning