Spark Streaming with Scala (Updated)
Разное | Автор: LeeAndro | Добавлено: 14-09-2020, 05:30 | Просмотров (6) | Комментариев (0) | Жалоба |
Spark Streaming with Scala (Updated)
MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHz, 2 Ch
Genre: eLearning | Language: English | Duration: 31 Lessons (11h 18m) | Size: 1.82 GB

Stream big data in real with Spark and integrate any data source, from Kafka to Twitter.

You'll learn how Spark Structured Streaming and "normal" Spark batch operations are similar and different

You'll work with new streaming abstractions (DStreams) for low-level, high-control processing

You'll integrate Kafka, JDBC, Cassandra and Akka Streams (!) so that you can later integrate anything you like

You'll work with powerful stateful APIs that only a few know how to properly use

And some extra perks:

You'll have access to the entire code I write on camera (2200+ LOC)

You'll be invited to our private Slack room where I'll share latest updates, discounts, talks, conferences, and recruitment opportunities

(soon) You'll have access to the takeaway slides

(soon) You'll be able to the videos for your offline view

Same comfort with Spark Structured Streaming APIs as with "normal" Spark batch:






High control over how data is processed with DStreams:

map, flatMap, filter


by-key operations

process each RDD individually

Ability to work with columns and window functions, both on structured and low-level streams

sliding windows

tumbling windows

reduce by window

reduce by window and key

Integration between Spark and other data sources, including

Kafka (structured and low-level)



and something that's not "natural" to Spark, like Akka

Ability to manually manage stateful data processing in ways SQL is incapable of



This course is for Scala and Spark programmers who need to process streaming data rather than one- or batch. If you've never done Scala or Spark, this course is not for you.

Project 1: Twitter

In this project we will integrate live data from Twitter. We will create a custom data source that we use with Spark, and we will do various analyses: tweet lengths, most used hashtags in real . You will be able to use this project as a blueprint for any data source that you might want to integrate. At the very end, we will use an NLP library from Stanford to do sennt analysis on tweets and find the general state of social media.

You will learn:

how to set up your own data receiver, that you can manage yourself and "pull" new data

how to create a DStream from your custom code

how to pull data from Twitter

how to aggregate tweets

how to use Stanford's coreNLP library for sennt analysis

how to apply sennt analysis on tweets in real

Project 2: A Science Project

In this project we will write a full-stack web application which will support multiple users that are test subjects of a scientific test. We will investigate the effects of alcohol/substances/insert_your_addictive_drug_like_Scala on reflexes and response s. We will send the data through a web UI connected to a REST endpoint, then the data will flow through a Kafka broker and finally to a Spark Streaming backend which will do the data crunching. You can use this application as a blueprint for any full-stack application that aggregates and processes data with Spark Streaming in real , from any number of concurrent users.

You will learn:

how to set up an HTTP server in minutes with Akka HTTP

how to manually send data through Kafka

how to aggregate data in a way that's almost impossible in SQL

how to write a full-stack application with a web UI, Akka HTTP, Kafka and Spark Streaming



Уважаемый посетитель, Вы зашли на сайт как незарегистрированный пользователь. Мы рекомендуем Вам зарегистрироваться либо войти на сайт под своим именем.
  • 0
Похожие новости:
Посетители, находящиеся в группе Гости, не могут оставлять комментарии к данной публикации.
Панель управления
На сайте
Пользователей Юзеры (1)
Гостей Гости (29)
Роботов Боты (2)
crawl Botbot
Всего Всего на сайте (32)
Не попавшее на главную
Архивы сайта