Back to Mid Level

Kafka Pub/Sub

Prerequisite

  • Prior experience or knowledge of Kafka
  • Prior experience or knowledge of streaming applications

Challenge

You’ve been hired by Storm Rebels Crop. Storm Rebels is a team of highly trained storm chasers driving around in a purpose built vehicle called the dominator. The dominator has gadgets on it that record temperature every second. You’ve been tasked to write a streaming pipeline that takes the temperature readings and stores them into a Kafka topic.

Tasks
  • Get Kafka and PostgreSQL running locally using docker compose. We’ve prepared a docker-compose.yml file for you that will spin up PostgreSQL and Kafka. It is available here. The YAML file includes instructions on how to access Kafka and PostgreSQL from inside/outside docker.
  • After you do docker-compose up you should have the following containers up
➜IccccpMoooooAnnnns~GfffftEllllgduuuuroeeeeecnnnnsktttt:eiiiilrnnnnacccctp////esccccsppppt-----esszfnceoothroreevkmrmeeaparetr-:pir7e"se.rteg0:a-i.7bcs0.lot0enr.ty0{r:{o7.l.I-0mc.ae0gnet}e}r\:t7{.{0..P0ortsP00020}O...8.}R00080\T...8.tS0000{...t.{000c0.:::p:N989,5a0004m27903e112.2s---0-}>>>.>}98905"000.428903112:2///2/ttt1tccc8cppp1p,,,-,>:::2::::1::::8:98915000/4279t3112c2---p->>>,>9895000:4289:3112:2///2/ttt1tccc8cppp1p,->02.108.10.t0c:p9,10318-8>89/1t0c1p/tcp,:::9101->9101/tcpNcsbzpAocrooMnhoosEtekktSrmeegoarerl-pe-resceregnitsetrry
Requirements
  • Write a microservice in language of your choice that publishes randomly generated temperature readings in Celcius to a kafka topic called (celcius-readings) every 1 second. The data in the topic should be: a double celcius reading and a long epoch timestamp. Example: 44.56:1638841414
  • Write a microservice in language of your choice that subscribes to the celcius-readings kafka topic, converts it to fahrenheit and inserts it into a PostgreSQL table called storm_chasers_data. Be creative about what columns should be in the table.
  • Write a SQL query that finds the two hottest and two coldest temperatures in a single query.
  • The entire solution (including microservices) should be in a single docker-compose.yml so we can run the entire thing with one command: docker-compose up

Deliverable

  • A GitHub repo with read permissions given to GitHub users rafty8s,bsneider, omnipresent07, and barakstout (how to invite collaborators)
  • The repo should have enough README.md instructions
  • A docker-compose.yml using which everything can be spun up
  • Any additional commands that need to be executed