Scaling the AI Hierarchy of Needs with Hopsworks

About

Hopsworks is an open-source data platform that integrates popular platforms for data processing such as Apache Spark, TensorFlow, Hops Hadoop, Kafka, and many others. All services provided by Hopsworks can be accessed using either a REST API or a User Interface. But the real value add of Hopsworks is that it makes Big Data and AI frameworks easier to use by introducing new concepts for collaborative data science (Projects, Users, and Datasets) and ubiquitous support for TLS certificates, opening the platform for integration with the outside world (IoT/mobile devices and external applications).

In this talk, we will discuss Hopsworks and how we are using it to provide Spark/TensorFlow/Hadoop-as-a-Service to hundreds of researchers in Sweden from the Rise ICE Data Center at www.hops.site. In particular, we will examine distributed TensorFlow both for parallel experimentation and distributed training, and how we built a cost-effective distributed deep learning platform-as-a-service on commodity Nvidia GeForce GPUs (1080Ti).

About the speaker

Jim Dowling is as an Associate Professor at KTH Royal Institute of Technology in Stockholm, the CEO of Logical Clocks AB, as well a Senior Researcher at SICS RISE. His research concentrates on building systems support for machine learning at scale. He is the lead architect of Hops Hadoop, the world's fastest and most scalable Hadoop distribution and only Hadoop platform with support for GPUs as a resource. He is a regular speaker at Big Data and AI industry conferences, and blogs at O'Reilly on AI. .

0:00:00 Slide 1

0:03:22 Slide 2

0:04:09 Slide 3

0:04:47 Slide 4

0:05:36 Slide 5

0:05:42 Slide 6

0:05:55 Slide 7

0:06:03 Slide 8