Unit 5_AI Software Ecosystems

Mlamu w3*sKsooa e.lemuos VIOIAU.

Outline vGPU Foundation Frameworks for Deep Learning Overview of the software stack CUDA-X A1 NGC NVIDIA A1 Enterprise Software Suite.

Objectives By the end of this unit, you will be able to: Understand vGPU as a foundational technology for A1 Describe the NVIDIA deep learning software stack and the NVIDIA CUDA-X ecosystem Define the steps in an A1 pipeline workflow Identify open-source, 3rd party, and NVIDIA frameworks Describe the benefits of the NGC and the Enterprise Catalog.

GPU Virtualization (vGPU).

NVIDIA Virtual GPU Software GPU virtualization for every workload Unprecedented challenges and how digital technologies have helped respond to them. End-user Computing $20B By 2030 Storage and Compute 40% Shift 000 t2:2 Improved User Experience and Security, IT Executive reports Increasing Investment 49% Looking to Improve.

• NVIDIA Virtual GPU Software GPU virtualization for every workload Nvidia's vGPU technology allows IT to deliver graphics-rich virtual experiences across their user base. Application and desktop virtualization solutions' point of failure is user experience. vGPUs virtualize a GPU and share it across multiple VMs, improving performance and creating new opportunities. Our portfolio includes: Browsers HD Video (Zoom, Webex, Skype) Workstations (CAD, Revit, Maya) GIS Apps (Esri ArcGIS Pro) 0 0 0 00 0 Manufacturing (CATIA, Siemens NX, Solidworks) Oil & Gas (petrel) Healthcare (Epic) Financial Apps (Bloomberg).

NVIDIA Virtual GPU Software NVIDIA Virtual PC Office Productivity, Knowledge Worker Workloads GPU virtualization for every workload NVIDIA Virtual Applications App Streaming with Citrix Virtual Apps or Other ROSH Solutions NVIDIA RTX Virtual Workstations Performance Graphics.

NVIDIA Virtual GPU NVIDIA virtual GPU delivers GPU acceleration to every visual workload With NVIDIA Virtual GPU CPU only Virtualization Apps and VMS Hypervisor Server 01011 11010 Apps and VMS NVIDIA Graphics Driver NVIDIA RTX Enterprise Driver ..„ONVIDIA Virtual GPU •••••••'NVIDIA Virtualization Software Hypervisor GPU Server.

Benefits of GPU Virtualization NVIDIA in virtualized environments—industry-leading innovations Bare-Metal Performance o— o— o— Business Continuity & Workload Balancing Operational Management Resource Sharing & Improved Utilization Insights & Tools Infrastructure & Data Security.

Frameworks for Deep Learning.

Labeled Training Data Deep Learning Approach Deep Neural Network "Model" Object Class Predictions Input Prediction Error (Label - Prediction) Back-propagate errors for parameter update O c Tractor Truck Car.

Steps in A1 Workflow Steps and tools for A1 workflow o a o o Data Processing Process of preparing raw data and making it more suitable for machine learning model NVIDIA RAPIDS NVIDIA RAPIDS Accelerator for Apache Spark Training Teaching A1 to accurately interpret and learn from data to perform a task with accuracy pyTorch NVIDIA TAO Toolkit TensorFlow Optimization Iteratively improving machine learning model accuracy to reduce error TensorRT Inferencing/Deployment Making models available to other systems so they can receive data and return predictions NVIDIA Triton Inference Server.

What are ML/DL Frameworks Essential tools for data scientists, researchers, and engineers Frameworks? Machine learning and deep learning frameworks are building blocks for designing, training, and validating machine learning models and deep neural networks, through a high-level programming interface. Framework use cases Computer Vision, Natural Language Processing, Speech and Audio Processing, Robot Learning, And more... Existing Frameworks Caffe @xnet nVlDlA. Isaac Lab Chainer PaddlePaddle Cognitive Toolkit PYTbRCH Spari( MATLAB TensorFlow.

NVIDIA Deep Learning Software Stack — NVIDIA's groundbreaking parallel • CUDA programming model Enables GPUs to be • NVIDIA Container Runtime used inside containers — Publicly available containers • NGC Containers optimized to run on NVIDIA GPUs • DL Frameworks — Popular deep learning frameworks available inside the containers • Provides essential optimizations for deep learning, machine learning, and high-performance computing (HPC) leveraging NVIDIA GPUs Containerized Tool Docker Engine NVIDIA Driver Host OS.

NVIDIA Deep Learning Software Stack — NVIDIA's groundbreaking parallel • CUDA programming model Enables GPUs to be • NVIDIA Container Runtime used inside containers — Publicly available containers • NGC Containers optimized to run on NVIDIA GPUs • DL Frameworks — Popular deep learning frameworks available inside the containers • Provides essential optimizations for deep learning, machine learning, and high-performance computing (HPC) leveraging NVIDIA GPUs A range of interfaces can be used Deep Learning Frameworks Deep Learning Libraries CUDA Toolkit Mounted NVIDIA Driver Container OS Containerized Tool NVIDIA Container Runtime For Docker Docker Engine NVIDIA Driver ...4 Host OS.

How Do I Build An A1 Platform? Two ways to build an A1 platform Do It Yourself (DIY) NVIDIA A1 Enterprise.

How Do I Build An A1 Platform? Two ways to build an A1 platform Do It Yourself (DIY) O O Leveraging open- source software Can be collaboratively shared and modified. Can be risky without support for production A1 Limited to current GPU architecture.

Build applications on top of dedicated NVIDIA Platform How Do I Build An A1 Platform? Two ways to build an A1 platform NVIDIA A1 Enterprise Provides hardware testing, support for enterprises Deployable on future GPU architectures.

NGC and the Enterprise catalog.

NGC Catalog - GPU Optimized hub for A1 & HPC Software Simplify and accelerate end-to-end workflows Containers HPC I ML On-Perm Pre-Trained Models COMPUTER VISION I NLP I DLRM Cloud Industry App Frameworks Ill, Hello O CLARA I RIVA I ISAAC Helm Charts O TRITON I GPU OPERATOR ARM POWER Hybrid Cloud Collections CLARA DISCOVERY I TLT-JARVIS I RECSYS Edge Get started with the NGC catalog at https://ngc.nvidia.com.

Fast Track A1 with Pre — nv101A NGC I CATAXOG PeopleNet Production Quality Trained and continuously updated by experts Model resumes to find the right fit PeopleNet Model Card trained Models from NGC 2020 Limitations Very Small Objects WI mA to OWtS l&ger than 1 Ox' O Therefore t Occluded Objects When objects t' that less is th" be and shov"ers e if the RfSOdS and/or are not Dark-lighting, Monochrome or Infrared Camera Images rhe trained on B in in Warped and Blurry Images not on Ot Face and Bag class kha.h bag •nd dass ate imiuded the mooa. the accuracy of these classes be much than to pog.protns.ed OBWAN N MS to and Wide Range Of Use Cases People Detection, Vehicle Detection & Gaze Estimation Intent Classification, Question-Answering, Speech Recognition and Text to Speech Adapt & Integrate Adapt your domain with custom data Integrate easily into industry SDKs.

Containers Enable You to Focus on Building A1 omn kin=tica CORE Enterprise-Ready Software Scanned for CVEs, malware, crypto Tested for reliability Backed by Enterprise support Performance Optimized Scalable Updated monthly Better performance on the same system Deploy Anywhere Docker I cri-o I Containerd I Singularity Bare metal, VMs, Kubernetes Multi-cloud, on-prem, hybrid, edge.

NVIDIA A1 Enterprise.

NVIDIA A1 Platform NVIDIA A1 enterprise is the software layer of the most advanced A1 platform NVIDIA A1 A1 Foundation Models & Services A1 Platform Software Accelerated Infrastructure NVIDIA A1 Enterprise.

NVIDIA A1 Enterprise End to end A1 software includes over 50 frameworks and pretrained models A1 Workflows, Frameworks and Pretrained Models* Medical Imaging Speech A1 Conversational A1 Recommenders Communi Video Logistics Analytics Robotics A1 and Data Science: Development and Deployment Tools Cloud Native Management and Orchestration Infrastructure Optimization Accelerated Infrastructure Autonomous Cybersecurity Vehicles Embedded 1 OFFERING Cloud Native, Hybrid Optimized Deploy anywhere - on-prem and in the cloud Reduce OSS development complexity Secure and Scalable Certifications with broad partner ecosystem • Improved A1 model accuracy Standard Support 9 x S, Premium 24x7 Cloud Data Center Edge *NVIDIA NGC public catalog provides a complete listing of over 50 supported frameworks and pretrained models..

Application Workflows SDKs, pre-trained models, and frameworks M LIN MODULUS MAXINE CLARA: A1 Applications and frameworks for healthcare and medical imaging. RIVA: Multilingual Speech and translation A1 software development kit. TOKKIO: Framework to build and deploy Al- powered digital assistants and avatars. MERLIN: Framework for building high- performing recommender systems at scale. MODULUS: Physics ML platform that blends physics with deep learning training data. MAXINE: A1 SDKs and cloud-native microservices for deploying A1 features that enhance audio, video and reality effects. . METROPOLI CUOPT NEMO' ISAAC DRIVE MORPHEUS METROPOLIS: Application framework to bring visual data and A1 together. CUOPT: Operations Research API using A1 to create complex, real-time fleet routing workflows. NEMO: Framework to build, customize, and deploy generative A1 models. ISAAC: Framework to build modular robotics applications. DRIVE: Framework to help collect data, train deep neural networks, test, validate and operate Autonomous Vehicles. MORPHEUS: Framework that enables cybersecurity developers to create Optimized pipelines for filtering, processing and classifying data..

NVIDIA A1 Workflows.

Terminology Explained Workload vs. workflow O o Workload O O $3 O Any application, microservice, or function that is . standalone, or as a part of a workflow, that uses compute resources to accomplish a task or output results. Data science, A1, and 3D graphics workloads can be accelerated by frameworks and libraries that leverage NVIDIA GPUs. Examples: Spark jobs, models doing video analytics, training a large language model, a text-to-speech function, video rendering Workflow Multi-step process to get from initiation to completion, where each step is a workload. For example, the generic workflow of A1 is data prep > training > simulation > inference. NVIDIA A1 workflows are assembled, tested, documented, and customizable to provide a partners and customers a head start in solving specific challenges. Examples: Audio transcription, digital fingerprinting to detect cybersecurity threats, contact center intelligent virtual assistant.

NVIDIA A1 Workflows Prepackaged reference applications to rapidly automate your business with A1 Intelligent Virtual Assistant 000 Engaging contact center assistance 24/7 for lower operational costs Cloud Audio Transcription World-class, accurate transcripts based on GPU-optimized models Digital Fingerprinting Threat Detection 6 Cybersecurity threat detection and alert prioritization to identify and act faster Next Item Prediction Personalized product recommendations for increased customer engagement and retention NVIDIA A1 Enterprise Data Center Route Optimization Vehicle and robot routing optimization to reduce travel times and fuel costs Edge o Generative A1 Knowledge base Embedded o.

A1 Workflows Accelerate the Path to A1 Outcomes Reduce the cost of developing and deploying A1 solutions Accelerate Development & Deployment Prepackaged, customizable reference applications include best-in-class A1 software with cloud-native deployable packaging Improve Accuracy & Performance Frameworks and containers performance-tuned and tested for NVIDIA GPUs Gain Confidence in A1 Outcomes Enterprise-grade support.

Unit Summary.

Summary Now that you have completed this unit, you should be able to: Define Virtual GPU (vGPU) Describe NVIDIA deep learning software stack and NVIDIA CUDA-X Ecosystem Define the steps in A1 pipeline workflow Define and identify open source, 3rd party, and NVIDIA frameworks Describe the benefits of NGC and Enterprise catalog Describe the benefits and use cases of NVIDIA A1 Enterprise Describe NVIDIA's A1 Workflows.

Accelerating A1 with GPUs Unit 4 Coming Up Next Continue the journey by taking the next unit! Data Center and Cloud Computing Unit 6 A1 Software Ecosystem Unit 5 Compute Platforms for A1 Unit 7.

nVlDlA nVlDIA.