FuriosaAI
FuriosaAI 图标

FuriosaAI

furiosa.ai

FuriosaAI designs high-performance, power-efficient AI accelerators (NPUs) used in data centers for computer vision, GenAI, LLMs, and demanding workloads.

inferenceefficiententerprisepowerfullycloud
收录于 1 年前更新于 1 年前正常收录
FuriosaAI 网站预览

详细内容

Frontpage

Redcard cropped

Furiosa RNGD - Gen 2 data center accelerator

Powerfully efficient AI inference for enterprise and cloud

View specs

Get notified first

# 1 E F F I C I E N T L L A M A I N F E R E N C E

token/s/W

Llama70 B 31

token/s/W

Llama8 B 31

Llama 3.1 70B

2,048 input tokens / 128 output tokens / x 8 cards

rngd

FuriosaSDK / FP8 / 957.05 token/s

h100 sxm

TensorRT-LLM 0.15.0 / FP8 / 2,064.53 token/s

l40s

TensorRT-LLM 0.15.0 / FP8 / 163.53 token/s

Llama 3.1 8B

128 input tokens / 4,096 output tokens / x 1 card

rngd

FuriosaSDK / FP8 / 3,935.25 token/s

h100 sxm

TensorRT-LLM 0.15.0 / FP8 / 13,222.06 token/s

l40s

TensorRT-LLM 0.15.0 / FP8 / 2989.17 token/s

RNGD H100 SXM L40S
Technology TSMC 5nm TSMC 4nm TSMC 5nm
BF16/FP8 (TFLOPS) 256/512 989/1979 362/733
INT8/INT4 (TOPS) 512/1024 1979/- 733/733
Memory Capacity (GB) 48 80 48
Memory Bandwidth (TB/s) 1.5 3.35 0.86
Host I/F Gen5 x16 Gen5 x16 Gen4 x16
TDP (W) 180 700 350

Disclaimer: Measurements by FuriosaAI internally on current specifications and/or internal engineering calculations. Nvidia results were retrieved from Nvidia website, https://github.com/NVIDIA/Tens..... /perf-overview.md, on Aug 25, 2024.

I N F E R E N C E W I T H O U T C O N S T R A I N T S

Performance

Deploy the most capable models with low latency and high throughput

Efficiency

Lower total cost of ownership with less energy, fewer racks, and air-cooled data centers of today

Programmability

Stay future-proof for tomorrow’s models and transition with ease

Learn more about RNGD

E F F I C I E N T A I I N F E R E N C E I S H E R E

RNGD (pronounced "Renegade") delivers high-performance LLM and multimodal deployment capabilities while maintaining a radically efficient 180W power profile.

512TFLOPS

64TFLOPS (FP8) x 8 processing elements

48GB

HBM3 memory capacity

2 x HBM3

CoWoS-S, 6.0Gbps

256MB SRAM

384TB/s on-chip bandwidth

1.5TB/s

HBM3 memory bandwidth

180W TDP

Targeting air-cooled data centers

PCIe P2P support for LLM
BF16, FP8, INT8, INT4 support

Multiple-instance and virtualization
Secure boot & model encryption

Tensor contraction, not matmul

Tensor Contraction Processor (TCP)

At the heart of Furiosa RNGD is Tensor Contraction Processor architecture (ISCA 2024), specifically designed for efficient tensor contraction operations.

The fundamental computation of modern day deep learning is tensor contraction, a higher dimensional generalization of matrix multiplication. However, most commercial deep learning accelerators today incorporate fixed-sized matmul instructions as primitives.

RNGD breaks away from that, unlocking powerful performance and efficiency.

Download TCP Paper

Learn more about TCP

Tensor mapping for max utilization

We elevate the programming interface between hardware and software to treat tensor contraction as a single, unified operation.

This fundamental design choice streamlines programming, maximizing parallelism and data reuse, while providing flexibility and reconfigurability of compute and maximizes memory resources based on tensor shapes.

Furiosa Compiler leverages this flexibility and reconfigurability of hardware to select the most optimized tactics, delivering powerful and efficient deep learning acceleration for all scales of deployment.

Tensor Contraction Processor

TCP is the compute architecture underlying Furiosa accelerators. With tensor operation as the first-class citizen, Tensor Contraction Processor (TCP) unlocks unparalleled energy efficiency.

S O F T W A R E F O R L L M D E P L O Y M E N T

Furiosa SW Stack consists of a model compressor, serving framework, runtime, compiler, profiler, debugger, and a suite of APIs for ease of programming and deployment. Available now.

Built for advanced inference deployment

Comprehensive software toolkit for optimizing large language models on RNGD. User-friendly APIs facilitate seamless state-of-the-art LLM deployment.

Maximizing data center utilization

Ensure higher utilization and flexibility for small and large deployments with containerization, SR-IOV, Kubernetes, as well as other cloud native components.

Robust ecosystem support

Effortlessly deploy models from library to end-user with PyTorch 2.x integration. Leverage the vast advancements of open-source AI and seamlessly transition models into production.

B E I N T H E K N O W

Sign up to be notified first about RNGD availability and product updates.

Sign up

Blog

See all posts

How PyTorch handles dynamic tensor shapes\
\
Our Viewpoints

Serving gpt-oss-120b at 5.8 ms TPOT with two RNGD cards: compiler optimizations in practice\
\
Our Viewpoints

Introducing Furiosa NXT RNGD Server: Efficient AI inference at data center scale\
\
News

See all posts

Get the latest from Furiosa AI

Sign up for news, announcements, and more.

First name

Last name

Organization name

Job title

Location

Select an optionAfghanistanAlbaniaAlgeriaAmerican SamoaAndorraAngolaAnguillaAntarcticaAntigua & BarbudaArgentinaArmeniaArubaAustraliaAustriaAzerbaijanBahamasBahrainBangladeshBarbadosBelarusBelgiumBelizeBeninBermudaBhutanBoliviaBosniaBotswanaBouvet IslandBrazilBritish Indian Ocean TerritoryBritish Virgin IslandsBruneiBulgariaBurkina FasoBurundiCambodiaCameroonCanadaCape VerdeCaribbean NetherlandsCayman IslandsCentral African RepublicChadChileChinaChristmas IslandCocos (Keeling) IslandsColombiaComorosCongo - BrazzavilleCongo - KinshasaCook IslandsCosta RicaCroatiaCubaCuraçaoCyprusCzechiaCôte d’IvoireDenmarkDjiboutiDominicaDominican RepublicEcuadorEgyptEl SalvadorEquatorial GuineaEritreaEstoniaEswatiniEthiopiaFalkland IslandsFaroe IslandsFijiFinlandFranceFrench GuianaFrench PolynesiaFrench Southern TerritoriesGabonGambiaGeorgiaGermanyGhanaGibraltarGreeceGreenlandGrenadaGuadeloupeGuamGuatemalaGuernseyGuineaGuinea-BissauGuyanaHaitiHeard & McDonald IslandsHondurasHong KongHungaryIcelandIndiaIndonesiaIranIraqIrelandIsle of ManIsraelItalyJamaicaJapanJerseyJordanKazakhstanKenyaKiribatiKuwaitKyrgyzstanLaosLatviaLebanonLesothoLiberiaLibyaLiechtensteinLithuaniaLuxembourgMacauMadagascarMalawiMalaysiaMaldivesMaliMaltaMarshall IslandsMartiniqueMauritaniaMauritiusMayotteMexicoMicronesiaMoldovaMonacoMongoliaMontenegroMontserratMoroccoMozambiqueMyanmarNamibiaNauruNepalNetherlandsNew CaledoniaNew ZealandNicaraguaNigerNigeriaNiueNorfolk IslandNorth KoreaNorth MacedoniaNorthern Mariana IslandsNorwayOmanPakistanPalauPalestinePanamaPapua New GuineaParaguayPeruPhilippinesPitcairn IslandsPolandPortugalPuerto RicoQatarRomaniaRussiaRwandaRéunionSamoaSan MarinoSaudi ArabiaSenegalSerbiaSeychellesSierra LeoneSingaporeSint MaartenSlovakiaSloveniaSolomon IslandsSomaliaSouth AfricaSouth Georgia & South Sandwich IslandsSouth KoreaSouth SudanSpainSri LankaSt. BarthélemySt. HelenaSt. Kitts & NevisSt. LuciaSt. MartinSt. Pierre & MiquelonSt. Vincent & GrenadinesSudanSurinameSvalbard & Jan MayenSwedenSwitzerlandSyriaSão Tomé & PríncipeTaiwanTajikistanTanzaniaThailandTimor-LesteTogoTokelauTongaTrinidad & TobagoTunisiaTurkiyeTurkmenistanTurks & Caicos IslandsTuvaluU.S. Outlying IslandsU.S. Virgin IslandsUKUSUgandaUkraineUnited Arab EmiratesUruguayUzbekistanVanuatuVatican CityVenezuelaVietnamWallis & FutunaWestern SaharaYemenZambiaZimbabweÅland Islands

Email Address *

Leave this field blank

Submit

Cancel

网站信息

域名
furiosa.ai
收录路径
furiosa