An Artificial Intelligence (AI) accelerator accelerates artificial intelligence applications such as artificial neural networks and machine learning. In the last decade, graphics processing units (GPUs) have seen increasing adoption for these applications since they efficiently perform image processing and mathematical neural network calculations. Fortunately for AI development, GPU manufacturers such as NVIDIA are making GPUs that greatly enhance AI performance, opening the door to many new computing products at the edge.  NVIDIA products have led the market with innovation and are widely used in AI computing. However, other specialty GPU providers offer enhanced performance on portions of AI computing, such as AI inferencing. 

What is a GPU?

A modern Graphical Processing Unit or GPU is similar to a CPU but makes use of parallel processing and is able to handle many processes and threads at the same time. Due to its parallel processing, a GPU is normally used for graphics processing and rendering.

GPUs have been around since the 1970s, primarily used for arcade games such as Sea Wolf and Space Invaders. Graphics cards were not commonly used in PCs until the mid-1980s, when the NEC μPD7220A became the first processor with a Large Scale Integration circuit chip, making it the most popular GPU. The next significant innovation was in the 1990s with S3 Graphics making the S3 86C911. This new GPU used 2D acceleration to gain a massive performance increase compared to its competitors. Today, a GPU is one of the most crucial hardware components of computer architecture.

Initially, the purpose of a video card was to take a stream of binary data from the central processor and render images to display. But modern graphics processing units are engaged in the most complex calculations, like big data research, machine learning, and AI.

While AI has existed since the 1990s in inference and training, AI accelerators did not enter the market until 10 years ago, when the workloads of AI processes became more intensive. Since 2010, AI accelerators such as field-programmable gate arrays (FGPA) and customized application-specific integrated circuits, (ASIC) are being replaced by commercial GPUs, which offer faster time to market and lower development costs. 

The GPU Market

Currently, the GPU industry is dominated by three large companies: NVIDIA, Intel, and AMD. These companies have been around for the longest, with NVIDIA pioneering the discrete GPU market since 2000 and putting them into a firm leadership position.  New entrants into the GPU market are picking a product niche and developing solutions that are efficient and designed to fill the appetite for higher performance with lower costs per watt of power. Up to 90% of the power consumed by an image processing application is accessing the RAM. Today, there are new entrants with differing approaches to building more efficient GPUs, including Mythic.AI, UntetherAI, Hailo, Blaize, etc., for edge AI computing.  These companies focus on avoiding data transfer between computing and memory to drive efficiency.  

GPU vs CPU

In the past, CPUs were used to process information for artificial intelligence. However, as GPUs have become more powerful over time, their ability to process more parallel information faster has made them a much better solution for AI. 

Integrating a GPU into an application:

GPUs are typically shipped as modules to be easily implemented into various applications. Chip-down GPU designs are intensive hardware and software projects; also, GPU vendors historically will only support Tier 1 customers and projects and encourage the rest of the applications to design a carrier that integrates their modules. 

Here are some common GPU module form factors:

PCIe

• This is the first and still most common form factor for GPU modules and is easily integrated with common PC motherboards with up to x16 Gen5 PCIe connections. However, the disadvantage is that they are large and bulky for many edge AI applications and typically require forced air to cool them.

MXM (Mobile Express Module)

• As the name indicates, the MXM form factor was developed to offer graphics processing module capabilities to smaller mobile computers such as laptops. It is also commonly used in edge SFF computing applications in markets such as military, medical, and transportation. The modules can be air cooled or conduction cooled.

M.2

• The M.2 standard replaces the mSATA standard and offers size and speed advantages for storage.  Gen4 PCIe x4 connections to the processor allow it to be also widely used for other computing functions, including GPS, LTE, IO, and smaller GPUs by vendors such as Hailo.  

E1.S EDSFF (Enterprise and Datacenter Small Form Factor)

• E1.S is the choice of next-generation storage modules.  It offers greater density and performance than the M.2 and other earlier form factors.

• Although it is being deployed in the datacenter server industry, it has not yet replaced M.2 as a standard in SFF Edge Computing.  
• Blaize is an example of a GPU being deployed in the ES.1 SFF, enabling up to 512 TOPS in a 1U server and 16-64 TOPS in an SFF computer.

SOM (System on Modules)

• System on Modules has been made popular for industrial edge computing by NVIDIA in products such as JETSON.
• Embedded ARM processors in the SOM enable a solutions provider to build a smaller, low-cost edge AI or graphics processing computer without needing a separate embedded CPU.

Chip-Down

• With many edge platforms, it is the most cost-effective to design a solution with a ‘chip-down’ GPU and a ‘chip-down’ CPU.  Tauro Technologies has designed these systems for several of the GPU vendors mentioned above.

GPU integration into a carrier board:

System design takes many factors into consideration, including cooling, power, I/O, storage, processing, etc., and each system requirement is different. Typically, GPU modules that are not chip-down are integrated into a main board or carrier board that has been customized to meet the application requirements.  

There are 2 main types of processors integrated into carrier boards:

• Intel x86: Due to the engineering complexity of Intel chip-down designs, many edge AI applications choose to use COMe or COM-HPC client-based modules to simplify their carrier design projects. Although the material cost of the module is higher than a chip-down solution, unless the product reaches modest volumes, it is more cost-effective to design with an x86-based module. 

• ARM-based processors: ARM processors are typically lower cost and take less power, making them ideal for many computer vision products when paired with a GPU.  Since there are few modules available based on open standards, most ARM-based products are custom designed with a chip-down processor such as NXP Cortex or Layerscape.  

Summary:

The advent of AI is opening the door for application-specific GPUs that are finely tuned to the objectives of the project. Tauro Technologies has broad experience implementing edge computers optimized for the application. If you wish to discuss a customized high-volume platform or a system which takes advantage of commercially available hardware and tools, reach out to us.

The post Use of GPUs in Edge AI Computing: appeared first on Tauro Technologies.

What are SiP, SoC, COM, and SOM devices and what are the pros and cons of each?  What is the best fit for each module and how do they compare to the more traditional ‘chip down’ approach?  In this post, we will take a short walk through each type, give product example links and suggest how these approaches can be useful to your next product development or technology refresh.  We will also outline how a design firm such as Tauro Technologies can enable you to choose the best path and migrate from one form factor to the next as it makes sense for your projects.

New designs in past product generations tended to be chip down and were application and product specific.  As computing silicon has become more robust and complex, industry bodies such as PICMG pioneered standards that enabled companies to produce modules of standardized computing elements, giving product management teams a much simpler task of developing carrier specific to the required application features.

System on Module (SOM) & Computer on Modules (COM)

The terms SOM & COM are used interchangeably and are widely accepted as synonyms.   There are many companies manufacturing to the industry specifications  and global distributor Avnet provides a product selection guide offering a wide range of vendors and applications.  Further, there are building blocks widely available for specialized applications such as SDR or IIoT that will save significant time and money.

SOMs provide the building blocks of the application computing thus saving time and money to get the application to market, they also require broad-based engineering skills to build the application specific carrier.  Tauro Technologies works closely with many COM vendors and creates the carrier boards to integrate application I/O into these modules and develop firmware and software to enable the hardware to fit into the final product.

FPGA based System on Chip (SoC)

Yesterday’s computing had separate blades connected over a backplane in a shelf or chassis.  The applications had complex hardware and software designs physically taking racks of computing space.  Today it is possible to have these massive systems of complex multi-board & multi-vendor products reduced to a single chip by vendors such as Intel and Xilinx.  The Xilinx SoC portfolio is an example of that integrates combinations of CPU, FPGA, GPU, Memory, RF and other peripherals on a single chip making it faster, easier and cheaper to create a new product.  The application is less prone to field failure and much simpler to upgrade with a software release vs a hardware swap.

Just as the development of systems in a multi-blade design required a large multi-discipline team, implementing SoC designs requires specialized knowledge and experience.  Tauro Technologies combines years of communications, RF, FPGA, and I/O experience and has deployed SoCs to communications, military, and industrial applications. 

System in Package (SiP)

SiP has been around since the 1980s and has recently been gaining traction – market researchers predict a high CAGR of this technology in the next 5 years.  Potential to integrate components such as DRAM, Flash and CPU into a single package makes it very attractive to application developers.  Where a SoC approach integrates these components into a single chip, the SiP approach combines separate chips into the same package which is then designed into the PCB.  The advantage again is that the design of the application specific portion of the printed circuit board is much simpler and lower cost.

Tauro Technologies designs systems based on SiPs for specialty applications where Size, Weight, and Power (SWaP) are critical for the application and can advise on the feasibility of this approach. 

Chip Down – Customized Design

The best design option for many large-scale projects or products that are deployed in harsh environments is a custom design that grants full control of the design and components.  Before SOM, SoC, and SiP were widely available, most products were designed by teams of engineers focused on hardware designs that were targeted to the efficiency, cost, and functionality requirements of the application.

SiP and SoC solutions essentially also are ‘chip-down’ solutions.  SoC combines many discrete components into a single chip whereas SiP is a variation of SoC packaging the discrete devices (Processor, DRAM, Power etc) onto a single IC package.

While fewer new designs will go a complete custom designed route, there are still many applications where this design approach is the best alternative.  Tauro Technologies’ provides experience in many such applications and can bring to fruition even your most complex concepts.

Summary

SOM, SoC, and SiP give many advantages to help get application hardware to market.  As the product matures, it can be advantageous to take the standardized elements and create a hybrid or complete chip-down design to maximize efficiency and optimize cost.  Each product and project is unique, thus no one strategy fits all. 

Reach out to us to discuss the feasibility and benefits of these technologies for your next project.

The post System Module Choices appeared first on Tauro Technologies.