Since the turn of the century, video frame interpolation systems have used motion compensation (MC) and Motion estimation (ME) systems. However, such solutions are not power-efficient and can introduce interpolation errors. Pixelworks, a video alchemist skunk-works in San Jose, has been quietly developing a new, back-end mobile processor that solved those problems for movies; and now for video games.
The result is an incredibly brilliant image, at super high resolutions and high frame rates, with minimal latency that can improve battery life as well—it sounds almost as if they have broken the law of physics. But no, says their CEO,
“we just know those laws very well and how to work within them”.
The company has pushed the bounds limiting performance or quality by using an expert-trained video motion estimation and compensation algorithm in silicon for video frame interpolation. Through a novel adaptive compensation scheme trained by the company’s picture quality experts (that the company doesn’t want to talk too much about), Pixelworks has integrated motion processing. They’ve done such things for the film and TV industry in software –now, a mobile gaming solution is possible in a smartphone.
Figure 1 . Pixelworks’ trained visual processor makes smartphones gaming phones
This secret sauce, the expert training by their best engineers and picture quality people, is differentiable. The flow and estimation are optimized jointly in real-time and quickly, with minimal latency. In addition to the Smart MEMC, the design is adapted seamlessly to several other video enhancement tasks such as super-resolution, denoising, and deblocking. Quantitative and qualitative evaluations and demonstrations have shown that the X7 post-processor outperforms state-of-the-art video frame interpolation and enhancement algorithms on many datasets. Compared to existing methods, Pixelwork’s magic is computationally efficient and able to generate more visually appealing results.
Gamers Understand
One segment that appreciates that is gaming, and as most people know, gaming has become a big business in units and dollars worldwide. It is massive in China, where there are 670 million gamers—20% of the world’s estimated 3.3 billion gamers. Most of those gamers play on a smartphone. Chinese gamers have adopted Genshin Impact as a benchmark. One of the elements in the game is the weather. Weather is an exploration mechanic. While exploring the world, the weather can shift. Depending on the new weather conditions, unique properties get applied to the environment, characters, and opponents.
Games like Genshin Impact and PC games like Cyberpunk 2077 were ported from the PC to smartphones that challenge the phone’s performance vs. battery and heat operating parameters. Big games demand everything the phone’s processor has and, in the process, run down the battery while heating the phone.
Pixelworks saw that as an opportunity to apply its magic and introduced the X7 chip architecture, post-processors that exploited some of the company’s techniques in the film and TV industry. The X7 chips created a new distributed computing/rendering chip market segment in mobile, like what Nvidia and AMD have done in the PC and Console segments. Chinese phone manufacturers have referred to X7 as a standalone display solution, a rendering accelerator, or a dual-chip gaming architecture.
MEMC and resolution scaling (SR) are very demanding on any platform, especially in mobile devices with limited cooling and power supply. The phone must deliver a high frame rate, with high color accuracy, high pixel resolution, no artifact or aliasing, and low latency, and accomplish all that while using very little power.
Pixelworks post-processor has proven to be just what the doctor ordered, a unique technology for motion compensation and SR that offloads processing from the phone’s SoC. For gaming specifically, by taking care of the motion vectors and working with the game engine developers, the X7 processor uses metadata to guarantee the best gaming experiences.
Figure 2. Pixelworks’ x7 puts HDR gaming in your hands
As a result, three out of four of the largest Chinese phone OEMs have been shipping their top phones with Pixelworks’ chips. The leading game engine providers (e.g., Unity) have partnered with Pixelworks. The big game studios in China (ByteCance, Tencent, miHoYo, etc.) have engaged with Pixelworks, and mobile SoC processor makers such as MTK and Qualcomm are working with the firm.
The X7 is a tiny chip and draws so little power the smartphone hardly knows it is running. Some smartphone suppliers in China have already added it to their phones for the game-crazy customers they have, and if one can satisfy that price-performance market segment, they must have something special.
This is a classic story of a smallish (~$70 million in revenue) company quietly working in close collaboration with some big names and colossally demanding customers and perfecting a scalable, efficient, and economical solution.
Pixelworks didn’t just stumble into this market. Founded in 1997, the company developed video processors for professional projectors. The video engineers and scientists at the company had a collective range of experience unmatched in companies many times their size. But, they were techno-geeks, not salespeople. You almost had to know someone who knew someone to find out about them. Their customers weren’t eager to reveal them either—Pixelworks was their secret sauce and how they differentiated themselves in a cut-throat take-no-prisoners business.
The mythology of the company is the CEO or CTO (the story varies depending on who is telling it) was chatting at a conference with a CTO or president of a phone company and said, “it’s my dream that someday we will have smartphones that you can watch a movie on. It will be as good as if you were in the theater.” The Pixelworks guy went home thinking about that, and the next day he told his team, I’ve got an idea.
A few years later, visiting the lab (if you could get in), one could see a big 4k screen showing video and games as if an enthusiast-level PC drove it—but it wasn’t—it was driven by a two-year-old smartphone and not even a top-of-the-line smartphone.
The term ‘skunk-works’ isn’t used lightly here. Pixelworks has managed to beat out the top SoC builders in Silicon Valley and San Diego in video processing. In one sense, that’s not too surprising—video is all they do. Today’s smartphones are a marvel of technology, but they can’t be the best at everything they do. Product cycles and R&D budgets mitigate how much they can invest in time and dollars. Pixelworks is a specialty company—a pixel polisher. They can also polish pixels coming from the sensor but acknowledge the big SoC builders had done an excellent job with their integrated ISPs—you climb one mountain at a time, the CEO told me.
The tantalizing end of this story is the future. Pixelworks has now proven their bespoke approach to video processing—pixel polishing works; in fact, it works almost better than they thought it would. And it’s scalable. It will ride Moore’s law and just get better with each generation. Smartphones have (or will) never have looked so good.
