The big PlayStation 5 Pro tech interview with Mark Cerny and Mike Fitzgerald

This tech interview covers a lot of ground through its 30-minute runtime, including the challenges and opportunities of PSSR upscaling, the thinking behind the hardware design of the new console and how developers have approached the new options - and performance - afforded to them. This piece arrives hand in glove witha deep dive presentation into the architecture of PS5 Prowith the interview taking place on October 30th, so while the topics covered are very extensive, elements like the variable nature of PSSR implementations across different titles weren't apparent at the time of the conversation.

The discussion also touches on where Sony is looking to go in the future. As Rich postulated in hisPS5 Pro review, as well as being a powerful machine in the here and now, this console is also a great opportunity for Sony and game developers alike to get to grips with the RT and ML tech that will define the next generation of games - and that's echoed in the conversation here. If you're looking for some hints as to the direction Sony is taking in PlayStation 6, there's plenty to think about in this piece. As always, the text has been lightly edited for length and clarity.

Mark Cerny: My preference is that the game developers do what they feel is best for their titles, so this is really a question for Mike.

Mike Fitzgerald: When we make a game and we work as a whole studio and team on delivering it, we're making some really tough trade-offs between what we want to present at a higher frame rate and what we can present at 30fps. The nice thing about being approached with new hardware is that suddenly those compromises we made while shipping the game go away. We're able to bring together a lot of what made the fidelity mode the fidelity mode and what made the performance mode the performance mode on the PS5 Pro. After spending some time doing that and having a really nice performance pro mode together, attention starts to turn to, well, maybe some people still want to play at 30fps and we do have this nice new piece of hardware, so what other fun things can we do with it?

Mark Cerny: Yeah. I knew there would be a lot of different strategies. I was upfront about this in that first video I did, with six games that were shown and the first three do the "almighty" mode and the other three titles do something completely different.

Mark Cerny: I'll talk a little bit about the tech we're developing and then Mike can talk about how they're using it. There's a lot of research going on there. We support one mode that covers a range of 2.5:1 to 1:1. And whether that's an ideal long-term strategy or not, we're definitely talking about it. We're looking at the low end of the range and the issues we see there versus the high end of the range. So I really wouldn't take that current 2.5:1 as any sort of indication about where we'll be years from now.

Mike Fitzgerald: Yeah, I would say a lot of it depends too on what you're moving from. So the TAA that we had been leveraging as a studio - and still do - was really well suited for about 1440p up to 4K, but really struggled when it got down closer to the 1080p to 1280p range. PSSR behaves very well at that range, so for us, it's a big improvement and gives us an opportunity to spend our time differently there.

Mark Cerny: Yeah, that fur on performance RT.

Mike Fitzgerald: Yeah [laughter]. It really struggled for me playing at home in the performance modes that we had on the base PS5. And so to see that quality come back...

Mark Cerny: We're definitely continuing with development of these ML-based libraries for game graphics. It's just that there's so many targets to go up against. Like in that talk today, I was also going over frame generation, frame extrapolation, ray tracing, denoising and the like. So what order and what focus? Those are really good questions.

Additionally, complicating this is that we announced Project Amethyst with AMD today. So it's not just us developing the networks; we have a partner that we're doing it with. And of course, what the partner feels about that technology direction and where the focus should be is also very important.

Mark Cerny: Definitely. Well, it wouldn't be PSSR - PSSR is PlayStation Spectral Super Resolution. Spectral is our branding for the ML libraries for graphics, just like Tempest is our branding for our 3D audio. And so it's going to be Spectralsomethingfor all of that. And yeah, we're definitely looking at what it would be that would be that Spectralsomething, if not Super Resolution. But Super Resolution is a very good place to start.

Mark Cerny: Well, there are a lot of interesting possibilities there. And honestly, I don't know at this time. It's wonderful to think that the game could say it's okay to update or a user could make a decision to update. Just how all of those pieces fit together, I don't know.

Mike Fitzgerald: From a developer perspective, I think we can look at our games differently a lot of the time. And I know some developers would say, "I don't want the system to touch what I have confirmed looks good to me and players." Other developers would say, "Oh, your system is going to go and make my game look better for people without me having to do anything? Great." So there's options there.

Mark Cerny: You're not part of the problem. I've heard you talk. I've spent too much time on DF. But I've heard you talk about how you really shouldn't be looking at internal rendering resolution and drawing too much in the way of conclusions from that. You really need to be looking at image quality. And I was trying to communicate exactly the same message today, that we need to change the way we're talking about resolution. Maybe we shouldn't be talking about it at all. The difficulty is if you don't have a number, it's very hard to have that conversation.

Mike Fitzgerald: Ray-traced reflections are an interesting challenge because they are a composite of multiple images in the frame, right? And so with the information you feed into an algorithm like PSSR, you're limited in how much you can describe about each pixel.

An example would be if you're passing the depth buffer in to tell PSSR how far things are from the camera, it can't tell it how far the window is and how far the reflecting thing is and how far the thing behind the window, right? As you turn the camera, you have some velocity information about how the window is changing, but maybe not information about how the reflecting thing in the mirror is changing. This can be really hard to encapsulate in an analytical algorithm like TAA, so that's one of the areas we were hoping to see improvement in PSSR as it potentially has more capability to recognise that type of situation or adapt to it.

Mark Cerny: It's a mid-generation release. So really the target is better graphics and it felt like the big three were the right things to go after: larger GPU, more advanced ray tracing, and ML. Not that complex of an approach!

Personally speaking, it's been just great digging into the ML aspects of this and gaining some knowledge there. Because we know that's the future we're heading into. And so what an opportunity to gain all that knowledge with the Pro, rather than waiting for the start of a true generation and having to get everything right with no experience whatsoever.

Mark Cerny: It's really just a question of where you put your resources. And I wasn't seeing - and I didn't think I would see -thatmany titles that couldn't run at 60. That's kind of my litmus test. I'm not asking, "do all the games run at 120?" at this point in time. I'm asking, "do all the games run at 60?" You can count the AAA exceptions on one hand, I think, for games that didn't eventually run at 60. And so it seemed if we threw pretty much all of our resources into the graphics, that that would be the best approach.

Mike Fitzgerald: Right. I would say as a developer, the day Mark comes and says, "Hey, we want to give you guys a presentation about what we're thinking about for the next hardware." That's a fun meeting for everyone and we were very happy with where the focus was here.

Mark Cerny: Yeah. The strategies that we had up through PS4 Pro or the like... I wouldn't say it's reached a limit, but mostly it's about making the GPU bigger or memory faster. And so, as we look to the future, the improvements will be ray tracing. I think we're going to see a lot happening there. And then everything we can get done with ML.

Mark Cerny: That's a great question. It's pretty much identical. So, PlayStation 5 capped at 2.23GHz, PlayStation 5 Pro caps at 2.35GHz, which I did not talk about today because it's just such a distraction. I don't want anybody doing flop math based on 2.35GHz because, at the end of the day, when we do the set design, we tune the set design so that the games are running at about 2.17GHz. We don't want it to get too bulky. We don't want to have the fan have to be too large or the like. So, I mean, that 2.35GHz is really a red herring. I don't think too many games will get up to that too often.

Mark Cerny: We're not making PCs. We're making a few very specific things, right? The Pro consoles are very tightly targeted around GPU. But when a new generation starts, then it gets opened up - what are we doing with the CPU? What are we doing with the amount of memory? What are we doing with haptics? It's a whole new experience for developers. I think it's really helpful, even if we drop words like "revolution" from the conversation, that developers can ask the question, "OK, this game will ship on one and the next, but how can it be a different creature for each of those?" Not gameplay-wise, but in terms of the visual experience.

Mark Cerny: Well, it's not all going to be rainbows and path tracing, so...

Mark Cerny: Sure. What was that like Mike? Because you had a moderate boost for the raw graphics performance, and then you had a much better boost for the ray tracing. How did you tweak the engine?

Mike Fitzgerald: Well, the nice part was that those things stacked together in a really constructive way. So we were able to use the faster RT performance to get some time to spend to drive the upscaler further. The more sizable GPU came back to cut frame-times down. So I think they stacked well for a game that was designed for the base PS5 and already had little bits of each of those pieces.

Mark Cerny: Well, I liked it because very simplistically looking at it, it's faster, and so you've just sped up your rendering, and now you have 5ms to spend on whatever. And so why not spend them on improving your RT or adding a new widget or whatever? If you're making an engine, nobody is ever coming to you and saying, "you now have 5ms to do whatever you want to do that's just gravy." Right? That doesn't happen. You're desperately trying to get to 60fps.

Mark Cerny: There's two components. It's a deep collaboration. It's multi-year. Don't expect some massive hardware announcement immediately coming out of this. One target is more optimal architectures for machine learning. And I look at that through a very narrow lens because we're making a console. And so I really want to hear about, number one, does it support these lightweight CNNs that we need for the graphics? But I'd also like to hear about, people want to do their AI or something on it. Is it something that is portable? Is it something that's easy to create? Because you never want to lock people into just one hardware architecture when they make that stuff. So working on this generalised architecture that's particularly good at the ML needs of consoles. I suspect AMD might frame it slightly differently [laughter].

Mark Cerny: That's my personal focus. AMD, of course, is incredibly broad and they have big machinery out in the cloud and they have laptops. They're in so many different spaces. But I look at it as, yes, I'd love to see these things happen. And the other part of it is for these neural networks, they are time consuming to make. I think, with no exaggeration, PSSR is the toughest technical project I've ever been in in my life. And it's just marvelous to have a partner who also has resources and expertise that can be going into very much the same targets. And what are those targets? It is super resolution. It is. I guarantee it's possible to do better than PSSR. And that's just scratching the surface of where we can head. I mean, I mentioned frame generation and the like, but there are uses I know are out there that we haven't even started talking about yet. So it's to develop those together.

Now, to clarify, we're talking about developing components, not developing libraries. Because our customers are different and when it comes down to it, the hardware is a little bit different. I hope it rung true when I was saying super resolution on PCs is a little bit different from consoles, because we tend to use a variable upscaling ratio and PCstendbe fixed upscaling ratios. So having this ability to grab those co-developed algorithms and then integrate them separately is really going to be a boost across the board.

Mark Cerny: It's a great question. We learned a lot from PS4 Pro - going into that, I thought we had to replicate pretty much the exact timing that the game was running... and it turns out you don't need to. The timing can drift a bit and the game will still work, though there's a limit to how far that can go. We did see a few titles - like 11, out of thousands - that wouldn't respond well when did PS4 Pro to the radically different timing. But in general, the games are pretty robust when it comes to the timing. So the shorter answer is that on PS5 Pro we just didn't worry about it, and the games are working. We still need binary compatibility - all of the registers that the game was hitting need to be there. But we don't need whatever was taking a thousand cycles before to still take a thousand cycles. It's okay if it takes 600 or 900 or even occasionally 1100.

Mike Fitzgerald: Well, I think we've already seen it. We've already seen that consistency of ray tracing performance pay off even in the Spider-Man 2 patch we've done. So we're able to put in RTAO. We have some RT shadows in there and places where we were bumping up against those divergence limits. In fact, for the water reflections in Spider-Man 2, we cheated them all so that they were less divergent than would be physically accurate in the water to sort of scrape that performance back. And so we're already seeing opportunities where it's paying off for us.

Mark Cerny: I think path tracing on PS5 Pro would be a little tricky to do. But, you know, I wouldn't put it past the developers. There are some amazing engineers out there.

Mike Fitzgerald: It sounds like a challenge.

Mark Cerny: I'm happy with the launch line-up, to be honest. We view it very simply - we're just making tools for the developers. How the developers use those tools, that's up to them. We've seen a couple different ways of doing it, I think right now. Part of the fun will be, you know, in about three or five years, there will be a lot more different ways to use these tools that we hadn't anticipated. Look forward to it.