CPU Spikes to Red Causing Intermittent Audio Artifacts

[svenr]

Thank you Martin,

GOTCHA!

* I set the affinity of the window manager (dwm.exe) manually to CPU0-3. Result: spikes and crackles are gone.
* I login via remote desktop: a second dwm appears, as expected, and, voilà, spikes/crackles are back.
* I set its affinity again to CPU0-3 ==> spikes/crackles gone.

A little powershell script helps: edit a file somewhere called "Set_DWM_Affi.ps1":

Code: Select all

$process = get-process dwm |Where-Object {$_.SI -eq 1}
$process |fl ProcessorAffinity
pause
$process.ProcessorAffinity=15
$process |fl ProcessorAffinity
pause

Make a link to the desktop, by dragging it to the desktop holding ALT key.
Right-click the new icon, select properties, change target to

Code: Select all

powershell -EP ByPass -File "(old target)"

Double clicking this icon opens a console that shows the old affinity of the DWM process, waits for "enter", set the new affinity (15 = cores 0-3) and displays it, waits for "enter", and crackles are gone.
After testing it, the "pause" lines can be removed.

The script acts only on the first desktop manager, so don't expect it to work using a remote desktop.

Yeah it's still a workaround, but the next step would have been to trash peu-a-peu the hardware and replace it, a lot of work and cost.

Cheers,
Sven

[mdyde]

Splendid -- thanks, Sven. Excellent work! That's very useful to know.

[abaymajr]

- Try running LatencyMon ( https://resplendence.com/latencymon ) when Hauptwerk is running, and see if it reports any problems when the glitches occur.

One other thought: given that you mention that the glitches occur when clicking on different windows, it might suggest that either the graphics driver, touch-screen driver, or the USB/motherboard driver (assuming your touch-screen/mouse are connected via USB) is the culprit. See whether any updates are available for your BIOS or for any of those drivers.

If none of that helps, and you do want the special build that doesn't use CPU affinities, then please send an email to Francois at support [at] hauptwerk.com asking for it and referring him to this reply and mentioning that your CPU has AVX2, so that he has the relevant background information and knows which CPU-type build you'd need.

Hi mdyde,

Since two days ago, I've been testing a new host system exclusively built for Hauptwerk and found that, at least for Hauptwerk VSTi / Cantabile VST workflow, all combinations of buffer size and number of audio buffers with sound delays less than 5.3 ms (256*1 or 128*2 at 48 KHz, 512*1 or 256*2 at 96 KHz) produce CPU spikes, even with Haupwerk.exe process set to realtime. Setting dwm.exe affinity to CPU 0-3, this trick doesn't work for me. Also, LatencyMon does not report any latency problems and the spikes on HW CPU meter usage don't seem to produce audio glitches, at least I couldn't notice them. But in the case of more adverse circunstances in the future, and to be able to make usage of 2.7 ms sound delay (256 buffer size at 96 KHz) with confidence, I'd like to test that workaround of a Hauptwerk 7 compilation made exclusively for CPUs with AVX2 instructions. The CPU is a i7-13700H. May I also contact support@hautpwerk.com for this request?

[mdyde]

Hello abaymajr,

all combinations of buffer size and number of audio buffers with sound delays less than 5.3 ms (256*1 or 128*2 at 48 KHz, 512*1 or 256*2 at 96 KHz) produce CPU spikes, even with Haupwerk.exe process set to realtime

As long as you have reliable audio with 256x1 at 48k, or 512x1 at 96 kHz, then I would regard your PC as performing well. Using less buffering than that would be considered extreme and 'experimental', and all PCs (even fast modern ones) will show a certain amount of constantly-varying background load on Hauptwerk's "audio CPU" meter even when no pipes with smaller buffer sizes, and especially so with less than 5.3ms of buffering. A small number of green bars (e.g. 1-3) showing erratically on Hauptwerk's CPU meter is normal and doesn't indicate a problem. I would highly recommend always using at least 5.3ms of buffering anyway. With most audio drivers, using a single larger buffer (e.g. 256x1) will usually give better performance (better resilience to audio glitches) than the equivalent total amount of buffering from several smaller buffers (e.g. 128x2).

But in the case of more adverse circunstances in the future, and to be able to make usage of 2.7 ms sound delay (256 buffer size at 96 KHz) with confidence, I'd like to test that workaround of a Hauptwerk 7 compilation made exclusively for CPUs with AVX2 instructions. The CPU is a i7-13700H.

Just to clarify -- the standard build of Hauptwerk v7 that you already have installed is compiled and fully optimised exclusively for the AVX2 instruction set anyway. (Hauptwerk's installer automatically selects and installs the appropriately-optimised executable, based on the host computer's CPU, from several different executables that the installer contains.)

The special executable that I mentioned is exactly the same with regard to how it is compiled and optimised, and the only relevant difference is that it doesn't set thread CPU core affinities at all, specifically for people who have problems with other processes (probably dwm.exe, if any) hogging specific cores. On PCs that don't have that problem, it's likely to give *worse* performance, since Windows often causes audio glitches whenever it moves an audio thread from one CPU core to another (which it is allowed to do for that special executable, given that it doesn't set thread CPU core affinities), and also since Windows may well then put audio threads on the same CPU cores as the background model threads (wind supply model, etc.).

In the other forum thread on your new PC ( https://www.forum.hauptwerk.com/viewtop ... 85#p156212 ) you mentioned that the CPU load isn't evenly distributed between the CPU's performance (P) and efficiency (E) cores. That's to be expected with Hauptwerk v7, since v7 doesn't have specific functionality for distributing load optimally across cores of differing performances (as found on the 12th+ generation Intel i7/i9 models). However, performance is still excellent on those CPUs anyway, and Hauptwerk will have dedicated support for handing such CPUs optimally in the near future.

I would recommend simply waiting for that to be available (and sticking with at least 5.3ms of buffering in any case).

[abaymajr]

Hello abaymajr,

all combinations of buffer size and number of audio buffers with sound delays less than 5.3 ms (256*1 or 128*2 at 48 KHz, 512*1 or 256*2 at 96 KHz) produce CPU spikes, even with Haupwerk.exe process set to realtime

As long as you have reliable audio with 256x1 at 48k, or 512x1 at 96 kHz, then I would regard your PC as performing well. Using less buffering than that would be considered extreme and 'experimental', and all PCs (even fast modern ones) will show a certain amount of constantly-varying background load on Hauptwerk's "audio CPU" meter even when no pipes with smaller buffer sizes, and especially so with less than 5.3ms of buffering. A small number of green bars (e.g. 1-3) showing erratically on Hauptwerk's CPU meter is normal and doesn't indicate a problem. I would highly recommend always using at least 5.3ms of buffering anyway. With most audio drivers, using a single larger buffer (e.g. 256x1) will usually give better performance (better resilience to audio glitches) than the equivalent total amount of buffering from several smaller buffers (e.g. 128x2).

But in the case of more adverse circunstances in the future, and to be able to make usage of 2.7 ms sound delay (256 buffer size at 96 KHz) with confidence, I'd like to test that workaround of a Hauptwerk 7 compilation made exclusively for CPUs with AVX2 instructions. The CPU is a i7-13700H.

Just to clarify -- the standard build of Hauptwerk v7 that you already have installed is compiled and fully optimised exclusively for the AVX2 instruction set anyway. (Hauptwerk's installer automatically selects and installs the appropriately-optimised executable, based on the host computer's CPU, from several different executables that the installer contains.)

The special executable that I mentioned is exactly the same with regard to how it is compiled and optimised, and the only relevant difference is that it doesn't set thread CPU core affinities at all, specifically for people who have problems with other processes (probably dwm.exe, if any) hogging specific cores. On PCs that don't have that problem, it's likely to give *worse* performance, since Windows often causes audio glitches whenever it moves an audio thread from one CPU core to another (which it is allowed to do for that special executable, given that it doesn't set thread CPU core affinities), and also since Windows may well then put audio threads on the same CPU cores as the background model threads (wind supply model, etc.).

In the other forum thread on your new PC ( https://www.forum.hauptwerk.com/viewtop ... 85#p156212 ) you mentioned that the CPU load isn't evenly distributed between the CPU's performance (P) and efficiency (E) cores. That's to be expected with Hauptwerk v7, since v7 doesn't have specific functionality for distributing load optimally across cores of differing performances (as found on the 12th+ generation Intel i7/i9 models). However, performance is still excellent on those CPUs anyway, and Hauptwerk will have dedicated support for handing such CPUs optimally in the near future.

I would recommend simply waiting for that to be available (and sticking with at least 5.3ms of buffering in any case).

The problem is not only the internal imbalance of one or some P core loads, but the raise of all cores' frequencies even on those which are near to 0% of utilization.

Thank you very much for your attention and patience.

[mnailor]

The load distribution may be affecting you more because of additional threads from your DAW competing for cores that Hauptwerk threads are bound to, or because HW audio and convolution threads are ending up on E cores instead of P cores.

You might actually get a boost from the 7 build that doesn't bind threads to cores so Windows can distribute the load better in this special case.

[abaymajr]

The load distribution may be affecting you more because of additional threads from your DAW competing for cores that Hauptwerk threads are bound to, or because HW audio and convolution threads are ending up on E cores instead of P cores.

You might actually get a boost from the 7 build that doesn't bind threads to cores so Windows can distribute the load better in this special case.

In fact, this imbalance occurs even without the DAW host, with Hauptwerk outputting directly to the audio interface. At idle, just the the organ blowers sounding, The E cores are all (near to) 0%, and so are most of P cores, except for 2 of them at 10-15%, and one at 90%+. I think that one is the culprit for raising frequency of all cores and triggering active cooling at 3000 RPM.

The 7 built is that one that doesn't associate Hauptwerk process to CPU cores by affinity? I wrote to support [at] hauptwerk.com requesting it for test purpose but have no answer until now. The e-mail is not delivered and an error message comes back as following: TLS Negotiation failed: FAILED_PRECONDITION: starttls error (71): 22154348234048:error:10000438:SSL routines:OPENSSL_internal:TLSV1_ALERT_INTERNAL_ERROR:third_party/openssl/boringssl/src/ssl/tls_record.cc:592:SSL alert number 80

[mdyde]

Hello abaymajr,

In fact, this imbalance occurs even without the DAW host, with Hauptwerk outputting directly to the audio interface. At idle, just the the organ blowers sounding, The E cores are all (near to) 0%, and so are most of P cores, except for 2 of them at 10-15%, and one at 90%+

Hauptwerk's background models (wind supply model, etc.) will be running constantly on a small number of cores, even if no pipes are sounding, giving a certain amount of CPU load.

The 7 built is that one that doesn't associate Hauptwerk process to CPU cores by affinity?

Yes -- that's what Mark Nailor was referring to.

I wrote to support [at] hauptwerk.com requesting it for test purpose but have no answer until now. The e-mail is not delivered and an error message comes back as following: TLS Negotiation failed: FAILED_PRECONDITION: starttls error (71): 22154348234048:error:10000438:SSL routines:OPENSSL_internal:TLSV1_ALERT_INTERNAL_ERROR:third_party/openssl/boringssl/src/ssl/tls_record.cc:592:SSL alert number 80

I've emailed Francois now to bring it to his attention, and so that he can get back to you.

[mnailor]

I don't load the blower noise, but I never have much CPU activity when an organ is loaded but not actually playing -- at 96k I see no green bars on the CPU meter when not playing. 90% + 10% of a P core is a lot of activity when the blower is the only polyphony voice. But maybe that's what it takes to deliver audio at 96k with a 256 sample buffer. I'm away on vacation so I can't test it.

Are you sure all the Windows tuning recommendations have been done and haven't been reversed by a Windows update? The idle load sounds like something else is running at realtime priority besides Hauptwerk.

It's probably a good idea to change to 48k in this case, start with a 1024 sample buffer, and reduce it to find the best latency you can get without the CPU meter going red.

I do run the unbound thread version, and it does help polyphony noticeably on the i9-12900K, but it does have the risk, as Martin has pointed out, that model threads placed on E cores by Windows could fall behind on their timing. The models are normally bound to P cores because they are the low numbered cores. But any timer- or I/O-driven thread is going to be pushed toward E cores when Thread Director identifies them as class 3 instruction streams. E cores with Turbo Boost aren't all that slow, and I haven't had problems with this in my uses, but it is a risk. I assume you might notice heavy wind sags/spikes, oscillations, or weird tremulants if the models don't get enough attention.

[micdev]

Good afternoon Antonio,

I've emailed you from support. We apologize for the problem you had emailing support direct.

Don't hesitate to reply to the email I sent you if needed, that one will reach me.

[abaymajr]

Good afternoon Antonio,

I've emailed you from support. We apologize for the problem you had emailing support direct.

Don't hesitate to reply to the email I sent you if needed, that one will reach me.

No problems, François. It was nothing I couldn't wait to fix. This is the type of problem that can only be known by customers, by those who demand the service from outside. Thanks for your attention and prompt solution.

[abaymajr]

The 7 built is that one that doesn't associate Hauptwerk process to CPU cores by affinity?

Yes -- that's what Mark Nailor was referring to.

Thanks, Martin. At this moment, preliminary tests point to a (much) better utilization of cores with such version 7. The single core with 92% utilization is gone. Now, at idle (with just the blower sounding), there are 3 or 4 P cores, with their utilizations between 20 and 50%. Under the same circumstances, P-core frequencies dropped a lot (from 100% to 60%), and also the E-core ones, with small percentual of 10 or 20%. Also, the usage of the remaining CPU cores also scales more equally with increasing polyphony. As I suspected it could happen, the most audible consequence of this is the CPU fan RPM having dropped by about 600 RPM, from the 3000-3200 RPM range, to the 2400-2600 RPM range (a huge percentual difference), which puts the MiniPC at (almost) inaudible noise level, even with a polyphony reaching 4000 (Prytanée de la Flèche sampleset) with an unrealistic combination of registers (made only for test purposes) and a dense execution of chords and trills.

If the achievements pointed out so far are confirmed and do not produce undesirable side effects in the medium term, can the same result be achieved by running the ordinary "Hauptwerk.exe" version and setting no affinity in the Hauptwerk.exe process? I ask this because I also use "Hauptwerk (alt config 1).exe", when I alternate using the MiniPC with a second and different console.

PS: Putting the MiniPC in its ordinary place, the fan RPM increased by about 200 RPM (to the 2600-2800 RPM range), but all the considerations I made are still valid. Also, the highest CPU usage on the Hauptwerk meter became even lower as well as its variance. That's for buffer of 512 at 96 KHz. I have just tested it now with a buffer of 256 at 96 KHz and the improvement is substantial. In no time has the CPU meter reached its maximum and its flickering is smaller and much less erratic as it was before with this buffer size. Huge improvement.

[mdyde]

Thanks, abaymajr.

The single core with 92% utilization is gone.

In that case, some process other than Hauptwerk must also have been using that particular core heavily, since with no pipes sounding, the only CPU-intensive Hauptwerk thread would be the wind supply model (regardless of whether CPU affinities are set), so if thread affinities make any difference then presumably some other process was using the same core heavily that the wind model was using.

can the same result be achieved by running the ordinary "Hauptwerk.exe" version and setting no affinity in the Hauptwerk.exe process? I ask this because I also use "Hauptwerk (alt config 1).exe", when I alternate using the MiniPC with a second and different console.

Yes, but you would need to open, then just OK, Windows' thread affinity window for the process every time after audio/MIDI starts in Hauptwerk (since it only affects threads that are already running at the point in time you do it),

You could, however, use the executable to replace all four of those in:

C:\Program Files\Hauptwerk Virtual Pipe Organ

... keeping their original names exactly.

[abaymajr]

Thanks, abaymajr.

The single core with 92% utilization is gone.

In that case, some process other than Hauptwerk must also have been using that particular core heavily, since with no pipes sounding, the only CPU-intensive Hauptwerk thread would be the wind supply model (regardless of whether CPU affinities are set), so if thread affinities make any difference then presumably some other process was using the same core heavily that the wind model was using.

After a few days of testing, Hauptwerk 7.0.1.005 is absolutely stable, rising the use of CPU cores according to demand in a distributed and homogeneous way. I'm not sure if this is a good score but polyphony test at 96KHz/higher/higher settings is supporting 11 (14 if for just 10-20 seconds) keys * 500 pipes/key before sound generation begins to fail.

But I was curious to know what kind of application or service would be disputing with Hauptwerk v7.0.0.136 a single core to the point of making it stable at 92% load, even with Hauptwerk in idle, so I could optimize this Hauptwerk setup even further. Do you know any software that could perform this diagnosis?

[mdyde]

Thanks, abaymajr.

After a few days of testing, Hauptwerk 7.0.1.005 is absolutely stable, rising the use of CPU cores according to demand in a distributed and homogeneous way.

Just to clarify, that isn't a standard Hauptwerk v7.0.1.005 build -- it's a special executable that has all thread-affinity temporarily commented out (so that affinity is never set), solely meant to be sent to people as a work-around of last resort for PCs for which it isn't possible to stop some other process hogging a specific CPU core that Hauptwerk would normally use.

But I was curious to know what kind of application or service would be disputing with Hauptwerk v7.0.0.136 a single core to the point of making it stable at 92% load, even with Hauptwerk in idle, so I could optimize this Hauptwerk setup even further. Do you know any software that could perform this diagnosis?

Unfortunately I'm not aware of any way within Windows to query a process's thread affinities, so probably the only way to do it would be to look at the processes on the Details tab in Task Manager, paying particular attention to any that have a higher-than-normal base priority, or are running with elevated privileges, and are using quite a bit of CPU time (sporadically or constantly), then try right-clicking on them, selecting "Set affinity" then just clicking OK (which evidently causes Windows to reset their thread affinities back to those of the process, i.e. 'all cores' by default), until you find the process that's causing the problem.