Loading times very slow HW7
Loading times very slow HW7
Hi,
I know I can upgrade to ver8 but I would like to get ver7 working properly first....!!
My first foray into HW and upgrading from a dual xeon HP z800....
New pc build back at start of new year.... obviously some things have changed now and do again would do different ddr5 etc... but very expensive when i did this.
I separated the drives for maximising speed across the different needs... made sense at the time and meant I could have multiple 1tb drives as opposed to a very expensite 2/3tb m2 having the speed also for what counts cache drive for fast load times....
i7 13700kf
Asus B660M-E D4
128gb corsair vengeance 3200 ddr4
Crucial MX sata SSD for HW storage install
Crucial M2 gen 4 superfast for cache
Crucial m2 gen 3 for OS win 11
HDD for downloads
96khz 24bit
Load times from cache are extremely slow.... Alessandria for eg around 56gb takes around 5 minutes.... it did it faster once by starting HW in admin mode around 2.5min from memory but generally getting slower... having to re-cache also is common with Alessandria being the 1 sampleset that i regularly use. Nothing generally happens to warrant it just gets to end and of loading and gives up saying have to reload and redo the cache....!!! no change to anything and it just does it randomly....
I don't think its hardware related.... as am able to load same sampleset from the cache drive in GO and it loads it in 54seconds... A time that i was much more thinking that should be possible...heck even loading same data from the sata ssd is faster just under 2 mins in GO...!! This is on par or slower than the old z800 running ssds slowed down by mb bus of sata 3 instead of 6??
HW doesn't seem to be utilizing the power of the super fast cache drive and cpu.... Noticed loading 0.1gb at a time... looking at the ram monitor in HW also cpu speed to load it is very low.... A very old macbook pro 15" i7 2011 with a MX crucial ssd loads at 0.4 chunks every second... if that has anything to do with it..>!?!? Once loaded everything is fine...
i know there has been some changes with 8 and improvements but other people with slower and older machines are loading a hell of a lot quicker than me with 7....!!!
I've even thought to buy another m2 to load everything on the one drive OS install cache etc... to see if that makes any difference.... but GO loads it super fast same amount of data...
Any ideas? Things to check....?!?! secret settings...>!?!?
I know I can upgrade to ver8 but I would like to get ver7 working properly first....!!
My first foray into HW and upgrading from a dual xeon HP z800....
New pc build back at start of new year.... obviously some things have changed now and do again would do different ddr5 etc... but very expensive when i did this.
I separated the drives for maximising speed across the different needs... made sense at the time and meant I could have multiple 1tb drives as opposed to a very expensite 2/3tb m2 having the speed also for what counts cache drive for fast load times....
i7 13700kf
Asus B660M-E D4
128gb corsair vengeance 3200 ddr4
Crucial MX sata SSD for HW storage install
Crucial M2 gen 4 superfast for cache
Crucial m2 gen 3 for OS win 11
HDD for downloads
96khz 24bit
Load times from cache are extremely slow.... Alessandria for eg around 56gb takes around 5 minutes.... it did it faster once by starting HW in admin mode around 2.5min from memory but generally getting slower... having to re-cache also is common with Alessandria being the 1 sampleset that i regularly use. Nothing generally happens to warrant it just gets to end and of loading and gives up saying have to reload and redo the cache....!!! no change to anything and it just does it randomly....
I don't think its hardware related.... as am able to load same sampleset from the cache drive in GO and it loads it in 54seconds... A time that i was much more thinking that should be possible...heck even loading same data from the sata ssd is faster just under 2 mins in GO...!! This is on par or slower than the old z800 running ssds slowed down by mb bus of sata 3 instead of 6??
HW doesn't seem to be utilizing the power of the super fast cache drive and cpu.... Noticed loading 0.1gb at a time... looking at the ram monitor in HW also cpu speed to load it is very low.... A very old macbook pro 15" i7 2011 with a MX crucial ssd loads at 0.4 chunks every second... if that has anything to do with it..>!?!? Once loaded everything is fine...
i know there has been some changes with 8 and improvements but other people with slower and older machines are loading a hell of a lot quicker than me with 7....!!!
I've even thought to buy another m2 to load everything on the one drive OS install cache etc... to see if that makes any difference.... but GO loads it super fast same amount of data...
Any ideas? Things to check....?!?! secret settings...>!?!?
Re: Loading times very slow HW7
Recaching Alessandria over and over indicates a failure of some sort, and it may very well be hardware. I would scan and repair your filesystems to check. And check your disk space, and whether Alessandria is running low on memory (RAM) during the load, which it might if you're loading all ranks uncompressed at 24 bits.
After fixing whatever is causing that, upgrade to HW 8. Loads are much faster.
(I think we're not supposed to mention Pokemon GO here...)
After fixing whatever is causing that, upgrade to HW 8. Loads are much faster.
(I think we're not supposed to mention Pokemon GO here...)
Re: Loading times very slow HW7
Just loaded up again to check and watch things....
Observed in task manager 100% utilised cache drive and loading at 240mb p sec a lot of times at 34mb !!! occasional spike up to 480mb thats it..... and 4.30minutes to load Alessandria....
Fresh win 11 turned off all the usuals if anything to do with it... hybernate and pagefile etc would have turned off indexing also on all drives..... latest up to date patched....
HW is the only installed app...
All the 1tb drives are largely empty due to only having a few samplesets installed....
Loading compressed 24bit
not sure what you mean by running low on mem during the load...?
Its just noted recaching is consistently Alessandria being the 1 mainly used all year certainly done it a few times but also triggered by me updating to latest 1.11 etc but definitely has done it a few times unexpectantly .....
I might just do a fresh install onto a spare ssd everything all in 1 drive just to see what speed i get.... if faster on a ssd than yes something astray....
Really got me stumped....!!
Observed in task manager 100% utilised cache drive and loading at 240mb p sec a lot of times at 34mb !!! occasional spike up to 480mb thats it..... and 4.30minutes to load Alessandria....
Fresh win 11 turned off all the usuals if anything to do with it... hybernate and pagefile etc would have turned off indexing also on all drives..... latest up to date patched....
HW is the only installed app...
All the 1tb drives are largely empty due to only having a few samplesets installed....
Loading compressed 24bit
not sure what you mean by running low on mem during the load...?
Its just noted recaching is consistently Alessandria being the 1 mainly used all year certainly done it a few times but also triggered by me updating to latest 1.11 etc but definitely has done it a few times unexpectantly .....
I might just do a fresh install onto a spare ssd everything all in 1 drive just to see what speed i get.... if faster on a ssd than yes something astray....
Really got me stumped....!!
Re: Loading times very slow HW7
Would formatting allocation size have anything to do with it....? can't remember if i changed that from default...thinking cache very large chunks of data instead of small....?!!
Re: Loading times very slow HW7
Hello mattrix,
Welcome to the forum.
I don't have Piotr Grabowski's Alessandria sample set here, but I see from his website ( https://piotrgrabowski.pl/alessandria/ ) that it takes about 56 GB of RAM to load all ranks/perspectives in 24-bit compressed, which should fit easily within your 128 GB.
As a comparison, I loaded another large sample set which I happen to have here (Sonus Paradisi's Portland Reuter) in Hauptwerk v7. After loading the sample set from cache three times in a row, then using "Help | View activity log" and taking the fastest of the three loads (although all three were very similar), I got the following figures on my laptop with an i9-12950HX CPU and fast NVME SSD:
Welcome to the forum.
I don't have Piotr Grabowski's Alessandria sample set here, but I see from his website ( https://piotrgrabowski.pl/alessandria/ ) that it takes about 56 GB of RAM to load all ranks/perspectives in 24-bit compressed, which should fit easily within your 128 GB.
As a comparison, I loaded another large sample set which I happen to have here (Sonus Paradisi's Portland Reuter) in Hauptwerk v7. After loading the sample set from cache three times in a row, then using "Help | View activity log" and taking the fastest of the three loads (although all three were very similar), I got the following figures on my laptop with an i9-12950HX CPU and fast NVME SSD:
Also for comparison, I then tried exactly the same test in Hauptwerk v8 on the same laptop and got the following figures:Total time: overall: 73.702 sec.
Total time: sample data: 68.877 sec.
...
Sample loader: loaded from data cache: Y.
Sample loader: data cache total disk size: 40129.14 MB.
Sample loader: buffers: 9.
Sample loader: approx. loader peak mem. usage during audio loading: 331.15 MB.
Sample loader: loader def. mem. usage during audio loading: 239.41 MB.
Sample loader: approx. loader mem. usage during trem. loading: 29.26 MB.
Sample loader: approx. avg. overall data read rate: 582.62 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 4776.60 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 12.20.
Sample loader: pct. sample loading time stage busy: 1st proc.: 98.91.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 35.37.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 16.55.
Sample loader: pct. sample loading time stage busy: 4th proc.: 47.81.
Sample loader: pct. sample loading time stage busy: 5th proc.: 33.28.
Sample loader: pct. thread activity due to stage: disk I/O: 5.00.
Sample loader: pct. thread activity due to stage: 1st proc.: 40.52.
Sample loader: pct. thread activity due to stage: 2nd proc.: 14.49.
Sample loader: pct. thread activity due to stage: 3rd proc.: 6.78.
Sample loader: pct. thread activity due to stage: 4th proc.: 19.59.
Sample loader: pct. thread activity due to stage: 5th proc.: 13.63.
The most important figure is "Sample loader: approx. avg. overall data read rate" above. What values do you get for that figure if you try loading the Alessandria three times in a row?Total time: overall: 31.820 sec.
Total time: sample data: 29.101 sec.
...
Sample loader: loaded from data cache: Y.
Sample loader: data cache total disk size: 40244.78 MB.
Sample loader: buffers: 16.
Sample loader: approx. loader peak mem. usage during audio loading: 567.16 MB.
Sample loader: loader def. mem. usage during audio loading: 425.62 MB.
Sample loader: approx. loader mem. usage during trem. loading: 52.02 MB.
Sample loader: approx. avg. overall data read rate: 1382.93 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 3014.70 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 45.87.
Sample loader: pct. sample loading time stage busy: 1st proc.: 97.91.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 42.22.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 22.33.
Sample loader: pct. sample loading time stage busy: 4th proc.: 75.15.
Sample loader: pct. sample loading time stage busy: 5th proc.: 30.53.
Sample loader: pct. thread activity due to stage: disk I/O: 14.61.
Sample loader: pct. thread activity due to stage: 1st proc.: 31.18.
Sample loader: pct. thread activity due to stage: 2nd proc.: 13.45.
Sample loader: pct. thread activity due to stage: 3rd proc.: 7.11.
Sample loader: pct. thread activity due to stage: 4th proc.: 23.93.
Sample loader: pct. thread activity due to stage: 5th proc.: 9.72.
Best regards, Martin.
Hauptwerk software designer/developer, Milan Digital Audio.
Hauptwerk software designer/developer, Milan Digital Audio.
Re: Loading times very slow HW7
Hi Martin thankyou for having a look...
wow so much faster than what im getting....
Total time: overall: 267.816 sec.
Total time: sample data: 264.342 sec.
Sample loader: loaded from data cache: Y.
Sample loader: data cache total disk size: 56304.56 MB.
Sample loader: buffers: 9.
Sample loader: approx. loader peak mem. usage during audio loading: 464.07 MB.
Sample loader: loader def. mem. usage during audio loading: 239.41 MB.
Sample loader: approx. loader mem. usage during trem. loading: 29.26 MB.
Sample loader: approx. avg. overall data read rate: 213.00 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 216.09 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 98.57.
Sample loader: pct. sample loading time stage busy: 1st proc.: 30.67.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 10.03.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 4.01.
Sample loader: pct. sample loading time stage busy: 4th proc.: 10.96.
Sample loader: pct. sample loading time stage busy: 5th proc.: 8.39.
Sample loader: pct. thread activity due to stage: disk I/O: 60.61.
Sample loader: pct. thread activity due to stage: 1st proc.: 18.86.
Sample loader: pct. thread activity due to stage: 2nd proc.: 6.17.
Sample loader: pct. thread activity due to stage: 3rd proc.: 2.47.
Sample loader: pct. thread activity due to stage: 4th proc.: 6.74.
Sample loader: pct. thread activity due to stage: 5th proc.: 5.16.
I checked and turned off page file for c: (thought i had) was already off for cache and install drives...
and got down to 4mins 240secs consistently - but thats as fast as it will go...
Trawling through the info dump log I see once back in august 25th with no changes to the system other than win11 and bios updates along the way... i had a much faster ONE time event! Wouldn't have even noticed it but rings a bell as used to it being so slow that i set and come back....
Total time: overall: 83.812 sec.
Total time: sample data: 80.731 sec.
Sample loader: loaded from data cache: Y.
Sample loader: data cache total disk size: 56307.81 MB.
Sample loader: buffers: 9.
Sample loader: approx. loader peak mem. usage during audio loading: 464.34 MB.
Sample loader: loader def. mem. usage during audio loading: 239.41 MB.
Sample loader: approx. loader mem. usage during trem. loading: 29.26 MB.
Sample loader: approx. avg. overall data read rate: 697.47 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 5850.85 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 11.92.
Sample loader: pct. sample loading time stage busy: 1st proc.: 99.79.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 33.82.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 15.98.
Sample loader: pct. sample loading time stage busy: 4th proc.: 36.88.
Sample loader: pct. sample loading time stage busy: 5th proc.: 28.52.
Sample loader: pct. thread activity due to stage: disk I/O: 5.25.
Sample loader: pct. thread activity due to stage: 1st proc.: 43.98.
Sample loader: pct. thread activity due to stage: 2nd proc.: 14.90.
Sample loader: pct. thread activity due to stage: 3rd proc.: 7.04.
Sample loader: pct. thread activity due to stage: 4th proc.: 16.25.
Sample loader: pct. thread activity due to stage: 5th proc.: 12.57.
This is/WAS amazing looking at the data - this was even faster than even yours being i did 57gb in 80secs vs your 40gb in 69secs - but a one time thing!! The very next morning 26th Aug loaded up 300secs 190mb average! (i play 3 services a week) the following week Sept 2nd 485secs 119mb/s !!!! even slower !! looking through the rest of the past couple months all in the 267-400secs with the only change the other week to do a Alessandria update 1.11 which forced re-cache and now consistently getting 267secs...
Could this suggest a bad m2 cache drive? Or perhaps just a not very good design style version by crucial.... !?? Depending on where it moves the cache to on the drive seems to be very dependent on what speed HW can utilize. How on earth to get warranty when it works and passes all testing crucial software and generally all says it ok....!!!??
The weird thing is though no matter how slow HW pulls info off it GO pulls the same data in 54seconds! ???
I note that Disk i/o busy is high on mine when its bad up at 98 and 60 i/o for pct activity due to stage - notice on yours down at 12 and 5 and my good one run wonder also down same at 12 and 5 i/o - this is interesting.... do you know what this all means....?! The drive is so busy that its not coping? or?! Again how is HW pulling the data off differently to GO that it makes such a huge difference.... I guess this is what's been addressed in ver8 now even faster looking at your data than go!!
Could there be some funny issue with pci express shared and the graphics card in that lane down grading the m2 slot... i think i have tried in the alternate spot without any change.... will check... but again... the consistent thing is HW...go still pulls the data off no matter what setting or location of hardware.... MB spec says 2 x m2 pcie 4 - i have the cache drive in the top high performance spot near cpu...
I might tonight after work go and connect a brand new sitting around 960 samsung sata ssd to motherboard and delete all the cache on the m2 and rerun hw setup and move cache to the ssd to see what it pulls off of that once cached Alesssandria.... (i presume no other easier sneeky way?) With the idea that if its greatly improved with a ssd that I may need to go and buy another 1tb m2 get a samsung gen 4 980pro instead perhaps...??
wow so much faster than what im getting....
Total time: overall: 267.816 sec.
Total time: sample data: 264.342 sec.
Sample loader: loaded from data cache: Y.
Sample loader: data cache total disk size: 56304.56 MB.
Sample loader: buffers: 9.
Sample loader: approx. loader peak mem. usage during audio loading: 464.07 MB.
Sample loader: loader def. mem. usage during audio loading: 239.41 MB.
Sample loader: approx. loader mem. usage during trem. loading: 29.26 MB.
Sample loader: approx. avg. overall data read rate: 213.00 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 216.09 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 98.57.
Sample loader: pct. sample loading time stage busy: 1st proc.: 30.67.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 10.03.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 4.01.
Sample loader: pct. sample loading time stage busy: 4th proc.: 10.96.
Sample loader: pct. sample loading time stage busy: 5th proc.: 8.39.
Sample loader: pct. thread activity due to stage: disk I/O: 60.61.
Sample loader: pct. thread activity due to stage: 1st proc.: 18.86.
Sample loader: pct. thread activity due to stage: 2nd proc.: 6.17.
Sample loader: pct. thread activity due to stage: 3rd proc.: 2.47.
Sample loader: pct. thread activity due to stage: 4th proc.: 6.74.
Sample loader: pct. thread activity due to stage: 5th proc.: 5.16.
I checked and turned off page file for c: (thought i had) was already off for cache and install drives...
and got down to 4mins 240secs consistently - but thats as fast as it will go...
Trawling through the info dump log I see once back in august 25th with no changes to the system other than win11 and bios updates along the way... i had a much faster ONE time event! Wouldn't have even noticed it but rings a bell as used to it being so slow that i set and come back....
Total time: overall: 83.812 sec.
Total time: sample data: 80.731 sec.
Sample loader: loaded from data cache: Y.
Sample loader: data cache total disk size: 56307.81 MB.
Sample loader: buffers: 9.
Sample loader: approx. loader peak mem. usage during audio loading: 464.34 MB.
Sample loader: loader def. mem. usage during audio loading: 239.41 MB.
Sample loader: approx. loader mem. usage during trem. loading: 29.26 MB.
Sample loader: approx. avg. overall data read rate: 697.47 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 5850.85 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 11.92.
Sample loader: pct. sample loading time stage busy: 1st proc.: 99.79.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 33.82.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 15.98.
Sample loader: pct. sample loading time stage busy: 4th proc.: 36.88.
Sample loader: pct. sample loading time stage busy: 5th proc.: 28.52.
Sample loader: pct. thread activity due to stage: disk I/O: 5.25.
Sample loader: pct. thread activity due to stage: 1st proc.: 43.98.
Sample loader: pct. thread activity due to stage: 2nd proc.: 14.90.
Sample loader: pct. thread activity due to stage: 3rd proc.: 7.04.
Sample loader: pct. thread activity due to stage: 4th proc.: 16.25.
Sample loader: pct. thread activity due to stage: 5th proc.: 12.57.
This is/WAS amazing looking at the data - this was even faster than even yours being i did 57gb in 80secs vs your 40gb in 69secs - but a one time thing!! The very next morning 26th Aug loaded up 300secs 190mb average! (i play 3 services a week) the following week Sept 2nd 485secs 119mb/s !!!! even slower !! looking through the rest of the past couple months all in the 267-400secs with the only change the other week to do a Alessandria update 1.11 which forced re-cache and now consistently getting 267secs...
Could this suggest a bad m2 cache drive? Or perhaps just a not very good design style version by crucial.... !?? Depending on where it moves the cache to on the drive seems to be very dependent on what speed HW can utilize. How on earth to get warranty when it works and passes all testing crucial software and generally all says it ok....!!!??
The weird thing is though no matter how slow HW pulls info off it GO pulls the same data in 54seconds! ???
I note that Disk i/o busy is high on mine when its bad up at 98 and 60 i/o for pct activity due to stage - notice on yours down at 12 and 5 and my good one run wonder also down same at 12 and 5 i/o - this is interesting.... do you know what this all means....?! The drive is so busy that its not coping? or?! Again how is HW pulling the data off differently to GO that it makes such a huge difference.... I guess this is what's been addressed in ver8 now even faster looking at your data than go!!
Could there be some funny issue with pci express shared and the graphics card in that lane down grading the m2 slot... i think i have tried in the alternate spot without any change.... will check... but again... the consistent thing is HW...go still pulls the data off no matter what setting or location of hardware.... MB spec says 2 x m2 pcie 4 - i have the cache drive in the top high performance spot near cpu...
I might tonight after work go and connect a brand new sitting around 960 samsung sata ssd to motherboard and delete all the cache on the m2 and rerun hw setup and move cache to the ssd to see what it pulls off of that once cached Alesssandria.... (i presume no other easier sneeky way?) With the idea that if its greatly improved with a ssd that I may need to go and buy another 1tb m2 get a samsung gen 4 980pro instead perhaps...??
Re: Loading times very slow HW7
Thanks, mattrix.
From your log excerpt, when loading slowly, I can see that it's disc I/O-bound on your PC:
Hence for some reason the drive itself (or Windows) is being slow to read the data from the SSD. A hardware or driver performance problem would seem the most likely culprit, or some other processing running at the same time as Hauptwerk trying to do something with the file or drive. With regard to why large files might be read slower in Hauptwerk than another application, I can only think that either:
- Perhaps differences in chunk sizes in which the data are read between the two applications are affecting performance on your system somehow, or:
- Perhaps the fact that Hauptwerk's cache is encrypted is relevant somehow. E.g. perhaps some virus scanner software is trying to scan the file as it's being read, and is taking longer to do so because the data in the file have greater entropy. (However, as above, although Hauptwerk's data is encrypted, note that the bottleneck on your system isn't any processing/decryption within Hauptwerk itself -- it's external to Hauptwerk -- the rate at which the drive is supplying data to Hauptwerk.)
- Perhaps the file's position/size on the SSD is somehow affecting performance.
I'd suggest:
- Make sure that all available motherboard BIOS updates are applied.
- Make sure that all available Windows updates are applied.
- Make sure all available firmware and driver updates for hardware devices are applied, and especially so for audio/MIDI interfaces, the graphics card, and the motherboard. Check on their manufacturers' websites if needed.
- Test your RAM for errors.
- Test the drive for errors.
- Make absolutely sure that no virus scanner software, or similar security software, is running. Completely disable or uninstall any that you might have.
- Make sure that Windows drive indexing is disabled for all drives.
- Make sure Windows Superfetch/sysmain is disabled.
- Make sure that no other unnecessary applications or processes are running in the background. Look at the Processes tab in Windows Task Manager, see what is running, and especially watch whether any processes apart from Hauptwerk show significant levels of CPU or disc activity as the sample set is loading in Hauptwerk.
- Try running LatencyMon ( https://resplendence.com/latencymon ) when Hauptwerk is running, and see whether it reports any hardware/driver performance problems, especially as the sample set is loading in Hauptwerk.
- Make sure the drive itself isn't encrypted.
- Perhaps try reformatting the drive.
From your log excerpt, when loading slowly, I can see that it's disc I/O-bound on your PC:
(Hauptwerk's processing stages 1-5 are spending most of their time waiting for the data to be read from the drive.)Sample loader: pct. data loading time each stage busy: disk I/O: 98.57.
Sample loader: pct. sample loading time stage busy: 1st proc.: 30.67.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 10.03.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 4.01.
Sample loader: pct. sample loading time stage busy: 4th proc.: 10.96.
Sample loader: pct. sample loading time stage busy: 5th proc.: 8.39.
Hence for some reason the drive itself (or Windows) is being slow to read the data from the SSD. A hardware or driver performance problem would seem the most likely culprit, or some other processing running at the same time as Hauptwerk trying to do something with the file or drive. With regard to why large files might be read slower in Hauptwerk than another application, I can only think that either:
- Perhaps differences in chunk sizes in which the data are read between the two applications are affecting performance on your system somehow, or:
- Perhaps the fact that Hauptwerk's cache is encrypted is relevant somehow. E.g. perhaps some virus scanner software is trying to scan the file as it's being read, and is taking longer to do so because the data in the file have greater entropy. (However, as above, although Hauptwerk's data is encrypted, note that the bottleneck on your system isn't any processing/decryption within Hauptwerk itself -- it's external to Hauptwerk -- the rate at which the drive is supplying data to Hauptwerk.)
- Perhaps the file's position/size on the SSD is somehow affecting performance.
I'd suggest:
- Make sure that all available motherboard BIOS updates are applied.
- Make sure that all available Windows updates are applied.
- Make sure all available firmware and driver updates for hardware devices are applied, and especially so for audio/MIDI interfaces, the graphics card, and the motherboard. Check on their manufacturers' websites if needed.
- Test your RAM for errors.
- Test the drive for errors.
- Make absolutely sure that no virus scanner software, or similar security software, is running. Completely disable or uninstall any that you might have.
- Make sure that Windows drive indexing is disabled for all drives.
- Make sure Windows Superfetch/sysmain is disabled.
- Make sure that no other unnecessary applications or processes are running in the background. Look at the Processes tab in Windows Task Manager, see what is running, and especially watch whether any processes apart from Hauptwerk show significant levels of CPU or disc activity as the sample set is loading in Hauptwerk.
- Try running LatencyMon ( https://resplendence.com/latencymon ) when Hauptwerk is running, and see whether it reports any hardware/driver performance problems, especially as the sample set is loading in Hauptwerk.
- Make sure the drive itself isn't encrypted.
- Perhaps try reformatting the drive.
Best regards, Martin.
Hauptwerk software designer/developer, Milan Digital Audio.
Hauptwerk software designer/developer, Milan Digital Audio.
Re: Loading times very slow HW7
P.S.
N.B. First of all, do also make sure that the caches are actually on your fast NVMe SSD (as opposed to on a slow HDD/SSD, for example): after launching Hauptwerk, use "Help | View activity log" and look in the "INF:4165 Welcome to Hauptwerk" message, and check that the "Hauptwerk temporary files folder" is on your fast SSD. Also check that there's plenty of free space on that drive.
For best performance of the PC in general, it's also a good a idea for the "Operating system temporary files folder" to be on a reasonably fast SSD, as well as Windows itself (typically C:), and for there to be ample free space on that/those drives.
N.B. First of all, do also make sure that the caches are actually on your fast NVMe SSD (as opposed to on a slow HDD/SSD, for example): after launching Hauptwerk, use "Help | View activity log" and look in the "INF:4165 Welcome to Hauptwerk" message, and check that the "Hauptwerk temporary files folder" is on your fast SSD. Also check that there's plenty of free space on that drive.
For best performance of the PC in general, it's also a good a idea for the "Operating system temporary files folder" to be on a reasonably fast SSD, as well as Windows itself (typically C:), and for there to be ample free space on that/those drives.
Best regards, Martin.
Hauptwerk software designer/developer, Milan Digital Audio.
Hauptwerk software designer/developer, Milan Digital Audio.
Re: Loading times very slow HW7
Ok so i installed a new ssd via sata 6 port samsung 860 evo as the cache drive - redid the install and moved it....
bam loads in 2min! but still high i/o but half the time of the m2....
Total time: overall: 123.573 sec.
Total time: sample data: 120.261 sec.
Sample loader: loaded from data cache: Y.
Sample loader: data cache total disk size: 56304.56 MB.
Sample loader: buffers: 9.
Sample loader: approx. loader peak mem. usage during audio loading: 456.24 MB.
Sample loader: loader def. mem. usage during audio loading: 239.41 MB.
Sample loader: approx. loader mem. usage during trem. loading: 29.26 MB.
Sample loader: approx. avg. overall data read rate: 468.19 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 469.93 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 99.63.
Sample loader: pct. sample loading time stage busy: 1st proc.: 67.85.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 22.52.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 10.34.
Sample loader: pct. sample loading time stage busy: 4th proc.: 24.95.
Sample loader: pct. sample loading time stage busy: 5th proc.: 20.23.
Sample loader: pct. thread activity due to stage: disk I/O: 40.58.
Sample loader: pct. thread activity due to stage: 1st proc.: 27.64.
Sample loader: pct. thread activity due to stage: 2nd proc.: 9.17.
Sample loader: pct. thread activity due to stage: 3rd proc.: 4.21.
Sample loader: pct. thread activity due to stage: 4th proc.: 10.16.
Sample loader: pct. thread activity due to stage: 5th proc.: 8.24.
I see that the install has created a temp file on the ssd as there was one same on the m2
yes f drive the new ssd has all the caches on it.... Hauptwerk temporary files folder: F:/HauptwerkInternalWorkingFiles (disk free: 145.7 GB).
that is very much maxing out the ssd 860 chrystal disk mark8 i tested on m2 read 5120 on the ssd now says 560 which is the max as reported by samsung and HW is getting 469 so close enough...
All drivers up to date no virus protection all disabled from fresh install start of year. sysmain disabled latest bios latest drivers for focusrite 18i20 etc....
being that m2 is empty now will format guid ntfs default allocation size but clearly the ssd is twice as fast but still have issues....
nothing else running
nothing else has access to the cache drive,,,, nothing else installed on it or running....
Just disabled indexing on c
Just disabled defender will see....
bam loads in 2min! but still high i/o but half the time of the m2....
Total time: overall: 123.573 sec.
Total time: sample data: 120.261 sec.
Sample loader: loaded from data cache: Y.
Sample loader: data cache total disk size: 56304.56 MB.
Sample loader: buffers: 9.
Sample loader: approx. loader peak mem. usage during audio loading: 456.24 MB.
Sample loader: loader def. mem. usage during audio loading: 239.41 MB.
Sample loader: approx. loader mem. usage during trem. loading: 29.26 MB.
Sample loader: approx. avg. overall data read rate: 468.19 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 469.93 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 99.63.
Sample loader: pct. sample loading time stage busy: 1st proc.: 67.85.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 22.52.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 10.34.
Sample loader: pct. sample loading time stage busy: 4th proc.: 24.95.
Sample loader: pct. sample loading time stage busy: 5th proc.: 20.23.
Sample loader: pct. thread activity due to stage: disk I/O: 40.58.
Sample loader: pct. thread activity due to stage: 1st proc.: 27.64.
Sample loader: pct. thread activity due to stage: 2nd proc.: 9.17.
Sample loader: pct. thread activity due to stage: 3rd proc.: 4.21.
Sample loader: pct. thread activity due to stage: 4th proc.: 10.16.
Sample loader: pct. thread activity due to stage: 5th proc.: 8.24.
I see that the install has created a temp file on the ssd as there was one same on the m2
yes f drive the new ssd has all the caches on it.... Hauptwerk temporary files folder: F:/HauptwerkInternalWorkingFiles (disk free: 145.7 GB).
that is very much maxing out the ssd 860 chrystal disk mark8 i tested on m2 read 5120 on the ssd now says 560 which is the max as reported by samsung and HW is getting 469 so close enough...
All drivers up to date no virus protection all disabled from fresh install start of year. sysmain disabled latest bios latest drivers for focusrite 18i20 etc....
being that m2 is empty now will format guid ntfs default allocation size but clearly the ssd is twice as fast but still have issues....
nothing else running
nothing else has access to the cache drive,,,, nothing else installed on it or running....
Just disabled indexing on c
Just disabled defender will see....
Re: Loading times very slow HW7
After disabling defender entirely .... Same well 1 sec quicker 119sec but this is very much max of the ssd so now have reformatted the m2 and will xfer back to that... still the very high i/o 99
C is on another m2 crucial gen 3 os temp folder by default on there... chrystal reporting 2400mb/s read 1400m/b write from c and plenty of space 200gb free
Have run checkers on the drives all good.
will check the ram - but given the ssd is performing now twice as fast as the m2 nvme its not ram surely....!??
Maybe should try a clean OS install on a ssd for the C drive take out the m2 c drive to test.... try win10 instead of 11?! I can't be bothered doing that right now.... not too sure what to do now.... Go back to the m2 for the cache i guess but the high i/o suggests what>? even though getting twice the speed loading with the ssd... reformatted crystal again still showing 5138 read and 3574 write interestingly the second tab of testing on the ssd seems to be exactly what HW 469mb/s got and on the m2 testing it says 3099 if at all relevant and real world in out i should be getting 3099 in HW perhaps or just a fluke SEQ1M Q1T1 seems to be what we are now getting....
C is on another m2 crucial gen 3 os temp folder by default on there... chrystal reporting 2400mb/s read 1400m/b write from c and plenty of space 200gb free
Have run checkers on the drives all good.
will check the ram - but given the ssd is performing now twice as fast as the m2 nvme its not ram surely....!??
Maybe should try a clean OS install on a ssd for the C drive take out the m2 c drive to test.... try win10 instead of 11?! I can't be bothered doing that right now.... not too sure what to do now.... Go back to the m2 for the cache i guess but the high i/o suggests what>? even though getting twice the speed loading with the ssd... reformatted crystal again still showing 5138 read and 3574 write interestingly the second tab of testing on the ssd seems to be exactly what HW 469mb/s got and on the m2 testing it says 3099 if at all relevant and real world in out i should be getting 3099 in HW perhaps or just a fluke SEQ1M Q1T1 seems to be what we are now getting....
Re: Loading times very slow HW7
Hello mattrix,
Thanks for the update, and glad to hear you're making worthwhile progress.
Do still also try these, if you haven't already done so:
Thanks for the update, and glad to hear you're making worthwhile progress.
The problem probably isn't the RAM, but it would be worth checking anyway, for good measure.mattrix wrote:will check the ram - but given the ssd is performing now twice as fast as the m2 nvme its not ram surely....!??
Do still also try these, if you haven't already done so:
mdyde wrote:- Make sure that no other unnecessary applications or processes are running in the background. Look at the Processes tab in Windows Task Manager, see what is running, and especially watch whether any processes apart from Hauptwerk show significant levels of CPU or disc activity as the sample set is loading in Hauptwerk.
- Try running LatencyMon ( https://resplendence.com/latencymon ) when Hauptwerk is running, and see whether it reports any hardware/driver performance problems, especially as the sample set is loading in Hauptwerk.
Best regards, Martin.
Hauptwerk software designer/developer, Milan Digital Audio.
Hauptwerk software designer/developer, Milan Digital Audio.
Re: Loading times very slow HW7
P.S.
Yes -- I think you are now getting about as good a performance as is likely to be possible from a Samsung 860 EVO, given that Samsung's specification for it is a sequential read rate of 550 MB/s. Hence there doesn't appear to be a problem there. However, an NVME SSD should be able to perform faster if it's working properly (i.e. with loading performance not being I/O-bound, which would result in a lower figure for "Sample loader: pct. data loading time each stage busy: disk I/O").mattrix wrote:Ok so i installed a new ssd via sata 6 port samsung 860 evo as the cache drive - redid the install and moved it....
bam loads in 2min! but still high i/o but half the time of the m2....
...
Sample loader: approx. avg. overall data read rate: 468.19 MB/s.
Best regards, Martin.
Hauptwerk software designer/developer, Milan Digital Audio.
Hauptwerk software designer/developer, Milan Digital Audio.
Re: Loading times very slow HW7
Still on 860 ssd
Latencymon:
hauptwerk.exe
Hauptwerk (Intel 64-bit, AVX2)
48
1364
Milan Digital Audio LLC
Hauptwerk
So 48 Hard page faults and 1364 PID if that means anything....
All green latency mon whilst HW running and loading....
Without anything running its a hot mess of nvidea and directx reading online the usual culprits i updated graphics drivers no change...
Nothing running up cpu etc whilst loading... just hw 10-15% cpu whilst loading everything else 0
um so this just happened as im writing above.... had taskmanager open monitoring thats it....
Organ ID: 002511.
Organ ver.: 1.11.
Organ min. Hauptwerk ver.: 6.0.2.000.
Mem.: total raw audio/trem. sample data mem. (excl. pre-proc. audio, images, or other data): 55703.05 MB.
Mem.: overall compression ratio for compressed ranks: 47.26 pct.
Total time: overall: 91.332 sec.
Total time: sample data: 87.522 sec.
Total time: all except sample data: 3.810 sec.
Time: prep.: 0.132 sec.
Time: organ def. loading/parsing: 0.522 sec.
Time: organ def. object/key validation: 0.148 sec.
Time: organ def. prep.: 0.087 sec.
Time: prep. for loading settings: 0.003 sec.
Time: settings loading/parsing: 0.020 sec.
Time: settings object/key validation: 0.007 sec.
Time: settings prep.: 0.004 sec.
Time: data cache validation/prep.: 0.284 sec.
Time: audio data: 87.522 sec.
Time: trem. data: 0.000 sec.
Time: images: 2.340 sec.
Time: comb. and temper. files: 0.263 sec.
Sample loader: loaded from data cache: Y.
Sample loader: data cache total disk size: 56304.56 MB.
Sample loader: buffers: 9.
Sample loader: approx. loader peak mem. usage during audio loading: 456.24 MB.
Sample loader: loader def. mem. usage during audio loading: 239.41 MB.
Sample loader: approx. loader mem. usage during trem. loading: 29.26 MB.
Sample loader: approx. avg. overall data read rate: 643.32 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 5475.09 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 11.75.
Sample loader: pct. sample loading time stage busy: 1st proc.: 97.15.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 35.55.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 15.38.
Sample loader: pct. sample loading time stage busy: 4th proc.: 37.99.
Sample loader: pct. sample loading time stage busy: 5th proc.: 28.70.
Sample loader: pct. thread activity due to stage: disk I/O: 5.19.
Sample loader: pct. thread activity due to stage: 1st proc.: 42.89.
Sample loader: pct. thread activity due to stage: 2nd proc.: 15.69.
Sample loader: pct. thread activity due to stage: 3rd proc.: 6.79.
Sample loader: pct. thread activity due to stage: 4th proc.: 16.77.
Sample loader: pct. thread activity due to stage: 5th proc.: 12.67.
HOW? how is it suddenly going faster with the little ssd 860? and all nice i/o as well....
I ran it havn't changed anything and just loaded it again whilst typing this....!!!
i changed in bios some cpu things lowered wattage and such for long and short... but that didn't change anything..... closed HW and ran again and then this....!!!???
Latencymon:
hauptwerk.exe
Hauptwerk (Intel 64-bit, AVX2)
48
1364
Milan Digital Audio LLC
Hauptwerk
So 48 Hard page faults and 1364 PID if that means anything....
All green latency mon whilst HW running and loading....
Without anything running its a hot mess of nvidea and directx reading online the usual culprits i updated graphics drivers no change...
Nothing running up cpu etc whilst loading... just hw 10-15% cpu whilst loading everything else 0
um so this just happened as im writing above.... had taskmanager open monitoring thats it....
Organ ID: 002511.
Organ ver.: 1.11.
Organ min. Hauptwerk ver.: 6.0.2.000.
Mem.: total raw audio/trem. sample data mem. (excl. pre-proc. audio, images, or other data): 55703.05 MB.
Mem.: overall compression ratio for compressed ranks: 47.26 pct.
Total time: overall: 91.332 sec.
Total time: sample data: 87.522 sec.
Total time: all except sample data: 3.810 sec.
Time: prep.: 0.132 sec.
Time: organ def. loading/parsing: 0.522 sec.
Time: organ def. object/key validation: 0.148 sec.
Time: organ def. prep.: 0.087 sec.
Time: prep. for loading settings: 0.003 sec.
Time: settings loading/parsing: 0.020 sec.
Time: settings object/key validation: 0.007 sec.
Time: settings prep.: 0.004 sec.
Time: data cache validation/prep.: 0.284 sec.
Time: audio data: 87.522 sec.
Time: trem. data: 0.000 sec.
Time: images: 2.340 sec.
Time: comb. and temper. files: 0.263 sec.
Sample loader: loaded from data cache: Y.
Sample loader: data cache total disk size: 56304.56 MB.
Sample loader: buffers: 9.
Sample loader: approx. loader peak mem. usage during audio loading: 456.24 MB.
Sample loader: loader def. mem. usage during audio loading: 239.41 MB.
Sample loader: approx. loader mem. usage during trem. loading: 29.26 MB.
Sample loader: approx. avg. overall data read rate: 643.32 MB/s.
Sample loader: approx. avg. data read rate during disk reader activity: 5475.09 MB/s.
Sample loader: pct. data loading time each stage busy: disk I/O: 11.75.
Sample loader: pct. sample loading time stage busy: 1st proc.: 97.15.
Sample loader: pct. sample loading time stage busy: 2nd proc.: 35.55.
Sample loader: pct. sample loading time stage busy: 3rd proc.: 15.38.
Sample loader: pct. sample loading time stage busy: 4th proc.: 37.99.
Sample loader: pct. sample loading time stage busy: 5th proc.: 28.70.
Sample loader: pct. thread activity due to stage: disk I/O: 5.19.
Sample loader: pct. thread activity due to stage: 1st proc.: 42.89.
Sample loader: pct. thread activity due to stage: 2nd proc.: 15.69.
Sample loader: pct. thread activity due to stage: 3rd proc.: 6.79.
Sample loader: pct. thread activity due to stage: 4th proc.: 16.77.
Sample loader: pct. thread activity due to stage: 5th proc.: 12.67.
HOW? how is it suddenly going faster with the little ssd 860? and all nice i/o as well....
I ran it havn't changed anything and just loaded it again whilst typing this....!!!
i changed in bios some cpu things lowered wattage and such for long and short... but that didn't change anything..... closed HW and ran again and then this....!!!???
Re: Loading times very slow HW7
oh its not loading it from the ssd it must be in memory still or somewhere as the ssd shows no activity..... 0%
unloaded the organ and did it again not pulling it from the ssd!! its virtual somewhere!!
Lol even closing HW and reopening loading alessandria and its super fast no activity on the ssd...
unloaded the organ and did it again not pulling it from the ssd!! its virtual somewhere!!
Lol even closing HW and reopening loading alessandria and its super fast no activity on the ssd...
Re: Loading times very slow HW7
Hello mattrix,
When you load any file, the operating system will attempt to keep some or all of the file cached in 'spare' RAM, so that if you load the file again it may be able to read some or all of it from RAM, avoiding the need to load it from the drive. Hence if you load a large sample set, but still have a reasonable amount of 'free' RAM remaining, and you then load the sample set again, the OS may actually read the cached file system data from RAM (not from the drive), thus allowing it to load faster. That's normal OS behaviour (file system caching).
When you load any file, the operating system will attempt to keep some or all of the file cached in 'spare' RAM, so that if you load the file again it may be able to read some or all of it from RAM, avoiding the need to load it from the drive. Hence if you load a large sample set, but still have a reasonable amount of 'free' RAM remaining, and you then load the sample set again, the OS may actually read the cached file system data from RAM (not from the drive), thus allowing it to load faster. That's normal OS behaviour (file system caching).
If the LatencyMon result is green then it hasn't found any problems, so that should be fine.mattrix wrote:till on 860 ssd
Latencymon:
hauptwerk.exe
Hauptwerk (Intel 64-bit, AVX2)
48
1364
Milan Digital Audio LLC
Hauptwerk
So 48 Hard page faults and 1364 PID if that means anything....
All green latency mon whilst HW running and loading....
Best regards, Martin.
Hauptwerk software designer/developer, Milan Digital Audio.
Hauptwerk software designer/developer, Milan Digital Audio.