DXT1 sequence + NTFS compression

Hey all!

As some of you know, the HAP codec is 2-stage compression:

  1. DDS-style (block based / domain specific) compression on GPU using DXT1
  2. Zip-style (entropy based / general purpose) compression on CPU using Snappy

I’m currently playing back 819201080 in DXT1 at 24fps, and i’d like to get up to 60fps. With HAP this seems to work, and it makes me wonder if i can maybe get similar results simply by enabling NTFS compression on the SSD which it’s running on.

NTFS compression means that the OS stores the files on the disk in a compressed format (like zip codec), and uses the CPU to transparently transform the files into their uncompressed form as they are being read from disk. This seems theoretically to offer the same functionality as the second layer of HAP, unless NTFS compression has some quirks which cause it to slow down (e.g. it presumes it will be faster if it’s uncompressed on disk, so it creates disk-stored caches of files and uses those instead)

Has anybody tried this?

Note I’d love to have either:

  • M.2 SSD
  • Raid 0 SSD
  • PCIe based SSD

but I don’t :). So looking for alternatives

Hi. I know vux experimenting with something like dds compression… not sure it’s ready yet.
In any case you better be careful with that stuff. Dds is a lot of small files, there is a known issue with data segmentation… So if u rewrite sequence or part of it many times u might end up getting broken frames…
Also if u are using black and white footage, i would try to export in format with only one channel like R32, this should improve speed/reduce size for like 4 times.

I do not suppose NTFS compression brings something.
M2 / PCIe is faster than SATA but it depends on the SSD.
I use either RAID1 or RAID10.

I can confirm with my test case that NTFS compression DOES OFFER SPEED BENEFITS.

Test setup:

  • 15360 x 1080
  • DXT1 DDS image stream
  • ~10,000 frame duration
  • Samsung 850 Evo drive
  • Dual Xeon E5-2630 v3 @ 2.4GHz
  • Quadro M4000 (seem to get exact same results on GeForce 1080 also)
  • CrystalDiskMark read speed ~ 540 MB/s

Results

Without NTFS compression

  • 26fps (Buffer underrun occurs with consistent personality at higher playback speeds)

  • 92.6 GB on disk

  • Effective read speed ~ 234 MB/s

With NTFS compression

  • <1000 frames - 60fps OK

  • <3000 frames - 30fps OK

  • All frames (10,600) - 29fps OK

  • 23.8GB on disk

  • Reported read speed ~ 740MB/s

  • Effective read speed at 29fps ~ 261 MB/s

Notes

The “System and compressed memory” process only uses 4% CPU generally (approximately 3 threads on my setup). With HAP’s “chunked” option, you could theoretically use more threads for decompression, but perhaps we’re limited more by disk read speed than by CPU (suggesting more aggressive CPU compression might not be helpful).

I’m not sure exactly what causes the buffer underruns to appear in the latter parts of playback. They tend to appear all of a sudden, and sometimes the playback recovers (up to 100% buffer) within 500 frames. I suggest playing the whole sequence back many times to guarantee your performance speed.

The symptom of failing at larger frame counts is that it can’t read from the disk fast enough. My theories for why this happens:

  1. Windows is running out of memory at this point, and starts unloading old frames whilst loading new ones (it’s been keeping a cache of all the uncompressed frames in memory, which is unnecessary)
  2. It caches the frames to uncompressed disk space (making readback slower)

Most likely the cause is (1).

Final notes

When evaluating your setup, the essential number to look at is the disk usage speed in Task Manager. By using disk compression we’re effectively increasing our disk read speed, and that speed is still the limiting factor of playback. (You might notice that the number only rises after you’ve started playing back for a little while, this is because Windows automatically creates a RAM cache of files).

By ‘effective disk speed’ I mean the equivalent speed at which we can read the DDS’s from disk if they were uncompressed. In the case of NTFS compression, this number can be read from task manager.

This thread is about trying to mimic the personality of HAP codec by using NTFS compression. Although I understand that RAID, M.2, etc are great, there’s plenty of other threads about those options for increasing effective disk speed. Please note that using a faster SSD in combination with NTFS compression will further increase your effective disk speed.

When performing the compression, the disk speed is ~ 200MB/s and CPU is 0.9% (process name is COM Surrogate). This speed is a combination of read and write together.

The HAP compressed version of the same image sequence is 16GB, suggesting that Snappy (HAP’s CPU codec) is more aggressive than the NTFS compression codec, and will allow for better effective disk read speeds (higher fps playback than NTFS compression alone). I don’t know what codec NTFS compression uses.

so HAP with NTFS compression and shitloads of Ram could make it work even better?

@sunep - even without available RAM you still get a performance increase of 10%

I guess NTFS compression also opens up the option of playing from an NTFS compressed RAMdisk (getting ~4x capacity with my DXT1 stream, i.e. RAM disk is effectively 4x larger)

ah sorry! misread that :)
HAP performance will NOT increase if you use NTFS compression, since HAP is already entropy compressed (zipping a zip doesn’t make it smaller on the disk)