New NVMe as vSAN caching device
The first tests and benchmarks of my Homelab Version 1.5 vSAN cluster showed a bitter truth, the performance was really disappointing.
I tested the vSAN with the VMware Fling HCIBench and also various other synthetic IO generators that are freely available.
Especially the write performance showed a big gap to my expectation. Even my NAS system which was built from 4 spindles and connected with 10Gbit via NFS to my vSphere cluster could write faster than my all-flash vSAN.
My vSAN at this point consisted of 3x Samsung MZVKW512HMJP (equal to SM961) 512GB NVMe drives for cache & 9x Samsung 870 QVO 1TB SATA SSDs for capacity.
Therefore, I expected quite a bit more performance as the benchmarks showed.
M.2 Samsung NVMe with PCIe Adapter 
NVMe more in detail
Results of the benchmark with Samsung SM961 NVMe:
Tested with the following configuration:
Results (4k Blocksize, 70% Read, 30% Write, 100% Random IO):
Conclusion:
Since the write performance in a vSAN all-flash cluster is largely dependent on the cache drives used and the NVMe’s I selected Samsung MZVKW512HMJP (equal to SM961) 512GB NVMe drives do not deliver the 4k write performance I expect, they need to be replaced.
After a long time of looking through datasheets of different NVMe drives and a price analysis, because my budget for my Homelab was already exhausted, I found what I was looking for and decided to buy Intel DC P4600 NVMe drives at a price that was just acceptable. These drives had a very good 4k write performance in the datasheet.
New NVMe for vSAN cache is Intel DC P4600:
Intel DC P4600 in Server with PCIe Adapter
Results of the benchmark with Intel DC P4600 (4k Blocksize, 70% Read, 30% Write, 100% Random IO)::
Final conclusion:
The benchmark with the new Intel DC P4600 NVMe cache drives was at nearly least about twice as fast in write performance and IOps as before with the Samsung NVMes. However, I was still not completely satisfied, and by a lucky circumstance, I found 8x 960GB SAS-SSDs on eBay, which should then form the capacity tier for my vSAN in the future. How it goes on, you can read in Homelab Version 1.7 🙂