The following are basically benchmarks of the different sparse matrix/vector multiplication methods of Eigen and cuSPARSE.

These benchmarks are supposed to be worst-case scenarios when doing candidate search, e.g. these benchmarks assume that every peptide would yield 100 ions and every spectrum 1000 peaks, while also performing normalization and gaussian peak modeling.

We ran benchmarks for different database sizes (different number of candidate peptides to be considered) to assess how that influences performance of the different methods. Furthermore, every benchmark is run five times to get a more comprehensive overview of computation times. The averages are plotted below, with error bars denoting standard deviation.

For all benchmarks we search 1001 spectra (this is specifically selected to see if batched multiplication has influence on performance) and return the top 100 candidates. All benchmarks were conducted during light background usage (e.g. open browser, text editor, etc.).

Abbreviations

The following terms are used synonymously throughout the document:

• f32CPU_SV: Float32-(CPU-)based sparse matrix * sparse vector search (using Eigen)
• i32CPU_SV: Int32-(CPU-)based sparse matrix * sparse vector search (using Eigen)
• f32CPU_DV: Float32-(CPU-)based sparse matrix * dense vector search (using Eigen)
• i32CPU_DV: Int32-(CPU-)based sparse matrix * dense vector search (using Eigen)
• f32CPU_SM: Float32-(CPU-)based sparse matrix * sparse matrix search (using Eigen)
• i32CPU_SM: Int32-(CPU-)based sparse matrix * sparse matrix search (using Eigen)
• f32CPU_DM: Float32-(CPU-)based sparse matrix * dense matrix search (using Eigen)
• i32CPU_DM: Int32-(CPU-)based sparse matrix * dense matrix search (using Eigen)
• f32GPU_DV: Float32-(GPU-)based sparse matrix * dense vector search (using cuSPARSE)
• f32GPU_DM: Float32-(GPU-)based sparse matrix * dense matrix search (using cuSPARSE)
• f32GPU_SM: Float32-(GPU-)based sparse matrix * sparse matrix search (using cuSPARSE)

System 1 - Standard Office PC

The first system we tested this on was a standard office laptop with the following hardware:

• Model: Dell Precision 3560
• CPU: Intel Core i7-1185G7 [4 cores @ 1.8 GHz base / 3.0 GHz boost]
• RAM: 16 GB DDR4 RAM [3200 MT/s, NA CAS]
• GPU: Nvidia T500 [2 GB VRAM]
• SSD/HDD: 512 GB NVMe SSD
• OS: Windows 10 Education 64-bit (10.0, Build 19045)

10 000 Candidates

A * B = C where A[10000, 500000] and B[500000, 1001]

Using a database of 10 000 peptide candidates the methods yield the following runtimes:

Figure 1: Float32-based sparse matrix * dense vector search using Eigen yields the fastest computation time of only 1.02 seconds.

100 000 Candidates

A * B = C where A[100000, 500000] and B[500000, 1001]

Using a database of 100 000 peptide candidates the methods yield the following runtimes:

Figure 2: Float32-based sparse matrix * sparse matrix search using Eigen yields the fastest computation time of only 5.08 seconds. Note that f32GPU_SM has been excluded from the plot since computation times exceeded all other methods by more than 10-fold. The raw data is available below.

1 000 000 Candidates

A * B = C where A[1000000, 500000] and B[500000, 1001]

Using a database of 1 000 000 peptide candidates the methods yield the following runtimes:

Figure 3: Int32-based sparse matrix * sparse matrix search using Eigen yields the fastest computation time of only 45.04 seconds. Note that f32GPU_SM has been excluded from the plot since the method ran out of memory. The raw data is available below.

2 500 000 Candidates

A * B = C where A[2500000, 500000] and B[500000, 1001]

Using a database of 2 500 000 peptide candidates the methods yield the following runtimes:

Figure 4: Float32-based sparse matrix * sparse matrix search using Eigen yields the fastest computation time of only 101.42 seconds. Note that f32GPU_DM has been excluded from the plot since the computation time exceeded that of other methods by more than 10-fold and f32GPU_SM has been excluded from the plot since the method ran out of memory. The raw data is available below.

5 000 000 Candidates

A * B = C where A[5000000, 500000] and B[500000, 1001]

Using a database of 5 000 000 peptide candidates the methods yield the following runtimes:

Figure 5: Float32-based sparse matrix * sparse matrix search using Eigen yields the fastest computation time of only 210.98 seconds. Note that all GPU-based methods have been excluded from the plot since their computation times exceeded that of CPU-based methods by more than 10-fold or because they ran out of memory. The raw data is available below.

System 2 - High Performance PC

The second system we tested this on was a more powerful desktop PC with the following (more recent) hardware:

• MB: ASUS ROG Strix B650E-I
• CPU: AMD Ryzen 7900X [12 cores @ 4.7 GHz base / 5.6 GHz boost]
• RAM: Kingston 64 GB DDR5 RAM [5600 MT/s, 36 CAS]
• GPU: ASUS Dual [Nvidia] GeForce RTX 4060 Ti OC [16 GB VRAM]*
• SSD/HDD: Corsair MP600 Pro NH 2 TB NVMe SSD [PCIe 4.0]
• OS: Windows 11 Pro 64-bit (10.0, Build 22631)

*_Note:_ Dual is part of the name, this is a single graphics card!

10 000 Candidates

A * B = C where A[10000, 500000] and B[500000, 1001]

Using a database of 10 000 peptide candidates the methods yield the following runtimes:

Figure 6: Float32-based sparse matrix * dense matrix search using cuSPARSE yields the fastest computation time of only 0.59 seconds. The raw data is available below.

100 000 Candidates

A * B = C where A[100000, 500000] and B[500000, 1001]

Using a database of 100 000 peptide candidates the methods yield the following runtimes:

Figure 7: Float32-based sparse matrix * dense matrix search using cuSPARSE yields the fastest computation time of only 1.75 seconds. Note that GPU-based sparse matrix * sparse matrix search has been excluded from the plot since its computation time exceeded that of all other methods by almost 20-fold. The raw data is available below.

1 000 000 Candidates

A * B = C where A[1000000, 500000] and B[500000, 1001]

Using a database of 1 000 000 peptide candidates the methods yield the following runtimes:

Figure 8: Float32-based sparse matrix * dense vector search using cuSPARSE yields the fastest computation time of only 13.53 seconds. Note that GPU-based sparse matrix * sparse matrix search was not measured due to its extremely long computation time already evident from the 100 000 candidate benchmark. The raw data is available below.

2 500 000 Candidates

A * B = C where A[2500000, 500000] and B[500000, 1001]

Using a database of 2 500 000 peptide candidates the methods yield the following runtimes:

Figure 9: Float32-based sparse matrix * dense vector search using cuSPARSE yields the fastest computation time of only 33.93 seconds. Note that GPU-based sparse matrix * sparse matrix search was not measured due to its extremely long computation time already evident from the 100 000 candidate benchmark. The raw data is available below.

5 000 000 Candidates

A * B = C where A[5000000, 500000] and B[500000, 1001]

Using a database of 5 000 000 peptide candidates the methods yield the following runtimes:

Figure 10: Float32-based sparse matrix * dense vector search using cuSPARSE yields the fastest computation time of only 68.27 seconds. Note that GPU-based sparse matrix * sparse matrix search was not measured due to its extremely long computation time already evident from the 100 000 candidate benchmark. The raw data is available below.

Beyond 5 000 000 Candidates

Here are some single benchmarks of the best performing CPU-based search i32CPU_SM and the best best performing GPU-based search f32GPU_DV:

You might notice that the maximum number of tested candidates is 21 474 835, this is because this results in 2 147 483 500 non-zero elements, close to the maximum of a signed int32 type value. Going beyond that is impossible with the provided implementation. In case you really need to go beyond that, please adapt the implementation accordingly (e.g. using unsigned int32 or int64 data types instead). See also this issue.

Conclusions

CPU-based sparse matrix * sparse matrix search is generally a good choice, no matter the system configuration. Choosing an Int32- or Float32-based approach usually does not make a considerable difference, we recommend going with the Int32 variant as it ensures better reproducibility of results, eliminating any kind of deviations due to floating point shenanigans. If a decent GPU (e.g. anything comparable to an Nvidia GeForce RTX 4060 Ti 16 GB or better) is available, running a GPU-based search is the more performant choice. We recommend going with the sparse matrix * dense vector approach as it requires less GPU memory. The GPU-based sparse matrix * sparse matrix search appears to be the worst of the tested methods, yielding very long computation times and exceedingly high memory usage. This is most likely due to the fact that the algorithm assumes that the resulting matrix is also sparse, which almost never will be the case.