4

Is there a way to use my main hard drive which is a NVMe pci3 SSD as a cache?

The use case is this:

When I sent large files from my desktop computer, to my Ubuntu server 21.04, I want my Ubuntu server to first dump the files I'm transferring to the NVMe drive and then to its intended destination to one of my mechanical 3.5 Hard drives.

Reason? To max out my 10 Gbit network connect I have between the two devices and to also (future-proof) max out 25 Gbit network connection when I upgrade.

Is this possible?

So basically how it goes is this:

I have sent a 10 GB video file from Computer A, and I want to transfer it to Server PC at location /somelocation/mechanicalHDD/videos and the way I envision it working in the background is that my server PC will first receive the files from my network connection directly to my main NVMe SSD drive (/home/cache) and then my server transfers from /home/cache to /somelocation/mechanicalHDD.

My main PC doesn't have to wait until my home server transfers it from /home/cache to /somelocation/mechanicalHDD. It only cares that I have successfully transferred it somewhere in my server for quick transfer speeds.

Pablo Bianchi
  • 17,371
Jono
  • 541
  • 3
  • 11
  • 19

2 Answers2

5

ZFS allows you to add your SSD as a cache device for your HDD (needs to be in ZFS as well) by configuring what is called a storage pool ... There are other caching solutions as well.

That being said, caching is not actually what you want because your copying command/program will still wait until the file has been fully written to the destination directory i.e. caching role is very minimal in ending the copying process earlier.

To achieve your requirement:

I have say a 10gb video file from Computer A, and I want to transfer it to Server PC at location /somelocation/mechanicalHDD/videos and the way I invision it working in the background is that my server PC will first receive the files from my network connection directly to my main nvme drive (/home/cache) and then my server transfers from /home/cache to /somelocation/mechanicalHDD.

My main PC doesn't have to wait until my home server transfers it from /home/cache to /somelocation/mechanicalHDD. It only cares that i have successfully transferred it somewhere in my server for quick transfer speeds.

I would suggest that you just copy files straight to your server's SSD and run a script to check and move whatever it finds on the SSD to the HDD ... This more reliable (files are permanently saved to the SSD in the first place) and faster (will finish copying process from your desktop to the server faster than write-caching to the HDD) than any caching workaround you might implement other than using ZFS ... the script would be as simple as:

#!/bin/bash

source_d="/home/cache/" # Specify the source directory.
destination_d="/somelocation/mechanicalHDD/videos/" # Specify the destination directory.


inotifywait -m -q -e close_write "$source_d" |


while read -r path action file; do
    echo mv -n -- "$path$file" "$destination_d$file" # "echo" is there for a dry-run(simulation) ... remove it to enable moving files
  done
Raffa
  • 34,963
3

It looks like bcache would do what you need.

Here's an answer that, while a bit outdated, will give you the basics:

How to setup bcache?

There's also Arch Linux's excellent documentation as a reference.

roadmr
  • 34,802