Always copy FROM TreeSize!!

1202 (alarm) I'm having my first go at updating the backups. This consists of copying the latest load of downloads from the Living Room PC (LRPC) (where I do most downloading) to an external HDD; an important next step is sorting the load of downloads into folders which match the sorted folders in the backup locations.

Then, this ExtHDD gets plugged into various other locations and the new data gets copied across to the relevant folders in the backup locations.

I've copied to a couple of locations so far, but have made a bit of a mistake: I've been using File Mangler to do the copying…

As previously detailed, FM (Windows Explorer) doesn't like copying folders it hasn't got permission to access, even though I can give it access without a check or a thought. It's extremely stupid.

What I should be doing is using TreeSize to do the copying, which just needs UAC permission once when the program is started.

I shall use it from the next location onwards.

…or should I go back and redo the places I've already done…?

Can I be zarked to do it?

Maybe I'll just try looking at the properties in FM & TS and see if they agree. The easy way.

An electronic, virtual appendectomy.

1001 The (much) older Lenovo AIO which is in use as my workbench computer has always been a bit dodgy; drivers just aren't available for many of its devices under newer Windowses.

One particular annoyance has been the trying-to-work-but-definitely-doesn’t-work touchscreen. Any touch on the screen is read as a touch in a tiny portion on the lower left side, and sometimes it keeps thinking touches are happening there and the mouse can't be used.

Yesterday, I was finally bothered enough by it to do something about it (N.B. not bothered enough to try to make the touchscreen work - that would be a painful & pointless exercise).

I went into Device Manager - Human Input Devices and simply disabled the two entries which mentioned touchscreen.

There I go again, putting the pro in I.T. professional.

No progress, but good intentions paving the road...

1549 Haven’t done much on the data/NAS/back-up front lately. Actually, what have I been doing?

I really should check it’s all still working and much more importantly, organise an off-site back-up…

OMV IP address troubles haven't gone away; OMV + external HDD = not a quick in 'n' out

1437 I'm attempting to reinstate the ‘missing’ IP addresses using OMV-FIRSTAID.

It seems to have worked for ethernet (eno1) by simply answering ‘yes’ to all the questions it asks; I think previously I had said no, out of fear & uncertainty, so it didn't work (i.e. lost all the IP address stuff).

Trying it again for WiFi (wlo1) which has the extra steps of inputting the wireless network’s name & password) but not only has no IP address appeared, it also seems to have vanquished the eno1 IP address… zark.

Let's try the process for eno1 again.

Ok, that's back, and I can access it remotely with the new IP address. 192.168.1.223

Let's try some mad gibble: configuring wlo1 with the ethernet cable unplugged!!!1!

…nothing; still no IP address. Looks like we'll just have to continue with ethernet. Luckily I have all the cables everywhere.

…although replugging in the network cable and running ipaddr shows no IP address for eno1. Let's run OMV-FIRSTAID again.

Ok, that restores the IP address and I can access folders remotely again.

Tried configuring WiFi access via the web UI: still doesn't set an IP address. I think I can choose one manually, but it also asks for ‘netmask’ and I'm like ¯\_(ツ)_/¯ 

Next challenge: external HDD.

Just plugging it back in doesn't work (it shows up remotely (because it never left) but cannot be accessed). A system restart seems to sort it.

Just taking my time. (Are you bored of "copying is slow" posts yet?)

1151 Boringly, not a lot to update.

The copying of data from a portable harddrive onto the harddrives of the NAS continues.

…extremely slowly.

It's glacial at the best of times, but because it's unattended (I'm not going to sit and watch it for days on end), any error means the process is delayed until I'm next in attendance. Usually this simply consists of me clicking ‘Continue’, which always means the ‘error’ is a pathetically petty little hiccup.

But this is a rod I made myself for my own back.

Time is all that's needed, and I have ample time.

1438

Classic example today:

Started the copying of about 450GB of data last night, but didn't see it actually start the copying because the file-counting takes a long time, so I left it to it.

Came to look at it this afternoon (should have checked it first thing…) and it transpires that my mental arithmetic was slightly off, so I was short on space by 2GB (less than 0.5%).

This meant the whole copying was delayed until I turned up again to cancel that copying and start an ever-so-slightly smaller copying.

Is that the best way of doing it? Microsoft must think so. Perhaps another way is to just start the damn copying, and then throw up the error when the space is full, rather than just sitting there doing absolutely zark-all until a human looks at it again…

Interestingly, this copying is proceeding at 30-40MB\s, which is (relatively) astonishingly fast.

Yet more slow transferring; OMV access issues; failed network cable

0718 I'm sitting in bed this morning and can use the File Mangler app on my phone to move data between two harddrives on another computer. Wot a time to be alive.

It occurred to me that that was something I could (almost) usefully do without having to get on a proper computer, and might be a useful rate test. It's nearly 8GB of data (so a tiny fraction of the whole) and started out my moving at 5MB\s, which is fairly reasonable.

It’s about a third of the way through, and the rate has steadily dropped down to below 4MB\s. The time remaining has also steadily increased (to above the starting estimate) suggesting that this transfer will never actually finish; a Paradox of Zeno.

0820

Tried with another folder, around 21GB of mostly MP3s, and am getting a transfer rate of 8MB\s dropping to 7MB\s. Not terrible, I guess.

0850

Trying a load of 2GB files and getting 4MB\s.

Perhaps I just have to go back to accepting that this process will take A Long Time.

1121

A stupid disaster has struck: I can't login to OMV on the laptop anymore.

I have the password ‘written down’ in a disguised format, so I know what the password should be, but it won't accept it.

Hang on…

I tried logging in with ‘root’ instead of ‘admin’ and it works, so maybe the ‘admin’ user doesn't exist…

WTZ is going on?

1433

Today is proving to be a total shitshow of a day.

Figured out the password: admin is for the web interface; root is for logging in on the actual machine.

Tried Midnight Commander and it wasn't great. The harddrives seem inaccessible through it. CBA.

Then the saga began: the internet decided to fail, seemingly spontaneously.

I had been messing around with the network settings on OpenMediaVault in an attempt to access it, so I assumed the problem was that at first. It's being weird, but that's a problem for later on.

Hive hub was no longer connected so I went to look at the router: flashing orange light which apparently means “trying to connect to broadband”. Tried turning it on & off, as well as the fibre converter box: nothing.

Unplugged everything and moved the hub to the fibre box and plugged it in with a different network cable: works. Moved it back and tried the (luckily existant) other network cable running in the same direction to near the fibre box: works. Dug out network-cable-tester thing: cable which had been connecting fibre box to router was faulty… It must have happened when I was moving stuff around there recently, but only spontaneously failed somehow today. I had hoped to not have to terminate any more network cables in my life (after having done loads when initially wiring the house) because it’s fiddly AZ.

…but the network cable is already in place, and I don’t fancy running another one through all the holes through walls.

Anyway, chopped the crappy ends off the network cable and finnickied about getting all the inner wires in the right order and the right length, crimped new plugs on (forgetting to put the shroud on one end, of course) and it works again.

It was all such a waste of time on a day when I just wanted to get on with copying stuff and experimenting to find out transfer rates.

OMV is also annoying, probably due to my lack of knowledge & experience. I think I’ve zarked up the IP addresses somehow, because I can’t find them listed anywhere by using any of the methods I’ve found online. Thankfully, I can still log in via the web interface, and the network drive still shows up in File Mangler. No idea how or why.

But back to file transfers: turns out using my phone to initiate the transfers was a bit of a bad idea because it absolutely ate the phone’s battery.

Currently copying Films, which is a biggie, of mostly large files. The rate hovers between 5 and 15MB/s, with the usual inexplicable pauses at 0.

As noted earlier today, this will evidently take A Long Time, and I should just get on with it.

More slow copying; and that age-old debate: Raspberry Pi Vs old laptop!

0905 Results so far:

• Copying over network from another computer to USB-connected HDD on NAS: slow as zark.

• Copying over network from another computer to internally-mounted HDD on NAS: slow A.Z.

• Copying from internal HDD to USB-connected HDD: slow A.Z.
  
  

So the first experiment is a bit of a bust; I still don’t know what the bottleneck is.

Onto the second experiment:

• Copying from mounted ext HDD to USB HDD: sporadically fast, but spent much time doing nothing at 0MB/s

• Copying large files from mounted external HDD to internal HDD: 70-80MB/s

1420

So, I still don't know what the quickest way is of getting data onto an OpenMediaVault machine. Search results don't seem to produce anything useful (or anything I can understand as useful). Maybe I'll just have to accept that getting stuff on there will be a slow process.

One recurring theme I keep seeing during my ‘research’ is Raspberry Pi; a lot of people seem to be running OMV on their Pis. Perhaps if I'd persevered with my Pi(s) earlier in this journey, I would have eventually stumbled upon OMV. Perhaps I would have realised that my RAID attempts were a folly (especially over USB…) and just had the drives connected simply, as they are now. Perhaps I'd have a system that consumes 10W instead of 20W.

…but I'm still glad I divested myself of the Pi paraphernalia. It wasn't a happy time for me towards the end, and I don't miss being constantly told that the problem was low power. Zarkwonks.

The laptop is a bit easier, having a screen & keyboard (and trackpad) permanently attached, it runs OMV at least as well a Pi would, plus it has one major advantage: if I get too annoyed with Linux, I can just put Windows back on there! :-D

1513

Looking back at the experiment results, I think I have an inkling of a ‘strategy’: if I need to get stuff onto PavNAS in a hurry, copying from the mounted, external drive (MED, from now on) to the CD-drive-bay-adaptor-mounted HDD (CDHDD) seems to be the quickest way. It's effectively a buffer, of sorts, and, like any buffer, has a limited capacity (456GB in this case). 

Any other time (like leaving it overnight, or when I go to work), I can run the slower job of copying stuff onto one of the permanent stores (USBHDDs), from either the CDHDD buffer, or straight from the MED.

I need to do some more testing to find out if there are any more differences in transfer rates to be discovered, particularly when copying files of different sizes (i.e. large like Films, or small like Music).

Where it gets really ‘interesting’ is when it comes to retrieval: having a NAS as merely a back-up is a bit pointless; a simple harddrive would do the same job. The idea of the NAS, as it has been since the start of this project\log, is to be able to retrieve the data, at any time, easily, from any machine in the house (or even further afield…)

If retrieval over the network is as slow as uploading has been, it might all be a bust. It might be similar to past situations where I know I have a copy of a program, or film, or song, but it's just easier & quicker to redownload it, or play it on youtube, than it is to find. In this new case, it might be quicker to retorrent something than it is to copy over (or even stream) from PavNAS.

To try next: streaming a film from CDHDD and also from USBHDD.

2151

The other thing I need to try: copying from another computer to PavNAS but with both machines ethernetted into the same network switch.

Sloowwwww copying; OMV with external HDD

1807 OpenMediaVault seems to be going ok, apart from one minor issue: the speed; it's slow.

Yesterday I started copying over about 110GB of data (out of 2.1TB that needs moving onto PavNAS) from one of the desktops, and it still isn't finished, well over 24 hours later. The speed hovers around 1.5MB\s, with occasional bursts at up to 10MB\s. I'm not sure exactly what the bottleneck is: network or laptop or USB. When this transfer is over (I don't want to interrupt it and slow it further) I've got a couple of experiments to do:

Firstly, try copying data onto the harddrive that's mounted internally to the laptop in a CD drive bay adaptor; this will eliminate the USB part of the transfer.

The second part of this experiment is then transferring data from that internal drive to one of the USB-connected drives. This should eliminate the network portion of the transfer… but since I am controlling all this from another computer on the network via the browser interface, does the data have to go via that machine…?

I've also done a bit more digging and found that I can actually attach an external harddrive with existing data on it and mount that in OMV (though the instructions have filled another page…)

So, the other experiment will be to transfer from that mounted external drive onto the laptop. Again, this will be via USB, so we'll see what data transfer rate we get.

During this research, I've stumbled upon people talking about transferring data using the command line, but I've kinda glossed over that as I don't really want to go there. I remember well the days of copying files & directories around in MS-DOS… and how much easier it was to do in Windows 3.1. I'll see how the experiments go, and then decide if I need to learn some terminal commands…

…but maybe Midnight Commander is the way to go?

2048

Trying the first experiment: copying data across the network to one of the laptop's internal harddrive.

Same speed: ~1 and a bit MB\s. Although this is lots of much smaller files, which probably affects things; I really ought to be doing these tests with the same data, but I can't be arsed to copy the same stuff more than once.

Switching; finding a home for the NAS; first 'regular' backup

1700 I've made a couple of ‘system’ diagrams lately, one of the components inside the PSXPC, and the other of the constituent parts of the PavNAS. It’s very satisfying and really helps me to remember what's connected to what.

I need to do another one for the home network, especially after today: a third switch has been added (well, fourth, if you count the router itself), this time in the cellar. I guess I just like the reässurance afforded by a wired connection!

The plan today was to put a switch in (network) place of the kitchen PC, and then run cables ‘back’ to the kitchen, the old Rock desktop, and to the PavNAS when it's back in its place. In the grand scheme of routing network cables around the house, it wasn't too arduous today: I simply taped the end of one cable to another and persuaded it down the hole from kitchen to cellar. The switch got attached to (‘dangled from’) the wall and all the cables go into it - done. In a world where setting things up is often an utter zarkache (see OMV), it always amazes me how simply & reliably this works.

Hmm, had a thought: the PavNAS had a home in the cellar in order to separate it as far as possible from other repositories of data in the house; that way, a disaster that afflicts one might not wipe out all of them. But… now that the old Rock desktop has its new home on the cellar shelves, there's no point putting the PavNAS there, in terms of separation; it's too close to make a difference, disaster-wise.

Perhaps the PavNAS can live on the shelf near the router where the Stone used to live when it was the Living Room PC. Obviously, an OMV machine won't be any good at being a normal PC, but… if I can learn a bit more about OMV plugins, perhaps the PavNAS could run Kodi, or something like that.

Then again, what's the point? The mini PC on the back of the telly that's the current Living Room PC (LRPC) is supremely capable, and both LE and I are capable of navigating files on a PC, i.e. we don't need a media server with a pretty front-end.

…and the plan is to swap-out that mini PC for the (literally identical) mini PC contained in the PSXPC in order to use that as the LRPC, and as a console emulator.

Perhaps that's the project I need when all this copying malarkey is over!

Talking of LRPC, part of that grand backing-up project today includes copying all the latest downloads from the LRPC onto an E(xternal)HDD and then distributing those files across the organised directories in the ‘Sorted Definitive’ folder …which is (almost) copied to four separate places. There's probably a more efficient way of doing it, but I think I'll just copy it all onto the old Rock desktop, and then distribute it to the two main EHDDs, and PavNAS. Then I'll have a test at copying it from PavNAS onto the old Big Box desktop, before taking that PC off to live in an off-site place.

…so this project's not quite over yet.

All data sorted...? Also: crazy step-by-step for setting up OMV.

1726 I think that maybe… all the data is sorted.

This is the most confident I've been that I have a definitive collection of directories containing all the data I've ever accumulated. 2 terabytes of stuff. Yikes.

Anyway, back to smashing my head against a wall, also known as configuring the Linux-by-another-name that is OpenMediaVault.

…but, to be fair, it hasn't been horrendous yet.

1857

Just in case it disappears, because the internet is fickle:
https://archive.ph/ev7l1

2029

…when I say “all the data is sorted”, obviously not all the data is sorted; there's still the messes that are Music and Comics, and the folder containing all my old work, pictures, etc. is a bit (lot) of a mess, but I'm fairly certain it's all there. I’ve got to the level of confidence that I'm being a bit gung ho with the data now, almost to the point that if it hasn't been swept up in my (not-very-fine-meshed) net whilst combing the piles of data, it wasn't meant to be.

I’d backed up a lot of stuff to a stack of loose 3.5” drives and had planned to copy the newly-sorted data over that old stuff, but I think I might keep it as-is, just in case things were lost, and, crucially, I ever actually noticed that something is missing.

Changing the subject back to OMV, here's a treat:

This is my version of those linked-to instructions for setting up shared directories. Again, I am struck by how very long-winded the process is. I do hope this system proves more reliable than the disappearing shared folders under Windows, i.e. I hope it's worth it. To lose the astonishing ease with which it was possible to share directories AND have it be unreliable would be most irritating.

But the general consensus online is that OMV is a fine option.

BIOS battery; Winamp to Wacup; taking lots of time to setup OMV

1056 Decided to open up the PavNAS to replace the CMOS battery. It's such a pain in the arse compared to a lovely ThinkPad; ALL the back panel screws need removing, and then the crappy plastic back panel still needs spudging off anyway. Such a cheap PoS.

…but it works! Possibly.

The BIOS is awfully basic, so doesn't have the option to auto-power-on after a power cut; I've toggled some settings so it auto-turns-on at midnight everyday (might mod that to be something more sensible like five in the morning, to catch power cuts in the wee hours, though I don't know if this feature actually works yet).

Had a funny moment just now when I couldn't remotely access the harddrive I had mapped yesterday from Windows’s File Mangler, and when I logged into the system via a browser it said the drive was “Missing”. Uh oh.

…then I noticed the CD bay adaptor containing a HDD lying on the desk which I had not yet replaced after opening up the laptop. Derp.

1128

Completely unrelated aside: I've been using Winamp to play MP3s since 2000, when I first bought my first, own PC. There was a whole thing with Winamp (that I never learned about and don't have the bandwidth to care about) and some forkage which resulted in me using 5.666 for the last few years.

I'm not entirely sure what prompted the transition a few weeks ago, but I've stumbled upon Wacup as an alternative. It has some kind of lyrics plug-in, which seems to sometimes work, but the greatest improvement I've found so far is being able to use the spacebar for pausing. ZXCVB controls still work, but the spacebar is soooo much easier to hit to pause, and it's intuitive after so long using space to pause in e.g. youtube and VLC. I'm finding it particularly useful as I'm transitioning away from youtube towards listening to audiobooks instead.

1227

One side-effect of replacing the battery was the BIOS resetting to defaults, including turning secure boot back on, and turning off legacy boot. This then resulted in the message “The selected boot devices failed” when trying to boot.

1802

That whole thing of Windows not copying everything in a directory because of pErMisSiOnS has come back to bite me on the butt: I've found some stuff that isn’t in the supposedly “definitive” sorted back-up folder of everything…

It seems that TreeSize ignores such nonsense, so I need to use that when copying in the future.

In other news, setting up shares in OMV is quite time-consuming; the process is long-winded with many, many steps, and the disk initialisation (or whatever it does) takes an aaaaaage, especially for a 1.8TB drive, but it does seem to work. 

…or does it…? The web interface says “allocating group tables” and the tables tick up very slowly (it's going to take until at least tomorrow morning to finish for the 1.8TB drive) but the screen on the actual laptop has a loooooong list of entries saying “usb 3-2.1: device descriptor read/64, error -110”. Some googling suggests this might be a power issue…

But the table allocating whatever is still ticking up, so I'll let it do its thing.

2028

I've now had a chance to compare two versions of the Backup\BAckup\BACKUP\BACKUP TEMP\BACKUP TEST directory and have found that, indeed, a whole swathe of stuff is missing from the “definitive” version. When I have some time (i.e. when work & the real world isn't taking up my valuable time) I'll collate two versions of the folder and ensure I have everything backed up.

OpenMediaVault!!!1!

1113 I’d like to write in here one day and not have to detail a weird thing happening. But that day is not today.

As I have greatly reduced the amount of space taken up by the DataHorde™, I need less space to store it, so I thought I'd slightly reduce the amount of storage connected to PavNAS. Its main storage is a 2TB 3.5” drive in a powered external enclosure, which is nearly enough for the 2.1TB taken up by the DataHorde™ (yeah, I know the actual size of the drive is only about 1.8TB, but it's still capable of holding the vast majority of it). There's also a 465GB drive in a CD bay adaptor, and two more drives (465GB and 931GB) connected via ‘naked’ USB caddies. My plan was to take out the 1TB drive for use in another portable caddy, and fit a mere 320GB drive instead. 

So, I swapped the hardware, and booted the PavNAS.

First problem was a time\date error. I vaguely remember this being an issue before; it seems the BIOS battery is dead, but I ignored because the system would always be on.

…Until I have to unplug it to move it or whatever, and then I need to put in the time & date again. Oopslol. Need to wedge a new 2032 in there at some point.

Next bit of fun: the 931GB drive has left behind a very persistent ghost. I've definitely unplugged it and replaced it with a 320GB drive, but File Mangler, Fdisk, AND Diskpart all report the 931GB drive as still being present.

Karabast to it, I thought, Windows is being a twonk, and I'm going to over-write it all anyway because…

Next, I proceeded to install OpenMediaVault (more on this later, don't worry; I won't skip this momentous movement) and during installation it asks where it wants installing to. This also reports the 1TB drive as still being connected. I figure we're still in the realms of this being an artefact of the Windows installation.

So I install OMV, and there are the usual Linux learning curve issues (compounded by there not being a GUI at all this time, thus I am UTTERLY reliant on googling for commands), but after some piddling about I am, astonishingly, actually able to log on to the new system in a browser on another computer!!

And guess what I find there…?

The 931GB drive is still showing up.

What the actual zark is going on…?

Anyway….

https://www.reddit.com/r/OpenMediaVault/comments/g3q47o/how_to_keep_laptop_from_turning_off/

https://forum.openmediavault.org/index.php?thread/33773-unable-to-prevent-laptop-suspend-shutdown-when-closing-the-lid/

I think that's stopped it turning off or sleeping when I shut the lid. Check: lid closed, and can still work on the system via web.

I tried plugging in the 298GB (“931GB”) harddrive again, and this time nothing at all showed up, which might be an improvement. Then I tried unplugging the USB cables (power from USB PSU, and power\data from USB hub) and replugging. Fdisk (thrilled to find that on Linux, though probs shouldn't be surprised) then reports a 298GB drive - success! 

It also appears on the drive list in the OMV browser interface on another machine. A surprise, to be sure, but a welcome one.

Right, this entry has gone a bit zarz-backwards, but what's going on is that I'm attempting to use OpenMediaVault instead of Win11 on the PavNAS. This will mean yet one fewer Windows machine and make the PavNAS an actual NAS, rather than a Windows machine with a load of shared folders. True, it'll make it a one-trick-act, but it is definitely a good idea to learn this stuff, and get more Linux experience.

I'm following this guide:

https://www.xda-developers.com/building-a-nas-with-openmediavault/

…which is not flawless, and, as with many of these things, isn't exactly the same version as the one running in front of me, but I've muddled through.

Astonishingly, apart from the harddrive weirdness (something to do with it being connected via a USB hub?), it's gone really rather smoothly. I've only got one drive\share\folder online so far, but I've been able to access it using File Mangler on two Windows PCs, and copy files to and from it!

1511

…and managed to access it from a mobile phone! Working as well as Windows so far, so nothing lost yet. Seems to be using about 20W of power with lid closed and doing stuff.

A proper stretch goal would be to be able to access the files whilst not on the local network, and divest myself from reliance on a big tech company…