15-1/2 years ago I braved the early days of video streaming and its constant buffering to watch the transit of Venus in the middle of the night. Athens Greece was providing the video and the Exploratorium in San Francisco the science-y narration. I had first heard about the transit from, of all places, Thomas Pynchon’s book Mason & Dixon.
Continue reading The Great Conjunction of Jupiter and SaturnCategory: Science and technology
Egregious lies believed by those in technology
Over the past few years, probably longer, I’ve encountered in the tech industry many too many people who are technologists in name only who believe ideas that even 12-year-old me, wide-eyed at Erich von Daniken‘s wondrous mania (which fancifully inspires my music), would have been embarrassed to admit. These are generally older individuals, sadly my age, and not any of the new kids coming into tech. That’s cause for hope. But old white men are still force majeure and so quirky flaws are amplified.
Continue reading Egregious lies believed by those in technologyHome backup
One year ago, I had a major cleanup of my home computers and server. Part of that was to get good off-site backup and so a co-worker recommended IDrive. They worked fine, and for a good price, but for Ubuntu there was only a web-based UI and it didn’t save previous versions of the backed up files. Not horrible, but annoying.
A week or so ago I discovered that I didn’t have a crucial folder in the backup set (!) so I ran their local Python scripts to add it. Script spits out a generic Script error along with a message to upgrade to a new version. New version installed but was even more broken:
Use of uninitialized value in string ne at Helpers.pm line 265.
I contacted them and after a few days (holiday season) they emailed back that I could schedule a time to get with their back end team. Again, not horrible, but it pushed me to look for a more home-grown solution.
Step 1: Copy local files to a network drive
Web server pages and wiki content gets copied daily to a backup folder on a Drobo DAS volume. Four 2 TB drives RAIDed to ~5-1/2 TB. Check their handy capacity calculator. I’ll need to upgrade to an actual NAS someday.
Step 2: Export databases to a network drive
MySQL databases exported with mysqldump every few hours, named with the hour and gzipped (hour_database.db.gz). The hour thing was used because IDrive didn’t give file history, so that can probably be skipped, but it’s nice to have quick access if needed.
Step 3: Generate versioned backups
I use duplicity to take the web server folders, database exports, and various document, photo, development project, and music recording folders and back them up to another location on the Drobo volume. First full backup is slow, subsequent incremental ones are quick.
Step 4: Copy versioned backups offsite
This is the goal. Duplicity can backup direct to an offsite location, but my initial set up uses rclone to do the copying. They support a billion types of storage (Amazon, FTP, Azure, …) but I decided to upgrade my Google Drive to 100 GB for ~$24 a year and use the rclone Google Drive commands.
Each step is a crontab-ed script with taged log output:
2019-01-02 05:00:29 [db_backup] Exporting database-a to 00_database-a.db.gz
2019-01-02 05:00:29 [db_backup] Finished database backup
2019-01-02 05:05:16 [www_backup] Backing up /var/www/site1 to /drobo/site1
2019-01-02 05:10:32 [www_backup] Finished www backup
2019-01-02 06:00:01 [duplicity_backup] Starting duplicity backup from /source to file:///dest
There’s room for improvement, but as a New Year’s upgrade I’m pretty happy.