You might not even like rsync. Yeah it's old. Yeah it's slow. But if you're working with Linux you're going to need to know it. In this video I walk through my favorite everyday flags for rsync. Su...

You might not even like rsync. Yeah it’s old. Yeah it’s slow. But if you’re working with Linux you’re going to need to know it.

In this video I walk through my favorite everyday flags for rsync.

Support the channel:
https://patreon.com/VeronicaExplains
https://ko-fi.com/VeronicaExplains
https://thestopbits.bandcamp.com/

Here’s a companion blog post, where I cover a bit more detail: https://vkc.sh/everyday-rsync

Also, @BreadOnPenguins made an awesome rsync video and you should check it out: https://www.youtube.com/watch?v=eifQI5uD6VQ

Lastly, I left out all of the ssh setup stuff because I made a video about that and the blog post goes into a smidge more detail. If you want to see a video covering the basics of using SSH, I made one a few years ago and it’s still pretty good: https://www.youtube.com/watch?v=3FKsdbjzBcc

Chapters:
1:18 Invoking rsync
4:05 The --delete flag for rsync
5:30 Compression flag: -z
6:02 Using tmux and rsync together
6:30 but Veronica… why not use (insert shiny object here)

  • dohpaz42@lemmy.worldEnglish
    22·
    2 months ago

    Here’s how I approach old and slow:

    1. Older software is mature and battle tested. It’s been around long enough that the developers should know what they’re doing, and have built a strong community for help and support.
    2. Slow is okay when it comes to accuracy. Would I love to back up my gigabytes (peanuts compared to some of you folks out there with data centers in your attics) in seconds? Yes. But more importantly, I’d rather have my data be valid for if I ever need to do any kind of restore. And I’ve been around the block enough times in my career to see many useless backups.
  • mesa@piefed.socialOPEnglish
    49·
    2 months ago

    Ive personally used rsync for backups for about…15 years or so? Its worked out great. An awesome video going over all the basics and what you can do with it.

    • confusedpuppy@lemmy.dbzer0.comEnglish
      3·
      2 months ago

      I use rsync for many of the reasons covered in the video. It’s widely available and has a long history. To me that feels important because it’s had time to become stable and reliable. Using Linux is a hobby for me so my needs are quite low. It’s nice to have a tool that just works.

      I use it for all my backups and moving my backups to off network locations as well as file/folder transfers on my own network.

      I even made my own tool (https://codeberg.org/taters/rTransfer) to simplify all my rsync commands into readable files because rsync commands can get quite long and overwhelming. It’s especially useful chaining multiple rsync commands together to run under a single command.

      I’ve tried other backup and syncing programs and I’ve had bad experiences with all of them. Other backup programs have failed to restore my system. Syncing programs constantly stop working and I got tired of always troubleshooting. Rsync when set up properly has given me a lot less headaches.

    • Eager Eagle@lemmy.worldEnglish
      71·
      2 months ago

      It works fine if all you need is transfer, my issue with it it’s just not efficient. If you want a “time travel” feature, your only option is to duplicate data. Differential backups, compression, and encryption for off-site ones is where other tools shine.

      • bandwidthcrisis@lemmy.worldEnglish
        4·
        2 months ago

        I have it add a backup suffix based on the date. It moves changed and deleted files to another directory adding the date to the filename.

        It can also do hard-link copied so that you can have multiple full directory trees to avoid all that duplication.

        No file deltas or compression, but it does mean that you can access the backups directly.

        • koala@programming.devEnglish
          3·
          2 months ago

          Thanks! I was not aware of these options, along with what other poster mentioned about --link-dest. These do turn rsync into a backup program, which is something the root article should explain!

          (Both are limited in some aspects to other backup software, but they might still be a simpler but effective solution. And sometimes simple is best!)

      • suicidaleggroll@lemmy.worldEnglish
        1·
        2 months ago

        If you want a “time travel” feature, your only option is to duplicate data.

        Not true. Look at the --link-dest flag. Encryption, sure, rsync can’t do that, but incremental backups work fine and compression is better handled at the filesystem level anyway IMO.

        • Eager Eagle@lemmy.worldEnglish
          1·
          2 months ago

          Isn’t that creating hardlinks between source and dest? Hard links only work on the same drive. And I’m not sure how that gives you “time travel”, as in, browsing snapshots or file states at the different times you ran rsync.

          Edit: ah the hard link is between dest and the link-dest argument, makes more sense.

          I wouldn’t bundle fs and backup compression in the same bucket, because they have vastly different reqs. Backup compression doesn’t need to be optimized for fast decompression.

          • BCsven@lemmy.caEnglish
            1·
            2 months ago

            Snapper and BTRFS. Its only adjusts changes in data, so time travel is just pointing to what blocks changed and when, and not building a duplicate of the entire file or filesystem. A snapshot is instant, and new block changes belong to the current default.

      • NekuSoul@lemmy.nekusoul.deEnglish
        1·
        2 months ago

        Agree. It’s neat for file transfers and simple one-shot backups, but if you’re looking for a proper backup solution then other tools/services have advanced virtually every aspect of backups so much it pretty much always makes sense to use one of those instead.

  • Landless2029@lemmy.worldEnglish
    6·
    2 months ago

    I need a breakdown like this for Rclone. I’ve got 1TB of OneDrive free and nothing to do with it.

    I’d love to setup a home server and backup some stuff to it.

  • clif@lemmy.worldEnglish
    14·
    2 months ago

    I’ll never not upvote Veronica Explains. Excellent creator and excellent info on everything I’ve seen.

  • surph_ninja@lemmy.worldEnglish
    81·
    2 months ago

    Use borg/borgmatic for your backups. Use rsync to send your differentials to your secondary & offsite backup storage.

  • Jessica@discuss.tchncs.deEnglish
    3·
    2 months ago

    If you’re trying to back up Windows OS drives for some reason, robocopy works quite similarly to rsync.

  • atk007@lemmy.worldEnglish
    221·
    2 months ago

    Rsnapshot. It uses rsync, but provides snapshot management and multiple backup versioning.

    • Tja@programming.devEnglish
      172·
      2 months ago

      Yes, but a few hours writing my own scripts will save me from several minutes of reading its documentation…

    • BonkTheAnnoyed@lemmy.blahaj.zoneEnglish
      4·
      2 months ago

      Yah, I really like this approach. Same reason I set up Timeshift and Mint Backup on all the user machines in my house. For others rsync + cron is aces.

  • Appoxo@lemmy.dbzer0.comEnglish
    4·
    2 months ago

    Veeam for image/block based backups of Windows, Linux and VMs.
    syncthing for syncing smaller files across devices.

    Thank you very much.

    • okamiueru@lemmy.worldEnglish
      9·
      2 months ago

      That part threw me off. Last time i used it, I did incremental backups of a 500 gig disk once a week or so, and it took 20 seconds max.

    • HereIAm@lemmy.worldEnglish
      81·
      2 months ago

      Compared to something multi threaded, yes. But there are obviously a number of bottlenecks that might diminish the gains of a multi threaded program.

  • vext01@lemmy.sdf.orgEnglish
    4·
    2 months ago

    I used to use rsnapshot, which is a thin wrapper around rsync to make it incremental, but moved to restic and never looked back. Much easier and encrypted by default.

  • calliope@retrolemmy.comEnglish
    9·
    2 months ago

    Tangentially, I don’t see people talk about rclone a lot, which is like rsync for cloud storage.

    It’s awesome for moving things from one provider to another, for example.

    • Landless2029@lemmy.worldEnglish
      1·
      2 months ago

      I tried rclone once because I wanted to sync a single folder from documents and freaked out when it looked like it was going to purge all documents except for my targeted folder.

      Then I just did it via the portal…

      • calliope@retrolemmy.comEnglish
        2·
        2 months ago

        rsync can sometimes look similarly scary! I very clearly remember triple-checking what it’s doing.

        rclone works amazingly well if you have hundreds of folders or thousands of files and you can’t be bothered to babysit a portal.

    • Eldritch@piefed.worldEnglish
      42·
      2 months ago

      It’s fine. But yes in the Linux space. We tend to want to host ourselves. Not have to trust some administrator of some cloud we don’t know/trust.

        • Eldritch@piefed.worldEnglish
          51·
          2 months ago

          I mention in the Linux space only because it’s what I’m familiar with and didn’t want to make assumptions about groups I’m not familiar with. Unlike you who’s looking for a way to take umbridge and talk passed people. I went to college for IT and have done it for 30 years.

          In network and IT planning. The cloud is the wider network outside your own. That you don’t have mapped. Often depicted by a “cloud”. If I have a personal data pool on one of my own networks. And need it from another. It may transmit via the “cloud”. But it isn’t IN the cloud. It’s on a personal server. If the server is in your house, and you can point exactly to where your data is. Then the rule of thumb is that it is in your house. Not the cloud. If it’s hosted on a system you couldn’t directly point to on a network you have no knowledge of. Especially a shared system. Then things literally and figuratively are getting cloudier.

          That said, marketing as it often does. Appropriates and misuses words based around buzz. And I am not about to admonish hobbyist who use it in the marketing sense. I understand, I get it.

          If you host in OSX on Apple Silicon, that’s great. If you host on a 68k Mac or Amiga you’re a fucking mad lad! If you’re hosting under Windows, any TCP port in the storm mate. If you are hosting from a Linux distribution that is not God’s chosen, cool how is it working out? If you are hosting from BeOS. or Haiku, you are a glorious oddball and absolutely my sort of person. And if you are hosting from an appliance that you really don’t know what it’s running, welcome to the hobby. It’s a good starting point. And a lill data in the cloud isn’t a crime. We all have some. But if you can’t easily point to it. Can you really know you have it?

          • calliope@retrolemmy.comEnglish
            13·
            2 months ago

            I’m not reading all that. Sorry for your issue, or I’m happy for you. Whichever you prefer.

            • Bo7a@lemmy.caEnglish
              1·
              2 months ago

              Partakes in text-based medium. Refuses to read well written and polite comment that is four whole paragraphs. Proceeds to think they are the intelligent one in the conversation. Are you huffing glue right now?

            • Ardent@kbin.earth
              3·
              2 months ago

              you should. they were polite unlike you. explained the origin of the term and how it was used. explaining that they were aware of how hobbyists have changed the definition etc. it was a decent post. frankly I’m kind of curious why your so hateful. but not enough to really care.

      • calliope@retrolemmy.comEnglish
        2·
        2 months ago

        It has been remarkably useful! I keep trying to tell people about it but apparently I am just their main use case or something.

        I would have loved it when I was using Samba to share files on my local network decades ago. It’s like a Swiss Army knife!