6.8 KiB
DebMirror Manager
This tool is for streamlining the mirroring of public Debian Repos to a local server on your network or in the cloud.
You can set what CPU Architecture you want to mirror, as well as for what Debian Distributions you want to mirror.
It is currently a work in progress, but I am working fulltime on it as I have a lot of old legacy hardware in my homelab that for various reason I will not get into, can only run Debian 11. Since Debian 11 is about to reach EOL, and there are various public Repos that I use frequently that will have their Debian 11 support removed once Debian 11 reaches EOL, I decided I need a way to easily manage what repos I mirror, and then have a cronjob that runs at a set time everyday to update the repos automatically.
Since the repos in question will lose their Debian 11 support, I plan to also create daily .tar.gz files of each repo once they have synced. That way, if a repo updates, and Debian 11 has been removed, I am covered incase this removes all the Debian 11 files from my local repos. So if you are doing the same as me, then make a .tar.gz after each update. You only need to keen a few days, depending on how often you intend to check in on things.
DebMirror Manager does not assume you have experience with creating a mirror of a repo. Even if you have never done this before, DebMirror Manager does all the heavy lifting for you. All you need to do is start with a vanilla Debian server, and download the installdebmirrormanager.sh file to your server, say in /root/ and then run it with Bash.
The installer will assume that if you are going to mirror your chosen repos on your local network, then you have a local DNS server on your network. Setup a fully qualified domain name on your local DNS server, say deb.mydomain.tld and point it as an A record to the IP address of the local server. If you do not have a local DNS server, and you do not wish to set one up, then just use any FQDN, example deb.homelab.lan, and then when you add the repos to a Debian machine, instead of using http://deb.homelab.lan rather use http://ipaddress.
Why you should consider having mirrors of Debian repos?
If you are like me, and you have a large homelab, or a lot of Debian servers or machines on your home or business network, you can greatly speed up the installation of Debian as well as the time it takes for APT to install packages or update already installed packages. Instead of each system having to download packages and update over your Internet connection, the Mirror Server downloads the changes to each repo and then each system does package downloads over the local network.
For example, my home network is running 1Gbps LAN connection to all servers, hypervisors, desktops and laptops. I only us WiFi for devices that can't be connected by cable, such as Phones, Tablets and TVs. But my Internet connection is only 50Mbps. If I am running updates on all our desktops and laptops, then it saturates the Internet connection and the updates become slow. So once DebMirror Manager is finished, I plan to setup a dedicated machine with about 10TB of harddrive space so that I can keep local mirrors of all of Debians official Repos as well as the other public repos that I use often. That way, updates are handled locally at 1Gbps.
The other advantage for me for doing this, is I develop a lot of server code, and when I need to test, I often spin up a new Debian Server VM on my local machine using VirtualBox. DebMirror Manager supports Debian Install repos as well, so I can just manually put in my local repo server when I install Debian to a new VM. This will vastly speed up the install process for me as well.
The other advantage is, if you are like me, and you have dozens of servers, desktops, laptops and VMs all access the same repos, then you are reducing the load on those repos as you are only downloading once and then storing it on your local mirror. This helps the various people who are funding their public repos out of their own pocket, such as Sury. Just because something is free for you to use, doesn't mean it is free to run. Free has two meanings, Gratis and Libre. Gratis means free as in free beer. Libre means free as in freedom.
Open Source software as well as free public services such as Sury are all Libre. But they are not truly Gratis. Someone is paying, either in their time or out of their pocket. So if you can afford a few dollars, then make a small donation to the projects you use frequently. If you are a business that makes use of a lot of open source and free public services, then consider donating more and frequently. Even if you pledge $20 per month, or even just $5 per month, it goes a long way to support the projects that help your business run.
My primary source of income is from donations I receive from people that find my various scripts and tools, that I host on this Git server, useful. Most people donate around $5 once off, some people donate $20 or more once or twice a year. With that money, I pay my rent, I buy my food, and I survive. The rest of my income comes from offering one on one support and training for other open source software such as Nextcloud and ISP Config to name just two.
But back to the reasons why you should consider having mirrors of Debian repos. If you are the SysAdmin for a company with several branches, then you can sync the public repos to one local repo, and then sync the repos at the other branches from that one local repo. Also, if you have a big network with hundreds of machines all running Debian, then consider this: You can run several Debian Mirror Servers. Simply put, each server has it's own IP address. But you setup each server with the same FQDN. On your company DNS servers, you setup multiple A records all for the same FQDN, but each record pointing to a unique server's IP address.
You then have one server that syncs from the public repos, and then when that is completed, you simply sync all your other servers from that one server, or you can use RSync to do it even quicker. I will eventually add some scripts on this repo that will help set that up and automate it. But here is the kicker... Desktop 1 asks the company DNS server for the IP address for your FQDN, and the DNS server will return the IP address at the first A record. Then Desktop 2 asks and gets the IP address at the second A record. And this continues until the last A record is served, then it goes back to the first one. Drop me a message on Reddit (u/thisiszeev) if you have an urgent need for this, and I will get the scripts done faster.
Hell, I will even set it all up for you on your company network for a very reasonable support fee. But I do it different. I remote desktop to you, then I call you, and over the phone, I walk you through what I am doing, why I am doing it, and what the other options are... that way you learn and next time, you know.