|
|
|
|
|
|
| Author |
Message |
David Brown
*nix forums beginner
Joined: 02 Mar 2005
Posts: 34
|
 Posted: Tue Jul 18, 2006 7:26 am Post subject:
Re: Backup Software
|
|
|
Michael Schnell wrote:
| Quote: | Really great !
Thanks a lot for taking the time to explain dirvish to me.
-Michael
|
I hope it works out for you (and anyone else who was following this
thread - I know it is off-topic in an embedded group, but backups are
useful for everyone). |
|
| Back to top |
|
 |
Michael Schnell
*nix forums Guru Wannabe
Joined: 21 Feb 2005
Posts: 110
|
| Posted: Mon Jul 17, 2006 2:30 pm Post subject:
Re: Backup Software
|
|
|
Really great !
Thanks a lot for taking the time to explain dirvish to me.
-Michael |
|
| Back to top |
|
|
David Brown
*nix forums beginner
Joined: 02 Mar 2005
Posts: 34
|
| Posted: Mon Jul 17, 2006 7:31 am Post subject:
Re: Backup Software
|
|
|
Michael Schnell wrote:
| Quote: |
A more usual arrangement for your expiry times is to have a variety -
for example, this is an excerpt from my dirvish "master.conf" file:
# Most backups kept for 30 days
# Monday backups kept for 6 months
# First Monday in month kept for 3 years
# Quad-yearly backups kept forever
expire-default: +30 days
expire-rule:
# Min Hr DOM Mon DOW Expire
* * * * 2 +6 months
* * 1-7 * 2 +3 year
* * 1-7 1,4,7,11 2 never
Of course all files will need to stay on the backup unless they are
overwritten on the working disk.
|
All files stay on the backup even if they are overwritten on the
original disk. That's the point of a backup. When dirvish runs and
makes a directory called "20060717" for the 17th July, 2006 backup, then
that directory contains a snapshot copy of the original disk when the
backup was run. The contents of the 20060717 backup directory never
change after that, unless it is "expired" (using the dirvish-expire
script) according to the expire rules used when the backup was taken.
When it is "expired", the directory is completely removed.
Remember, all this stuff about hard links and rsync differential backups
is just to make the process more efficient (both in terms of disk space
and network bandwidth). The backups appear as simple full copies of the
original source.
| Quote: |
(How) Does this setting provide for this ?
Thanks,
-Michael |
|
|
| Back to top |
|
|
Michael Schnell
*nix forums Guru Wannabe
Joined: 21 Feb 2005
Posts: 110
|
| Posted: Wed Jul 12, 2006 2:18 pm Post subject:
Re: Backup Software
|
|
|
| Quote: | A more usual arrangement for your expiry times is to have a variety -
for example, this is an excerpt from my dirvish "master.conf" file:
# Most backups kept for 30 days
# Monday backups kept for 6 months
# First Monday in month kept for 3 years
# Quad-yearly backups kept forever
expire-default: +30 days
expire-rule:
# Min Hr DOM Mon DOW Expire
* * * * 2 +6 months
* * 1-7 * 2 +3 year
* * 1-7 1,4,7,11 2 never
|
Of course all files will need to stay on the backup unless they are
overwritten on the working disk.
(How) Does this setting provide for this ?
Thanks,
-Michael |
|
| Back to top |
|
|
Michael Schnell
*nix forums Guru Wannabe
Joined: 21 Feb 2005
Posts: 110
|
| Posted: Tue Jul 11, 2006 2:32 pm Post subject:
Re: Backup Software
|
|
|
Thanks for the pointer
-Michael |
|
| Back to top |
|
|
Captain Dondo
*nix forums Guru Wannabe
Joined: 28 Feb 2005
Posts: 250
|
| Posted: Tue Jul 11, 2006 12:51 pm Post subject:
Re: Backup Software
|
|
|
On Mon, 10 Jul 2006 17:10:56 +0200, Mikey Quick wrote:
| Quote: | All this sounds really promising. (As the unsolved Backup problem is the
cause why I did not yet migrate the server to Linux, I now have hope
that it can be done soon.)
|
Also check out Amanda... It's an enterprise backup solution. It backs up
to tapes primarily. Its biggest advantage is that restores can be handled
via ordinary tools - tar and friends - so a bare metal restore is easier.
Although dirvish sounds really nice for a small, single server system.
--Yan
--
o__
,>/'_ o__
(_)\(_) ,>/'_ o__
Yan Seiner, PE (_)\(_) ,>/'_ o__
Certified Personal Trainer (_)\(_) ,>/'_ o__
Licensed Professional Engineer (_)\(_) ,>/'_
Who says engineers have to be pencil necked geeks? (_)\(_) |
|
| Back to top |
|
|
Michael Schnell
*nix forums Guru Wannabe
Joined: 21 Feb 2005
Posts: 110
|
| Posted: Tue Jul 11, 2006 11:43 am Post subject:
Re: Backup Software
|
|
|
| Quote: | I have not used NAS (or NFS, for that matter), so I couldn't tell you. I
also don't know how LVM would work with removable drives. But there is
no requirement to use LVM or reiserfs - these are just my personal
favourites for such systems (I've set up two such backup arrangements).
|
I do follow your argument preferring raiser, of course
| Quote: | I find the backup over ADSL to be far more convenient than using
removable media - quite simply, it is far less effort and therefore far
more reliable. But of course, that requires you having two locations
linked by broadband.
|
For this small site, I would find it secure enough to install the backup
drive in the basement, which is very secure against fire. But it's a
little damp so I don't want to install the server there.
-Michael |
|
| Back to top |
|
|
David Brown
*nix forums beginner
Joined: 02 Mar 2005
Posts: 34
|
| Posted: Tue Jul 11, 2006 10:19 am Post subject:
Re: Backup Software
|
|
|
Michael Schnell wrote:
| Quote: | Still another question.
To move the the backup drive location away from the server, would it be
possible to use a NAS device (instead of an USB or FireWire disk, as I
intended before) as a backup medium. I understand that this NAS would
need to use a Network protocol that supports hard links (NFS ? SSH ?)
and should to use Raiser as a file system.
-Michael
|
I have not used NAS (or NFS, for that matter), so I couldn't tell you.
I also don't know how LVM would work with removable drives. But there
is no requirement to use LVM or reiserfs - these are just my personal
favourites for such systems (I've set up two such backup arrangements).
I find the backup over ADSL to be far more convenient than using
removable media - quite simply, it is far less effort and therefore far
more reliable. But of course, that requires you having two locations
linked by broadband. |
|
| Back to top |
|
|
Michael Schnell
*nix forums Guru Wannabe
Joined: 21 Feb 2005
Posts: 110
|
| Posted: Tue Jul 11, 2006 10:09 am Post subject:
Re: Backup Software
|
|
|
Still another question.
To move the the backup drive location away from the server, would it be
possible to use a NAS device (instead of an USB or FireWire disk, as I
intended before) as a backup medium. I understand that this NAS would
need to use a Network protocol that supports hard links (NFS ? SSH ?)
and should to use Raiser as a file system.
-Michael |
|
| Back to top |
|
|
Michael Schnell
*nix forums Guru Wannabe
Joined: 21 Feb 2005
Posts: 110
|
| Posted: Tue Jul 11, 2006 9:55 am Post subject:
Re: Backup Software
|
|
|
| Quote: |
A more usual arrangement for your expiry times is to have a variety -
for example, this is an excerpt from my dirvish "master.conf" file:
# Most backups kept for 30 days
# Monday backups kept for 6 months
# First Monday in month kept for 3 years
# Quad-yearly backups kept forever
expire-default: +30 days
expire-rule:
# Min Hr DOM Mon DOW Expire
* * * * 2 +6 months
* * 1-7 * 2 +3 year
* * 1-7 1,4,7,11 2 never
|
Thanks a lot !
I definitely will do the server migration soon and use dirvish.
-Michael |
|
| Back to top |
|
|
David Brown
*nix forums beginner
Joined: 02 Mar 2005
Posts: 34
|
| Posted: Mon Jul 10, 2006 5:23 pm Post subject:
Re: Backup Software
|
|
|
Mikey Quick wrote:
| Quote: | All this sounds really promising. (As the unsolved Backup problem is the
cause why I did not yet migrate the server to Linux, I now have hope
that it can be done soon.)
Just the question of deleted files is not clear to me yet. My hope is
that they are handled like versions: if I set the expire time to say 10
days (or whatever a suitable setting is), a deleted file should stay
accessible on the backup for 10 days, too, and get deleted at day 11.
|
That is correct.
| Quote: | Thanks again,
-Michael
|
A more usual arrangement for your expiry times is to have a variety -
for example, this is an excerpt from my dirvish "master.conf" file:
# Most backups kept for 30 days
# Monday backups kept for 6 months
# First Monday in month kept for 3 years
# Quad-yearly backups kept forever
expire-default: +30 days
expire-rule:
# Min Hr DOM Mon DOW Expire
* * * * 2 +6 months
* * 1-7 * 2 +3 year
* * 1-7 1,4,7,11 2 never |
|
| Back to top |
|
|
Mikey Quick
*nix forums beginner
Joined: 10 Jul 2006
Posts: 4
|
| Posted: Mon Jul 10, 2006 3:10 pm Post subject:
Re: Backup Software
|
|
|
All this sounds really promising. (As the unsolved Backup problem is the
cause why I did not yet migrate the server to Linux, I now have hope
that it can be done soon.)
Just the question of deleted files is not clear to me yet. My hope is
that they are handled like versions: if I set the expire time to say 10
days (or whatever a suitable setting is), a deleted file should stay
accessible on the backup for 10 days, too, and get deleted at day 11.
Thanks again,
-Michael |
|
| Back to top |
|
|
David Brown
*nix forums beginner
Joined: 02 Mar 2005
Posts: 34
|
| Posted: Mon Jul 10, 2006 2:08 pm Post subject:
Re: Backup Software
|
|
|
Michael Schnell wrote:
| Quote: | David, Thanks a lot for answering
If you are backing up to DVD or CD, you have to be aware that these
media have limited lifetimes (although no one is quite sure what that
lifetime might be...).
Of course you are right. That is true for tapes, as well. I even read
the recommendation to use DVDs instead of tapes, as the small likelihood
that in some decades functional drives for reading the tapes used will
not be available is the largest danger to be considered
Before thinking of a backup strategy, think about your restore
strategy. People often forget that, and come up with a system that
requires searching through a dozen "incremental" backup CDs to get the
files they are really looking for.
Again you are right. This concept is primarily meant for recovering
versions lost by human actions and only secondarily meant for disaster
recovery. So fully automatic backup and the possibility to restore
single files is the _primary_ intention.
|
Most disasters are caused by human actions (hopefully the reverse is not
true...)
| Quote: | There are lots of backup solutions for disaster recovery that might be
use additionally. In case of disaster you would need to set up a working
system before you can use the backup media, so this is not a quick
solution anyway. To be fast, a mirrored system is the way to go. Of
course a restore strategy also suitable for disaster and other
mass-restores might be a good enhancement if the system is doing it's
primary purpose OK.
|
One of the great advantages of dirvish is that you have a mirror, and
thus quick and easy restores of either single files or whole trees.
| Quote: | The other key point for your backups is that they must be kept off
site - your backups are of little value if a thief or a fire destroys
the backups as well as the originals.
Right again. Same issue with any backup solution. And very often
forgotten. Doing the backup remotely over a Network helps independently
from the software used. Doing an incremental ("intelligent") strategy
reduces the Network traffic greatly. IMHO the strategy I consider is
very useful for "over the wire" backups.
|
Dirvish lets you use incremental backups, with the underlying rsync even
supporting incremental backups of changes to large files, so that things
like mailboxes are handled efficiently. An incremental backup of our
office data normally takes under an hour - the single original full
backup would have taken weeks over the same wire, so I cheated and
plugged the backup server into our local network for the first run.
| Quote: |
The backup system I use is "dirvish" ( http://www.dirvish.com/ ). It
is basically a wrapper around rsync
Thanks a lot for the pointer. I'll take a look at dirvish/rsync. (If I
can find something usable for me, I'm not going to do it myself)
|
Dirvish gives you a higher level viewpoint - it uses the goodies from
rsync, without you having to read the details in the man page, or write
your own scripts. But it's also worth reading about rsync, to see
what's happening underneath.
| Quote: | How does this rsync handle versions of modified files ? any chanced to
get back a previous version after some days ?
|
Using dirvish (or your own rsync scripts), yes, no problem. All the old
backup snapshots are easily available (depending on your "expire"
policies for backup snapshots). When a file has not changed between
backups, extra copies are hard linked to the old copies, avoiding wasted
disk space. Changed versions get their own files - but the network copy
may only send the changed parts, reducing network traffic.
| Quote: | How does this rsync handle deleted files ? Any chanced to get one back
after some days ? Don't they clutter the backup disk ?
|
If I delete a file today, the original remains in yesterday's backup
snapshot, and is gone from this evening's backup run. Files eventually
disappear off the disk if all their links are removed during "expire"
runs to clear out old backups.
One thing to remember about dirvish (or other similar rsync based backup
systems, as there are several around), is that you end up with a lot of
files on the disk, and a huge number of links. It's also much more
efficient if you have a single file system rather than spread the
backups (for any given "vault") over separate file systems, since you
can then use hard links. You can also expect the backup system to grow
larger than you thought when you first setup the system.
The best way to set up the system (IMHO), therefore, is to use LVM and
reiserfs. Reiserfs copes better with large numbers of files and links
than most file systems, and has no (realistic) limits. It also supports
on the fly size changes, with no issues about limited inode tables. So
when you start getting low on disk space, you simply connect up a new
hard disk, make it an LVM physical volume, link it to your volume group,
and then add as much space as you need to your backup logical volume. I
use LVM for most of my data volumes - any time I need more space,
assuming there is free space left on the disk, I can grow the file
systems in 10 seconds with a couple of commands without even umounting
the filesystems.
| Quote: | Thanks again,
-Michael
|
|
|
| Back to top |
|
|
Mikey Quick
*nix forums beginner
Joined: 10 Jul 2006
Posts: 4
|
| Posted: Mon Jul 10, 2006 10:55 am Post subject:
Re: Backup Software
|
|
|
David, Thanks a lot for answering
| Quote: | If you are backing up to DVD or CD, you have to be aware that these
media have limited lifetimes (although no one is quite sure what that
lifetime might be...).
|
Of course you are right. That is true for tapes, as well. I even read
the recommendation to use DVDs instead of tapes, as the small likelihood
that in some decades functional drives for reading the tapes used will
not be available is the largest danger to be considered
| Quote: |
Before thinking of a backup strategy, think about your restore strategy.
People often forget that, and come up with a system that requires
searching through a dozen "incremental" backup CDs to get the files they
are really looking for.
|
Again you are right. This concept is primarily meant for recovering
versions lost by human actions and only secondarily meant for disaster
recovery. So fully automatic backup and the possibility to restore
single files is the _primary_ intention.
There are lots of backup solutions for disaster recovery that might be
use additionally. In case of disaster you would need to set up a working
system before you can use the backup media, so this is not a quick
solution anyway. To be fast, a mirrored system is the way to go. Of
course a restore strategy also suitable for disaster and other
mass-restores might be a good enhancement if the system is doing it's
primary purpose OK.
| Quote: | The other key point for your backups is that
they must be kept off site - your backups are of little value if a thief
or a fire destroys the backups as well as the originals.
|
Right again. Same issue with any backup solution. And very often
forgotten. Doing the backup remotely over a Network helps independently
from the software used. Doing an incremental ("intelligent") strategy
reduces the Network traffic greatly. IMHO the strategy I consider is
very useful for "over the wire" backups.
Thanks a lot for the pointer. I'll take a look at dirvish/rsync. (If I
can find something usable for me, I'm not going to do it myself)
How does this rsync handle versions of modified files ? any chanced to
get back a previous version after some days ?
How does this rsync handle deleted files ? Any chanced to get one back
after some days ? Don't they clutter the backup disk ?
Thanks again,
-Michael |
|
| Back to top |
|
|
Michael Schnell
*nix forums Guru Wannabe
Joined: 21 Feb 2005
Posts: 110
|
| Posted: Mon Jul 10, 2006 10:54 am Post subject:
Re: Backup Software
|
|
|
David, Thanks a lot for answering
| Quote: | If you are backing up to DVD or CD, you have to be aware that these
media have limited lifetimes (although no one is quite sure what that
lifetime might be...).
|
Of course you are right. That is true for tapes, as well. I even read
the recommendation to use DVDs instead of tapes, as the small likelihood
that in some decades functional drives for reading the tapes used will
not be available is the largest danger to be considered
| Quote: |
Before thinking of a backup strategy, think about your restore strategy.
People often forget that, and come up with a system that requires
searching through a dozen "incremental" backup CDs to get the files they
are really looking for.
|
Again you are right. This concept is primarily meant for recovering
versions lost by human actions and only secondarily meant for disaster
recovery. So fully automatic backup and the possibility to restore
single files is the _primary_ intention.
There are lots of backup solutions for disaster recovery that might be
use additionally. In case of disaster you would need to set up a working
system before you can use the backup media, so this is not a quick
solution anyway. To be fast, a mirrored system is the way to go. Of
course a restore strategy also suitable for disaster and other
mass-restores might be a good enhancement if the system is doing it's
primary purpose OK.
| Quote: | The other key point for your backups is that
they must be kept off site - your backups are of little value if a thief
or a fire destroys the backups as well as the originals.
|
Right again. Same issue with any backup solution. And very often
forgotten. Doing the backup remotely over a Network helps independently
from the software used. Doing an incremental ("intelligent") strategy
reduces the Network traffic greatly. IMHO the strategy I consider is
very useful for "over the wire" backups.
Thanks a lot for the pointer. I'll take a look at dirvish/rsync. (If I
can find something usable for me, I'm not going to do it myself)
How does this rsync handle versions of modified files ? any chanced to
get back a previous version after some days ?
How does this rsync handle deleted files ? Any chanced to get one back
after some days ? Don't they clutter the backup disk ?
Thanks again,
-Michael |
|
| Back to top |
|
|
Google
|
|
| Back to top |
|
|
|
|
The time now is Sat Nov 22, 2008 3:53 am | All times are GMT
|
|
Mortgages | Gas Suppliers | Mortgage Calculator | Loans | Secured Loans
|
|
Copyright © 2004-2005 DeniX Solutions SRL
|
|
|
|
Other DeniX Solutions sites:
Unix/Linux blog |
electronics forum |
medicine forum |
science forum |
|
|
Privacy Policy
|
Powered by phpBB © 2001, 2005 phpBB Group
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
Forum index
»
*nix
»
Linux
»
embedded
