|
|
|
|
|
|
| Author |
Message |
Paul Theodoropoulos
*nix forums beginner
Joined: 19 Jul 2005
Posts: 17
|
 Posted: Thu Jul 06, 2006 6:48 pm Post subject:
writing directly to Maildir/new
|
|
|
I'm faced with migrating all of my customers and their email to a new
server in a remote location. The location is far enough away that
simply shutting down the pop/imap server and transporting it would
incur likely in excess of twelve hours of 100% downtime, which is not
acceptable.
the idea i'm toying with is to run ufsdump/ufsrestore over the WAN to
the new server. from what i understand, the initial pass of
ufsrestore writes out all of the directory structure first before
writing out the files. in that way, after a brief period during which
connections would be locked out, customer pop/imap traffic could be
redirected to the new server once the dirs are written. initially
customers would see no mail, but as the hours pass, their Maildirs
would be populated with their mail from the old server (new mail being
redirected from the MX servers to the new server after that initial
interval as well). The customers would at least be able to send and
retrieve new mail while the transfer is taking place, rather than
being locked out of their email for many, many hours during the
transfer over the WAN.
my concern is with what happens if there's a 'collision' with a
client connecting to their account while the ufsrestore is in the
process of writing an existing message into Maildir/new. my
understanding is that qmail's path is to write to Maildir/tmp then
atomically mv the files to Maildir/new, which mitigates partial file
retrieval. however, if ufsrestore is writing a message with
attachment that's say 10mb in size to Maildir/new, i could see the
potential for a POP client 'seeing' the new message in the initial
LIST, then trying to download it before the write is complete,
resulting in a partial file being downloaded to their client, or
possibly multiple copies as the system tries to move the file to
Maildir/cur before the write's complete.
If this is the case, it pretty much puts the kibosh on the whole
ufsdump/ufsrestore over the WAN idea. that then would suggest a very
tedious manual migration, customer by customer, moving their data by
hand, having them update their client software to connect to the new
server with a new name (rather than just repointing the dns entries
to the new server's IP). That's doable too - but much, much more
labor intensive.
sorry for the length. any insights appreciated.
Paul Theodoropoulos
http://www.anastrophe.com |
|
| Back to top |
|
 |
Charles Cazabon
*nix forums Guru
Joined: 08 Jan 2005
Posts: 805
|
| Posted: Thu Jul 06, 2006 7:00 pm Post subject:
Re: writing directly to Maildir/new
|
|
|
Paul Theodoropoulos <paul@anastrophe.com> wrote:
| Quote: |
the idea i'm toying with is to run ufsdump/ufsrestore over the WAN to
the new server.
[...]
my concern is with what happens if there's a 'collision' with a
client connecting to their account while the ufsrestore is in the
process of writing an existing message into Maildir/new. my
understanding is that qmail's path is to write to Maildir/tmp then
atomically mv the files to Maildir/new, which mitigates partial file
retrieval. however, if ufsrestore is writing a message with
attachment that's say 10mb in size to Maildir/new, i could see the
potential for a POP client 'seeing' the new message in the initial
LIST, then trying to download it before the write is complete,
resulting in a partial file being downloaded to their client, or
possibly multiple copies as the system tries to move the file to
Maildir/cur before the write's complete.
|
Yes, exactly.
If you can simply disable service while the transfer is in place (either
explicitly, by refusing connections, or trying to hide it with a dummy service
that always reports "no mail"), re-enabling normal service once the transfer
is complete, that will eliminate the problem.
Charles
--
--------------------------------------------------------------------------
Charles Cazabon <qmail@discworld.dyndns.org>
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
My services include qmail consulting. See http://pyropus.ca/ for details.
-------------------------------------------------------------------------- |
|
| Back to top |
|
|
Jeremy Kitchen
*nix forums Guru Wannabe
Joined: 14 Jan 2005
Posts: 212
|
| Posted: Thu Jul 06, 2006 7:13 pm Post subject:
Re: writing directly to Maildir/new
|
|
|
On Thursday 06 July 2006 11:48, you wrote:
| Quote: | I'm faced with migrating all of my customers and their email to a new
server in a remote location. The location is far enough away that
simply shutting down the pop/imap server and transporting it would
incur likely in excess of twelve hours of 100% downtime, which is not
acceptable.
|
can you use rsync to make the amount of data to be transferred during downtime
smaller?
You should be able to rsync the mail store and get a good idea of how much
downtime you'll have.. then right before you do the final switchover, run
rsync again (to get as much mail as possible transferred before downtime),
then shut services off, run rsync a final time, and voila! :)
-Jeremy
--
Jeremy Kitchen ++ kitchen@scriptkitchen.com
http://www.pirate-party.us/ -- defend your rights |
|
| Back to top |
|
|
Paul Theodoropoulos
*nix forums beginner
Joined: 19 Jul 2005
Posts: 17
|
| Posted: Thu Jul 06, 2006 7:19 pm Post subject:
Re: writing directly to Maildir/new
|
|
|
At 12:00 PM 7/6/2006, Charles Cazabon wrote:
| Quote: | Yes, exactly.
If you can simply disable service while the transfer is in place (either
explicitly, by refusing connections, or trying to hide it with a dummy service
that always reports "no mail"), re-enabling normal service once the transfer
is complete, that will eliminate the problem.
Charles
|
Thank you. the customers would be after my head on a platter if they
were prevented from retrieving new mail for XX hours during the xfer
over the WAN, so manual migration appears to be the only option.
Although.....perhaps if customers are alerted to these very caveats,
and advised that if they do find they've downloaded a partial message
they can ask us to retrieve the complete message from the old server
for them manually....that might do the trick. We plan on doing this
in the wee hours of the morning on a weekend, when the vast majority
of customers wouldn't be affected, however we have a substantial
customer base in the UK, which means some proportion of customers
will be affected no matter when we do it.
I've always in the past been adamantly opposed to mass migrations due
to other, worse pitfalls that can happen when moving to systems
different from the original (having been through them twice
unwillingly in the past), but the *near* elegance of this idea has me
captivated, so to speak.
Paul Theodoropoulos
http://www.anastrophe.com |
|
| Back to top |
|
|
Charles Cazabon
*nix forums Guru
Joined: 08 Jan 2005
Posts: 805
|
| Posted: Thu Jul 06, 2006 7:32 pm Post subject:
Re: writing directly to Maildir/new
|
|
|
Paul Theodoropoulos <paul@anastrophe.com> wrote:
| Quote: | At 12:00 PM 7/6/2006, Charles Cazabon wrote:
Yes, exactly.
If you can simply disable service while the transfer is in place (either
explicitly, by refusing connections, or trying to hide it with a dummy
service that always reports "no mail"), re-enabling normal service once
the transfer is complete, that will eliminate the problem.
Thank you. the customers would be after my head on a platter if they
were prevented from retrieving new mail for XX hours during the xfer
|
Okay, that'd be a "no" to the "if", then.
| Quote: | Although.....perhaps if customers are alerted to these very caveats,
and advised that if they do find they've downloaded a partial message
they can ask us to retrieve the complete message from the old server
for them manually....
|
Ouch. You're in for a lot of manual work if you do it that way.
Suggestion instead:
-tell customers incoming/new mail will be unaffected, but that legacy/old
mail will take some hours to transfer.
-point MX records (or however the mail is getting to the new servers) at
approximately the same time that you tell customers to point their MUAs at
the new server. At the same time, redirect or proxy attempted connections
to the "old" server.
-transfer the "old" mail to a place on the new server on the same filesystem
as the users' maildirs -- but not actually in their maildirs.
i.e., the new server would have, in one filesystem,
/mail/domain1/users/foo/inbox/
bar/inbox/
[...]
domain2/users/qux/inbox/
[...]
/oldmail/domain1/users/foo/inbox/
bar/inbox/
[...]
-then either when the transfer is done (if users can wait til the end for
their "old" mail) or periodically (if it would be better for it to show up
a bit at a time as the transfer proceeds) move (not copy -- move, so it's
atomic -- that's why it has to be in the same filesystem) messages from
/oldmail/domainX/users/Y/inbox/{cur,new}/ to
/mail/domainX/users/Y/inbox/{cur,new}/
. You can cron this to happen every five minutes while the transfer
proceeds, for instance.
Charles
--
--------------------------------------------------------------------------
Charles Cazabon <qmail@discworld.dyndns.org>
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
My services include qmail consulting. See http://pyropus.ca/ for details.
-------------------------------------------------------------------------- |
|
| Back to top |
|
|
Paul Theodoropoulos
*nix forums beginner
Joined: 19 Jul 2005
Posts: 17
|
| Posted: Thu Jul 06, 2006 8:23 pm Post subject:
Re: writing directly to Maildir/new
|
|
|
At 12:32 PM 7/6/2006, Charles Cazabon wrote:
| Quote: | Paul Theodoropoulos <paul@anastrophe.com> wrote:
Although.....perhaps if customers are alerted to these very caveats,
and advised that if they do find they've downloaded a partial message
they can ask us to retrieve the complete message from the old server
for them manually....
Ouch. You're in for a lot of manual work if you do it that way.
|
maybe. with the majority of our customers asleep and not checking (we
can ask in advance that customers close their email client programs
that evening), that leaves about ten percent of our customers
potentially checking for new mail during the process. of that ten
percent, the odds that a new message would 'collide' with their
checking for mail at that very moment i would guess are fairly low -
say, 30% chance at any given moment? so i'm thinking perhaps a few
dozen instances of partial mail.... I just added them up, we have
about 4,000 mailboxes spread across about 500 domains (yes, we're
tiny). so using my guestimates, that's perhaps 400 mailboxes active
during the xfer, so 120 mailboxes possibly experiencing collisions
during the process. again, they're just guesses. but perhaps modestly informed.
| Quote: | Suggestion instead:
-tell customers incoming/new mail will be unaffected, but that legacy/old
mail will take some hours to transfer.
-point MX records (or however the mail is getting to the new servers) at
approximately the same time that you tell customers to point their MUAs at
the new server. At the same time, redirect or proxy attempted connections
to the "old" server.
-transfer the "old" mail to a place on the new server on the same
filesystem
as the users' maildirs -- but not actually in their maildirs.
i.e., the new server would have, in one filesystem,
/mail/domain1/users/foo/inbox/
bar/inbox/
[...]
domain2/users/qux/inbox/
[...]
/oldmail/domain1/users/foo/inbox/
bar/inbox/
[...]
-then either when the transfer is done (if users can wait til the end for
their "old" mail) or periodically (if it would be better for it to show up
a bit at a time as the transfer proceeds) move (not copy -- move, so it's
atomic -- that's why it has to be in the same filesystem) messages from
/oldmail/domainX/users/Y/inbox/{cur,new}/ to
/mail/domainX/users/Y/inbox/{cur,new}/
. You can cron this to happen every five minutes while the transfer
proceeds, for instance.
|
Interesting idea as well. Perhaps i could use rsync to generate the
customer directory structure in advance of the process, so that when
the cutover happens the customers won't be faced with non-existent
mailboxes (i was going to rely on ufsrestore building the directory
heirarchies in advance of the data being copied over).
Paul Theodoropoulos
http://www.anastrophe.com |
|
| Back to top |
|
|
Joshua Megerman
*nix forums beginner
Joined: 29 Mar 2006
Posts: 9
|
| Posted: Thu Jul 06, 2006 10:02 pm Post subject:
Re: writing directly to Maildir/new
|
|
|
| Quote: | Interesting idea as well. Perhaps i could use rsync to generate the
customer directory structure in advance of the process, so that when
the cutover happens the customers won't be faced with non-existent
mailboxes (i was going to rely on ufsrestore building the directory
heirarchies in advance of the data being copied over).
I would do the following: |
1) Create a tarball of the existing client directory tree and extract it
in the new server's "live" location - don't worry about it changing
between now and the cutover, you'll resync it a couple times before then.
2) Use rsync to update the tree, including the --delete option to clear
out stuff that's been deleted from the source. It's a little less
efficient thatn it would otherwise be since Maildirs rename files to show
status, but it's still pretty clean. Do this sync a few times to get a
feel for how long it will take when you're ready for the final cutover.
3) Crank your DNS TTL way down - ~300 seconds or so is a good "switch"
time. You'll put it back up when you're finished with the move. You need
to do this at least old-TTL before you try to cut over, preferably 2x just
in case.
4) Do a final pre-cutover rsync to get it as close to current as possible.
This is actually the first part of the move, so continue onward
immediately once the rsync finishes.
5) Shut down the qmail-send portion on both the old and new servers - you
can have qmail-smtp running to queue-up new mail outbound. Also, shut
down all pop/imap servers until the transfer is done.
6) Do the LAST rsync, to catch any last-minute changes. If you've kept up
pretty well, this hopefully won't take more than 15 minutes or so. People
checking email won't be able to get in at this time, but sending email
should still work.
7) Change DNS while the rsync is running so that it propagates.
 Start qmail-send and pop/imap on the new server. If you have webmail,
you can configure the old webmail to talk to the new imap server until you
shut it down as well.
9) Configure qmail-send on the old server to forward ALL email to the new
server by IP address (using smtproutes). Specify that the old server is
allowed to relay through the new server until it's shut down. Then start
qmail-send and flush the queue (svc -a) to forward all mail to the new
server's qmail-smtp instance. You may want to reduce the
concurrencyremote value before doing this to avoid overloading the new
SMTP server.
10) Once you're sure that everything is moved and DNS is pointing to the
new server, as well as the queue being flushed, shut down qmail on the old
one.
It's fairly involved, and does include some total downtime, but nowhere
near as much as a physical move, and has the advantage of being very safe
since you effectively run 2 parallel servers until the last second.
Josh
--
Joshua Megerman
SJGames MIB #5273 - OGRE AI Testing Division
You can't win; You can't break even; You can't even quit the game.
- Layman's translation of the Laws of Thermodynamics
josh@honorablemenschen.com |
|
| Back to top |
|
|
Paul Theodoropoulos
*nix forums beginner
Joined: 19 Jul 2005
Posts: 17
|
| Posted: Thu Jul 06, 2006 10:49 pm Post subject:
Re: writing directly to Maildir/new
|
|
|
At 03:02 PM 7/6/2006, Joshua Megerman wrote:
| Quote: | [elided for brevity]
It's fairly involved, and does include some total downtime, but nowhere
near as much as a physical move, and has the advantage of being very safe
since you effectively run 2 parallel servers until the last second.
|
thanks. this - in somewhat modified form - is essentially what i was
planning. a giant tarball won't work, as this old server doesn't have
enough space - but that's where doing a ufsdump/ufsrestore over the
WAN as an initial step would work okay.
i've managed in my long sysadmin career to not use rsync much. i
reread the man pages today, and it seems that the --temp-dir= command
line option could be the saving grace - by forcing the tempdir onto
the same filesystem as the maildirs, it would populate the
directories via atomic moves, even while the system's live. at least,
that's my interpretation. that would mitigate even more downtime.
Paul Theodoropoulos
http://www.anastrophe.com |
|
| Back to top |
|
|
Paul Theodoropoulos
*nix forums beginner
Joined: 19 Jul 2005
Posts: 17
|
| Posted: Thu Jul 06, 2006 11:51 pm Post subject:
Re: writing directly to Maildir/new
|
|
|
At 12:13 PM 7/6/2006, Jeremy Kitchen wrote:
| Quote: | can you use rsync to make the amount of data to be transferred
during downtime
smaller?
You should be able to rsync the mail store and get a good idea of how much
downtime you'll have.. then right before you do the final switchover, run
rsync again (to get as much mail as possible transferred before downtime),
then shut services off, run rsync a final time, and voila! 
|
yes, it's looking like rsync may do the trick, particularly by using
the --temp-dir command line option to force the temp files to be
written to the same filesystem, for those delicious atomic moves.
Paul Theodoropoulos
http://www.anastrophe.com |
|
| Back to top |
|
|
Russell Nelson
*nix forums beginner
Joined: 15 Mar 2005
Posts: 38
|
| Posted: Mon Jul 10, 2006 3:30 pm Post subject:
Re: writing directly to Maildir/new
|
|
|
Paul Theodoropoulos writes:
| Quote: | customers would see no mail, but as the hours pass, their Maildirs
would be populated with their mail from the old server (new mail being
redirected from the MX servers to the new server after that initial
interval as well).
|
That's how I always do a transition.
| Quote: | my concern is with what happens if there's a 'collision' with a
client connecting to their account while the ufsrestore is in the
process of writing an existing message into Maildir/new.
|
Oh, yes, rather. It will be bad. You could indulge in a bit of
hackery to qmail-pop3d, and cause it to ignore files in Maildir/new
that have "recently" been written to. It already stat()s files, so
you're not adding any significant overhead. But as noted, that's
hackery. Much more reliable to move files into Maildir/tmp and
rename.
--
--my blog is at http://blog.russnelson.com | When immigration is
Crynwr sells support for free software | PGPok | outlawed, only criminals
521 Pleasant Valley Rd. | +1 315-323-1241 | will immigrate. Illegal
Potsdam, NY 13676-3213 | Sheepdog | immigration causes crime. |
|
| Back to top |
|
|
Google
|
|
| Back to top |
|
|
|
|
The time now is Sun Nov 23, 2008 10:27 am | All times are GMT
|
|
Car Finance | Dutch Bodybuilding Forums | Electricity Suppliers | Web Games | Loans
|
|
Copyright © 2004-2005 DeniX Solutions SRL
|
|
|
|
Other DeniX Solutions sites:
Unix/Linux blog |
electronics forum |
medicine forum |
science forum |
|
|
Privacy Policy
|
Powered by phpBB © 2001, 2005 phpBB Group
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
Forum index
»
Apps
»
Qmail
