|
|
|
|
|
|
| Author |
Message |
Garrett Wollman
*nix forums addict
Joined: 08 May 2002
Posts: 59
|
 Posted: Mon Jul 08, 2002 4:05 am Post subject:
Re: Package system flaws?
|
|
|
In article <1A55D91B-921F-11D6-AACD-0003938C7B7E@queasyweasel.com>
Jordan writes:
[what is that, a GUID in that Message-ID field?!]
| Quote: | Having to seek to the end is, indeed, one of the major draw-backs of
zip. I have no idea why the originators, in their infinite wisdom, put
it there.
|
I do. (Was I the last person to still be using an IBM PC when this
happened back in 1988?)
Recall that Phil Katz was under legal pressure to create something
that was as entirely unlike System Enhancement Associates' ARC as
possible. ARC used a `distributed directory' model (stolen from tar)
because that made it trivial to append to an archive -- just overwrite
the ``end of archive'' block at the end of the file with new data, and
eventually a new ``end of archive'' block.
Katz wanted to preserve this ability without using the `distributed
directory' model, because floppy disks were and are unspeakably slow,
and even seeking through an archive to list all the files was painful
compared to reading a single sector with all the directory information
in it. If the directory is at the beginning, it can never be expanded
without making a copy of the entire archive (painfully inconvenient
for a 320K archive on a 360K floppy). So, he put the directory at the
end, where it was easy to find (in a maximum of two seeks) and could
be overwritten when extending the archive, just like the old `end of
archive' header in ARC format.
-GAWollman
--
Garrett A. Wollman | [G]enes make enzymes, and enzymes control the rates of
wollman@lcs.mit.edu | chemical processes. Genes do not make ``novelty-
Opinions not those of| seeking'' or any other complex and overt behavior.
MIT, LCS, CRS, or NSA| - Stephen Jay Gould (1941-2002)
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
 |
Garrett Wollman
*nix forums addict
Joined: 08 May 2002
Posts: 59
|
| Posted: Mon Jul 08, 2002 4:10 am Post subject:
Re: Package system flaws?
|
|
|
In article <200207080334.g683YQsg042547@dotar.thuvia.org> you write:
| Quote: | It made me sit up and think, but you almost certainly have to extend pax(1)
to realise the extensions (correct me if I'm wrong), and I rather like a
format I can script with existing utilities.
|
Not at all. You certainly *could* do it that way, but whether it's
worth it to do so depends greatly on what it is that you want to do
(that you have declined to share) with the archives other than
installing them. Using the pax headers to store package
metainformation is no more difficult to deal with than using the
comment field in a ZIP archive's directory to do the same thing. If
you're just examining the package, you can use a purpose-built tool to
get the metainformation out, and if you don't need that, you don't
need the tool either.
The pax specification provides that unrecognized extension headers are
to be ignored.
-GAWollman
--
Garrett A. Wollman | [G]enes make enzymes, and enzymes control the rates of
wollman@lcs.mit.edu | chemical processes. Genes do not make ``novelty-
Opinions not those of| seeking'' or any other complex and overt behavior.
MIT, LCS, CRS, or NSA| - Stephen Jay Gould (1941-2002)
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Doug Barton
*nix forums addict
Joined: 24 Apr 2002
Posts: 91
|
| Posted: Mon Jul 08, 2002 6:35 am Post subject:
Re: Package system flaws?
|
|
|
On Sun, 7 Jul 2002, Terry Lambert wrote:
| Quote: | Doug Barton wrote:
On Sun, 7 Jul 2002, Terry Lambert wrote:
We want to be able to install a package from a non-rewindable source
without storing a temporary copy on disk. This means the metadata
must without fail be at the very beginning of the package.
Ok, then what about storing the meta data as a seperate file? Why do they
have to be in the same package?
For the same reason the fact that root owns /etc/master.passwd is
not in a seperate file?
|
ENOPARSE
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Doug Barton
*nix forums addict
Joined: 24 Apr 2002
Posts: 91
|
| Posted: Mon Jul 08, 2002 6:47 am Post subject:
Re: Package system flaws?
|
|
|
As much as I love zip, it's probably not appropriate for compressing the
binary packages we actually distribute. I did a quick test with zip vs.
bzip2, both on max compression, and the bzip'ed tarball was 40% smaller
than the .zip file. Not only is bandwidth an issue for our users, but
there is currently a problem just pushing the packages from the cluster
out to ftp-master. Not to mention pushing them around to the mirrors.
One of the reasons I suggested the "compress the binaries, then compress
the metadata plus binary tarball" method is that it is more space
efficient.
I think that we should start asking the questions about how we can end up
with the smallest packages, then go backwards from there to decide how to
beat that format into doing what we want.
Doug
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Mark Valentine
*nix forums addict
Joined: 13 Jun 2002
Posts: 62
|
| Posted: Mon Jul 08, 2002 7:14 am Post subject:
Re: Package system flaws?
|
|
|
| Quote: | From: Doug Barton <DougB@freebsd.org
Date: Sun 7 Jul, 2002
Subject: Re: Package system flaws?
One of the reasons I suggested the "compress the binaries, then compress
the metadata plus binary tarball" method is that it is more space
efficient.
|
Compressing the "metadata + binary tarball" just lost you the ability to
access the metadata without uncompressing the whole caboodle.
It isn't _all_ about space (but space helps a lot). See my earlier
suggestion which was basically "compress the binaries, then _archive_
the metadata plus binary tarball".
Cheers,
Mark.
--
Mark Valentine, Thuvia Labs <mark@thuvia.co.uk> <http://www.thuvia.co.uk>
"Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses
"We're kind of stupid that way." *munch* *munch* and endorses FreeBSD
-- <http://www.calvinandhobbes.com> <http://www.freebsd.org>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Terry Lambert
*nix forums Guru
Joined: 19 Mar 2002
Posts: 434
|
| Posted: Mon Jul 08, 2002 7:29 am Post subject:
Re: Package system flaws?
|
|
|
Doug Barton wrote:
| Quote: | On Sun, 7 Jul 2002, Terry Lambert wrote:
Doug Barton wrote:
On Sun, 7 Jul 2002, Terry Lambert wrote:
We want to be able to install a package from a non-rewindable source
without storing a temporary copy on disk. This means the metadata
must without fail be at the very beginning of the package.
Ok, then what about storing the meta data as a seperate file? Why do they
have to be in the same package?
For the same reason the fact that root owns /etc/master.passwd is
not in a seperate file?
ENOPARSE
|
Metadata is metadata.
If it's good to put metadata in a file seperate from the data it
describes, then the judgement of "goodness" is universal.
We store the metadata for individual files in most file systems in
a shared structure which contains both the references to the data
and to the metadata (exceptions are FS's which do not support doing
this intrinsically, e.g. MSDOSFS, or FS's which have to store it
seperately in order to achieve minimal POSIX semantics, e.g. Udo
Walter's "UMSDOSFS" under Linux).
We do this because it makes the data and metadata non-severable,
it relieves us of having to consider synchronization issues which
would otherwise arise, and we do it because it's convenient in
terms of speed of operations involving both, and convenient in
terms of locality of reference. It also gets rid of the implied
graph edge for locking of data and metadata, which can lead to an
undetectable deadly embrace deadlock .
All of these arguments apply equally well to bundling package
metadata with package data: conceptually, that metadata is no
different than file ownership, flags, or permissions.
-- Terry
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Doug Barton
*nix forums addict
Joined: 24 Apr 2002
Posts: 91
|
| Posted: Mon Jul 08, 2002 7:32 am Post subject:
Re: Package system flaws?
|
|
|
On Mon, 8 Jul 2002, Mark Valentine wrote:
| Quote: | From: Doug Barton <DougB@freebsd.org
Date: Sun 7 Jul, 2002
Subject: Re: Package system flaws?
One of the reasons I suggested the "compress the binaries, then compress
the metadata plus binary tarball" method is that it is more space
efficient.
Compressing the "metadata + binary tarball" just lost you the ability to
access the metadata without uncompressing the whole caboodle.
|
Well, if people are dead set on having both things in the same package (I
still think two seperate files is a cleaner solution) then as long as the
binaries are compressed efficiently (using tar + bzip or some such) then
we can use a less efficient, though more friendly alternative to compress
the metadata + binary bit. Assuming that we use the most efficient means
possible to compress the binaries (and by binaries I of course mean "what
the package is designed to install") then it's not going to compress
further, no matter what method we use to squish the two together.
| Quote: | It isn't _all_ about space (but space helps a lot).
|
Don't underestimate this factor. Pushing the packages out to ftp-master
from bento is already a limiting factor on how often we can update the
package set. We also have to take foreign mirrors, and users who pay for
every byte they download into account.
There is also another reason to consider seperating the binary tarball and
the metadata that I haven't mentioned yet. And actually, now that I think
of it more it's another good reason to seperate the two things into
different files. If I have package A that depends on package B, under the
current system if we up the version of package B, we have to re-roll
package A altogether just to update the dependency data, even though
nothing about the binary has, or needs to change. By seperating the
metadata and the binaries we can just update the metadata with the new
dependency and push just that.
| Quote: | See my earlier suggestion which was basically "compress the binaries,
then _archive_ the metadata plus binary tarball".
|
Sorry to say I skimmed that too rapidly. It sounds like we agree roughly
on this point, although I'd love for us to use a non-GNU tool for this
(sorry, blatant prejudice there).
Doug
--
"We have known freedom's price. We have shown freedom's power.
And in this great conflict, ... we will see freedom's victory."
- George W. Bush, President of the United States
State of the Union, January 28, 2002
Do YOU Yahoo!?
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Artem Tepponen
*nix forums beginner
Joined: 08 Jul 2002
Posts: 7
|
| Posted: Mon Jul 08, 2002 12:38 pm Post subject:
RE: Package system flaws?
|
|
|
| Quote: | 1) Allows random access AND compression
At the expense of having to seek to the end first? What
about access to the metadata over a slow data stream,
like you might have got with tar --fast-read?
Having to seek to the end is, indeed, one of the major draw-backs of
zip. I have no idea why the originators, in their infinite
wisdom, put it there.
|
They optimized for adding files to huge archive.
Artem Tepponen
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Artem Tepponen
*nix forums beginner
Joined: 08 Jul 2002
Posts: 7
|
| Posted: Mon Jul 08, 2002 1:18 pm Post subject:
RE: Package system flaws?
|
|
|
| Quote: | From: Doug Barton [mailto:DougB@FreeBSD.org]
Sent: Monday, July 08, 2002 10:47 AM
I think that we should start asking the questions about how
we can end up with the smallest packages, then go backwards
from there to decide how to beat that format into doing what we want.
|
You will achieve best compression with tar+(bzip2|gzip),
cause format that compresses individual pieces and makes
archive usually loses to make archive and compress to one
file kind of format. Exceptions are very rare.
The only problem is how to handle metadata piece that
have to have at least two properties:
1. It should be available without whole thing being decompressed.
2. It should be available asap in case of slow link.
Artem Tepponen
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Dan Moschuk
*nix forums beginner
Joined: 06 Jul 2002
Posts: 7
|
| Posted: Mon Jul 08, 2002 1:42 pm Post subject:
Re: Package system flaws?
|
|
|
| > I think that we should start asking the questions about how
| > we can end up with the smallest packages, then go backwards
| > from there to decide how to beat that format into doing what we want.
|
| You will achieve best compression with tar+(bzip2|gzip),
| cause format that compresses individual pieces and makes
| archive usually loses to make archive and compress to one
| file kind of format. Exceptions are very rare.
|
| The only problem is how to handle metadata piece that
| have to have at least two properties:
| 1. It should be available without whole thing being decompressed.
| 2. It should be available asap in case of slow link.
I think the meta data should be available uncompressed prepended to the
..tar.bz2 archive. Zipping would also work, however as someone earlier
pointed out zip compression is sub-optimal compared to bzip (or gzip for
that matter). Considering the sheer amount (and size) of some of our
packages I don't think that this is a minor issue.
I'll be preparing a summary of this whole thread later on today.
-Dan
--
That poverty is no disaster is understood by everyone who has not yet
succumbed to the madness of greed and luxury that turns everything topsy-turvy.
-- Seneca
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Artem Tepponen
*nix forums beginner
Joined: 08 Jul 2002
Posts: 7
|
| Posted: Mon Jul 08, 2002 2:01 pm Post subject:
RE: Package system flaws?
|
|
|
| Quote: | From: Dan Moschuk [mailto:dan@FreeBSD.ORG]
Sent: Monday, July 08, 2002 7:43 PM
| The only problem is how to handle metadata piece that
| have to have at least two properties:
| 1. It should be available without whole thing being decompressed.
| 2. It should be available asap in case of slow link.
I think the meta data should be available uncompressed
prepended to the .tar.bz2 archive. Zipping would also work,
however as someone earlier pointed out zip compression
is sub-optimal compared to bzip (or gzip for that matter).
Considering the sheer amount (and size) of some of our
packages I don't think that this is a minor issue.
|
I guess that two files solution has some nice properties like
not having to update the whole shebang only when dependencies
are changed. It just needs a secure way to ensure that two
files are paired (MD5?). But it loses convenience and beauty
of single file scheme.
Artem Tepponen
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Mark Valentine
*nix forums addict
Joined: 13 Jun 2002
Posts: 62
|
| Posted: Mon Jul 08, 2002 2:30 pm Post subject:
Re: Package system flaws?
|
|
|
| Quote: | From: drosih@rpi.edu (Garance A Drosihn)
Date: Mon 8 Jul, 2002
Subject: Re: Package system flaws?
I do find the ports collection one of the most useful things in
the freebsd community, but the more comfortable you get with
depending on it, the more you see little cracks where various
problems crop up -- over and over again.
|
Indeed the system is very useful already. But here are my main gripes
anyway...
1. Packages install to /usr/local by default; however, they do not
(and cannot) follow my long-standing (since 4.2BSD) cross-platform
administrative policies for /usr/local. In fact, just attempting
a package install on those systems generally screws up the
permissions on /usr/local fatally.
The next generation system should install to /usr/pkg or /opt or
whatever.
(Policy differences here: my /usr/local is generally administered
by group "staff", not user "root"; I put in a bit of effort to
generally _not_ require the co-existence of multiple versions of
a package - a lot of the ports patches are there to accomplish just
this, renaming libraries and data directories - I don't want it).
2. I'd like to see improved support for building/installing packages
as non-root. Installing as root should only be necessary if local
policy requires it, or if there are set[ug]id components to install
(or similar permission requirements). Even in the latter case it
would be nice if the install created a simple script to run as root
to fix up the permissions.
I especially don't like having to build ports as root (I consider
bsd.port.mk unauditable, and I know a thing or two about make(1)).
3. There's no way to have pkg_add -r use a local package cache so
that only the first install needs to fetch the package _and its
dependencies_ over the Internet (see Cygwin's setup.exe for an
example of how this can be done reasonably well).
I went to add this feature and my stumbling block was that pkg_add
doesn't know the versioned name of the package archive to store
(but only for the primary package, the dependencies are fine).
That's to say that "pkg_add -r XFree86" should fetch and store (if
necessary) "XFree86-4.2.0_1,1.tgz" and its dependencies; alternatively,
"pkg_add -r XFree86-4.2.0_1,1.tgz" should do the right thing.
Currently, to install a package on multiple systems, I have to jump
through hoops to get the package and its dependencies into my local
package area.
| Quote: | Separate from that, one thing I think we need is some mechanism
which makes certain that the generated package-list for a port
is exactly correct.
|
I like your idea for creating packages from a chroot'd area, though
I'd probably prefer this to be an optional testing feature than the
default method (but yes, nobody would use it then...).
Cheers,
Mark.
--
Mark Valentine, Thuvia Labs <mark@thuvia.co.uk> <http://www.thuvia.co.uk>
"Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses
"We're kind of stupid that way." *munch* *munch* and endorses FreeBSD
-- <http://www.calvinandhobbes.com> <http://www.freebsd.org>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Mark Valentine
*nix forums addict
Joined: 13 Jun 2002
Posts: 62
|
| Posted: Mon Jul 08, 2002 2:41 pm Post subject:
Re: Package system flaws?
|
|
|
| Quote: | From: Garrett Wollman <wollman@lcs.mit.edu
Date: Mon 8 Jul, 2002
Subject: Re: Package system flaws?
In article <200207080334.g683YQsg042547@dotar.thuvia.org> you write:
It made me sit up and think, but you almost certainly have to extend pax(1)
to realise the extensions (correct me if I'm wrong), and I rather like a
format I can script with existing utilities.
Not at all. You certainly *could* do it that way, but whether it's
worth it to do so depends greatly on what it is that you want to do
(that you have declined to share) with the archives other than
installing them. Using the pax headers to store package
metainformation is no more difficult to deal with than using the
comment field in a ZIP archive's directory to do the same thing. If
you're just examining the package, you can use a purpose-built tool to
get the metainformation out, and if you don't need that, you don't
need the tool either.
|
Now I'm confused.
I'd assumed you were suggesting using pax(1) extensions to provide
some means of direct access to the archive members, not for storing
metadata...
What advantage is there in storing the metadata as extended pax(1) headers
instead of as the first file(s) in the archive?
And how does this help solve the problem of direct access to metadata
while keeping the package's files compressed?
Cheers,
Mark.
--
Mark Valentine, Thuvia Labs <mark@thuvia.co.uk> <http://www.thuvia.co.uk>
"Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses
"We're kind of stupid that way." *munch* *munch* and endorses FreeBSD
-- <http://www.calvinandhobbes.com> <http://www.freebsd.org>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Mark Valentine
*nix forums addict
Joined: 13 Jun 2002
Posts: 62
|
| Posted: Mon Jul 08, 2002 2:55 pm Post subject:
Re: Package system flaws?
|
|
|
| Quote: | From: Doug Barton <DougB@freebsd.org
Date: Mon 8 Jul, 2002
Subject: Re: Package system flaws?
On Mon, 8 Jul 2002, Mark Valentine wrote:
Compressing the "metadata + binary tarball" just lost you the ability to
access the metadata without uncompressing the whole caboodle.
Well, if people are dead set on having both things in the same package (I
still think two seperate files is a cleaner solution)
|
My earlier suggestion actually said just about the same thing, with the
_option_ of storing the two (or more) parts in an uncompressed archive
instead of a directory, for ease of handling.
In light of Wes' comments on storing the metadata, I'd modify my examples
as follows.
Example 1: simple package, not sub-packaged.
$ ls /var/spool/pkg/foo-x.y
base.bz2 package.xml
Example 2: package with optional development and documentation components
$ ls /var/spool/pkg/bar-m.n
base.bz2 devel.bz2 doc.bz2 package.xml
(In an archive, of course, package.xml would be the first member.)
| Quote: | then as long as the
binaries are compressed efficiently (using tar + bzip or some such) then
we can use a less efficient, though more friendly alternative to compress
the metadata + binary bit.
|
Bingo!
| Quote: | It isn't _all_ about space (but space helps a lot).
Don't underestimate this factor.
|
I should have said a _lot_. ;-)
| Quote: | There is also another reason to consider seperating the binary tarball and
the metadata that I haven't mentioned yet. And actually, now that I think
of it more it's another good reason to seperate the two things into
different files. If I have package A that depends on package B, under the
current system if we up the version of package B, we have to re-roll
package A altogether just to update the dependency data, even though
nothing about the binary has, or needs to change. By seperating the
metadata and the binaries we can just update the metadata with the new
dependency and push just that.
|
Indeed.
| Quote: | See my earlier suggestion which was basically "compress the binaries,
then _archive_ the metadata plus binary tarball".
Sorry to say I skimmed that too rapidly. It sounds like we agree roughly
on this point, although I'd love for us to use a non-GNU tool for this
(sorry, blatant prejudice there).
|
Ideally you use POSIX archive formats and any tool which implements them.
Cheers,
Mark.
--
Mark Valentine, Thuvia Labs <mark@thuvia.co.uk> <http://www.thuvia.co.uk>
"Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses
"We're kind of stupid that way." *munch* *munch* and endorses FreeBSD
-- <http://www.calvinandhobbes.com> <http://www.freebsd.org>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Jefferson Harlough
*nix forums beginner
Joined: 08 Jul 2002
Posts: 1
|
| Posted: Mon Jul 08, 2002 3:12 pm Post subject:
Re: Package system flaws?
|
|
|
| Quote: | I think the meta data should be available uncompressed prepended to the
.tar.bz2 archive. Zipping would also work, however as someone earlier
pointed out zip compression is sub-optimal compared to bzip (or gzip for
that matter). Considering the sheer amount (and size) of some of our
packages I don't think that this is a minor issue.
|
Would it be possible to replace the current compression algorithm(s) used in
zip with the bzip2 compression algorithms? Would that allow for a greater
degree of compression while still offering the benefits of the zip archive
format?
Package size is quite an issue to me personally. I do not yet have a
broadband connection, and I much prefer to download archives compressed with
the bzip2 software as they usually smaller than archives compressed with
gzip or compress. I'd have to ask that package size is considered as a very
important part of the decision, and hopefully a system that allows for a
high degree of compression will be used in the end.
Sincerely,
Jefferson
_________________________________________________________________
Send and receive Hotmail on your mobile device: http://mobile.msn.com
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Google
|
|
| Back to top |
|
|
|
|
The time now is Thu Jan 08, 2009 5:44 am | All times are GMT
|
|
Debt Help | Credit Card Advice | Debt Consolidation | Looking for Credit Cards? | Online Advertising
|
|
Copyright © 2004-2005 DeniX Solutions SRL
|
|
|
|
Other DeniX Solutions sites:
Unix/Linux blog |
electronics forum |
medicine forum |
science forum |
|
|
Privacy Policy
|
Powered by phpBB © 2001, 2005 phpBB Group
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
Forum index
»
*nix
»
BSD
»
FreeBSD
»
mail-lists
»
Architecture
