|
|
|
|
|
|
| Author |
Message |
Munish Chopra
*nix forums beginner
Joined: 06 Jul 2002
Posts: 4
|
 Posted: Sat Jul 06, 2002 8:18 pm Post subject:
Re: Package system flaws?
|
|
|
On 2002-07-06 18:05 +0000, Dan Moschuk wrote:
| Quote: |
I've been doing some thinking lately about our ports system versus what
other systems have adopted and was curious as to what people think on
the subject? What does FreeBSD do well? Where can we improve? How does it
rate against the umpteen Linux flavours?
I'm not sure that this classifies as an architectual discussion (for now)
so if you feel its appropriate please feel free to redirect to ports@.
Cheers,
-Dan
|
A while back I heard some 'rumours' (oh boy, rumours...) that FreeBSD
was either going to switch to OpenPackages or to a new XML-based system.
This was around the same time jkh posted something on the subject of
ports or sysinstall or in that direction. In any case, OpenPackages has
lost wind for now, though I've been told they just need someone to pick
up the slack and get everyone rolling again.
As for our current ports system, many people (including me, though I'm
by no means an expert on the subject) feel that it's very limiting and
has some substantial flaws (concept is great, implementation is
lacking). Most of these come up when you're porting an app and you need
to make some ugly hacks for stuff to work the way you want it to. Also,
looking at the 'big picture', a few other concerns come up.
I'm sure some other people have more concrete things to say.
--
Munish Chopra The FreeBSD NVIDIA Driver Initiative
http://nvidia.netexplorer.org
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
 |
Terry Lambert
*nix forums Guru
Joined: 19 Mar 2002
Posts: 434
|
|
| Back to top |
|
|
Wes Peters
*nix forums addict
Joined: 09 Apr 2002
Posts: 90
|
| Posted: Sat Jul 06, 2002 11:31 pm Post subject:
Re: Package system flaws?
|
|
|
Munish Chopra wrote:
| Quote: |
On 2002-07-06 18:05 +0000, Dan Moschuk wrote:
I've been doing some thinking lately about our ports system versus what
other systems have adopted and was curious as to what people think on
the subject? What does FreeBSD do well? Where can we improve? How does it
rate against the umpteen Linux flavours?
I'm not sure that this classifies as an architectual discussion (for now)
so if you feel its appropriate please feel free to redirect to ports@.
Cheers,
-Dan
A while back I heard some 'rumours' (oh boy, rumours...) that FreeBSD
was either going to switch to OpenPackages or to a new XML-based system.
|
OpenPackages has some interesting, sticky problems to solve that are not
necessarily problems for FreeBSD. For instance, if you've decided to
store archives of files in tar format, which tar do you use? Do you write
the code so it can recognize and use the native tar utility on every system,
or do you mandate an OP-approved tar utility and require the sysadmin to
install OPtar on the system via some method other than OPpkg_add, which
only supports OPtar?
| Quote: | This was around the same time jkh posted something on the subject of
ports or sysinstall or in that direction. In any case, OpenPackages has
lost wind for now, though I've been told they just need someone to pick
up the slack and get everyone rolling again.
|
What they really need is somebody to just make the current system work,
rather than worrying about the global package architecture of the 22nd
century.
| Quote: | As for our current ports system, many people (including me, though I'm
by no means an expert on the subject) feel that it's very limiting and
has some substantial flaws (concept is great, implementation is
lacking). Most of these come up when you're porting an app and you need
to make some ugly hacks for stuff to work the way you want it to. Also,
looking at the 'big picture', a few other concerns come up.
|
Have you followed up by making some improvements or suggesting an improved
architecture? If this is the start of that process, thank you in advance.
Please be sure to scan the archives for the past 3 or 4 iterations of this
conversation so you'll truly be ready to address the issues.
| Quote: | I'm sure some other people have more concrete things to say.
|
I hope so. ;^)
--
"Where am I, and what am I doing in this handbasket?"
Wes Peters Softweyr LLC
wes@softweyr.com http://softweyr.com/
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Wes Peters
*nix forums addict
Joined: 09 Apr 2002
Posts: 90
|
| Posted: Sun Jul 07, 2002 12:08 am Post subject:
Re: Package system flaws?
|
|
|
Dan Moschuk wrote:
| Quote: |
I've been doing some thinking lately about our ports system versus what
other systems have adopted and was curious as to what people think on
the subject? What does FreeBSD do well?
|
FreeBSD does well at a number of important things:
o Provides a framework for porting applications to FreeBSD. Don't belittle
this point, the ports collection is a huge advantage.
o Provides dependency checking that pretty much works. This is certainly
far better than RPMs, for instance, and a LOT better than doing it
manually.
o Provides the ability to wrap a "port" that has been built into a binary
"package," and the associated tools to install and deinstall packages.
The dependency checking mechanism works as well for packages as it does
for ports, modulo a few nits like having packages on multiple CD-ROMs
and only one CD-ROM drive.
o The ports team and the FreeBSD infrastructure provide a very up-to-date
collection of ports and pre-compiled packages. This is especially
valuable for those who use binary packages and need critical security
updates for them in a timely manner.
| Quote: | Where can we improve?
|
In all of the above areas, plus all the ones we haven't addressed yet.
Some of the areas are extensions to problems we have with binary system
distributions, some are unique to the ports/packages system. The issue
of generational control of system configurations bites every OS; having
nearly 9,000 "free" applications magnifies the problem significantly.
Maintaining configuration files through the installation and de-
installation of binary ports is a challenge we've not yet risen to.
| Quote: | How does it rate against the umpteen Linux flavours?
|
Hard to say, because the upteen Linux flavors have umpteen different ways
of addressing the same problem. They have different tool sets, such as
RPMs and dpkg/apt-get, they have different staff levels dedicated to
porting traditional UNIX applications and such, and they have different
ways to of communicating such to their users. Some lower-end Linux
distros do none of the above, relying on you the user to obtain whatever
applications you want via your own means.
| Quote: | I'm not sure that this classifies as an architectual discussion (for now)
so if you feel its appropriate please feel free to redirect to ports@.
|
I'd certainly like to see it evolve into an architectural discussion, even
if all it accomplishes is to layout the work that needs to be done and
provide some sort of road map of what the next step or two might be. I
see a lot of interest in improving our application packaging and building
tools, but no concrete idea of what to improve next.
I'm more interested in the binary package side personally, a holdover
from a previous professional involvement in this area. I have a number
of ideas for things that could be improved in relatively small projects
if someone wants to discuss those with me.
For the ports building infrastructure, I lack enough knowlege about the
ports infrastructure to be able to contribute meaningfully, but I am sure
the portsmgr team and all active ports maintainers are interested in any
such discussion.
Please, anyone who is interested in this aspect of FreeBSD, read the
archives to learn about the issues that have been brought up in the past,
think a bit, and start a discussion.
--
"Where am I, and what am I doing in this handbasket?"
Wes Peters Softweyr LLC
wes@softweyr.com http://softweyr.com/
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Dag-Erling Smorgrav
*nix forums Guru Wannabe
Joined: 23 Mar 2002
Posts: 110
|
| Posted: Sun Jul 07, 2002 8:09 am Post subject:
Re: Package system flaws?
|
|
|
Wes Peters <wes@softweyr.com> writes:
| Quote: | Dan Moschuk wrote:
Where can we improve?
In all of the above areas, plus all the ones we haven't addressed yet.
|
*grin*
One thing we could improve a lot is the package file format. We
currently use gzipped tarballs, which have to be completely unpacked
before processing can begin. One improvement we can make is to use an
archive format such as zip that allows us to extract individual files
quickly, so we can extract the metadata and check dependencies
etc. without unpacking the entire package. This would save both time
and space (the current system can fail if the temp directory doesn't
have room for the unpacked package, even if the installation directory
has room to spare). A further improvement is to use a custom archive
format that always stores the metadata at the beginning of the archive
so we can install packages from the net without having to download and
store them locally first (zip isn't suitable for this as it stores the
content directory at the end of the archive, and the files within the
archive can appear in any order).
DES
--
Dag-Erling Smorgrav - des@ofug.org
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Nik Clayton
*nix forums beginner
Joined: 09 Apr 2002
Posts: 20
|
| Posted: Sun Jul 07, 2002 12:51 pm Post subject:
Re: Package system flaws?
|
|
|
On Sun, Jul 07, 2002 at 12:09:05PM +0200, Dag-Erling Smorgrav wrote:
| Quote: | One thing we could improve a lot is the package file format. We
currently use gzipped tarballs, which have to be completely unpacked
before processing can begin. One improvement we can make is to use an
archive format such as zip that allows us to extract individual files
quickly, so we can extract the metadata and check dependencies
etc. without unpacking the entire package.
|
gzipped UFS filesystem images that can be mounted on a vn/md device. . .
N
--
FreeBSD: The Power to Serve http://www.freebsd.org/ (__)
FreeBSD Documentation Project http://www.freebsd.org/docproj/ \\\'',)
\/ \ ^
--- 15B8 3FFC DDB4 34B0 AA5F 94B7 93A8 0764 2C37 E375 --- .\._/_) |
|
| Back to top |
|
|
Dan Moschuk
*nix forums beginner
Joined: 06 Jul 2002
Posts: 7
|
| Posted: Sun Jul 07, 2002 1:34 pm Post subject:
Re: Package system flaws?
|
|
|
| > Dan Moschuk wrote:
| > > Where can we improve?
| > In all of the above areas, plus all the ones we haven't addressed yet.
|
| *grin*
|
| One thing we could improve a lot is the package file format. We
| currently use gzipped tarballs, which have to be completely unpacked
| before processing can begin. One improvement we can make is to use an
| archive format such as zip that allows us to extract individual files
| quickly, so we can extract the metadata and check dependencies
| etc. without unpacking the entire package. This would save both time
| and space (the current system can fail if the temp directory doesn't
| have room for the unpacked package, even if the installation directory
| has room to spare). A further improvement is to use a custom archive
| format that always stores the metadata at the beginning of the archive
| so we can install packages from the net without having to download and
| store them locally first (zip isn't suitable for this as it stores the
| content directory at the end of the archive, and the files within the
| archive can appear in any order).
I don't think using an archive format like zip would be a step in the
right direction. If the package file format were to be redesigned, I would
vote for a custom header prepended to a bziped tarball.
Cheers,
-Dan
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Cy Schubert - CITS Open S
*nix forums beginner
Joined: 14 May 2002
Posts: 8
|
| Posted: Sun Jul 07, 2002 2:00 pm Post subject:
Re: Package system flaws?
|
|
|
In message <20020706220511.GA88651@scoobysnax.jaded.net>, Dan Moschuk
writes:
| Quote: |
I've been doing some thinking lately about our ports system versus what
other systems have adopted and was curious as to what people think on
the subject? What does FreeBSD do well? Where can we improve? How does it
rate against the umpteen Linux flavours?
|
Comparing our package system with others, such as RH's RPM. Sun's pkg*,
and DEC's... er Compaq's... er HP's setld, the FreeBSD pkg_* utilities
are one of the better package applications in the industry. Actually I
like our package system better than Suns. However, I have been spoiled
by 19 years of using IBM's SMP/E (and it's predecessor SMP) for the MVS
(mainframe) operating system prior to my 10 years in the UNIX world.
Unfortunately I don't think we have enough people on the project to
build the Cadillac (some might argue monstrosity), with all of its
bells, whistles, prereq's, coreq's, functions, dependent functions,
PTF's, APARFIXES, USERMODS, zones, recovery, and other handy features
that IBM has built for MVS over the past 40 years. Interestingly IBM
has not ported SMP/E to AIX, probably because it would take them an ice
age to port and SMP/E isn't the easiest application to understand --
most people hated it while the few of us who fully understood it loved
it. Excluding the IBM "anomaly," I think we probably have the best
package system in the market.
Can we improve our package system? Yes. Can I think of any? Well, if
we want to keep it simple, I cannot think of any enhancements that
would improve our lives, except for possibly a GUI front end.
--
Cheers, Phone: 250-387-8437
Cy Schubert Fax: 250-387-5766
Team Leader, Sun/Alpha Team Email: Cy.Schubert@osg.gov.bc.ca
Open Systems Group, CITS
Ministry of Management Services
Province of BC
FreeBSD UNIX: cy@FreeBSD.org
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Mark Valentine
*nix forums addict
Joined: 13 Jun 2002
Posts: 62
|
| Posted: Sun Jul 07, 2002 3:46 pm Post subject:
Re: Package system flaws?
|
|
|
| Quote: | From: des@ofug.org (Dag-Erling Smorgrav)
Date: Sun 7 Jul, 2002
Subject: Re: Package system flaws?
One thing we could improve a lot is the package file format. We
currently use gzipped tarballs, which have to be completely unpacked
before processing can begin. One improvement we can make is to use an
archive format such as zip that allows us to extract individual files
quickly, so we can extract the metadata and check dependencies
etc. without unpacking the entire package.
|
My idea for this is to use a hybrid format.
The package itself would be an uncompressed archive in any available
simple format. Non-sequential access to the members of the archive
is non-critical for local storage where seeks are "cheap" (so tar
archives would be acceptable for that purpose, or for when the
entire archive will be transferred, e.g. from an ftp server), but
compressing the entire archive would be a significant efficiency loss
due to having to uncompress before seeking.
For package installs over slow data streams, however, sequential access
to the archive is important.
A _directory_ makes a good direct-access archive (whether stored locally
or via ftp/web/whatever server). If that's unpalatable, then a server-
side CGI/PHP script could serve up member elements from a sequential-
access archive. In any case the tools should accept a directory name as
a package archive.
However, the archive members are _not_ the individual files comprising
the package. Instead, the package is split into logical sub-packages
(there may be only one, that's fine), such as the core run-time package,
language sets, development headers and libraries, end-user documentation.
Each sub-package is stored as a compressed archive (.tar.bz2 or whatever).
The _package_ archive comprises the package metadata files (optionally
compressed - useful for the larger files such as +CONTENTS) and the
sub-package(s), each as a package archive member.
The idea here is to allow individual access to those parts of the package
that are needed individually, whilst gaining the benefits of compressing
major portions of the package as collections of files (to which individual
access is not generally required).
While the division of an existing port into sub-packages requires effort,
it's necessary only to gain the benefits of optionally-installable sub-
packages.
For existing ports, and for the majority of ports which will never likely be
sub-package, pkg_create simple creates the package archive using the existing
metadata files and a single compressed archive of the packages files.
Examples 1: simple package, not sub-packaged.
$ ls /var/spool/pkg/foo-x.y
+COMMENT +DESC +INSTALL
+CONTENTS +DISPLAY base.tgz
Example 2: package with optional development and documentation components
$ ls /var/spool/pkg/bar-m.n
+COMMENT +DESC +INSTALL devel.tgz
+CONTENTS +DISPLAY base.tgz doc.tgz
The package install tools would allow the user to select between optional
sub-packages. The base sub-package may be marked as "required". If there
are only "required" sub-packages, or the user elects to install "all" or
"only required" sub-packages, no dialog is necessary. And so on..
Cheers,
Mark.
--
Mark Valentine, Thuvia Labs <mark@thuvia.co.uk> <http://www.thuvia.co.uk>
"Tigers will do ANYTHING for a tuna fish sandwich." Mark Valentine uses
"We're kind of stupid that way." *munch* *munch* and endorses FreeBSD
-- <http://www.calvinandhobbes.com> <http://www.freebsd.org>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Wes Peters
*nix forums addict
Joined: 09 Apr 2002
Posts: 90
|
| Posted: Sun Jul 07, 2002 5:23 pm Post subject:
Re: Package system flaws?
|
|
|
Dag-Erling Smorgrav wrote:
| Quote: |
Wes Peters <wes@softweyr.com> writes:
Dan Moschuk wrote:
Where can we improve?
In all of the above areas, plus all the ones we haven't addressed yet.
*grin*
One thing we could improve a lot is the package file format. We
currently use gzipped tarballs, which have to be completely unpacked
before processing can begin. One improvement we can make is to use an
archive format such as zip that allows us to extract individual files
quickly, so we can extract the metadata and check dependencies
etc. without unpacking the entire package.
|
I'd rather go one step further and separate the metadata from the files
completely. I'm thinking an XML-ish format for the metadata, with
blobs of files stored in whatever format makes the most sense inter-
spersed. Zip format makes a lot of sense and is available in library
format, but I could easily tolerate tar or cpio, as well as a variety
of compression formats. The metadata should not be compressed for
speed of package processing.
This would also allow us to separate the files in the package into
distinct pieces, and apply metadata to the collections of files.
Picture, for instance, a package that installs an X application. We
have executable files that need to go into $PKG_BIN, perhaps a library
or two that needs to in $PKG_LIB, an application resource file that
needs to go into $X_APPRESDIR and some fonts that need to be installed
in $X_FONTDIR. We provide defaults for each of these, let the user
override them as necessary.
| Quote: | This would save both time
and space (the current system can fail if the temp directory doesn't
have room for the unpacked package, even if the installation directory
has room to spare). A further improvement is to use a custom archive
format that always stores the metadata at the beginning of the archive
so we can install packages from the net without having to download and
store them locally first (zip isn't suitable for this as it stores the
content directory at the end of the archive, and the files within the
archive can appear in any order).
|
Yes, an excellent idea. With a format as I've proposed, it would be
possible to specify the file collections as URLs rather than encoded
archives. You could download the entire package collection in about
the same amount of space as the current ports collection. Of course
this means pkg_add would have to be linked with libfetch, too. ;^)
--
"Where am I, and what am I doing in this handbasket?"
Wes Peters Softweyr LLC
wes@softweyr.com http://softweyr.com/
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Dag-Erling Smorgrav
*nix forums Guru Wannabe
Joined: 23 Mar 2002
Posts: 110
|
| Posted: Sun Jul 07, 2002 6:26 pm Post subject:
Re: Package system flaws?
|
|
|
Wes Peters <wes@softweyr.com> writes:
| Quote: | Of course
this means pkg_add would have to be linked with libfetch, too. ;^)
|
It already is.
DES
--
Dag-Erling Smorgrav - des@ofug.org
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Giorgos Keramidas
*nix forums Guru
Joined: 10 May 2002
Posts: 330
|
| Posted: Sun Jul 07, 2002 6:50 pm Post subject:
Re: Package system flaws?
|
|
|
On 2002-07-07 12:09 +0000, Dag-Erling Smorgrav wrote:
| Quote: | Wes Peters <wes@softweyr.com> writes:
Dan Moschuk wrote:
Where can we improve?
In all of the above areas, plus all the ones we haven't addressed yet.
*grin*
One thing we could improve a lot is the package file format. We
currently use gzipped tarballs, which have to be completely unpacked
before processing can begin. One improvement we can make is to use an
archive format such as zip that allows us to extract individual files
quickly, so we can extract the metadata and check dependencies
etc. without unpacking the entire package.
|
Err, that can be done with tar too. Although one might argue that
reading the entire archive (instead of a small TOC at the beginning or
end of the archive) is a bit clumsier...
% cat archive | ( cd /tmp; tar xzvf - PKG_METADATA )
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Dag-Erling Smorgrav
*nix forums Guru Wannabe
Joined: 23 Mar 2002
Posts: 110
|
| Posted: Sun Jul 07, 2002 6:55 pm Post subject:
Re: Package system flaws?
|
|
|
Giorgos Keramidas <keramida@ceid.upatras.gr> writes:
| Quote: | On 2002-07-07 12:09 +0000, Dag-Erling Smorgrav wrote:
Wes Peters <wes@softweyr.com> writes:
Dan Moschuk wrote:
Where can we improve?
In all of the above areas, plus all the ones we haven't addressed yet.
*grin*
One thing we could improve a lot is the package file format. We
currently use gzipped tarballs, which have to be completely unpacked
before processing can begin. One improvement we can make is to use an
archive format such as zip that allows us to extract individual files
quickly, so we can extract the metadata and check dependencies
etc. without unpacking the entire package.
Err, that can be done with tar too. Although one might argue that
reading the entire archive (instead of a small TOC at the beginning or
end of the archive) is a bit clumsier...
|
That's precisely what I want to avoid. With tar, you always have to
uncompress the entire package to find even just one file (unless you
use the GNU-specific --fast-read option, in which case it stops once
it finds the file you asked for)
DES
--
Dag-Erling Smorgrav - des@ofug.org
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Doug Barton
*nix forums addict
Joined: 24 Apr 2002
Posts: 91
|
| Posted: Sun Jul 07, 2002 7:15 pm Post subject:
Re: Package system flaws?
|
|
|
Dan Moschuk wrote:
| Quote: | I don't think using an archive format like zip would be a step in the
right direction. If the package file format were to be redesigned, I would
vote for a custom header prepended to a bziped tarball.
|
What is it about working on ports that makes people automatically
gravitate to the most hideous, complicated solutions? Why not just
create a tarball with the stuff to install, create the metadata file (in
whatever format), then create a tarball with the two together? Then,
step one is to unroll the tarball, read the metadata, grab dependencies
if needed, then proceed to do the rest of the install.
Before you start saying, "THAT WON'T WORK," please understand that I've
already used this exact technique in 3 different projects, including
windows, OS/2, and unix platforms.
Doug
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Dag-Erling Smorgrav
*nix forums Guru Wannabe
Joined: 23 Mar 2002
Posts: 110
|
| Posted: Sun Jul 07, 2002 7:28 pm Post subject:
Re: Package system flaws?
|
|
|
Doug Barton <DougB@FreeBSD.org> writes:
| Quote: | What is it about working on ports that makes people automatically
gravitate to the most hideous, complicated solutions? Why not just
create a tarball with the stuff to install, create the metadata file (in
whatever format), then create a tarball with the two together? Then,
step one is to unroll the tarball, read the metadata, grab dependencies
if needed, then proceed to do the rest of the install.
|
We want to be able to install a package from a non-rewindable source
without storing a temporary copy on disk. This means the metadata
must without fail be at the very beginning of the package.
DES
--
Dag-Erling Smorgrav - des@ofug.org
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Google
|
|
| Back to top |
|
|
|
|
The time now is Thu Jan 08, 2009 5:52 am | All times are GMT
|
|
Credit Cards | Loans | Car Finance | Internet Advertising | McDonalds
|
|
Copyright © 2004-2005 DeniX Solutions SRL
|
|
|
|
Other DeniX Solutions sites:
Unix/Linux blog |
electronics forum |
medicine forum |
science forum |
|
|
Privacy Policy
|
Powered by phpBB © 2001, 2005 phpBB Group
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
Forum index
»
*nix
»
BSD
»
FreeBSD
»
mail-lists
»
Architecture
