|
|
|
|
|
|
| Author |
Message |
Bruce Evans
*nix forums Guru Wannabe
Joined: 22 Mar 2002
Posts: 190
|
 Posted: Sun Jun 02, 2002 3:22 pm Post subject:
Re: Removing wait union
|
|
|
On Sun, 2 Jun 2002, Mike Barcroft wrote:
| Quote: | Does anyone have any objections to removing the deprecated 4.2/4.3BSD
wait union in <sys/wait.h>? It's been deprecating since Rev 1.1 and
there are only a few consumers in the base system. Attached are two
patches, one to removing it from <sys/wait.h> and the other to remove
its consumers. Changes to lpd( sent directly to its maintainer.
|
I think the only potential problem is use of the compatibility cruft
in deprecware outside the base system. It would be useful to have a
quick way to determine how many ports a change in a standard header
affects.
Bruce
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
 |
Mike Barcroft
*nix forums beginner
Joined: 03 Apr 2002
Posts: 25
|
| Posted: Mon Jun 03, 2002 9:08 pm Post subject:
Re: Removing wait union
|
|
|
Bruce Evans <bde@zeta.org.au> writes:
| Quote: | On Sun, 2 Jun 2002, Mike Barcroft wrote:
Does anyone have any objections to removing the deprecated 4.2/4.3BSD
wait union in <sys/wait.h>? It's been deprecating since Rev 1.1 and
there are only a few consumers in the base system. Attached are two
patches, one to removing it from <sys/wait.h> and the other to remove
its consumers. Changes to lpd( sent directly to its maintainer.
I think the only potential problem is use of the compatibility cruft
in deprecware outside the base system. It would be useful to have a
quick way to determine how many ports a change in a standard header
affects.
|
Kris is going to do a test run of the ports build for me.
Best regards,
Mike Barcroft
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Kris Kennaway
*nix forums Guru
Joined: 28 Apr 2002
Posts: 634
|
| Posted: Mon Jun 03, 2002 9:25 pm Post subject:
Re: Removing wait union
|
|
|
On Mon, Jun 03, 2002 at 01:22:14AM +1000, Bruce Evans wrote:
| Quote: | On Sun, 2 Jun 2002, Mike Barcroft wrote:
Does anyone have any objections to removing the deprecated 4.2/4.3BSD
wait union in <sys/wait.h>? It's been deprecating since Rev 1.1 and
there are only a few consumers in the base system. Attached are two
patches, one to removing it from <sys/wait.h> and the other to remove
its consumers. Changes to lpd( sent directly to its maintainer.
I think the only potential problem is use of the compatibility cruft
in deprecware outside the base system. It would be useful to have a
quick way to determine how many ports a change in a standard header
affects.
|
I'm always happy to test proposed patches on the ports cluster.
Kris |
|
| Back to top |
|
|
Terry Lambert
*nix forums Guru
Joined: 19 Mar 2002
Posts: 434
|
| Posted: Mon Jun 03, 2002 9:50 pm Post subject:
Re: Removing wait union
|
|
|
Kris Kennaway wrote:
| Quote: | On Mon, Jun 03, 2002 at 01:22:14AM +1000, Bruce Evans wrote:
I think the only potential problem is use of the compatibility cruft
in deprecware outside the base system. It would be useful to have a
quick way to determine how many ports a change in a standard header
affects.
I'm always happy to test proposed patches on the ports cluster.
|
I think the problem needs something other than the ports cluster.
The ports cluster is designed to keep dependencies seperate, so it
serializes a lot of things that could otherwise be done in parallel,
to ensure dependency order is maintained.
I think that, for the most part, what you care about in this case is
not catching broken dependencies, but catching breakage caused by
system cahnges, not inter-port dependency changes.
This really argues that you could install the ports sources on a
machine, and then build out all of the ports from local copies of
all the packages, in parallel -- if you had already installed the
things on which they dependend on the machine.
This would let you basically re-build in parallel, without needing
to enforce ports dependency order, on a single machine instead of
a cluster, and much faster, because all the dependencies would
already be there (from the previous "install" of all ports, rather
than from the build process for an individual port). All the builds
could be done in parallel in their own subdirectories (not installed),
so the process could be much fater than using the cluster the way the
cluser is supposed to be used.
-- Terry
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Garance A Drosihn
*nix forums Guru Wannabe
Joined: 21 Mar 2002
Posts: 190
|
| Posted: Mon Jun 03, 2002 10:51 pm Post subject:
Re: Removing wait union
|
|
|
At 4:50 PM -0700 6/3/02, Terry Lambert wrote:
| Quote: | All the builds could be done in parallel in their own
subdirectories (not installed), so the process could be
much faster than using the cluster the way the cluser is
supposed to be used.
|
This implies the building of a second cluster, one that is
setup for doing these parallel builds. That sounds like
it would be more trouble (to setup, and to keep setup)
than it's worth. Might as well just use the tools we
already have.
--
Garance Alistair Drosehn = gad@gilead.netel.rpi.edu
Senior Systems Programmer or gad@freebsd.org
Rensselaer Polytechnic Institute or drosih@rpi.edu
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Kris Kennaway
*nix forums Guru
Joined: 28 Apr 2002
Posts: 634
|
| Posted: Mon Jun 03, 2002 11:15 pm Post subject:
Re: Removing wait union
|
|
|
On Mon, Jun 03, 2002 at 04:50:01PM -0700, Terry Lambert wrote:
| Quote: | Kris Kennaway wrote:
On Mon, Jun 03, 2002 at 01:22:14AM +1000, Bruce Evans wrote:
I think the only potential problem is use of the compatibility cruft
in deprecware outside the base system. It would be useful to have a
quick way to determine how many ports a change in a standard header
affects.
I'm always happy to test proposed patches on the ports cluster.
I think the problem needs something other than the ports cluster.
The ports cluster is designed to keep dependencies seperate, so it
serializes a lot of things that could otherwise be done in parallel,
to ensure dependency order is maintained.
|
In practise this isn't an issue. I haven't obtained accurate
statistics, but the cluster runs at full capacity for pretty much the
entire build.
Kris |
|
| Back to top |
|
|
Terry Lambert
*nix forums Guru
Joined: 19 Mar 2002
Posts: 434
|
| Posted: Mon Jun 03, 2002 11:24 pm Post subject:
Re: Removing wait union
|
|
|
Garance A Drosihn wrote:
| Quote: | At 4:50 PM -0700 6/3/02, Terry Lambert wrote:
All the builds could be done in parallel in their own
subdirectories (not installed), so the process could be
much faster than using the cluster the way the cluser is
supposed to be used.
This implies the building of a second cluster, one that is
setup for doing these parallel builds. That sounds like
it would be more trouble (to setup, and to keep setup)
than it's worth. Might as well just use the tools we
already have.
|
No, you don't need a cluster.
I think people are not understanding the problem.
The problem is that a change to the system breaks the build
of a package.
What this means is that you should be able to:
1) Build all the packages on a system
2) Install everything that wasn't automatically installed as
a dependency
This gives you a system where all the *port* dependencies of any
randomly selected port are pre-satisfied, so the building can take
place independently from any other port.
It's about getting to the point where the dependencies are presatisfied.
The ports cluster is not about that. It's all about ensuring that
no dependencies are lost, or implicitly satisfied.
Effectively, you could get very close to a system that could be
used to build all the ports in parallel, in their own directory,
by simply installing all the packages in existance. You really
don't give a damn that the installed *binaries* of a port on
which you depend for actual operation would end up being different,
too. So you break the need for dependency order enforcement between
different ports.
In very simple terms:
o The point of the ports cluster is to make sure that
dependencies on other ports aren't broken
o The point of this system would be to make sure that
dependencies on the system aren't broken
Two very different problems. The solution of the first needs a
ports cluster to be able to solve it within about 8 hours. But
the much of that 8 hours is based on dependency ordering that
you can throw away, if you are trying to solve the "system changed"
problem, rather than the "ports changed" problem.
Make sense?
-- Terry
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Terry Lambert
*nix forums Guru
Joined: 19 Mar 2002
Posts: 434
|
| Posted: Mon Jun 03, 2002 11:33 pm Post subject:
Re: Removing wait union
|
|
|
Kris Kennaway wrote:
| Quote: | I think the problem needs something other than the ports cluster.
The ports cluster is designed to keep dependencies seperate, so it
serializes a lot of things that could otherwise be done in parallel,
to ensure dependency order is maintained.
In practise this isn't an issue. I haven't obtained accurate
statistics, but the cluster runs at full capacity for pretty much the
entire build.
|
You need to keep statistics of how many times each individual port
is built, in the cluster case (i.e. you need a combinatoric dependency
map and a way to factor out permutations that are necessary for
solving the ports problem, but not necessary for solving the "what
breaks?" problem).
When you build a given port on its own little virtual box, you
also build all the ports on which it depends. This is by design,
since it avoid the problem of finding dependencies which are not
explicitly called out.
In the case of a system change, you aren't concerned about that;
in fact, you *want* it to happen, since it reduces the total build
time.
How many machines are in the ports cluster? Knowing how many total
cycles are required, and how many redundant dependency builds happen,
would be a good way to ballpark whether it's feasible to build a
single machine that can tell what is broken by a proposed base OS
change, before the change is checked in.
-- Terry
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Kris Kennaway
*nix forums Guru
Joined: 28 Apr 2002
Posts: 634
|
| Posted: Mon Jun 03, 2002 11:37 pm Post subject:
Re: Removing wait union
|
|
|
On Mon, Jun 03, 2002 at 06:24:45PM -0700, Terry Lambert wrote:
| Quote: | No, you don't need a cluster.
|
Um, yes you do.
| Quote: | I think people are not understanding the problem.
The problem is that a change to the system breaks the build
of a package.
What this means is that you should be able to:
1) Build all the packages on a system
2) Install everything that wasn't automatically installed as
a dependency
|
Um, Terry, we have over 7000 ports now. It takes our 8-way cluster 20
hours to build them (sorry, the 8 hours I said earlier was wrong); as
already stated it runs at close to 100% capacity the entire time.
Therefore a good estimate of the number of equivalent CPU-hours is
8*20 hours = 6 2/3 CPU-days.
Very little time is wasted with the cluster sitting idle waiting for
dependencies to build (and I can probably get that down to 0 time
wasted if I just reorder the makefile a bit to start the build of long
dependency branches like GNOME first - GNOME and KDE are the only
"choke points" where the cluster ends up idling for a few minutes
except for one or two machines: by that time everything else is
finished anyway).
| Quote: | Two very different problems. The solution of the first needs a
ports cluster to be able to solve it within about 8 hours. But
the much of that 8 hours is based on dependency ordering that
you can throw away, if you are trying to solve the "system changed"
problem, rather than the "ports changed" problem.
Make sense?
|
No, because the above is wrong ;-)
Please accept that I actually know what I'm talking about (since I've
been managing the ports cluster for about 8 months now) and just drop
the matter.
Kris |
|
| Back to top |
|
|
Kris Kennaway
*nix forums Guru
Joined: 28 Apr 2002
Posts: 634
|
| Posted: Mon Jun 03, 2002 11:39 pm Post subject:
Re: Removing wait union
|
|
|
On Mon, Jun 03, 2002 at 06:33:57PM -0700, Terry Lambert wrote:
| Quote: | Kris Kennaway wrote:
I think the problem needs something other than the ports cluster.
The ports cluster is designed to keep dependencies seperate, so it
serializes a lot of things that could otherwise be done in parallel,
to ensure dependency order is maintained.
In practise this isn't an issue. I haven't obtained accurate
statistics, but the cluster runs at full capacity for pretty much the
entire build.
You need to keep statistics of how many times each individual port
is built, in the cluster case (i.e. you need a combinatoric dependency
map and a way to factor out permutations that are necessary for
solving the ports problem, but not necessary for solving the "what
breaks?" problem).
|
Here are your statistics.
Each port is built precisely this many times:
1
Jesus, Terry, please give us a little bit of credit for not being
completely stupid.
Kris |
|
| Back to top |
|
|
Will Andrews
*nix forums beginner
Joined: 04 Jun 2002
Posts: 21
|
| Posted: Tue Jun 04, 2002 1:42 am Post subject:
Re: Removing wait union
|
|
|
On Mon, Jun 03, 2002 at 06:33:57PM -0700, Terry Lambert wrote:
| Quote: | How many machines are in the ports cluster? Knowing how many total
cycles are required, and how many redundant dependency builds happen,
|
Terry, you just proved you don't actually know anything about the
software used to build packages on the bento cluster (and
potentially elsewhere).
I would suggest that you do a little resarch before talking and
leave the package build cluster details to the people who run it.
Regards,
--
wca |
|
| Back to top |
|
|
John Baldwin
*nix forums Guru Wannabe
Joined: 27 Mar 2002
Posts: 278
|
| Posted: Tue Jun 04, 2002 1:43 am Post subject:
Re: Removing wait union
|
|
|
On 04-Jun-2002 Terry Lambert wrote:
| Quote: | Kris Kennaway wrote:
I think the problem needs something other than the ports cluster.
The ports cluster is designed to keep dependencies seperate, so it
serializes a lot of things that could otherwise be done in parallel,
to ensure dependency order is maintained.
In practise this isn't an issue. I haven't obtained accurate
statistics, but the cluster runs at full capacity for pretty much the
entire build.
You need to keep statistics of how many times each individual port
is built, in the cluster case (i.e. you need a combinatoric dependency
map and a way to factor out permutations that are necessary for
solving the ports problem, but not necessary for solving the "what
breaks?" problem).
|
Each port should be built once.
| Quote: | When you build a given port on its own little virtual box, you
also build all the ports on which it depends. This is by design,
since it avoid the problem of finding dependencies which are not
explicitly called out.
|
Erm, nope, dependencies are pkg_add'd and then the port is built (IIRC).
If a port doesn't specify a dependency that it needs then that is an error.
--
John Baldwin <jhb@FreeBSD.org> <>< http://www.FreeBSD.org/~jhb/
"Power Users Use the Power to Serve!" - http://www.FreeBSD.org/
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Terry Lambert
*nix forums Guru
Joined: 19 Mar 2002
Posts: 434
|
| Posted: Tue Jun 04, 2002 4:23 am Post subject:
Avoiding unnecessary breakage (was Re: Removing wait union)
|
|
|
Will Andrews wrote:
[ ... ]
This thread is not about the ports cluster, per se.
We could actually care less about actually building the ports as
they will be used, in this case. It is a proposed means, not an
ends.
I have changed the subject to avoid any further confusion.
--
This whole thread started because Kris wanted developers to be
individually responsible for fixing the ports that broke because
of their commits to the OS, or, as a minimum second derivative,
for notifying the official maintainer of a port that was broken
as a result of such a commit.
This requires that developers making such changes have a-priori
knowledge of the effects of their changes on code that is probably
not, nor ever will be, installed on their machines.
The problem with just requiring this unilaterally is that I don't
know of one developer, other than Kris, who has a box with all of
the ports source code on it so that changes that might have this
impact can be tested.
Why am I posting at all in this thread?
I'm personally philosophically opposed to the idea of "bit rot":
when the ISODE and X.25 code "rotted" because it was not maintained
when the routing code was changed out from under it, and when the
LFS code was moved to the attic because the unified VM/buffer cache
code changes were checked in without maintaining the LFS code (a
client of the VM and buffer cache code), it really pissed me off.
It really, really, really pissed me off.
Kris has identified an analogous situation with base system changes
impacting ports. It's a natural extension of the same philosophy,
though it's harder to justify unilaterally, because it involved third
party code that's not "part of the OS".
My participation in this thread is therefore about whether or not
it's possible to come up with some method of determining what
ports will be broken by a proposed change, before that change is
made, and in a reasonable time frame, with a reasonable amount of
effort, which it would be reasonable to expect a developer who is
contemplating such a change to expend.
In effect, this means that, if this idea is to go forward, there
are only a few obvious courses of action:
1) Build another "ports cluster" so that changes can be tested
before they are committed
2) Find another way around the problem, which has less overhead
than #1
3) Run all such changes through the existing ports cluster by
having Kris apply them locally on the machines, without them
having been checked into CVS, and giving a thumbs up/down to
the change.
4) Tell Kris "lump it": breakage is going to happen, and it's
up to you as the ports guy to mop out the rest rooms after
the concert, after we have peed all over the fixtures (i.e.
leave things as they are now).
Call me crazy, but I think #4 is not an option. Kris would not
have raised the issue if it were OK with him to define the breakage
as "necessary".
I also don't think that #3 is an option. The anti-process people
would probably object to having to run their code through another
volunteer, who could not say "yes/no" until he could find the time.
It would, at the very least, be a definite bottleneck.
#1's a possibility, but it's unlikely to be a useful possibility,
and it almost implies #3, since it requires a gatekeeper, if only
for scheduling use of the cluster. Such resources would probably
be better spent on doing the primary work of the ports cluster,
anyway: building ports.
So that leaves #2.
If you can think of any other options, please chime in *now*.
So if you don't mind, I'd like to continue looking at ways to
attack implementing #2, unless you are prepared to find someone
else (or ar volunteering yourself, instead of implicitly volunteering
Kris, again) to muck out the restrooms after the next concert.
Thanks,
-- Terry
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Terry Lambert
*nix forums Guru
Joined: 19 Mar 2002
Posts: 434
|
| Posted: Tue Jun 04, 2002 4:43 am Post subject:
Avoiding unnecessary breakage
|
|
|
Kris Kennaway wrote:
| Quote: | On Mon, Jun 03, 2002 at 06:24:45PM -0700, Terry Lambert wrote:
No, you don't need a cluster.
Um, yes you do.
|
If this is the *only possible way* to tell if a port will be broken
by a FreeBSD change, then I hold little hope that your request that
people avoid breaking ports with their commits can ever be satisfied.
| Quote: | Um, Terry, we have over 7000 ports now. It takes our 8-way cluster 20
hours to build them (sorry, the 8 hours I said earlier was wrong); as
already stated it runs at close to 100% capacity the entire time.
Therefore a good estimate of the number of equivalent CPU-hours is
8*20 hours = 6 2/3 CPU-days.
Very little time is wasted with the cluster sitting idle waiting for
dependencies to build (and I can probably get that down to 0 time
wasted if I just reorder the makefile a bit to start the build of long
dependency branches like GNOME first - GNOME and KDE are the only
"choke points" where the cluster ends up idling for a few minutes
except for one or two machines: by that time everything else is
finished anyway).
|
Thanks for the real information. I was hoping, from my old
understanding of the ports build out from a past discussion
with Satoshi Asami, that it was possible to shave a factorial
off of the build time because the build-out itself was reponsible
for signalling missing package dependencies.
This message implies that, even if you installed packages for
previous versions of all ports so that all build dependencies bould
be satisfied for all ports, without building anything but the port
itself (without installing it), it would not significantly reduce
the overall build time.
I knew that some of the build-out was serialized, but I was hoping
the average dependency depth wasn't as shallow as it appears to
be. 8-(.
If this is all that can be done, it look like quadrupling the
number of machines, and making the cluster generally available
to attempts builds on "-current + patches" (to get the test time
for ports breakages caused by a base OS change to under 5 hours),
is probably not a workable solution to the problem. I rather
expect that if the resources for this existed, they would already
be in use as part of the "ports cluster".
Short of going on an "interface usage classification" binge, I
don't really see how it would be possible to automate the process
of identifying ports that would be broken by a proposed change,
other than breaking them, and letting you do the build that finds
the breakage, and the notifying the ports maintainers that they
need to "take care of it" on a per-individual-port basis.
| Quote: | No, because the above is wrong ;-)
Please accept that I actually know what I'm talking about (since I've
been managing the ports cluster for about 8 months now) and just drop
the matter.
|
It's ancillary to the matter you raised, about people breaking
ports with commits to the base OS.
I don't believe in "bit rot", but if, as everyone is claiming, it's
impossible to reduce the problem, I think you are going to have to
just live with the consequences of base OS changes meaning broken
ports which don't show up until you do your builds. Sorry. 8-(.
-- Terry
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Terry Lambert
*nix forums Guru
Joined: 19 Mar 2002
Posts: 434
|
| Posted: Tue Jun 04, 2002 4:44 am Post subject:
Re: Removing wait union
|
|
|
John Baldwin wrote:
| Quote: | Each port should be built once.
|
[ ... ]
| Quote: | Erm, nope, dependencies are pkg_add'd and then the port is built (IIRC).
If a port doesn't specify a dependency that it needs then that is an error.
|
No blood in that stone, then. Looks like option #4 wins. 8-(.
-- Terry
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message |
|
| Back to top |
|
|
Google
|
|
| Back to top |
|
|
|
|
The time now is Thu Jan 08, 2009 6:08 am | All times are GMT
|
|
Sprint Ringtones | Debt Consolidation | Problem Mortgage | Business Sales | Credit Counseling
|
|
Copyright © 2004-2005 DeniX Solutions SRL
|
|
|
|
Other DeniX Solutions sites:
Unix/Linux blog |
electronics forum |
medicine forum |
science forum |
|
|
Privacy Policy
|
Powered by phpBB © 2001, 2005 phpBB Group
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
Forum index
»
*nix
»
BSD
»
FreeBSD
»
mail-lists
»
Architecture
