niXforums Forum Index
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   PreferencesPreferences   Log in to check your private messagesLog in to check your private messages   Log inLog in 
·  nixdoc.net ·  man pages ·  Linux HOWTOs ·  FreeBSD Tips ·  Forums
navigation Forum index » Programming » shell
Multiple consumers scenario
Post new topic   Reply to topic Page 1 of 1 [8 Posts] View previous topic :: View next topic
Author Message
mrs
*nix forums beginner


Joined: 20 Jul 2006
Posts: 3

PostPosted: Thu Jul 20, 2006 7:37 am    Post subject: Multiple consumers scenario Reply with quote
Hello

I'm currently encountering difficulties to write a mutliple consumers
bash snippet.

The problem is the following : I want multiple scripts that consumes a
file line by line . Each line of the file is consumed only once. Each
consumer is autonomous and has no idea of the other consumers'
existence. Each consumer consumes only one line at a time. The consumer
is persistent between two consumptions until there is no more line to
consume.

Does someone know how to achieve this ? I tried using fifo but I had
very poor results.

Thanks for your kind help
regard,
mrs
Back to top
Chris F.A. Johnson
*nix forums Guru


Joined: 20 Feb 2005
Posts: 2268

PostPosted: Thu Jul 20, 2006 7:59 am    Post subject: Re: Multiple consumers scenario Reply with quote
On 2006-07-20, mrs wrote:
Quote:
Hello

I'm currently encountering difficulties to write a mutliple consumers
bash snippet.

The problem is the following : I want multiple scripts that consumes a
file line by line . Each line of the file is consumed only once. Each
consumer is autonomous and has no idea of the other consumers'
existence. Each consumer consumes only one line at a time. The consumer
is persistent between two consumptions until there is no more line to
consume.

Does someone know how to achieve this ? I tried using fifo but I had
very poor results.

Perhaps is you post what you tried, it will give us an idea of
what you are trying to do.

--
Chris F.A. Johnson, author <http://cfaj.freeshell.org>
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
===== My code in this post, if any, assumes the POSIX locale
===== and is released under the GNU General Public Licence
Back to top
Krimgelas
*nix forums beginner


Joined: 22 Jun 2005
Posts: 35

PostPosted: Thu Jul 20, 2006 8:46 am    Post subject: Re: Multiple consumers scenario Reply with quote
mrs wrote:

Quote:
Hello

I'm currently encountering difficulties to write a mutliple consumers
bash snippet.

The problem is the following : I want multiple scripts that consumes a
file line by line . Each line of the file is consumed only once. Each
consumer is autonomous and has no idea of the other consumers'
existence. Each consumer consumes only one line at a time. The consumer
is persistent between two consumptions until there is no more line to
consume.

Does someone know how to achieve this ? I tried using fifo but I had
very poor results.

Thanks for your kind help
regard,
mrs

Sounds like a homework assignment... Smile
Back to top
aryzhov@spasu.net
*nix forums addict


Joined: 17 Feb 2005
Posts: 64

PostPosted: Thu Jul 20, 2006 8:47 am    Post subject: Re: Multiple consumers scenario Reply with quote
I'd definitely prefere to implement this i C.
In shell,I see no option but using either a flag (lock) file,
or a master (frontend) process.

1. Each client checks the existance of lock file,
creates lock file,
takes a line
edits the file (removes a line), with "ed", for instance
removes lock

2. Each client creates 2 pipe files with unique names;
master process is aware of those names, for instance,
all pipe files may live in /tmp/pipe_in/$consumer-pid
and /tmp/pipe_out/$consumer-pid
As soon as consumer wants a line, he puts a unique
sequence number (call it "ticket") into his pipe-in
and goes on reading his pipe-out.

Master process reads a line, edits the file,
(or just reads the master pipe), and pools the pipe-ins
for "tickets". Feeds the line to the pipe_out of the clonsumer
whose
ticket has been just taken. Next line. Next consumer. Not ready?
Next.

Depending on how fast the consumers are, you may wish to awake the
master process with signals, rather than looping it in dumb pooling.

I doubt that master written in shell, can distinguish who has sent
the signal, otherwise picking the correct pipe to feed next line to,
would be much easier, and you wouldn't need the "ticket" pipes at all.

Regards,
Andrei
Back to top
mrs
*nix forums beginner


Joined: 20 Jul 2006
Posts: 3

PostPosted: Thu Jul 20, 2006 9:30 am    Post subject: Re: Multiple consumers scenario Reply with quote
The very basic (and not functional) code would be this :
http://pastebin.ca/93246

I'll try to implement the lock/edit/unlock case which look interesting
(Thanks ANdrei)

Regards
mrs
Back to top
Chris F.A. Johnson
*nix forums Guru


Joined: 20 Feb 2005
Posts: 2268

PostPosted: Thu Jul 20, 2006 9:36 am    Post subject: Re: Multiple consumers scenario Reply with quote
On 2006-07-20, mrs wrote:
Quote:
The very basic (and not functional) code would be this :
http://pastebin.ca/93246

Code to do what? Please include context in your follow-ups.

Post the code here unless it is extremely long.

Quote:
I'll try to implement the lock/edit/unlock case which look interesting
(Thanks ANdrei)


--
Chris F.A. Johnson, author <http://cfaj.freeshell.org>
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
===== My code in this post, if any, assumes the POSIX locale
===== and is released under the GNU General Public Licence
Back to top
mrs
*nix forums beginner


Joined: 20 Jul 2006
Posts: 3

PostPosted: Thu Jul 20, 2006 9:57 am    Post subject: Re: Multiple consumers scenario Reply with quote
Sorry Chris - I thought that replying to your post was enough for you
to understand the context.

on its first instance, the script is supposed to create a producer that
feeds a fifo. Then it recursively call itself to process one line at a
time until the list is finished.

Quote:
-------------------------------------------------------------------------------------------------
#!/bin/bash


max_session=5

my_wd=$(pwd)

if [ -z $1 ] ; then
my_session=1
else
my_session=$(( $1 + 1 ))
fi

create_producer(){
if [ ! -p $my_wd/iofifo ] ; then
mkfifo $my_wd/iofifo
fi
while read line ; do echo $line > $my_wd/iofifo ; done < liste &
}


init_session(){
# session initialisation w/ the processing server
echo "$my_session : init_session"
}

process_line(){
# arg : $1 : line to process
echo "my session processing : $1"
}

close_session(){
echo "[7m$my_session : close_session[0m"
exit
}

fct_cleanup(){
if [ $my_session -ne 1 ]; then
kill -s SIGUSR1 $PPID
fi
close_session
}
trap "fct_cleanup" SIGHUP SIGINT SIGQUIT SIGILL SIGABRT SIGFPE SIGUSR1
SIGUSR2 SIGPIPE SIGTERM


read_iofifo(){
if [ -f $my_pwd/all_lines_processed ] ; then
kill -s SIGUSR1 $PPID
fi
read line < iofifo
if [ $line == 'Finish' ] ; then # Finish is the last file's line to
process
kill -s SIGUSR1 $$
else
process_line $line
fi
}

if [ $my_session == 1 ] ; then
create_producer
fi

echo "Hello - my pid is $$"
echo "my parent is $PPID"

if [[ $my_session -lt $max_session ]] ; then
$0 $my_session $my_duty_max $_duty_marker
fi

init_session
while read myline ; do
process_line $myline
done < <(read_iofifo)
Quote:
-------------------------------------------------------------------------------------------------

Thanks for your help

Chris F.A. Johnson wrote:
Quote:
On 2006-07-20, mrs wrote:
The very basic (and not functional) code would be this :
http://pastebin.ca/93246

Code to do what? Please include context in your follow-ups.
Back to top
Chris F.A. Johnson
*nix forums Guru


Joined: 20 Feb 2005
Posts: 2268

PostPosted: Thu Jul 20, 2006 7:57 pm    Post subject: Re: Multiple consumers scenario Reply with quote
On 2006-07-20, mrs wrote:
Quote:
Sorry Chris - I thought that replying to your post was enough for you
to understand the context.

Please remember that you are posting to a Usenet newsgroup, not
Google Groups. There is no guarantee that the previous post is
still (or ever was) visible. If it was a reader may have read
hundreds of other posts since, and there is a very good chance
that the details will have been forgotten. Reading the previous
post[s] may or may not be a simple operation. It should be
unnecessary.

Also, don't top post. Put your reply after the appropriate text.

When posting from GG, please make sure your lines of code are
short enough that they don't wrap. That makes the code invalid. It
may be easy to fix (it is in this case), but often it isn't, and
most people will not take the time to decipher it,

Quote:
on its first instance, the script is supposed to create a producer that
feeds a fifo. Then it recursively call itself to process one line at a
time until the list is finished.

What problems are you having?

What is the purpose of the script?
What is it supposed to do?
What input does it expect?
What output do you want?

Quote:
-------------------------------------------------------------------------------------------------
#!/bin/bash

max_session=5

my_wd=$(pwd)

if [ -z $1 ] ; then
my_session=1
else
my_session=$(( $1 + 1 ))
fi

create_producer(){
if [ ! -p $my_wd/iofifo ] ; then
mkfifo $my_wd/iofifo
fi
while read line ; do echo $line > $my_wd/iofifo ; done < liste &
}


init_session(){
# session initialisation w/ the processing server
echo "$my_session : init_session"
}

process_line(){
# arg : $1 : line to process
echo "my session processing : $1"
}

close_session(){
echo "[7m$my_session : close_session[0m"
exit
}

fct_cleanup(){
if [ $my_session -ne 1 ]; then
kill -s SIGUSR1 $PPID
fi
close_session
}
trap "fct_cleanup" SIGHUP SIGINT SIGQUIT SIGILL SIGABRT SIGFPE SIGUSR1
SIGUSR2 SIGPIPE SIGTERM


read_iofifo(){
if [ -f $my_pwd/all_lines_processed ] ; then
kill -s SIGUSR1 $PPID
fi
read line < iofifo
if [ $line == 'Finish' ] ; then # Finish is the last file's line to
process
kill -s SIGUSR1 $$
else
process_line $line
fi
}

if [ $my_session == 1 ] ; then
create_producer
fi

echo "Hello - my pid is $$"
echo "my parent is $PPID"

if [[ $my_session -lt $max_session ]] ; then
$0 $my_session $my_duty_max $_duty_marker
fi

init_session
while read myline ; do
process_line $myline
done < <(read_iofifo)
-------------------------------------------------------------------------------------------------

Thanks for your help

Chris F.A. Johnson wrote:
On 2006-07-20, mrs wrote:
The very basic (and not functional) code would be this :
http://pastebin.ca/93246

Code to do what? Please include context in your follow-ups.



--
Chris F.A. Johnson, author <http://cfaj.freeshell.org>
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
===== My code in this post, if any, assumes the POSIX locale
===== and is released under the GNU General Public Licence
Back to top
Google
Back to top
Display posts from previous:   
Post new topic   Reply to topic Page 1 of 1 [8 Posts] View previous topic :: View next topic
The time now is Mon Dec 01, 2008 7:18 pm | All times are GMT
navigation Forum index » Programming » shell
Jump to:  

Similar Topics
Topic Author Forum Replies Last Post
No new posts Postfix Multiple Relay Hosts? blaze Postfix 0 Wed Jun 25, 2008 5:02 am
No new posts Deliver msg to multiple hosts jures Postfix 2 Fri Feb 29, 2008 4:21 pm
No new posts Multiple Inheritance colint C++ 5 Thu Jul 20, 2006 2:29 pm
No new posts Multiple inheritance junw2000@gmail.com C++ 3 Thu Jul 20, 2006 3:52 am
No new posts multiple InnoDB read-only server instances on same file s... pradhuman jhala MySQL 1 Tue Jul 18, 2006 11:48 pm

Loans | Credit Cards | Auto Loans | Car Credit | Daddy Yankee
Copyright © 2004-2005 DeniX Solutions SRL
 
Other DeniX Solutions sites: Unix/Linux blog |  electronics forum |  medicine forum |  science forum | 
Privacy Policy


Powered by phpBB © 2001, 2005 phpBB Group

[ Time: 0.2678s ][ Queries: 16 (0.1656s) ][ GZIP on - Debug on ]