serving millions of files

Axel-Stéphane SMORGRAV · *nix forums Guru Joined: 07 Feb 2005 Posts: 419

SetOutputFilter works very well, even in Location filters. There are lots of happy customers of this feature, using it for such things as for example compression of response bodies.

Are you quite sure your filter is NOT invoked? Could you possibly have it generate some output to stderr?

I copied your filter declarations into an Apache 2.0.54 configuration and was unable to reproduce the behaviour you describe. That said, the response was cached by the browser and when reloading the page, it issued a conditional request to which the server responded with a 304. Therefore, at first it appeared as if the filter was not invoked, but after forcing an unconditional request, the page was correctly updated.

Here are the relevant parts of the configuration I used:

LoadModule ext_filter_module /opt/apache2/modules/mod_ext_filter.so
Listen labelle16:80

NameVirtualHost *:80
<VirtualHost *:80>
ServerName labelle16.gsi.fr
DocumentRoot /www/apachetest/htdocs/labelle16
CustomLog /www/apachetest/logs/access16_log combined
ErrorLog /www/apachetest/logs/error16_log
LogLevel debug

ExtFilterDefine a mode=output intype=text/html outtype=text/html cmd="/bin/sed -e s/foo/bar/g"
ExtFilterDefine b mode=output intype=text/html outtype=text/html cmd="/bin/sed -e s/bing/boom/g"

<Location /test>
Header set Cache-Control no-cache
SetOutputFilter a
</Location>
</VirtualHost>

-ascs

-----Original Message-----
From: Andy Buckley [mailto:andy.buckley@durham.ac.uk]
Sent: Thursday, June 29, 2006 3:10 PM
To: users@httpd.apache.org
Subject: [users@httpd] SetOutputFilter doesn't work in <Location>s?

Hi,

(Before I begin: I sent two messages on a similar topic to this a couple of weeks ago but got no reply: can someone please give me some feedback on whether I should report this issue as a bug or if it is my fault?)

I have a long-standing problem with getting Apache 2 output filters to work on a per-<Location> basis. As far as I can tell, the SetOutputFilter directive either doesn't work in <Location> blocks, or is quite unpredictable in that context. I sent a test configuration in a previous email but recieved no response, so I've produced a simpler configuration and tested it on a different system, with the same results. Here's the configuration fragment:

<VirtualHost 80.68.95.15:80>
ServerAdmin webmaster@insectnation.org
DocumentRoot /www/filtertest
ServerName test.insectnation.org

#SetOutputFilter a;b

<Location />
Options Indexes FollowSymLinks MultiViews ExecCGI
AllowOverride None
Order allow,deny
Allow from all
DirectoryIndex index
</Location>

ExtFilterDefine a mode=output intype=text/html outtype=text/html cmd="/bin/sed -e s/foo/bar/g"
ExtFilterDefine b mode=output intype=text/html outtype=text/html cmd="/bin/sed -e s/bing/boom/g"
<Location /a>
SetOutputFilter a
</Location>
<Location /b>
SetOutputFilter b
#Order deny,allow
#Deny from all
</Location>
</VirtualHost>

The results of using this configuration are as follows:

* The sed replacements take place neither in test.insectnation.org/a or test.insectnation.org/b
* If the Location /b access is denied (by uncommenting the Order and Deny statements), this is correctly reflected in the browser behaviour
* If the SetOutputFilter at VirtualHost level (i.e not in any Location blocks) is uncommented, the sed replacements both take place as expected

So, the problem is not the Location matching because the access denial works as expected: there seems to be some problem specifically with SetOutputFilter used on a Location basis. This is observed using Apache
2.0.55 as packaged in Ubuntu Dapper:

andy@dirac:~$ apache2 -v
Server version: Apache/2.0.55
Server built: May 29 2006 01:52:53

I've also observed the same behaviour in Scientific Linux 4.x. Can anyone offer any suggestions / try this config for themselves and comment. If it's a genuine problem then it should be reported as a bug, but I'm not aware of any independent confirmations. Some assistance, please?

Thanks!
Andy

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Axel-Stéphane SMORGRAV · *nix forums Guru Joined: 07 Feb 2005 Posts: 419

That piece of s**t has made me tear all of my hair out, but if you really have to use it, please make yourself a favor and use the latest and greatest (SMWA 5QMR8). I think I counted something like 160 bug fixes in that release, as compared to appx. 40 in other releases. Given all the problems I have had with previous releases of the so-called framework agent, I think they have had huge quality problems that have hopefully been solved after CA acquired Netegrity.

What I believe is your problem is that libmod_sm20 depends on another library, and since it appears that /opt/siteminder2/webagent/lib/ is not in your SHLIB_PATH, the run-time linker can't find it.

The definitions below relate to Solaris. You probably want to replace LD_LIBRARY_PATH with SHLIB_PATH

I added the following to the Apache environment. There are several ways of doing that. I personally prefer adding to the end of apache/bin/envvars:

NETE_WA_ROOT=/u01/netegrity/siteminder/webagent
NETE_WA_PATH=$NETE_WA_ROOT/lib
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/u01/opt/lib:$NETE_WA_PATH"
PATH=$NETE_WA_ROOT/bin:$PATH
export NETE_WA_PATH NETE_WA_ROOT PATH LD_LIBRARY_PATH

You absolutely need to set both LD_LIBRARY_PATH (SHLIB_PATH) and PATH in order for the LLAWP process to be able to start.

Good luck!
-ascs

-----Original Message-----
From: Lindsey_Lepisko@cargill.com [mailto:Lindsey_Lepisko@cargill.com]
Sent: Monday, July 03, 2006 7:06 PM
To: users@httpd.apache.org
Subject: [users@httpd] Configuring Siteminder 5.5 for apache2.0 on HP-UX 11- can't find libmod_sm20.sl

Hi,

I'm very new to the apache/unix world and am currently in charge of switching the web server my department's web pages run on to apache (currently they are running on a sun one server). My first goal is to get apache and siteminder working together nicely. I installed apache 2.0 and compiled it with mod.so as the only non-default module. The siteminder installation went well and it is registered as a trusted host with our policy servers. I followed the netegrity installation/configuration manual for configuring siteminder for apache and added the following to apache's httpd.conf file:

PassEnv SHLIB_PATH /opt/siteminder2/webagent/config
LoadModule sm_module "/opt/siteminder2/webagent/lib/libmod_sm20.sl"
SmInitFile "\opt\sitemindre2\webagent\config\WebAgent.conf"
(also tried \opt\web\apache2\conf\WebAgent.conf...the manual is very confusing as to what <web_agent_home> means. )

Alias /siteminderagent/ .<web_agent_home>/samples/.
<Directory "/export/webagent/samples/">
Options Indexes MultiViews
AllowOverride None
Order allow,deny
Allow from all
</Directory>

AddHandler smcookieprovider-handler .ccc

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Axel-Stéphane SMORGRAV · *nix forums Guru Joined: 07 Feb 2005 Posts: 419

SetOutputFilter works very well, even in Location filters. There are lots of happy customers of this feature, using it for such things as for example compression of response bodies.

Are you quite sure your filter is NOT invoked? Could you possibly have it generate some output to stderr?

I copied your filter declarations into an Apache 2.0.54 configuration and was unable to reproduce the behaviour you describe. That said, the response was cached by the browser and when reloading the page, it issued a conditional request to which the server responded with a 304. Therefore, at first it appeared as if the filter was not invoked, but after forcing an unconditional request, the page was correctly updated.

Here are the relevant parts of the configuration I used:

LoadModule ext_filter_module /opt/apache2/modules/mod_ext_filter.so
Listen labelle16:80

NameVirtualHost *:80
<VirtualHost *:80>
ServerName labelle16.gsi.fr
DocumentRoot /www/apachetest/htdocs/labelle16
CustomLog /www/apachetest/logs/access16_log combined
ErrorLog /www/apachetest/logs/error16_log
LogLevel debug

ExtFilterDefine a mode=output intype=text/html outtype=text/html cmd="/bin/sed -e s/foo/bar/g"
ExtFilterDefine b mode=output intype=text/html outtype=text/html cmd="/bin/sed -e s/bing/boom/g"

<Location /test>
Header set Cache-Control no-cache
SetOutputFilter a
</Location>
</VirtualHost>

-ascs

-----Original Message-----
From: Andy Buckley [mailto:andy.buckley@durham.ac.uk]
Sent: Thursday, June 29, 2006 3:10 PM
To: users@httpd.apache.org
Subject: [users@httpd] SetOutputFilter doesn't work in <Location>s?

Hi,

(Before I begin: I sent two messages on a similar topic to this a couple of weeks ago but got no reply: can someone please give me some feedback on whether I should report this issue as a bug or if it is my fault?)

I have a long-standing problem with getting Apache 2 output filters to work on a per-<Location> basis. As far as I can tell, the SetOutputFilter directive either doesn't work in <Location> blocks, or is quite unpredictable in that context. I sent a test configuration in a previous email but recieved no response, so I've produced a simpler configuration and tested it on a different system, with the same results. Here's the configuration fragment:

<VirtualHost 80.68.95.15:80>
ServerAdmin webmaster@insectnation.org
DocumentRoot /www/filtertest
ServerName test.insectnation.org

#SetOutputFilter a;b

<Location />
Options Indexes FollowSymLinks MultiViews ExecCGI
AllowOverride None
Order allow,deny
Allow from all
DirectoryIndex index
</Location>

ExtFilterDefine a mode=output intype=text/html outtype=text/html cmd="/bin/sed -e s/foo/bar/g"
ExtFilterDefine b mode=output intype=text/html outtype=text/html cmd="/bin/sed -e s/bing/boom/g"
<Location /a>
SetOutputFilter a
</Location>
<Location /b>
SetOutputFilter b
#Order deny,allow
#Deny from all
</Location>
</VirtualHost>

The results of using this configuration are as follows:

* The sed replacements take place neither in test.insectnation.org/a or test.insectnation.org/b
* If the Location /b access is denied (by uncommenting the Order and Deny statements), this is correctly reflected in the browser behaviour
* If the SetOutputFilter at VirtualHost level (i.e not in any Location blocks) is uncommented, the sed replacements both take place as expected

So, the problem is not the Location matching because the access denial works as expected: there seems to be some problem specifically with SetOutputFilter used on a Location basis. This is observed using Apache
2.0.55 as packaged in Ubuntu Dapper:

andy@dirac:~$ apache2 -v
Server version: Apache/2.0.55
Server built: May 29 2006 01:52:53

I've also observed the same behaviour in Scientific Linux 4.x. Can anyone offer any suggestions / try this config for themselves and comment. If it's a genuine problem then it should be reported as a bug, but I'm not aware of any independent confirmations. Some assistance, please?

Thanks!
Andy

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

<Lindsey_Lepisko@cargi · *nix forums beginner Joined: 03 Jul 2006 Posts: 4

Still isn't working. I put what you suggested in the envvars file and it didn't help. I even tried explicitly setting PATH to include /opt/siteminder2/webagent/bin and SHLIB_PATH to include /opt/siteminder2/webagent/lib by using export and even with that it didn't work. It still has a problem with the line
'LoadModule sm_module "/opt/siteminder2/webagent/lib/libmod_sm20.sl" ' and says it can't find libmod_sm20.sl.

So I really don't know what the heck to do now. I spent more time searching the web for linking issues and solutions to those issues, but nothing was very helpful. Any other ideas?

-Lindsey

-----Original Message-----
From: Axel-Stéphane SMORGRAV
[mailto:Axel-Stephane.SMORGRAV@europe.adp.com]
Sent: Tuesday, July 04, 2006 1:04 AM
To: users@httpd.apache.org
Subject: RE: [users@httpd] Configuring Siteminder 5.5 for apache2.0 on
HP-UX 11- can't find libmod_sm20.sl

That piece of s**t has made me tear all of my hair out, but if you really have to use it, please make yourself a favor and use the latest and greatest (SMWA 5QMR8). I think I counted something like 160 bug fixes in that release, as compared to appx. 40 in other releases. Given all the problems I have had with previous releases of the so-called framework agent, I think they have had huge quality problems that have hopefully been solved after CA acquired Netegrity.

What I believe is your problem is that libmod_sm20 depends on another library, and since it appears that /opt/siteminder2/webagent/lib/ is not in your SHLIB_PATH, the run-time linker can't find it.

The definitions below relate to Solaris. You probably want to replace LD_LIBRARY_PATH with SHLIB_PATH

I added the following to the Apache environment. There are several ways of doing that. I personally prefer adding to the end of apache/bin/envvars:

NETE_WA_ROOT=/u01/netegrity/siteminder/webagent
NETE_WA_PATH=$NETE_WA_ROOT/lib
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/u01/opt/lib:$NETE_WA_PATH"
PATH=$NETE_WA_ROOT/bin:$PATH
export NETE_WA_PATH NETE_WA_ROOT PATH LD_LIBRARY_PATH

You absolutely need to set both LD_LIBRARY_PATH (SHLIB_PATH) and PATH in order for the LLAWP process to be able to start.

Good luck!
-ascs

-----Original Message-----
From: Lindsey_Lepisko@cargill.com [mailto:Lindsey_Lepisko@cargill.com]
Sent: Monday, July 03, 2006 7:06 PM
To: users@httpd.apache.org
Subject: [users@httpd] Configuring Siteminder 5.5 for apache2.0 on HP-UX 11- can't find libmod_sm20.sl

Hi,

I'm very new to the apache/unix world and am currently in charge of switching the web server my department's web pages run on to apache (currently they are running on a sun one server). My first goal is to get apache and siteminder working together nicely. I installed apache 2.0 and compiled it with mod.so as the only non-default module. The siteminder installation went well and it is registered as a trusted host with our policy servers. I followed the netegrity installation/configuration manual for configuring siteminder for apache and added the following to apache's httpd.conf file:

PassEnv SHLIB_PATH /opt/siteminder2/webagent/config
LoadModule sm_module "/opt/siteminder2/webagent/lib/libmod_sm20.sl"
SmInitFile "\opt\sitemindre2\webagent\config\WebAgent.conf"
(also tried \opt\web\apache2\conf\WebAgent.conf...the manual is very confusing as to what <web_agent_home> means. )

Alias /siteminderagent/ .<web_agent_home>/samples/.
<Directory "/export/webagent/samples/">
Options Indexes MultiViews
AllowOverride None
Order allow,deny
Allow from all
</Directory>

AddHandler smcookieprovider-handler .ccc

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Axel-Stéphane SMORGRAV · *nix forums Guru Joined: 07 Feb 2005 Posts: 419

Although this subject does not seem to be related to Apache, and I have not been near HP-UX for years...

I had a look at the Siteminder WebAgent Installation guide. There is a section called "Enabling SHLIB Path for an Agent on Apache2/HP-UX 11" which states: For the Web Agent to operate to operate on an Apache 2.0 web server running HP-UX 11, be sure the SHLIB_PATH is enabled in the Apache executable.

You can check if SHLIB_PATH is enabled by executing "chatr httpd". If enabled, you should find a line in the output saying "SHLIB_PATH enabled second". If it says "SHLIB_PATH enabled second" instead, execute "chatr +s enable httpd"

-ascs

-----Original Message-----
From: Lindsey_Lepisko@cargill.com [mailto:Lindsey_Lepisko@cargill.com]
Sent: Thursday, July 06, 2006 4:31 PM
To: users@httpd.apache.org
Subject: RE: [users@httpd] Configuring Siteminder 5.5 for apache2.0 on HP-UX 11- can't find libmod_sm20.sl

Still isn't working. I put what you suggested in the envvars file and it didn't help. I even tried explicitly setting PATH to include /opt/siteminder2/webagent/bin and SHLIB_PATH to include /opt/siteminder2/webagent/lib by using export and even with that it didn't work. It still has a problem with the line 'LoadModule sm_module "/opt/siteminder2/webagent/lib/libmod_sm20.sl" ' and says it can't find libmod_sm20.sl.

So I really don't know what the heck to do now. I spent more time searching the web for linking issues and solutions to those issues, but nothing was very helpful. Any other ideas?

-Lindsey

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Joost de Heer · *nix forums Guru Joined: 30 Mar 2005 Posts: 339

Lindsey_Lepisko@cargill.com wrote:

Mididoc Productions · *nix forums beginner Joined: 30 May 2005 Posts: 16

Hello,

my name is mike roland.
i'm not a unix expert.

we have a vps server with for virtual host domains.
everything is working fine so far:
the webs are accessible the mails can be sent and received.

but we want to install a blacklist to refuse emails from certain
email-addresses for the whole server, say one list for the whole server.

so we proceeded like in the spamassassin manual and edited the file:
/usr/local/etc/mail/spamassassin/local.cf

we inserted a blacklist entry: blacklist_from example@domain

we have also rebooted the server, but it is not working.
emails from the address: example@domain are stil beeing received.

could anybody please help us setting the blacklist?

thank you

best regards

mike roland

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Joshua Slive · *nix forums Guru Joined: 07 Feb 2005 Posts: 1647

On 7/11/06, Mididoc Productions <mail@mididoc.com> wrote:

Pid · *nix forums beginner Joined: 19 Jun 2006 Posts: 32

you would probably want the spamassassin mailing list for that, rather
than the httpd list.

try here:
http://wiki.apache.org/spamassassin/MailingLists

Mididoc Productions wrote:

Mididoc Productions · *nix forums beginner Joined: 30 May 2005 Posts: 16

thank you will do.

best regards

mike roland

Boyle Owen · *nix forums Guru Joined: 07 Feb 2005 Posts: 618

Kim Leng Goh · *nix forums beginner Joined: 07 Jul 2006 Posts: 2

Hi Marx,
Thanks for the quick reply. I'm cc-ing to the apache user's list for
the benefit of others.

On 7/12/06, Apache Security Response Team <security@apache.org> wrote:

Mario Pavlov · *nix forums beginner Joined: 12 Jul 2006 Posts: 2

sniedermeyer@cob.org · *nix forums beginner Joined: 29 Jun 2006 Posts: 4

I'm still chipping away at getting SSL working for our reverse proxy server
(RPS). Currently, it looks like our RPS is going secure:

https://www.cob.org

However, we're getting the following entry in the Apache HTTPD error log:

RSA server certificate CommonName (CN) 'www.cob.org' does NOT match server
name!?

Here are the settings in my httpd.conf file:

ServerName www.cob.org

<VirtualHost www.cob.org:443>

SSLEngine on
SSLProxyEngine on
SSLCertificateFile ...somepath/public.crt
SSLCertificateKeyFile ...somepath/private.key
SSLCertificateChainFile ...somepath/intermediate.crt
ProxyPass /webapp1 https://x.x.x.x/webapp1
ProxyPassReverse /webapp1 https://x.x.x.x/webapp1

</VirtualHost>

Once I get this working correctly, I'll move on to trying to get the
connection from the RPS to the webapp1 server secure without a browser
complaining ;)

Any help would be appreciated.

Thanks.

____________________________
Steven Niedermeyer
Information Technology Services
City of Bellingham
625 Halleck St
Bellingham, WA 98225
Phone: (360) 676-6671 x156
Fax: (360) 676-7693

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Boyle Owen · *nix forums Guru Joined: 07 Feb 2005 Posts: 618

Google