Mod waklog

From Syscore

mod_waklog is an apache loadable module, originally from the University of Michigan, used for enhancing Apache's functionality in an AFS environment. The UMich mod_waklog had the following features:

• Allows the Apache web server to run with the tokens of a special web-server user, and periodically renew them.
• Use the AFS credentials of an accessing user, by harnessing a feature of CoSign which dumps a user's KRB5 credentials into the web server environment.

The UMBC mod_waklog has been enhanced and expanded to provide the following extra functionality:

• Allows sections of the web tree (in Location blocks) to be served out using alternate AFS/Kerberos identities.
• Integrates with UMBC's WebAuth (adding back in CoSign compatibility would be trivial)
• Maintains a shared "token cache" of recently used tokens for increased performance and security. (more on that later)

Devolutions from the umich mod_waklog:

• Multiple cell support isn't really there right now. Don't worry, it can easily be re-added -- I just don't have any particular need for it.
• It'll take a *little* bit of work to get it working correctly under Apache 2.x again. Our central web environment is still 1.3, so I've only done my testing/development work on 1.3 servers.

The motivation for using this enhanced mod_waklog at UMBC is primarily for segmentation of various web services running from one central web server instance, such as the www.umbc.edu / virthost.umbc.edu. In our current environment, all of the departmental web sites and virtual hosts are running with the same AFS/Kerberos credentials, causing a serious potential for mayhem. Our largest installation of this configuration is our primary campus web server cluster, which is now serving up a multitude of virtual hosts and web directories with over 300 different security principals.

Contents 1 Configuration Directives 1.1 WaklogDefaultPrincipal <principal> <keytab> 1.2 WaklogProtected <on/off> 1.3 WaklogPrincipal <principal> <keytab> 1.4 WaklogUseUserTokens <on/off> 2 What It Does 2.1 Module Initialization 2.2 Child Intialization 2.3 Apache Request Phases 2.3.1 Phase 0 2.3.2 Phase 1,3,6,7 2.3.3 Phase 9 3 How to (really) use it 4 Other Important Stuff 4.1 Web Service Principal Nomenclature 4.2 Required ACL Settings 4.3 Security Hints 4.3.1 Be Minimalistic With Your Permissions 4.3.2 Put Your Secrets in the Right Place 4.3.3 Writing to Files 5 Administrative Tasks 5.1 Creating A New Kerberos Principal 5.2 Configuring the Web Server 5.2.1 Assigning An Identity to a Portion of a Tree / VirtualHost 5.2.2 Assigning an Identity to an entire VirtualHost 5.3 Debugging Stuff 5.3.1 Tokens Not Renewed 5.3.2 It gets really slow and dies 5.3.3 The Source 5.3.3.1 20070316 5.3.3.2 20070314a 5.3.3.3 20070314

Configuration Directives

WaklogDefaultPrincipal <principal> <keytab>

This sets the Kerberos5 principal (and keytab to get the credentials from) that this server instance will run as when it hasn't been instructed to run as anything else. This should only be configured once per server.

WaklogProtected <on/off>

Enable, or disable, Waklog operations while serving this web area. Put this in the server configuration area or a <Location> block. If it's off, and you have a DefaultPrincipal set, the server will retain those credentials.

WaklogPrincipal <principal> <keytab>

Select an alternate principal (and related keytab) to use when serving out elements in this page. Use this in a server configuration area or <Location> block.

WaklogUseUserTokens <on/off>

This causes waklog to search the environment for a user principal name and appropriate "secret", and use that information to serve those resources. For this to work with UMBC's WebAuth, the WebAuth authentication module must be available, and the following configuration parameters set:

    WebAuthEnabled Yes
    WebAuthPassAttrs Yes
    AuthType WebAuth
    require valid-user (or other require statement)

In addition, the service used for this must be configured by the WebAuth service to provide user credentials.

What It Does

Module Initialization

• Creates a shared (mmapped) file for the storage of token data. As mod_waklog.so is probably running as a shared object, this persists through module reloads.
• Forks off a child to fetch and renew credentials from keytabs (this runs as root.).
• Sets the credentials of the parent process to those of the DefaultPrincipal (if set).

Child Intialization

• Create a new PAG for the child.
• Set the tokens in that PAG to the ones of the DefaultPrincipal (if set).

Apache Request Phases

Phase 0

Apache is running with it's default credentials for the server instance or virtual host.

Phase 1,3,6,7

If this area is configured to use UserTokens, then the user's credentials are extracted from the environment and obtained. Otherwise, we continue on in the state described in Phase 0. We try at all of these phases, because at each one of them more configuration information may become available -- we'd like to be running as the target user for as much of the request as possible.

Phase 9

We switch back to the default state.

How to (really) use it

Here's some actual mod_waklog configuration stuff from a real web server.

LoadModule waklog_module /afs/umbc.edu/admin/www/uber/007/@sys/lib/mod_waklog.so

WaklogDefaultPrincipal www/umbc /etc/umbc/umbc_www.keytab

<Directory /afs/umbc.edu/public/www/oit/iss/syscore >
        WaklogProtected on
        WaklogPrincipal www/syscore /etc/umbc/www_syscore.keytab
</Directory>

<Directory /afs/umbc.edu/pubilc/www/oit/iss/runasauser >
        PerlSetVar ServiceName testauth
        PerlSetVar ServerSecretKey /etc/testauth.key
        WebAuthEnabled Yes
        WebAuthPassAttrs Yes
        AuthType WebAuth
        require valid-user
        WaklogProtected on
        WaklogUseUserTokens on
</Directory>

As yon probably guess, most of the time this web server is running under www/umbc. Unless you go to /oit/iss/syscore, then anything under there is running under the www/syscore credentials. A simple cgi that prints the output of 'tokens' will confirm that. And (if you've been paying attention), visiting /waklog/usertest will cause the request to be handled with your "user" tokens, with the credentials provided by WebAuth.

For keytabs -- as shown in the example, you can have multiple principlals stored in each keytab. You don't have to do it this way. It's also recommended that the keytabs be root owned, and only readable by root. That way they can't be inadvertently snarfed. mod_waklog will be happy with this, because the only process that reads & refreshes these credentials is the worker child that's forked off during module initialization.

Other Important Stuff

Web Service Principal Nomenclature

When creating new service principals for use by web servers and such, please use the following semantics:

www/<name>	Web Server Instance -- www/umbc would be the tokens for the web server in /afs/umbc.edu/admin/www/umbc
web/<name>	Web Site -- Might go with the name of the particular public/www web area, for example, web/aok might mean what's under /afs/umbc.edu/public/www/aok. Kind of free-form -- nothing we can really do about that -- but try to have it make some sense.
webapp/<name>	Web Application -- A particular web-based application, usually centrally administered, that might have control over various web areas. For example, Movable Type might run under webapp/mt, and the appropriate blog publishing areas in the web tree would be configured so that webapp/mt can write to them.

Required ACL Settings

Any directory that's being served must have, at minimum, read (rl) permissions for the web server user -- either by system:anyuser, or the principal that's set as the default. If any directory contains an ".htaccess" (or other per-directory configuration file), it must also be, at minimum, readable (rl) from the web server user.

The target user (or something like system:authruser or system:anyuser) for a directory must also have lookup rights (l) on all directories subordinate to the pages being served, or else you'll see a lot of these.

[Thu Feb  1 12:45:36 2007] [error] [client 12.110.112.30] (13)Permission denied: access to /oit/iss/syscore/wiki/EMT_and_depot failed because search permissions are missing on a component of the path

Security Hints

(see also Core Web Environment#Security Realms & Authentication)

For these examples, we'll assume the web server user is "www/umbc" (aka www.umbc) and the secruity principal our pages are using is www/syscore (www.syscore). You should consult with your friendly OIT representative to determine what settings should be used for your particular web application and server that its running on.

Be Minimalistic With Your Permissions

• Remove system:anyuser, or system:authuser from all of your web directories if you don't really need it.

bash% find . -type d -exec fs sa {} system:anyuser none \;

• Add your web principal to those directories -- read only, of course.

bash% find . -type d -exec fs sa {} www.syscore rl \;

• Remember, the web server user needs to be able to read stuff too, but not for serving content... (long story)

bash% find . -type d -exec fs sa {} www.umbc rl \;

Put Your Secrets in the Right Place

If you have scripts (php, etc) which include important information -- such as logs, passwords to databases, etc -- do NOT put them in a directory that has read permissions of the web server user. If you do this, chances are someone will find a way to access it, and that's bad. Make sure those sorts of things are somewhere with only www.umbc l, or better no access by the web server at all. If it's a directory of php includes that aren't referenced directly by apache -- only by your application -- they only need to be www.syscore rl -- as your application will be running with your application's permissions.

For example, if you had to give permission to www.umbc to read a .htaccess file, you'll want to put your super-secret files in a sub-directory that has more restricted permissions. (and remember to have that .htaccess file deny read access to those files!!!)

Writing to Files

If you've installed a pre-packaged php application (like mediawiki) that's going to need to write to files, you'll need to make those directories writable by your application's user:

bash% find . -type d -exec fs sa {} www.syscore write \;

...also, many of these applications have instructions on further securing the directory structure after installation. You'll probably have to do some "translation" from UNIX-style permission semantics to AFS-ACL semantics, but the principles will be the same.

Administrative Tasks

Creating A New Kerberos Principal

To create a new kerberos principal for use in this environment, you use the 'kadmin' tool. You should do this in a sudo/root environment so that you can write to the destination keytab(s).

bash% /usr/k5/sbin/kadmin
Password for banz@UMBC.EDU: nedm!
Authenticating as principal banz/admin@UMBC.EDU with password.
Password for banz/admin@UMBC.EDU: 12345
kadmin: ank -randkey www/testservice
Principal "www/testservice@UMBC.EDU" created.
kadmin: ktadd -k /etc/umbc/www_testservice.keytab
Entry for principal www/testservice with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/umbc/www_testservice.keytab.
Entry for principal www/testservice with kvno 3, encryption type AES-256 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/umbc/www_testservice.keytab.
Entry for principal www/testservice with kvno 3, encryption type AES-128 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/umbc/www_testservice.keytab.
Entry for principal www/testservice with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/umbc/www_testservice.keytab.
Entry for principal www/testservice with kvno 3, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/umbc/www_testservice.keytab.
kadmin:  

Also, don't forget to create the PTS entry...

bash% pts create www.testservice

If the web server that you're using for this service exists on multiple servers, make sure you distribute this new keytab to all of the hosts. *WARNING:* Don't do that by doing the last step (ktadd) on the other servers, ktadd actually re-randomizes the key, so the previous extraction keytab won't be valid anymore.

Configuring the Web Server

Confgurations for locations to be served under this security identity must be made to the actual web server configuration files, and cannot be done in per-directory-configuration files, such as .htaccess files. This is because keytab data is processed only by the parent process, as root, and we also don't want folks being able to select other security identities that haven't been assigned to them.

Assigning An Identity to a Portion of a Tree / VirtualHost

In order to assign an alternate security identity to a portion of an existing web tree or virtual host, you must place it within a <Location> block as shown below in this example.

<Directory /afs/umbc.edu/public/www/newtestservice >
        WaklogProtected on
        WaklogPrincipal www/testservice /etc/umbc/www_testservice.keytab
</Directory> 

Assigning an Identity to an entire VirtualHost

To assign a security identity to the entirety of a virtual host, place the appropriate directives directly within the particular VirtualHost definition, such as:

<VirtualHost *:80>
ServerName www.testservice.umbc.edu
ServerAlias testservice.umbc.edu
DocumentRoot /afs/umbc.edu/public/www/testservice
ErrorLog /local/umbc/vhost/testservice_umbc_edu.error_log
CustomLog /local/umbc/vhost/testservice_umbc_edu.access_log common
WaklogProtected on
WaklogPrincipal www/testservice /etc/umbc/www_testservice.keytab
</VirtualHost>

For new vhosts, you probably want to set the default permissions on that particular DocumentRoot to something appropriate as well, such as:

fs sa /afs/umbc.edu/public/www/testservice system:anyuser none www.umbc rl www.testservice read

For existing hosts, you'll probably have to work closely with the content manager for that particular site to configure their ACLs as discussed in the previous sections of this document.

Debugging Stuff

mod_waklog is entirely foolproof, and nothing ever will ever possibly ever go wrong.

However if something does break, here's what to look for.

Tokens Not Renewed

You'll see a lot of permission denied errors. This really shouldn't happen, the backround ticket-renewer thing. A kill -HUP/USR1 will cause a reload of the server config, and a forced-renew of all of the principals.

It gets really slow and dies

Well, what it's actually doing is really hurting your fileserver. Quick lesson:

Every time an AFS client switches credentials within an authentication context, it creates a new "client connection" instance on the fileservers it talks to -- and it doesn't do a good job at cleaning them up. In a typical request served by mod_waklog, authentication credentials switch twice. That's twice for every web request. That "client connection" list on the fileserver gets large rather fast; and the fileserver doesn't age out old ones until they've been idle for two hours(!).

There is only one way I've found to combat this problem -- decrease that timeout in the fileserver. This isn't a configurable option at runtime, though it should be one -- however it's rather easy to modify in src/viced/host.c

/*
     * Send a probe to the workstation if it hasn't been heard from in
     * 15 minutes
     */
    checktime = now - 15 * 60;
    clientdeletetime = now - 120 * 60;  /* 2 hours ago */

I cranked that clientdeletetime down to 15 minutes and it cleared up my problems...

The Source

The following is offered with no warranty, expressed or implied. You should look at the source and know what this thing is doing before you use it. It could get you p0wn3d. It might make your wife leave you. It could cause a rash. Do not take internally. It contains chemicals known to the State of California that cause cancer -- lots of it. Just because we're crazy enough to run it, doesn't mean you are.

20070316

Fixed a couple minor things:

• pthread fixes. (they might actually work now. I even tested them.)
• fixed a call to pr_Initialize that was still looking at "/usr/vice/etc"
• removed a #define AFS_CELL "umbc.edu" from the .c
• if your default cell has a service principal of afs/<cellname>@REALM, then uncomment out the appropriate line in the Makefile

umbc_mod_waklog-20070316.tar.bz2

20070314a

Fixed a minor bug that affected index.whatever files (internal redirects)

umbc_mod_waklog-20070314a.tar.bz2

20070314

Recent Changes:

• Integrated some changes by Adam Megacz from Berkeley which added pthready portability to my Solaris Threads specific implementation
• Fixed the Makefile and stuff so it's more fun to build
• Switched from an anonymous memory chunk to an mmap'd file for the shared token storage -- makes restarts faster.
• Apache doesn't listen on it's sockets until all of the tokens have been stashed
• Fixed a problem where pathinfo'd URLS (such as blahblah/thing.cgi/blahblah) weren't getting executed with appropriate tokens, because the sub requests weren't seeing the module configuration data.

umbc_mod_waklog-20070314.tar.bz2