Source Forge Release 2.5 Installation Addendum

Author: Yves Bardout
Date created : June 2001
Last modified : July 31st 2001

Introduction

PURPOSE

The goal is to collect the most useful bits of information on installing and putting to work SF and to update them for SF2.5, since the real nice instructions are out of date, or have a different starting point. The doc is not self-contained and is an addendum to the documents given in next section.

Seems like sisyphe since 2.6 will also mean a lot of change there. Also, I will present the procedure for Mandrake 7.2. You will need to adapt it for your distribution.

Unfortunately, I have not started with the installation in sf-genericinst, and I thought it might not be so easy, and in some case, an overkill. To be followed: I intend to review the procedure to use these scripts when all service are installed as standard by packages. Indeed, the procedure described here is chaotic, and I advise to use sf-genericinst if you can be satisfied by the basic assumptions. There is some valuable experience, but I am not yet able to point out which pieces.

CHOICE

Note that a number of candidate to SF installation whether stick to SF2.0 with mysql, or want the SF 2.5 features, and say that the 2.5 distribution was quite broken, so they get the CVS versions: a beta 2.6, or the last editions.

SF2.6 has a longer grocery list than 2.5, that is already quite long. To start with 2.5, I had to learn Postgres and SSL, which I wanted to use anyway.

Give the few time I have, I definitly want a stable release not to get lost, Also I want to adapt the code, so that starting with a moving target will force me to merge all the time, and have some real difficulties, which I prefer to consider only once I have validaet that this platform give me an adge in my development, versus other platform that provides some services, or doing it myself from good high level languages.

I expect to find a stable release good enough for a prototype use, so I can evaluate cost and benefit of the solution.

SUMMARY

These notes follows the experience of a first install of Source Forge with the 2.5 release, on top of an existing Mandrake 7.2 distribution of linux, with an existing web server, DNS server and so on.

The specifics are in SFmandrake.html, in the package selection and install, the comparison of using a packaged installation versus a source tar, and some problem I had that are specific to this platform.

This document list the information that should be generic with regard to the platform, but very likely to be dependant on this release of Source Forge.

Getting the platform installed is long, but not really problematic. A first real problem for me was to load the database, as documented below. The provided load of the database is partial and has errors. I found better sql files from several persons.

My main problem was that the backend in SF2.5 release is partly broken, and that usable information is difficult to get about the cronjobs to setup and the configuration in 2 different include.pl, and in several sources.

I looked at a patch without any documentation by SF legato that had some useful code.

Another issue is getting the service well configured and intergated: CVS, mailman, and so on...

 My current problem right now, is that :

  1. I didn't find any design document that would be a base for further evolution of this code, to add a feature or some table and fields. I don't know also how to get the user docs that were in SF database.
  2. following the information on the backend from [SCHULTZE] or [MORIN] on the release and platform I have.
Here is the information I gathered. I hope to have this used in the sf-genericinst project.

STATUS

I am currently able to run a majority of SF features using 2.5. I have access to web pages, using the domain defined, PHP scripting works, the database, I can create a project, a user, using SSL optionnaly, receive mail, log and so on.

I can access the project, create subproject, tasks, news items, post in forums, create release, add files in them.

There is still a lot to install, setup and configure. I am working on the backend, having a lot of changes in databasedump.pl, and then I still need to setup cvs, proftp...

 Comparing to Frank work in sf-genericinst, clearly I have still to work on the tools used and the database content. I shall try a 2.6 and these installation scripts on another Mandrake 7.2 host, and check result, to avoid messing up my poor result.

How much of the documentation is to be done manually, given the software that automates stuff ? The docs are not versionned!

ADAPTATION

The configuration for your system is done in local.inc and include.pl (both in backend and utils). Caution, there are some other place in the code, where there is some hardcoded path, that you'll need to adapt. A number of patch, detailed in the mandrake installation enhances the genericity of code, in a simpler way than sfgeneric-inst. The minimal adaptation is for the use of the domain name all over the pages and scripts. Here you will need more that scripts and instructions, but I wrote in a separate section SFadaptation.html, a check list, to follow from the out of the box source Forge to your own collaborative service.

A/ DOCUMENTATION

This is a review of existing documentation material. Unfortunately, this is not easy to find them at first.

1/ General Information to start

Source Forge USER DOC from the site project:

(most important ones):

DESIGN DOC:

2/ Installation Information

The first point to start is obviously the docs directory in the tar file. It has more or less usefull bits: The README links to the SF Offsite Talk From which there is all the rest of the information.

 Indeed, there are FOUR different efforts to make SourceForge easy to install: the guide from Franck, Guillaume morin's, the sfportable project (which bigdisk created, with no activity), and lo-lan-do's Debian package.

The two main source for me are :

As said by all people who did it, it is tough, but doable to varying degrees of success.

3/ Other resources

In this post are some links to other install information:

5/ NEEDED INFORMATION :

This is a check list of resources needed for me that I have not yet found.

B/ INSTALLATION NOTES

In the following:
mydomain = the name of domain you run your webserver
= the directory where you put the tar from SF. I kept it unmodified for reference.
$SF = the location of the root directory of your running code

 Mandrake 7.2 Notes are in SFmandrake.html
 
 

1/ Basic Start-Up

From Tim Purdue in forum Make your apache htdocs folder point to: $SF/www.
Set up your PHP include path:
php_value include_path $SF:$SF/include:.
mv alexandria/etc/local.inc to /etc/local.inc and edit all variables (database access, hostname, etc). Make sure /etc/local.inc is readable by your web server.
make an /sfcache/ directory (which is noted in the /etc/local.inc file) and make it owned/writable by the webserver.
create your sourceforge database and a database super-user that has full permissions to the database ("createdb sourceforge" "createuser www" and answer yes to all questions to create a super user). Modify /etc/local.inc to make connections to the db possible.
Set up the database structure: (see below for more)
Set up cronjobs using the crontab in $SF/cronjobs/README
(Some of those cronjobs contain SQL at the top of them - NO - you need to copy/paste that SQL into your database before they can be run the first time)
(see below for more on backend)
(Tim suggested to make a base document out of it, adding what is missing: A lot!)

2/ Configuration

PRINCIPLE OF FILE STRUCTURE :

This is both a question of ease of installation/update and quite important for security (have just the needed authorization), for disk space management in production (disk space, backup, memory image or other performance tuning).

 Source Forge has a lot of directories of different use :

Temporarily, we use :
ln -s $SF  /mydomain/
cd  /mydomain
ln -s /var/www/mydomain www
Path configuration You have to substitute paths to /root/bin/alexandria/ and such. (not like doc in sf-genericinst p40, kevin koch offsite forum ). This is one important task that sf-genericinst is designed to handle. But it is not long to do it manually, once you find the different locations to update:
cd $SF; grep -rH alexandria .
These definitions are reflected in /etc/local.inc, backend/include.pl utils/include.pl and some path are hardcoded in:
./utils/download/stats_logparse.sh
./utils/tmpfilemove (*)  see below
./backend/shell/apache.sh  
./utils/fileforge.c (*)
./www/userlist.php (just remove the word  alexandria in the text)

(*) tmpfilemove is a binary. fileforge.c is a small utility shall be built (as written in file release, below). The binary is likely to be an intermediate file to rm.

The shell files could use an ENV variable, e.g. set up for the cron user.
 
 

3/ Front End

Web server

Firewall

I have installed SourceForge in a host connected to the internet (and a local network). It runs a firewall. Here is how to configure a firewall to allow public access to SourceForge, through the firewall (see /etc/services).
service port
ftp  20 21 
dns (domain names) 53 
smtp (mail) 25 
http (web)  80 
https (ssl) 443
ssh 22 
cvs  2401 
ftp maybe optional, if uploading via https. Be wary to block telnet 23, mysql 1490, postgres 5432

Port used

If you ever need to change the port used to server SF, to be different from 80, this is not supported in the delivery. Two possible reasons: (1) You want to run separate web servers (two different apache release, or apache and tomcat) on the same host. In this case, you probably will proxy request to a given domain by one web server at 80 to another at another port. (2) For server development, you run behind a firewall that does not allow incoming packets on port 80. You may adapt the code. There are several places to modify:
  1. httpd.conf: Listen <port>
  2. Vhost.conf:
  3. /etc/local.inc:
  4. $sys_default_domain = <your domain>:<port>
    $sys_secure_domain =  <your domain>
  5. account/login/php: use $sys_secure_domain instead of $HTTP_HOST
  6.                 if ($stay_in_ssl) {
                            $ssl_='https://$sys_secure_domains';
                    } else {
                            $ssl_='http://$sys_default_domain';
                    }
                    if ($return_to) {
                            header ("Location: ".$ssl_ . $return_to);
                            exit;
                    } else {
                            header ("Location: ".$ssl_ ."/my/");
                            exit;
                    }
  7. database: update groups set homepage = homepage || '::<port>/';
  8. ./project/admin/editgroupinfo.php: $form_homepage='http://' .$sys_default_domain;
  9. :
  10. ./export/index.php: use $sys_default_domain
  11. ./project/admin/index.php: use $sys_default_domain
  12. ./tos/privacy.php: use $sys_default_domain
  13. ./tos/tos_text.php: use $sys_default_domain
There should be other ways to manage what you want, as Virtual Hosts,

Web Pages

There are some corrections needed in one or two PHP pages.

How do we get the pages: /project/myproject and /users/myuser ? They are not found. This is because "project" and "users" are php scripts without a .php extension, which is misleading to the webserver. you have to add an entry in your .htaccess file to treat "projects" as a php file, since it doesn't have the .php extension. here is the entry in my .htaccess file:

    <Files projects> 
    ForceType application/x-httpd-php 
    </Files>
(I repeated this for "foundry" and "users" as well) from forum .

 I could not get projects/myproject or users/myname to behave correctly. First I added the .htaccess that is missing to ForceType these files as php, the apache server did not take this in account.

 If a create a link named projects.php or users.php and use it in url, it worked. Then I modified the httpd config in Vhosts.conf to setup the directive AllowOveride .

images

The images on the project and personal 'homepages' wont load correctly. This has to do with the Apache setup. I have in /etc/local: '$sys_images_url = ""' The images for the main page are shown ok, but when I go to a project the icons for the 'Public Areas' don't show. The url for the images is 'http://<mydomain</projects/test1/images/ic/home16b.png'. As you can see the icons are references through the /projects fake directory, and they are directly under /images.

When i did this '$sys_images_url = "/";' in etc/local.inc, the icons showed perfectly, but all other images did not show. The url for the other images became 'http://themes/forged/images/tleft1.png' (or https in secured mode)

 Two solutions were given to this question (discussed several times) in offsite forum:

  1. put a absolute url (including the $sys_Default_domain)
  2. change all calls to html_image to include a "/"
None of these are good for me. The solution I recommend is to add this line in html.php. That assume no image path will be used as relative. If ever that was wrong, it will need some other trick.
        if ($src[0] != '/') { $src = "/" . $src; }

4/ Database setup

DATABASE CREATION

Creation of the database is documented in Local Admin Guide from source forge project doc area: -- url --

Note that if you had postgres already installed, you need to move data out of the way to have postgres created a new one. Follows instruction for data migration if you have databases to keep.

The setup of the database is not obvious, first because the installation guide bundled is for 1.04, and it is not obvious at first not to use mysql, but postgres.

The install notes 2.5 do not cover every thing. Since the *.sql provided are not complete and the fake.sql (on the doc page of alexandra) that allows to start for test is obvioulsy for before SF2.5. I partly modified the script, for the problems I could find.

Security

To enhance somehow the security, there is two possible variations on the SF instalaltion that I used, that are discussed here:
  1. I kept the defaut setup from the php package, that allows only socket connection (from a local host). The PHP server does NOT listen to a port, making ot harder to access the database without a backdoor on your box.
  2. I created the database as user postgres, as detailled in the next section. This is not a very good idea.
Note that I do not have enough experience to evaluate the effects of such changes.

Owner

The first question is who own the database. I decided to create it as postgres user, not apache. That way, I can restrict the possible actions allowed from the web, and keep administrative right separately. I have not yet evaluated which table I don't want to be editable via the web server. Maybe this is neither possible nor usefull, given the security on the web pages and PHP.
As a start, I gave ALL the right to the user apache that is really using it. To do this, I din't find the way to do a GRANT ALL. There fore I created a grant.sql with one command for each table, pasting the output of a \dt from a psql client to an editor and doing a global replace.
grant all on  activity_log                  to apache;
It is important to understand that both the web frontend and the backend use the database. The cronjobs use the same php classes as the web frontend, and call database.php, therefore all runs on the database as user apache. An undesirable consequence, is that the vaccum command, restricted to the owner of the database, cannot be run from the apache user. The solution I found was to distinguish in this class the caller type and used the postgres user.

users

First you shall create users with: createuser apache

If you want to setup specific permissions mode, you need to edit the pg_hba.conf.

# This default configuration allows any local user to connect as any
# PostgreSQL username, over either UNIX domain sockets or IP:
 
local        all                                           trust
host         all         127.0.0.1     255.255.255.255     trust
This was fine for me. If you define user so that it need a password to connect on localhost. You shall set the password with pg_passwd.

Connection Mode

I couldn't connect to the db with the default code, and default config. This is a config of postgres. connecting to localhost explicitely means an IP connection, unlike not specifying an host, that use the socket.
psql -h 127.0.0.1  mydomain
psql: PQconnectPoll() -- connect() failed: Connexion refusée
        Is the postmaster running (with -i) at '127.0.0.1'
        and accepting connections on TCP/IP port 5432?  
psql    mydomain
Welcome to psql, the PostgreSQL interactive terminal.
If you run apache and postgres on the same box, you better not enable IP connections. But to use source forge code as is, you can start postmaster with -i option:
postmaster -i -d 2 -D /var/lib/pgsql/data

-i     Allows  clients  to  connect  via  TCP/IP 
-d n   For debug level n
-D     For db path
My solution was more restrictive, and maybe less risk, to update the scripts include.pl and databasdatabase.php that comes with sourceforge. (Changed it to remove the host parameter.
 
 

Loading

This is the list of import files as provided in the SF2.5 release and the files I selected and edited to load the database
SF2.5/db                        mydomain notes
---------------------------------------------------------------------------
DefaultValues.sql               =               subst email. Adapt values.
SQL_diff-2.0-to-2.5.sql                         - (migration only)
SourceForge.sql                 =               errors corrected
                                fake.sql        (adapted from local install doc on site, see errors in fake.lst)
pgport/
trove_defaults.sql              trove.sql       adapt map for needs
user-to-users.sh                -               (migration only)        
user_rating.sql                 -               a lot of parser errors !!
  .#language.sql.1.9            language.sql    adapt for 2.5 and limit language set
                                 correct.sql     
                                 grant.sql      grant perm to user apache, if another user own tables
                                 more.sql       based on Morin's file (but only a part is still needed)
                                 sfdocs.sql     from sf-genericinst
                                 domain.sql     (specifics, for adaptation, as discussed in this memo).
(I checked also filldb.sh list from Morin on SF2.0) It had a filerelease-changes.sql on which I am not sure.

 The file init-extra.sql (from lo-lan-do) is quite equivalent to the sum of language.sql, DefaultValues.sql and more.sql.

 I loaded the following modified files :

# psql mydomain postgres 
\i SourceForge.sql      
\i DefaultValues.sql
\i fake.sql 
\i correct.sql
\i trove.sql    
\i langage.sql 
\i sfdocs.sql
The changes are reviewed below : NOTE: s/^#/--/g on the .sql files done as a dump from mysql, because postgresql doesnot accept the # for comment. remove the connect lines from the. you can import conencted as the apache user, or postgres user, if you give all rights to apache. See also the ADAPTATION. This is where you add your own default values, and extensions to the source forge features.

5/ Administration

To make an admin user: create a user admin, password admin1, then you need to actually give that admin user the proper permissions in the database itself, using the postgresql client to get him an admin flag of A; there is no SF interface to create the fully set up administrator account.
 
psql 
update admin flag = 'A' where user_name = 'admin';
use the information in this post. This will give him the privileges for administration.

To Have access to administration features : get a site admin menu in the side bar.
I used the menu from [MORIN] Doc p 7, modified from deprecated call to object call $HTML-> , checked the users and user_group table.

Note: There was an issue to get the user_is_super_user() to work. the idea is difficult to understand. (corrected: --- to be documented here ---)
 
 

Questions:

A site admin is defined as a user who's an administrator for the site admin group, group ID #1. To make more site admins, simply add them as admins for that group.

News and stats

There is a weak error handling in submit.php and forum_utils.php that I enhanced. The solution official recommandation is to create a project for news and one for stats. See this forum thread Then setup these number in /etc/local.inc as sf-genericisnt
// For us the NEWS and STATS Group are the first group. 
 $sys_news_group=1; 
 $sys_stats_group=1;

6/ BACKEND

CRON JOBS

crontab /mydomain/cronjobs/README
(substitute for real path in all the file)

 I used Bret's list of file for reference http://sourceforge.net/docman/display_doc.php?docid=2660&group_id=9193 There is no need to compile PHP as a standalone binary for packages (RedHat, Mandrake), that deliver both. See offsite forum for installation from source.

 Before starting, note that perl, including postgres module has to be present (see above platform packages).
 
 

List of modifications:

These are parts of a patch included in SF2.5 RPM that I proposed. local.inc: add $sys_mail_host = "ns.mydomain.net";

My local.inc is defined for a single host install. A template file for single host install could easily be provided, to substitute all occurence of the domain and single host.

 include.pl: use the code of J.F. Legato; define needed variables

 DatabaseDump.pl: a number of modifications!
including the content generated for httpd.conf, dns.host, mailalias...
new_parse.pl?/ --- to document --- db_top_groups_calc.sh As delivered, This script create thousands of useless records, because it assume the group_id are sequential from 1. It was modified for only creating records for the existing groups.
 
 

for ($i=1;$i<$max_group_id;$i++) {
  if ($top[$i][13]) {
...
}
# daily calc of the top groups /utils/underworld-root/db_top_groups_calc.pl

Setup of zones files

I used my original configuration from DNS, Apache and so on and copied these files under $SF/backend/zones/*.zone

For mail, shell and cvs access to work on a single host installation, you need these in backend/zones/dns.zone

mailhost  IN    A       127.0.0.1
www     IN      CNAME   @
shell1   IN     CNAME   @
cvs1     IN     CNAME   @

additions in DATABASE for test

I did this in order to get the correct dump files, based on the existing projects. I shall review this as generic modification needed in the configuration or php code. I was unable to set the log of all sql commands in postgres as it is by default in mysql. This is very usefull to document the development. So I am not sure this is complete.
 
 
CREATE TABLE "mailaliases" (
 "user_name" text DEFAULT '' NOT NULL,
 "group_id" integer DEFAULT '0' NOT NULL,
 "email_forward" text DEFAULT '' NOT NULL
 );

insert into mailaliases values('test',1,'yvesbardout@home.com');
insert into mail_group_list values (1234,16380,'testlist',1,'',120802,'','for test purpose');               

# corrections in DB for group administration
# was 'shell1'
update groups set unix_box = 'ns';  
# was empty   
 update groups set http_domain='' where http_domain='www.mydomain.net' ;              
 update groups set http_domain='www.mydomain.net' where  group_id = 1;
I found this is hardcoded to be modified in .php
 
 

System integration of the database dumps

One way or another, you should link the system files to the files dumped by SF. Depending on the service, and the setup you have already made, this is a matter of taste.

For example, what I did:

ln -s $SF/home/dummy/dumps/mydomain.hosts  /var/named/mydomain.hosts                                                   
edit httpd.conf to add
Include $SF/home/dummy/dumps/Vcosts.conf
+ crontab --- to document ---

 Pb of db connection to localhost: I had to change it temporarily (both in database.class and DataBaseDumpl.pl to use the real domain name. This access right problem existed as well by default with mySql. I had a idea it came from DNS or host file. I need to find the configuration.

 Also: who runs the DB Dump? this was not in the cronjobs!
Check the fron jobs run as scheduled (logs) and the result.
 
 

Changes for file release

CORRECTION TO SF php pages : Missing argument 7,8 for frsaddfile() in /var/www/SF2.5/include/frs.class on line 85 -> 0,0 for type_id, proc_id
 
 
File Release utilities:
For the moving of files you need to use the tmpfilemove.c and fileforge.c program. The scripts that use these are php scripts that are run by the web server. These are www/project/admin/qrs.php and www/project/admin/editreleases.php. Both binaries needs to be suid so that the user apache can call them and use root's privileges.

But you will need to edit it first to match your directories and then make sure that the file release system is calling 'fileforge' from the right place. edit the source utilities to change the path, compile it and suid (makes this program run as su)

 To simplify these changes, I put a define in both source files, for src_dir and dest_dir variables, previously hardcoded, so that the value is given at compile time. Of course it has to be the same as the $FTPINCOMING_DIR and $FTPFILES_DIR variables in /etc/local.inc. to match the(which can be changed and recompiled). Also, because it is concatenated with the group name, the $FTPFILES_DIR variable seemed to require a trailing slash, whereas the $FTPINCOMING_DIR did not.

cd /SF2.5/utils    
cc -o /usr/local/bin/fileforge -DFTPFILES_DIR="/SF2.5/www/released" -DFTPINCOMING_DIR="/SF2.5/ftp/incoming/" utils/fileforge.c
chmod +s /usr/local/bin/fileforge
For uploading file, they need to be moved in the ftp incoming dir. This is done by tmpfilemove.c program. The same manipulation needs to be done.
edit  tmpfilemove.c   
  char* dest_dir  = FTPINCOMING_DIR;
cc -o /usr/local/bin/tmpfilemove -DFTPINCOMING_DIR="/SF2.5/ftp/incoming/" tmpfilemove.c       
chmod +s /usr/local/bin/tmpfilemove     

Compilation of other utilities: no dependance on path. No know use in the web site or the backend. Need investigation.
 
 
cc -o /usr/local/bin/grap grap.c
gcc -s -I/usr/include/mysql/ -o /usr/local/bin/sffingerd sffingerd.c -L/usr/lib -L/usr/lib/mysql  -lmysqlclient
Notes on BACKEND (offsite forum): There are some things to take into account (mostly for include_path trickery) ?

Appendix

FAQ from offsite forum : -- to be provided ---