Source Forge Release 2.5 Installation Addendum

Author: Yves Bardout
Date created : June 2001
Last modified : July 31st 2001

Introduction

PURPOSE

The goal is to collect the most useful bits of information on installing and putting to work SF and to update them for SF2.5, since the real nice instructions are out of date, or have a different starting point. The doc is not self-contained and is an addendum to the documents given in next section.

Seems like sisyphe since 2.6 will also mean a lot of change there. Also, I will present the procedure for Mandrake 7.2. You will need to adapt it for your distribution.

Unfortunately, I have not started with the installation in sf-genericinst, and I thought it might not be so easy, and in some case, an overkill. To be followed: I intend to review the procedure to use these scripts when all service are installed as standard by packages. Indeed, the procedure described here is chaotic, and I advise to use sf-genericinst if you can be satisfied by the basic assumptions. There is some valuable experience, but I am not yet able to point out which pieces.

CHOICE

Note that a number of candidate to SF installation whether stick to SF2.0 with mysql, or want the SF 2.5 features, and say that the 2.5 distribution was quite broken, so they get the CVS versions: a beta 2.6, or the last editions.

SF2.6 has a longer grocery list than 2.5, that is already quite long. To start with 2.5, I had to learn Postgres and SSL, which I wanted to use anyway.

Give the few time I have, I definitly want a stable release not to get lost, Also I want to adapt the code, so that starting with a moving target will force me to merge all the time, and have some real difficulties, which I prefer to consider only once I have validaet that this platform give me an adge in my development, versus other platform that provides some services, or doing it myself from good high level languages.

I expect to find a stable release good enough for a prototype use, so I can evaluate cost and benefit of the solution.

SUMMARY

These notes follows the experience of a first install of Source Forge with the 2.5 release, on top of an existing Mandrake 7.2 distribution of linux, with an existing web server, DNS server and so on.

The specifics are in SFmandrake.html, in the package selection and install, the comparison of using a packaged installation versus a source tar, and some problem I had that are specific to this platform.

This document list the information that should be generic with regard to the platform, but very likely to be dependant on this release of Source Forge.

Getting the platform installed is long, but not really problematic. A first real problem for me was to load the database, as documented below. The provided load of the database is partial and has errors. I found better sql files from several persons.

My main problem was that the backend in SF2.5 release is partly broken, and that usable information is difficult to get about the cronjobs to setup and the configuration in 2 different include.pl, and in several sources.

I looked at a patch without any documentation by SF legato that had some useful code.

Another issue is getting the service well configured and intergated: CVS, mailman, and so on...

My current problem right now, is that :

  1. I didn't find any design document that would be a base for further evolution of this code, to add a feature or some table and fields. I don't know also how to get the user docs that were in SF database.
  2. following the information on the backend from [SCHULTZE] or [MORIN] on the release and platform I have.
Here is the information I gathered. I hope to have this used in the sf-genericinst project.

STATUS

I am currently able to run a majority of SF features using 2.5. I have access to web pages, using the domain defined, PHP scripting works, the database, I can create a project, a user, using SSL, receive mail, log and so on.

I can access the project, create subproject, tasks, news items, post in forums, create release, add files in them.

There is still a lot to install, setup and configure. I am working on the backend, having a lot of changes in databasedump.pl, and then I still need to setup cvs, proftp...

Comparing to Frank work in sf-genericinst, clearly I have still to work on the tools used and the database content. I shall try a 2.6 and these installation scripts on another Mandrake 7.2 host, and check result, to avoid messing up my poor result.

How much of the documentation is to be done manually, given the software that automates stuff ? The docs are not versionned!

ADAPTATION

The configuration for your system is done in local.inc and include.pl (both in backend and utils). Caution, there are some other place in the code, where there is some hardcoded path, that you'll need to adapt. The minimal adaptation is for the use of the domain name all over the pages and scripts. Here you will need more that scripts and instructions, but I wrote in a separate section adaptation.html, a check list, to follow from the out of the bnox source Forge to your own collaborative service.

A/ DOCUMENTATION

This is a review of existing documentation material. Unfortunately, this is not easy to find them at first.

1/ General Information to start

Source Forge USER DOC from the site project:

(most important ones):

DESIGN DOC:

2/ Installation Information

The first point to start is obviously the docs directory in the tar file. It has more or less usefull bits: The README links to the SF Offsite Talk From which there is all the rest of the information.

Indeed, there are FOUR different efforts to make SourceForge easy to install: the guide from Franck, Guillaume morin's, the sfportable project (which bigdisk created, with no activity), and lo-lan-do's Debian package.

The two main source for me are :

As said by all people who did it, it is tough, but doable to varying degrees of success.

3/ Other resources

In this post are some links to other install information:

5/ NEEDED INFORMATION :

This is a check list of resources needed for me that I have not yet found.

B/ INSTALLATION NOTES

In the following:
mydomain = the name of domain you run your webserver
= the directory where you put the tar from SF. I kept it unmodified for reference.
$SF = the location of the root directory of your running code

Mandrake 7.2 Notes are in SFmandrake.html

1/ Basic Start-Up

From Tim Purdue in forum Make your apache htdocs folder point to: $SF/www.
Set up your PHP include path:
php_value include_path $SF:$SF/include:.
mv alexandria/etc/local.inc to /etc/local.inc and edit all variables (database access, hostname, etc). Make sure /etc/local.inc is readable by your web server.
make an /sfcache/ directory (which is noted in the /etc/local.inc file) and make it owned/writable by the webserver.
create your sourceforge database and a database super-user that has full permissions to the database ("createdb sourceforge" "createuser www" and answer yes to all questions to create a super user). Modify /etc/local.inc to make connections to the db possible.
Set up the database structure: (see below for more)
Set up cronjobs using the crontab in $SF/cronjobs/README
(Some of those cronjobs contain SQL at the top of them - NO - you need to copy/paste that SQL into your database before they can be run the first time)
(see below for more on backend)
(Tim suggested to make a base document out of it, adding what is missing: A lot!)

2/ Configuration

PRINCIPLE OF FILE STRUCTURE :

This is both a question of ease of installation/update and quite important for security (have just the needed authorization), for disk space management in production (disk space, backup, memory image or other performance tuning).

Source Forge has a lot of directories of different use :

Temporarily, we use : 
ln -s $SF  /mydomain/
cd  /mydomain
ln -s /var/www/mydomain www
Path configuration You have to substitute paths to /root/bin/alexandria/ and such. (not like doc in sf-genericinst p40, kevin koch offsite forum ). This is one important task that sf-genericinst is designed to handle. But it is not long to do it manually, once you find the different locations to update: 
cd $SF; grep -rH alexandria .
These definitions are reflected in /etc/local.inc, backend/include.pl utils/include.pl and some path are hardcoded in: 

./utils/download/stats_logparse.sh
./utils/tmpfilemove (*)  see below
./backend/shell/apache.sh  
./utils/fileforge.c (*)
./www/userlist.php (just remove the word  alexandria in the text)
(*) tmpfilemove is a binary. fileforge.c is a small utility shall be built (as written in file release, below). The binary is likely to be an intermediate file to rm.

The shell files could use an ENV variable, e.g. set up for the cron user.

3/ Web Pages

There are some corrections needed in one or two PHP pages.

How do we get the pages: /project/myproject and /users/myuser ? They are not found. This is because "project" and "users" are php scripts without a .php extension, which is misleading to the webserver. you have to add an entry in your .htaccess file to treat "projects" as a php file, since it doesn't have the .php extension. here is the entry in my .htaccess file: 

    <Files projects> 
    ForceType application/x-httpd-php 
    </Files> 


(I repeated this for "foundry" and "users" as well)
from .


I could not get projects/myproject or users/myname to behave correctly.  First I added the .htaccess that is missing to ForceType these files as php, the apache server did not take this in account.

If a create a link named projects.php or users.php and use it in url, it worked. Then I modified the httpd config in Vhosts.conf to setup the directive AllowOveride.


    
images


The images on the project and personal 'homepages' wont load correctly. This has to do with the Apache setup. I have in /etc/local: '$sys_images_url = ""'
The images for the main page are shown ok, but when I go to a project the icons
for the 'Public Areas' don't show. The url for the images is
'https://sourceforge/projects/test1/images/ic/home16b.png'. As you can see the
icons are references through the /projects file, and they are directly under /images. 

i suppose the problem may be here? 
If i did this '$sys_images_url = "/";' in etc/local.inc, the icons showed perfectly, but all other images did not show. 
The url for the other images became 'https://themes/forged/images/tleft1.png'

(was a question unanswered in offline)

    
4/ Database setup


    
DATABASE CREATION

Creation of the database is documented in Local Admin Guide from source forge project doc area: -- url -- 


Note that if you had postgres already installed, you need to move data out of the way to have postgres created a new one. Follows instruction for data migration if you have databases to keep. 


The setup of the database is not obvious, first because the installation guide bundled is for 1.04, and it is not obvious at first not to use mysql, but postgres. 

The install notes 2.5 do not cover every thing.
Since the *.sql provided are not complete and the fake.sql (on the doc page of alexandra) that allows to start for test is obvioulsy for before SF2.5.
I partly modified the script, for the problems I could find. 


    
Security

The first question is who own the database. I decided to create it as postgres user, and as a start give ALL the right to the user apache that is really using it. That way, I can restrict the possible actions allowed from the web, and keep administrative right separately. I have not yet evaluated which table I don't want to be editable via the web server. Maybe this is neither possible nor usefull, given the security on the web pages and PHP.
To do this, I din't find the way to do a GRANT ALL. There fore I created a grant.sql with one command for each table, pasting the output of a  \dt from a psql client to an editor and doing a global replace. 
grant all on  activity_log                  to apache;


    
users

First you shall create users with:
createuser apache


If you want to setup specific permissions mode, you need to edit the pg_hba.conf. 

# This default configuration allows any local user to connect as any
# PostgreSQL username, over either UNIX domain sockets or IP:
 
local        all                                           trust
host         all         127.0.0.1     255.255.255.255     trust                                                             


This was fine for me. If you define user so that it need a password to connect on localhost. You shall set the password with pg_passwd. 



 Connection Mode
 I couldn't connect to the db with the default code, and default config. This is a config of postgres.  connecting to localhost explicitely means an IP connection, unlike not specifying an host, that use the socket. 


psql -h 127.0.0.1  mydomain
psql: PQconnectPoll() -- connect() failed: Connexion refusée
        Is the postmaster running (with -i) at '127.0.0.1'
        and accepting connections on TCP/IP port 5432?  
psql    mydomain
Welcome to psql, the PostgreSQL interactive terminal.  



If you run apache and postgres on the same box, you better not enable IP connections. 
But to use source forge code as is, you can start postmaster with -i option:


postmaster -i -d 2 -D /var/lib/pgsql/data

-i     Allows  clients  to  connect  via  TCP/IP 
-d n   For debug level n
-D     For db path


My solution was more restrictive, and maybe less risk, to update the scripts include.pl and databasdatabase.php that comes with sourceforge. (Changed it to remove the host parameter.

 
    
Loading

This is the list of import files as provided in the SF2.5 release and the files I selected and edited to load the database


SF2.5/db	    		mydomain notes
---------------------------------------------------------------------------
DefaultValues.sql		=		subst email. Adapt values.
SQL_diff-2.0-to-2.5.sql				- (migration only)
SourceForge.sql			=		errors corrected
				fake.sql  	(adapted from local install doc on site, see errors in fake.lst)
pgport/
trove_defaults.sql		trove.sql       adapt map for needs
user-to-users.sh		-		(migration only)	
user_rating.sql			- 		a lot of parser errors !!
  .#language.sql.1.9		language.sql    adapt for 2.5 and limit language set
				 correct.sql     
				 grant.sql      grant perm to user apache, if another user own tables)
				 more.sql       based on  Morin, but only a part is still needed)
				 sfdocs.sql     from sf-genericinst)
				 domain.sql (specifics, for adaptation, as discussed in this memo).



(I checked also filldb.sh list from Morin on SF2.0)
It had a filerelease-changes.sql on which I am not sure.

THe file init-extra.sql (from lo-lan-do) is quite equivalent to the sum of language.sql, DefaultValues.sql and more.sql.

I loaded the following modified files :


# psql mydomain postgres 
\i SourceForge.sql	
\i DefaultValues.sql
\i fake.sql 
\i correct.sql
\i trove.sql    
\i langage.sql 
\i sfdocs.sql  


The changes are reviewed below :
NOTE: s/^#/--/g on the .sql files done as a dump from mysql, because postgresql doesnot accept the # for comment.
remove the connect lines from the. you can import conencted as the apache user, or postgres user, if you give all rights to apache.

See also the ADAPTATION. This is where you add your own default values, and extensions to the source forge features.

    
5/ Administration


To make an admin user: create a user admin, password admin1, then you need to actually give that admin user the proper permissions in the database itself, using the postgresql client to get him an admin flag of A; there is no SF interface to create the fully set up administrator account.



 
psql 
update admin flag = 'A' where user_name = 'admin';

 use the information in this post.
This will give him the  privileges for administration. 


To Have access to administration features : get a site admin menu in the side bar.

I used the menu from [MORIN] Doc p 7, modified from deprecated call to object call $HTML-> , checked the users and user_group table. 

Note: There was an issue to get the user_is_super_user() to work.
the idea is difficult to understand.
(corrected: ---  to be documented here ---)






Questions: 


A site admin is defined as a user who's an administrator for the site admin group, group ID #1. To make more site admins, simply add them as admins for that group.



    
6/ BACKEND

  
CRON JOBS

 crontab /mydomain/cronjobs/README      

(substitute for real path in all the file)


I used Bret's list of file for reference  http://sourceforge.net/docman/display_doc.php?docid=2660&group_id=9193
There is no need to compile PHP as a standalone binary for packages (RedHat, Mandrake), that deliver both.
See offline forum for installation from source.


Before starting, note that perl, including postgres module has to be present (see above platform packages).




    
List of modifications:


local.inc:
add $sys_mail_host = "ns.mydomain.net";


My local.inc is defined for a single host install. A template file for single host install could easily be provided, to substitute all occurence of the domain and single host.


include.pl:
use the code of J.F. Legato; define needed variables


DatabaseDump.pl:
a number of modifications!

including the content generated for httpd.conf, dns.host, mailalias...


new_parse.pl?/
--- to document ---


    
Setup of zones files

I used my original configuration from DNS, Apache and so on and copied these fiels under  $SF/backend/zones/*.zone 


    
additions in DATABASE for test

I did this in order to get the correct dump files, based on the existing projects. I shall review this as generic modification needed in the configuration or php code.
I was unable to set the log of all sql commands in postgres as it is by default in mysql. This is very usefull to document the development. So I am not sure this is complete.



CREATE TABLE "mailaliases" (
 "user_name" text DEFAULT '' NOT NULL,
 "group_id" integer DEFAULT '0' NOT NULL,
 "email_forward" text DEFAULT '' NOT NULL
 );

insert into mailaliases values('test',1,'yvesbardout@home.com');
insert into mail_group_list values (1234,16380,'testlist',1,'',120802,'','for test purpose');               

# corrections in DB for group administration
# was 'shell1'
update groups set unix_box = 'ns';  
# was empty   
 update groups set http_domain='' where http_domain='www.mydomain.net' ;              
 update groups set http_domain='www.mydomain.net' where  group_id = 1;  


I found this is hardcoded to be modified in .php


    
System integration of the database dumps


One way or another, you should link the system files to the files dumped by SF. Depending on the service, and the setup you have already made, this is a matter of taste. 

For example, what I did:

ln -s $SF/home/dummy/dumps/mydomain.hosts  /var/named/mydomain.hosts                                                   
edit httpd.conf to add
Include $SF/home/dummy/dumps/Vcosts.conf


+ crontab --- to document ---


Pb of db connection to localhost:
I had to change it temporarily (both in database.class and DataBaseDumpl.pl to use the real domain name. This access right problem existed as well by default with mySql. I had a idea it came from DNS or host file.
I need to find the configuration.


Also: who runs the DB Dump? this was not in the cronjobs!

Check the fron jobs run as scheduled (logs) and the result.




    
Changes for file release

CORRECTION TO SF php pages :
Missing argument 7,8 for frsaddfile() in /var/www/cadproject/include/frs.class on line 85 
-> 0,0 for type_id, proc_id


File Release:
For the moving of files you need to use the fileforge.c program. It needs
to be suid so that apache can call it and use root's privileges. But you will
need to edit it first to match your directories and then make sure that the
file release system is calling 'fileforge' from the right place.
edit to change path, compile it and suid  (makes this program run as su)


cc -o /usr/local/bin/fileforge utils/fileforge.c
chmod +s /usr/local/bin/fileforge


Notes on BACKEND (offline forum): 



There are some things to take into account (mostly for include_path trickery) ?







Appendix


FAQ from offline forum : -- to be provided ---