Subversion

Below, in a blog-like fashion, some independent articles around Subversion.

(Story # 20120127.0121)

Repositories bulk processing helpers

I have to maintain about 600 repositories, means workspace-checkouts, commits and updates on the desktop, and svnadmin create, verify, dump, load etc. on serverside.

Doing this manually is between pain and death. The only way to survive are some little helpers to automatize the jobs. But those helpers must leave enough individual control for handling different pathes, special situations and exceptions.

Over the years, I developed a set of batchfiles to help me handling the bags of respositories on the server and the bags of corresponding workspaces on the desktop.

Batchfile number one process-repos.bat contains the definition of all existing repositories. It contains two jumps for selecting a portion to be processed. Then for each defined repository, it calls the second batchfile, which executes the wanted job. The file looks like this:

echo off
rem fileid : ncm # 20120125.1311
echo ***************************************************************
echo *** This is %~n0%~x0 by user %USERNAME%
echo *** This calls a processor for all existing repositories
echo *** thisdir   = %~dp0
echo *** actualdir = %cd%
echo *** option    = %1
echo ***************************************************************

rem Make the batchfile callable from everywhere, e.g. from a desktop icon
%~d0
cd %~dp0

rem Select one of various jobs by uncommenting the wanted line
set cmnd=call process-repos-DUMP-LOAD.bat
rem set cmnd=call process-repos-CHECKOUT.bat
rem set cmnd=call process-repos-VERIFY.bat

rem Define various locations on the drive where workspaces/repositories live
set path01gipsydri=G:\work\gipsydrive
set path02downtown=G:\work\downtown
set path03dowloads=G:\data\downloads
set path042gallery=G:\data\gallery.svn
rem
rem ... more pathes ...

echo Do you know what this file does?! Othwerwise cancel with CTRL-C.
pause

goto start
rem Shift the start label to where you want start the processing
:start

%cmnd% %path01downgi% app.amaya
%cmnd% %path01downgi% app.apache
%cmnd% %path01downgi% app.asm
%cmnd% %path01downgi% app.cakephp
%cmnd% %path01downgi% app.chrome
%cmnd% %path01downgi% app.clamwin
rem Shift the end label to where you want stop the processing
goto end
%cmnd% %path01downgi% app.codeblocks
%cmnd% %path01downgi% app.contao
%cmnd% %path01downgi% app.couchdb
%cmnd% %path01downgi% app.dia
%cmnd% %path01downgi% app.dotnet
%cmnd% %path01downgi% app.eclipse
%cmnd% %path01downgi% app.emacs
rem
rem ... many more ...
rem
%cmnd% %path01downgi% app.wireshark
%cmnd% %path01downgi% app.xampp
%cmnd% %path01downgi% app.xmlcopyeditor

:end
echo.
echo BATCH PROCESSING END.
pause

From above batchfile, for each workspace/repository one to execute the actual job is called, e.g. process-repos-CHECKOUT.bat:

echo off
rem # 20120125.1314
echo.
echo ***********************************************************
echo Process %1 %2
echo ***********************************************************

rem Uncomment next line to confirm each single job before execution
rem pause

set svnad=G:\work\gipsydrive\app.apache\trunk\svn-win32-1.7.2\bin\svn.exe

echo on

mkdir %1\%2
%svnad% checkout http://downtown.trilo.de/svn/%2 %1\%2

@echo off

echo.
echo Completed  %1 %2
echo.

Or the one for making a dump/load cycle with all repositories on the server, process-repos-DUMP-LOAD.bat:

echo off
rem ncm # 20120125.1314
echo.
echo ***********************************************************
echo Process %1 %2
echo ***********************************************************

set svnad=D:\work\gipsydrive\app.apache\trunk\svn-win32-1.7.2\bin\svnadmin.exe

echo on

%svnad% dump %1\%2 > D:\data\_dumps_\%2.dump
rename %1\%2 _OLDSVN144_.%2
%svnad% create %1\%2
%svnad% load %1\%2 < D:\data\_dumps_\%2.dump

@echo off

echo.
echo Completed  %1 %2
echo.
rem pause

You cannot blindly paste and copy above machinery. You have to understand it, then you can quickly adapt the files to very different demands and circumstances.

Have fun.

(Story # 20120126.1521)

Curious dump/load corruption

.

.

.

.

.

Bottom line: I solved the isssue. Fine. But still I don't know what it was! Probably it has to do with the basic Apache/Subversion configuration. So the remaining questions are: What exactly was it? Under which circumstances might it appear again?

(Story # 20100328.1701)

Minutes setting up Subversion on Gipsydrive

This is a developers log file, created on the fly, not meant to enjoy the reader, but to capture knowhow to be digged out in eventual later odd circumstances.

Make it Start

(Session # 20100328.1701)

The Subversion homepage is subversion.tigris.org. A good starting point to investigate how to provide Subversion repositories via an Apache webserver, is of course the book "Version Control with Subversion" (chapter 6 "Server Configuration", subchapter "httpd, the Apache HTTP server"). The book is available as PDF and other formats at svnbook.red-bean.com. And it is also in our Webstack's SVN program folder (app.tools/trunk/svn-1-6-3): file svn-book.pdf.

Another nice tutorial is http://svn.spears.at (png pdf mht) .

According to the Subversion book, we basically need those prerequisites:

• Get httpd 2.0 running with the mod_dav module
• Install the mod_dav_svn plugin to mod_dav, which uses Subversion's libraries to access the repository (mod_dav_svn is the filesystem provider module distributed with Subversion)
• Configure the httpd.conf file to export (or expose) the repository

Searching for "dav" in httpd.conf, we find the following lines

• LoadModule dav_module modules/mod_dav.so
  #LoadModule dav_fs_module modules/mod_dav_fs.so
  #LoadModule dav_lock_module modules/mod_dav_lock.so
  (=> We opened only the first line of those three. We didn't need to load mod_dav.so if we had it not as a shared object but if it were linked statically at compile time.)
   
• # Distributed authoring and versioning (WebDAV)
  #Include conf/extra/httpd-dav.conf
  (=> This lines we leave as is for now.)
   

In the Apache modules folder, we find four files with "dav".

Looking at the timestamps of the four files, we guess that three of them are native, and the fourth was coming later, probably from our svn program folder.

Then we need to include the following line in httpd.conf:

• LoadModule dav_svn_module modules/mod_dav_svn.so

Ready. Start walking!

Stumbling Stone

The walk finishes immediately.

Syntaxcheck error with the new line. What's wrong?.

Where does mod_dav_svn.so come from? Our repository log say at r22: "Add the two Subversion modules from an Subversion 1.6.9.1 installation". The two added modules were mod_authz_svn and mod_dav_svn.so from a regular Subversion 1.6.9.1 installation (download archives # 20100308.1701).

A regular local Subversion 1.6.9.1 installation (directory listing)

Note that the above Subversion 1.6.9.1 installation includes an Apache server. Sorrily, that one is of no use for us, since in our portable webstack, we want have only one Apache, and that one already exists.

In the internet, we find not much about. Only e.g. a 2006 thread at www.svnforum.org/2017/viewtopic.php?t=986 (png pdf mht) . There, they even talk about DependencyWalker. Perhaps not so bad idea, in such situation.

Dependency Walker shows files in red.

IESHIMS.DLL and WER.DLL seem irrelevant, they may be wanted by KERNEL.DLL or the like, and being delayed loaded. But LIBHTTPD.DLL should not light up here!

Recalling the exact error message, it does not say "module cannot be loaded" but "The specified procedure could not be found". So the file is here, but it is the wrong file! Forum post markmail.org/message/ubafqwd7jyigz5kj (png pdf mht) may well indicate the truth: they talk about version incompatibilities.

From the SVN mailinglist archives, this 2006 posting resembles quite exactly our situation: svn.haxx.se/users/archive-2006-02/1105.shtml (png pdf mht) . Version incompatibilities! Only, we have the latest versions of the both, and thought they were compatible. Subversion for long time required an older Apache. But now it caught up, and requires a new one -- but perhaps not too new?

Recall the versions: Apache 2.2.14 plus subversion modules from Subversion 1.6.9 (installation file CollabNetSubversion-server-1.6.9-1.win32.exe).

This session endet unfinished.

Before continuing, we need a version compatibility matrix. Found none in the internet. Let's make one with the versions we already had to do with.

Apache/Subversion Compatibility Matrix

(Session # 20100329.0021)

For convenience, here the two download sites: http://httpd.apache.org/download.cgi and http://www.open.collab.net/downloads/subversion.

The matrix below refers only to versions we somehow faced ourself. (So it is by far not complete.)

	SVN 1.4.3 (# 20070212.2201) (installed on Dow)	SVN 1.4.4 (# 20070814.2001)	SVN 1.5.4 (# 20081106.0401)	SVN 1.6.3 (# 20090806.1721)	SVN 1.6.9.1 (# 20100308.1701) released 2010-01-26 readme.txt
Apache 1.3.42 released 2010-02-02	.	.	.	.	.
Apache 2.0.45 (# 20030429.1401)	.	.	.	.	.
Apache 2.0.59 (# 20070613.0201)	.	(Run on Sua)	.	.	.
Apache 2.0.63 released 2008-01-19 (# 20080326.1901)	.	Running on Sua	.	.	.
Apache 2.2.4 (# 20070613.0221)	.	.	.	.	Running on Dow
Apache 2.2.14 released 2009-10-05 (# 20100303.0401)	.	.	.	.	Our actual problem
Apache 2.2.15 released 2010-xx-xx	.	.	.	.	.

The readme readme file (pdf) for Subversion 1.6.9.1 says, the Apache version is upgraded from 2.0 to 2.2. So one would think, it runs with all Apache 2.2.* versions. On the other hand, it mentions Apache 2.2.13. Might it really not run with 2.2.14?!

Seek Help

(Session # 20100330.1800)

Seeking help, we find forum http://www.svnforum.org (png pdf mht) and others. But those are vast, and existing postings containing our key words are sparse. We detect the CVS and Subversion Group existing on XING, subscribe, start talking, and of course drop our question. Great feedback! existing on XING, They return a bunch of sophisticated questions. The essential questions/notes are:

• Where from come the Apache files exactly?
  The files were copied to folder app.apache/trunk/Apache2.2 from a pristine Apache 2.2.14 installation on a local station. That installation was made from the Windows installer file apache_2.2.14-win32-x86-openssl-0.9.8k.msi, coming from http://httpd.apache.org/download.cgi (download archives # 20100303.0401 folder buddyfile listing).
• Where from come the Subversion files exactly?
  The file mod_dav_svn.so and mod_authz_svn.so was copied to folder app.apache/trunk/Apache2.2/modules from a pristine installation on a local station. That installation was made from the Windows installer file CollabNetSubversion-server-1.6.9-1.win32.exe, coming from http://www.open.collab.net/downloads/subversion (download archives # 20100308.1701 folder buddyfile listing).
• The release dates in above matrix are not correct.
  TODO - We have to check the dates.
• Subversion 1.6.9 must indeed run with Apache 2.2.14.

After thorough crosschecking, we find: the files are o.k., they are indeed all from Apache 2.2.14 and Subversion 1.6.9-1, a combination which must work, according to the specification.

That was homework anyway. The most important part of the reply from the german Subversion Group was: they confirmed me in the opinion, that Subversion 1.6.9 and Apache 2.2.14 must run together. We continue to tackle.

Spot the Culprit

Let's recall the problem: The Apache syntaxcheck complains about mod_dav_svn.so and mod_authz_svn.so, they do find the files they depend on (libhttpd.dll and others) but a procedure inside that files is not found. A clear version misfit! Only where?

Again we open Dependency Walker, this time investigating deeper and sharper, not only the red files, but also the black files:

Looking again and even sharper at Dependency Walker.

There is one unspectacular but curious notion: there are libraries called over a path to a TortoiseHg directory (c:\programme\tortoisehg\LIBSVN_REPOS-1.DLL and c:\programme\tortoisehg\LIBSVN_SUBR-1.DLL). This makes no sense. What folder is this?! The folder turns out to be an abandoned TortoiseHg installation on the machine.

The Windows environment path variable contains a path to this folder. We shut down this path, and look at Dependency Walker again. Now, files are not wrong but entirely missing.

Without the unfitting TortoiseHg path, Dependency Walker looks much redder!

Inspecting the files in the TortoiseHg folder brings to light Subversion 1.5.5 libraries!

Subversion 1.5.5 files in the TortoiseHg path!

Subversion 1.5.5 files would not cooperate with our Apache 2.2.14. The culprit seems to be found!

Only, today we work offline, and the machine we are sitting at, has no such files in a newer version. We inspect the directory listing from the Subversion 1.6.9 Server installation (a week ago on another machine) from download # 20100303.0401 (archives folder buddyfile ) and they are listed there (compare picture above).

We repeat the installation on the today machine, this time without checking the option for installing an Apache or SVNServe. So we get a folder containing the wanted files. We set a path to that folder and -- Success!

The Apache syntaxcheck is satisfied. Escaped from DLL hell!

Well, there is one complaint, but that is a minor taint, not a problem for today, we will care about it another time, perhaps.

Walk!

For today, we only have to make some cleanup: There is the path to the local Subversion 1.6.9 Server installation. This is not allowed for our portable installation! We need to pick the needed files and place them directly in the Apache USB folder. Which files this are, we read from latest Dependency Walker picture:

• libsvn_delta-1.dll
• libsvn_fs-1.dll
• libsvn_repos-1.dll
• libsvn_subr-1.dll

We copy this files to the Apache modules folder, remove the environment path to the original Subversion 1.6.9 Server folder, and run a short test with the Apache syntaxcheck. Fine!

In httpd.conf, we assign a directory on the drive as the one, Apache shall treat as a multiple Subversion repositories home, by adding the following lines :
 
<Location /svn>
   DAV svn
   SVNParentPath ../../../../z.localhost.svn
   SVNListParentPath on
   Deny from all
   Allow from 127.0.0.1
</Location>

Still one crucial component is missing: the Subversion administration program, which can create repositories, and do all the maintainance we need doing on them. It is svnadmin.exe with it's libraries. We already saw it in picture "A regular local Subversion 1.6.9.1 installation" high above.

We copy this plain folder, without the Autoupdate components and without the included Apache server to our app.apache/trunk folder:

The Subversion Admin folder to create and maintain repositories.

We create a repository:

The first repository on the USB drive is created. Don't mistake it for the workspace, you can later ceckout from it.

Session finished.