VisualSVN (Subversion) and Mantis Integration

This post is a bit off-topic for this blog, but since source control and bugtracking are important for Windows CE development as well I thought I'd post about my experiences for (my own) future reference (but I'm sure this will help others as well).

The current setup

At GuruCE we use Subversion (aka SVN) for source control and Mantis (a free popular web-based bugtracking system) for tracking bugs & features.

Subversion runs on a Windows Server through VisualSVN Server (VisualSVN Server makes the Subversion server easy and convenient to install and administer on Windows).

Mantis is running on a Linux server on the other side of the world.

The goal

The goal is to automatically update the status and notes of an issue when committing changes to the SVN server through the TortoiseSVN client.

I want to be able to write comments like:

Worked on issue #64. Robot now enters low power mode when in surveillance mode.
Fixed bug #65. No more unintentional kills. Hopefully...
Implemented feature #66. Robot is now self-aware.
Implementing feature #67 failed. Robot does not allow implementation of self-destruct command.

Ok, I apologize. That was really geeky... ;)

The solution

A lot of information can be found on the net about integrating Subversion and Mantis when they both run on the same Linux server. Our situation is a bit different, being that Subversion runs on a Windows server and Mantis runs on a Linux server on the other side of the world.

Step 1: Configuring Mantis

Using MantisBT version 1.2.0
The first step is to configure Mantis to filter comments so it can determine which issue to update and what to set the state of the issue to. To do that, open config_inc.php in your Mantis installation folder on your server and add the following items:

  • $g_source_control_regexp = '/\b(?:feature|bug|issue)\s*[#]{0,1}(\d+)\b/i';
  • This regular expression filters out the feature, issue or bug number (however you'd like to call it) so it can add a note to the feature/bug/issue.

  • $g_source_control_fixed_regexp = '/\b(?:implement(?:ed|s)|fix(?:ed|es))\s+(?:feature|bug|issue)?\s*[#]{0,1}(\d+)\b/i';
  • This regular expression filters out the feature/bug/issue number if you prepend the word "fixed", "fixes", "implemented" or "implements". When this regular expression evaluates to a number Mantis will use the settings below to set the state of the issue.

  • $g_source_control_set_status_to = RESOLVED;
  • If you fix a bug or implement a feature Mantis will set the issue to status "Resolved".

  • $g_source_control_set_resolution_to = FIXED;
  • If you fix a bug or implement a feature Mantis will set the resolution to "Fixed".

  • $g_source_control_notes_view_status = VS_PUBLIC;
  • Any note added to the issue will be "Public" by default. If you omit this statement all notes will be "Private".

Mantis v1.2.0 comes with a script that you can call from your source control application when committing (or checking in) files. The script can be found in the "scripts" subfolder in your Mantis installation and is called checkin.php.

This script originally uses g_source_control_account to specify the name of the user updating the issue. I didn't want to always specify the same user, but instead specify the name of the user actually committing the files. Therefore I changed line 31 so it reads:

  1. # Check that the username is set and exists
  2. $t_username = $argv[1]; # config_get( 'source_control_account' );
  3. if( is_blank( $t_username ) || ( user_get_id_by_name( $t_username ) === false ) ) {
  4.         echo "Invalid source control account ('$t_username').\n";
  5.         exit( 1 );
  6. }

Now the script accepts a username as the first parameter to the script. For this to work you need to make sure that the usernames you have configured in VisualSVN (subversion) are the same as the ones you configure in Mantis.

To see if Mantis is configured correctly you can test the script. First create a new test issue in Mantis and note the issue number. Now login to your server (through ssh) and run the script as follows (if your host doesn't have a simple way to log in using ssh read on to see how to use PuTTY for this):

path-to/php-cli path-to/checkin.php "username" <<< "This will be added as a note to issue #nn"

In our case, the server that runs Mantis is hosted by hostmonster, so for me the actual command to run is:

/ramdisk/bin/php5-cli ~/public_html/mantis/scripts/checkin.php "Michel" <<< "This will be added as a note to issue #46"

The above should add a note to your newly created issue (remember to change #46 with the issue number you created!).

If you want to test setting the issue to "Resolved" with resolution "Fixed", run the following command:

path-to/php-cli path-to/checkin.php "username" <<< "This fixes bug #nn"

Again, in my case:

/ramdisk/bin/php5-cli ~/public_html/mantis/scripts/checkin.php "Michel" <<< "This fixes bug #46"

Once you've verified the checkin script is working properly, you have to configure the machine running the VisualSVN Server so it can remote login on the server running Mantis over ssh. I've used PuTTY/Plink for that purpose:

Step 2: Configuring PuTTY/Plink

  1. Download the PuTTY suite
  2. Generate a public/private key pair using PuTTYgen
  3. DO NOT enter a passphrase for this private key. We want to use Plink from a script so we can not enter a passphrase!

  4. Import the public key into your server running Mantis and authenticate the key (if your Mantis server is hosted by Hostmonster you can simply use the "SSH/Shell Access" icon on the HM control panel to "Manage SSH Keys" and import & authenticate the public key that way, otherwise read this)
  5. Now start PuTTY, fill in the hostname and the ssh port of your server running Mantis and point to the private key in Connection->SSH->Auth->Private key file for authentication
  6. Test the session by clicking "Open". If this is the first time you access this server using PuTTY it will display a warning:

    Click yes to store the server's fingerprint in the registry (so you don't have to do this the next time).

    Now enter the username you use for ssh access to your Mantis server (in the case of hostmonster this is your account name). If you imported and authenticated the public key you created in (2) correctly on your server you should NOT have to enter a password. The server should have authenticated you using the key you pointed to in (4).

    You should now be able to run the checkin script through PuTTY using the same commands as above.

    The next step is to use Plink, a commandline version of PuTTY that can be used to run commands on a remote server using ssh without dialogs or user input:

  7. "path-to\plink.exe" -ssh -i "path-to-private-key" -batch sshusername@server.com path-to/php-cli path-to/checkin.php username <<< \"This will be added as a note to issue #nn\""
  8. If you've filled in the correct values for the placeholders above plink should return without outputting any warning or error, and a note should be added to the specified issue in Mantis.

    If not, specify -v on the plink commandline to get verbose output.

We're almost done! What we have accomplished so far is that we are able to update Mantis remotely, from the machine running the VisualSVN (or subversion) server. Now all we have to do is write a proper "hook" script that gets called right after we commit some files.

Step 3: Configuring the VisualSVN post-commit hook script

In VisualSVN (or subversion), create a "Test" repository that you'll be using to test the post-commit hook script.

VisualSVN offers a handy IDE to edit hook scripts. Go to the VisualSVN administration console, right click on the repository you want to install the hook into ("Test"), click "Properties" and select the "Hooks" tab. Here you will find templates for the various scripts SVN supports. We want to use the post-commit script, so select that script and press the "Edit" button.

Paste the following batch script into the VisualSVN edit window (or if you're not using VisualSVN just create post-commit.cmd) and update the PLINK, PLINK_CMDLINE, PHP, CHECKIN and SVNLOOK environment variables to the correct values on your system:

@ECHO off

SET REPOS=%1
SET REV=%2
SET PLINK="%ProgramFiles(x86)%\PuTTY\plink.exe"
SET PLINK_CMDLINE=-ssh -i "path-to-private-key" -batch <a href="mailto:sshusername@server.com">sshusername@server.com</a>
SET PHP=/server_path_to/php5-cli
SET CHECKIN=/server_path_to_mantis/scripts/checkin.php
SET SVNLOOK="%VISUALSVN_SERVER%\bin\svnlook.exe"

SET PLINKLOGFILE=%REPOS%\hooks\log.txt
SET LOGFILE=%REPOS%\hooks\log%REV%.txt
SET AUTHORFILE=%REPOS%\hooks\author%REV%.txt
SET OUTPUTFILE=%REPOS%\hooks\output%REV%.txt
SET PLINKBAT=%REPOS%\hooks\plinkbat.cmd

%SVNLOOK% log -r %REV% %REPOS% > %LOGFILE%
%SVNLOOK% author -r %REV% %REPOS% > %AUTHORFILE%

TYPE %LOGFILE% > %OUTPUTFILE%
ECHO  --- >> %OUTPUTFILE%
ECHO Checked into repository >> %OUTPUTFILE%
ECHO %REPOS% (Revision %REV%) >> %OUTPUTFILE%
ECHO by >> %OUTPUTFILE%
TYPE %AUTHORFILE% >> %OUTPUTFILE%

ECHO %PLINK% %PLINK_CMDLINE% ^"%PHP% %CHECKIN% > %PLINKBAT%
ECHO ^\^">> %PLINKBAT%
TYPE %AUTHORFILE% >> %PLINKBAT%
ECHO ^\^" >> %PLINKBAT%
ECHO ^<^<^< >> %PLINKBAT%
ECHO ^\^">> %PLINKBAT%
TYPE %OUTPUTFILE% >> %PLINKBAT%
ECHO ^\^">> %PLINKBAT%
ECHO ^">> %PLINKBAT%

setlocal enabledelayedexpansion
@echo off
set FINAL=
for /f "delims=" %%a in (%PLINKBAT%) do (
set FINAL=!FINAL!%%a
)
echo %FINAL% > %PLINKBAT%
endlocal

CALL %PLINKBAT% > %PLINKLOGFILE%

rem CALL DEL %PLINKBAT%
CALL DEL %LOGFILE%
CALL DEL %AUTHORFILE%
CALL DEL %OUTPUTFILE%

Step 4: Testing it all (and some bugfixing)

Now locally check out the repository "Test" and add a text file to it. Commit the changes with a comment "worked on issue #nn", of course replacing nn with the number of the Mantis test issue you created above, and the script should be automatically called.

After committing, go to Mantis and see if your note has been added.

No? Hmmm.... Strange...

Check the "hooks" subfolder in repository "Test" and open the plinkbat.cmd file:

"C:\Program Files (x86)\PuTTY\plink.exe" -ssh -i "path-to-private-key" -batch <a href="mailto:sshusername@server.com">sshusername@server.com</a> "/server_path_to/php5-cli /server_path_to_mantis/scripts/checkin.php \"Michel\" <<< \"Worked some more on issue #50 --- Checked into repository C:\Repositories\Test (Revision 23) by Michel\""

Well, that's strange. That command looks all good. (Of course your command has the right values filled in for "path-to-private-key" etc!)

Let's try to manually call the post-commit.cmd file and see if it works then...

Open a dos prompt command window (in Windows 7 shift-rightclick the folder in Explorer and choose "Open command window here") to your repository's hook subfolder (in my case "C:\Repositories\Test\hooks") and run the following command:

post-commit.cmd C:\Repositories\Test 23 (of course replacing '23' with your revision number!)

Now check the issue again in Mantis. Has it been updated? What the... Now it works!?

If you would take a look in plinkbat.cmd you'd see it is exactly the same as before (in fact, running plinkbat.cmd from the command line results in the note being added to Mantis as well of course!), so what is going on?

Well, that question has kept me busy for some time... Remember we started PuTTY once to store the server key in the registry? Well, it so happens that VisualSVN runs as a service, and as such executes the hook scripts under a different user name than yours (because you may not be logged in at all). Second, PuTTY stores the ssh host keys under HKEY_CURRENT_USER in the registry, and what's even worse DOES NOT return 1 when it fails to connect to the server due to the server's host key not being cached in the registry. I think that's a bug in plink. Plink should return 1 in that instance (if it would we would be able to see the error in TortoiseSVN).

So, what's the solution? The solution is to simply copy the stored ssh host key to all users in the system:

  1. Start regedit.exe
  2. Browse to "HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\SshHostKeys" and export that key
  3. Browse to HKEY_USERS and expand the key
  4. You should now see a list of users on your system similar to this:
    • .DEFAULT
    • S-1-5-18
    • S-1-5-19
    • ...etc...
  5. Open the exported key from (2) in notepad and duplicate the key for the list of users above
  6. Like this:

    [HKEY_USERS\.DEFAULT\Software\SimonTatham\PuTTY\SshHostKeys]
       hostkeyvalue_here

    [HKEY_USERS\S-1-5-18\Software\SimonTatham\PuTTY\SshHostKeys]
       hostkeyvalue_here

    [HKEY_USERS\S-1-5-19\Software\SimonTatham\PuTTY\SshHostKeys]
       hostkeyvalue_here

    ...etc...

    Do not duplicate the key for user keys that have more numbers behind S-1-5-XX, since that is the current user.

  7. Now save and close the registry file
  8. And finally double click it to add the hostkey registry information to the registry

If you now change a file in your Test repository again, commit it and add a comment containing "issue #nn", Mantis should update correctly now with a note added to the issue, like this:

Worked some more on issue 0000050 --- Checked into repository C:\Repositories\Test (Revision 25) by Michel

You could go on and write a more elaborate batch script, but we all know that batch files are not the most user friendly way of writing scripts. Nothing stops us from writing a simple C# application that gathers all information and updates Mantis for us though!

I've attached a simple C# application that updates Mantis using a nicely formatted note, like these ones (I committed 2 files at the same time with the comments "worked on issue #50 (file1.txt)" and "implemented feature #46 (file2.txt)":

worked on issue 0000050 (file1.txt)
Checked into repository "Test" (revision 26) by Michel on 2010-03-12 10:38:27 +1300 (Fri, 12 Mar 2010)
Changed files:
U trunk/file1.txt
U trunk/file2.txt
implemented feature 0000046 (file2.txt)
Checked into repository "Test" (revision 26) by Michel on 2010-03-12 10:38:27 +1300 (Fri, 12 Mar 2010)
Changed files:
U trunk/file1.txt
U trunk/file2.txt

As you can see my C# program supports updating multiple issues at the same commit, it gathers all information and formats it nicely.

Call the attached C# program, UpdateMantis.exe, like this from your post-commit.cmd:

"%ProgramFiles(x86)%\GuruCE Ltd\UpdateMantis\UpdateMantis.exe" "%1" %2 > "%1\hooks\UpdateMantis.log"

But, before you do you'll of course have to install UpdateMantis (a setup package is included in the attachment) and configure the UpdateMantis.exe.config file (note that in Windows 7 you'll have to configure security for the .config file so normal users can modify the file):

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="plink" value="path_to_PuTTY\plink.exe"/>
    <add key="plink_commandline" value="-l username -i privkey -ssh -batch"/>
    <add key="php-cli" value="/serverpath_to/php5-cli"/>
    <add key="checkin" value="/serverpath_to_mantis/scripts/checkin.php"/>
    <add key="svnlook" value="path_to\svnlook.exe"/>
  </appSettings>
</configuration>

Note that you can use environment variables in the xml values. My config file looks something like this:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="plink" value="%ProgramFiles(x86)%\PuTTY\plink.exe"/>
    <add key="plink_commandline" value="-ssh -i "C:\Data\MantisSVN\mantiskey.ppk" -batch <a href="mailto:username@guruce.com">username@guruce.com</a>"/>
    <add key="php-cli" value="/ramdisk/bin/php5-cli"/>
    <add key="checkin" value="~/public_html/mantis/scripts/checkin.php"/>
    <add key="svnlook" value="%VISUALSVN_SERVER%bin\svnlook.exe"/>
  </appSettings>
</configuration>

Note that the attached VS2008 C# project includes source and IS NOT OF PRODUCTION QUALITY. There is no error handling whatsoever, so if you want to use this program in a production environment you should add some code and make it more robust.

I hope this blog post will help some of you out there struggling to get Mantis integrated with SVN. Good luck!

Last but not least, I'd like to thank the writer of this blog post and all people that commented on that blog post. It has been my single source of information while trying to find a solution to my specific problem. Thanks!

AttachmentSize
Package icon UpdateMantis.zip354.89 KB

Comments

Michel, what version(s) of Windows do you have this setup running on? I am on Windows 7 and only have things running at 95% despite following your excellent instructions. Manual running of the CMD files works fine, but never via the SVN hooks. I even commented out the deletion of the temporary files for inspection and all looks fine. I don't expect you to fix this, just trying to confirm whether or not I have missed something! Thanks.

Hi Daniel,

I'm using Windows 7 64 bit running VisualSVN server. Your problem description sounds exactly like what I described in the article where SVN calls the hooks under a different user name, and plink can't find the cached ssh host key under that user's HKEY.

Did you follow the steps beneath "So, what's the solution?"?

Below is the script in my SVN post-hook; Fails to execute, and hangs at the psexec command (LAST LINE)

-----------------------------------------------------------------
@ECHO off

SET REPOS=%1
SET REV=%2

SET SVNLOOK=C:\Subversion1_6\bin\svnlook.exe

SET LOGFILE=C:\autofixmantisbtbugs\svnlog%REV%.txt
SET AUTHORFILE=C:\autofixmantisbtbugs\svnauthor%REV%.txt
SET OUTPUTFILE=C:\autofixmantisbtbugs\svnoutput%REV%.txt

%SVNLOOK% log -r %REV% %REPOS% > %LOGFILE%
%SVNLOOK% author -r %REV% %REPOS% > %AUTHORFILE%

TYPE %LOGFILE% > %OUTPUTFILE%
ECHO -------- >> %OUTPUTFILE%
ECHO Checked into repository >> %OUTPUTFILE%
ECHO %REPOS% (Revision %REV%) >> %OUTPUTFILE%
ECHO by >> %OUTPUTFILE%
TYPE %AUTHORFILE% >> %OUTPUTFILE%
CALL DEL %LOGFILE%
CALL DEL %AUTHORFILE%

net use P: \\192.168.1.91\svnmantisfiles xxxxxxx /USER:MANTISSERVER\Administrator

TYPE %OUTPUTFILE% > Q:\svnoutput%REV%.txt
CALL DEL %OUTPUTFILE%

ECHO D:\sgrcbugtracker\php\php.exe D:\sgrcbugtracker\apache\Apache2\htdocs\mantisbt\core\checkin.php ^< D:\sgrcbugtracker\svnmantisfiles\svnoutput%REV%.txt > Q:\svnautocheckin%REV%.bat
ECHO DEL /Q D:\sgrcbugtracker\svnmantisfiles\svnoutput%REV%.txt >> Q:\svnautocheckin%REV%.bat
ECHO DEL /Q D:\sgrcbugtracker\svnmantisfiles\svnautocheckin%REV%.bat >> Q:\svnautocheckin%REV%.bat
net use Q: /DELETE

C:\autofixmantisbtbugs\psexec.exe \\192.168.1.91 -u MANTISSERVER\Administrator -p XXXXXXX D:\sgrcbugtracker\svnmantisfiles\svnautocheckin%REV%.bat
-------------------------------------------------------------------------------------

This script creates a batch file and a log file on the remote MANTISSERVER. batch file is to run checkin.php with appropriate SVN log file. But the psexec within this script is not able to run the batch file.

But I am able to run the same batch file successfully from the SVN machine on cmd prompt as well as through a batch file.

Other commands with psexec (cmd.exe /C ipconfig /all) through SVN hook also fails.

Both machines (SVN and MANTISSERVER) are NOT part of any domain, both are Windows 2008 servers.

Any ideas?

Today we run into the same problem and found that there was a svn service user on our 2008 R2 machine for which we had to add the registry entry as described for local users, after that, it works perfect.

This information is EXACTLY what I needed. I guess I am not the only one to use this setup (VisualSVN / Mantis / Bluehost)

this found in mantisbt 1.1.8?

Not sure, 1.1.8 is a very old version of MantisBT. Can't you upgrade to the latest?