Kolab 3.1.5, Debian 7.5 and issue during setup process

After a longer period of time I have decided to install Kolab and use it as personal information manager. Installation process went as expected up to the point where setup process tried to install Roundcube database and failed miserably.

Source of the problem

The problem can be easily identified by the error messages returned by the setup process.

Follow the example below to see MySQL errors at the very end.

# setup-kolab 

Please supply a password for the LDAP administrator user 'admin', used to login
to the graphical console of 389 Directory server.

Administrator password [Sii79iQyU3gkHN3]: 

Please supply a password for the LDAP Directory Manager user, which is the
administrator user you will be using to at least initially log in to the Web
Admin, and that Kolab uses to perform administrative tasks.

Directory Manager password [nNQAwQVnSDRzSUq]: 

Please choose the system user and group the service should use to run under.
These should be existing, unprivileged, local system POSIX accounts with no
shell.

User [dirsrv]: 
Group [dirsrv]: 

This setup procedure plans to set up Kolab Groupware for the following domain
name space. This domain name is obtained from the reverse DNS entry on your
network interface. Please confirm this is the appropriate domain name space.

sleeplessbeastie.eu [Y/n]: 

The standard root dn we composed for you follows. Please confirm this is the root
dn you wish to use.

dc=sleeplessbeastie,dc=eu [Y/n]: 

Setup is now going to set up the 389 Directory Server. This may take a little
while (during which period there is no output and no progress indication).

[..]

Please supply a Cyrus Administrator password. This password is used by Kolab to
execute administrative tasks in Cyrus IMAP. You may also need the password
yourself to troubleshoot Cyrus IMAP and/or perform other administrative tasks
against Cyrus IMAP directly.

Cyrus Administrator password [qolmP-8qsQAiPRe]: 

Please supply a Kolab Service account password. This account is used by various
services such as Postfix, and Roundcube, as anonymous binds to the LDAP server
will not be allowed.

Kolab Service password [yp48IyJvn8rd12H]: 

[..]

What MySQL server are we setting up?
 - 1: Existing MySQL server (with root password already set).
 - 2: New MySQL server (needs to be initialized).
Choice: 1

Please supply the root password for MySQL, so we can set up user accounts for
other components that use MySQL.

MySQL root password: 

Please supply a password for the MySQL user 'kolab'. This password will be used
by Kolab services, such as the Web Administration Panel.

MySQL kolab password [LAOv8Gmn2l1Gs-G]: 

Please supply the timezone PHP should be using. You have to use a Continent or
Country / City locality name like 'Europe/Berlin', but not just 'CEST'.

Timezone ID [UTC]: 

Please supply a password for the MySQL user 'roundcube'. This password will be
used by the Roundcube webmail interface.

MySQL roundcube password [yP7R_m4wJVcNiDW]: 
ERROR 1005 (HY000) at line 9: Can't create table 'roundcube.kolab_alarms' (errno: 150)
ERROR 1146 (42S02) at line 179: Table 'roundcube.system' doesn't exist

[..]

These errors are generated due to the following code inside /usr/share/pyshared/pykolab/setup/setup_roundcube.py file.

    schema_files = []
    for root, directories, filenames in os.walk('/usr/share/doc/'):
        for directory in directories:
            if directory.startswith("roundcubemail"):
                for root, directories, filenames in os.walk(os.path.join(root, directory)):
                    for filename in filenames:
                        if filename.startswith('mysql.initial') and filename.endswith('.sql'):
                            schema_filepath = os.path.join(root,filename)
                            if not schema_filepath in schema_files:
                                schema_files.append(schema_filepath)

                break
        break

I assume that its goal is to recursively find every mysql.initial*.sql file inside /usr/share/doc/roundcubemail*/ directories.

Finding the solution

First approach - Problem identification

Copy the earlier mentioned code to the disctinct script and slightly modify it to output additional information.

$ cat first_code_check.py 
import os

schema_files = []
for root, directories, filenames in os.walk('/usr/share/doc/'):
  for directory in directories:
    if directory.startswith("roundcubemail"):
      print "-> " + directory + " (" + root + ")"
      for root, directories, filenames in os.walk(os.path.join(root, directory)):
        for filename in filenames:
          print "   \ " + filename
          if filename.startswith('mysql.initial') and filename.endswith('.sql'):
            schema_filepath = os.path.join(root,filename)
            if not schema_filepath in schema_files:
              schema_files.append(schema_filepath)

It is easy to notice the problem after the script execution - the root variable inside third for loop is messing with future iterations.

$ python first_code_check.py 
-> roundcubemail-plugins-kolab (/usr/share/doc/)
   \ changelog.Debian.gz
   \ copyright
-> roundcubemail-plugin-threadingasdefault (/usr/share/doc/roundcubemail-plugins-kolab)
-> roundcubemail (/usr/share/doc/roundcubemail-plugins-kolab)

Second approach - Python script

The most obious fix is to rename second occurence of the root variable.

$ cat second_code_check.py 
import os

schema_files = []
for root, directories, filenames in os.walk('/usr/share/doc/'):
  for directory in directories:
    if directory.startswith("roundcubemail"):
      print "-> " + directory + "(" + root + ")"
      for nested_root, directories, filenames in os.walk(os.path.join(root, directory)):
        for filename in filenames:
          print "   \ " + filename
          if filename.startswith('mysql.initial') and filename.endswith('.sql'):
            schema_filepath = os.path.join(nested_root,filename)
            if not schema_filepath in schema_files:
              schema_files.append(schema_filepath)
              print "     ! added schema file"

Simple test reveals that it works as expected.

$ python second_code_check.py
-> roundcubemail-plugins-kolab(/usr/share/doc/)
   \ changelog.Debian.gz
   \ copyright
-> roundcubemail-plugin-threadingasdefault(/usr/share/doc/)
   \ changelog.Debian.gz
   \ copyright
-> roundcubemail(/usr/share/doc/)
   \ mysql.initial.sql
    ! added schema file
   \ mssql.initial.sql
   \ README.md
   \ sqlite.initial.sql
   \ changelog.Debian.gz
   \ copyright
   \ postgres.initial.sql
-> roundcubemail-plugin-contextmenu(/usr/share/doc/)
   \ changelog.gz
   \ changelog.Debian.gz
   \ copyright

Second approach - Path

--- /usr/share/pyshared/pykolab/setup/setup_roundcube.py.orig	2014-06-21 18:43:40.975058719 +0200
+++ /usr/share/pyshared/pykolab/setup/setup_roundcube.py	2014-06-21 19:12:46.957149746 +0200
@@ -139,16 +139,13 @@
     for root, directories, filenames in os.walk('/usr/share/doc/'):
         for directory in directories:
             if directory.startswith("roundcubemail"):
-                for root, directories, filenames in os.walk(os.path.join(root, directory)):
+                for nested_root, directories, filenames in os.walk(os.path.join(root, directory)):
                     for filename in filenames:
                         if filename.startswith('mysql.initial') and filename.endswith('.sql'):
-                            schema_filepath = os.path.join(root,filename)
+                            schema_filepath = os.path.join(nested_root,filename)
                             if not schema_filepath in schema_files:
                                 schema_files.append(schema_filepath)
 
-                break
-        break
-
     if os.path.isdir('/usr/share/roundcubemail'):
         rcpath = '/usr/share/roundcubemail/'
     elif os.path.isdir('/usr/share/roundcube'):

Third approach - Python script

This could be simplified a bit by replacing problematic os.walk call.

$ cat third_code_check.py 
import os
import fnmatch

schema_files = []
for root, directories, filenames in os.walk('/usr/share/doc/'):
  for directory in directories:
    if directory.startswith("roundcubemail"):
      print "-> " + directory + "(" + root + ")"
      for filename in os.listdir(os.path.join(root, directory)):
        if fnmatch.fnmatch(filename, 'mysql.initial*.sql'):
          schema_filepath = os.path.join(root,directory,filename)
          if not schema_filepath in schema_files:
            schema_files.append(schema_filepath)
            print "  \ " + filename
$ python third_code_check.py 
-> roundcubemail-plugins-kolab (/usr/share/doc/)
-> roundcubemail-plugin-threadingasdefault (/usr/share/doc/)
-> roundcubemail (/usr/share/doc/)
  \ mysql.initial.sql
-> roundcubemail-plugin-contextmenu (/usr/share/doc/)

Third approach - Patch

--- /usr/share/pyshared/pykolab/setup/setup_roundcube.py.orig	2014-06-21 18:43:40.975058719 +0200
+++ /usr/share/pyshared/pykolab/setup/setup_roundcube.py	2014-06-21 19:43:14.235162418 +0200
@@ -20,6 +20,7 @@
 from Cheetah.Template import Template
 import hashlib
 import os
+import fnmatch
 import random
 import re
 import subprocess
@@ -139,15 +140,11 @@
     for root, directories, filenames in os.walk('/usr/share/doc/'):
         for directory in directories:
             if directory.startswith("roundcubemail"):
-                for root, directories, filenames in os.walk(os.path.join(root, directory)):
-                    for filename in filenames:
-                        if filename.startswith('mysql.initial') and filename.endswith('.sql'):
-                            schema_filepath = os.path.join(root,filename)
-                            if not schema_filepath in schema_files:
-                                schema_files.append(schema_filepath)
-
-                break
-        break
+                for filename in os.listdir(os.path.join(root, directory)):
+                    if fnmatch.fnmatch(filename, 'mysql.initial*.sql'):
+                        schema_filepath = os.path.join(root,directory,filename)
+                        if not schema_filepath in schema_files:
+                            schema_files.append(schema_filepath)
 
     if os.path.isdir('/usr/share/roundcubemail'):
         rcpath = '/usr/share/roundcubemail/'

Fourth approach - Python script

This code can be simplified further by not reading directories recursively.

$ cat fourth_code_check.py
import os
import fnmatch
import glob

schema_files = []
for directory in glob.glob('/usr/share/doc/roundcubemail*'):
  print "-> " + directory
  for filename in os.listdir(directory):
    if fnmatch.fnmatch(filename, 'mysql.initial*.sql'):
      schema_filepath = os.path.join(directory,filename)
      if not schema_filepath in schema_files:
        schema_files.append(schema_filepath)
        print "  \ " + filename
$ python fourth_code_check.py 
-> roundcubemail-plugins-kolab (/usr/share/doc/)
-> roundcubemail-plugin-threadingasdefault (/usr/share/doc/)
-> roundcubemail (/usr/share/doc/)
  \ mysql.initial.sql
-> roundcubemail-plugin-contextmenu (/usr/share/doc/)

Fourth approach - Patch

--- /usr/share/pyshared/pykolab/setup/setup_roundcube.py.orig	2014-06-21 18:43:40.975058719 +0200
+++ /usr/share/pyshared/pykolab/setup/setup_roundcube.py	2014-06-21 19:59:42.380040035 +0200
@@ -20,6 +20,8 @@
 from Cheetah.Template import Template
 import hashlib
 import os
+import glob
+import fnmatch
 import random
 import re
 import subprocess
@@ -136,18 +138,12 @@
             fp.close()
 
     schema_files = []
-    for root, directories, filenames in os.walk('/usr/share/doc/'):
-        for directory in directories:
-            if directory.startswith("roundcubemail"):
-                for root, directories, filenames in os.walk(os.path.join(root, directory)):
-                    for filename in filenames:
-                        if filename.startswith('mysql.initial') and filename.endswith('.sql'):
-                            schema_filepath = os.path.join(root,filename)
-                            if not schema_filepath in schema_files:
-                                schema_files.append(schema_filepath)
-
-                break
-        break
+    for directory in glob.glob('/usr/share/doc/roundcubemail*'):
+        for filename in os.listdir(directory):
+            if fnmatch.fnmatch(filename, 'mysql.initial*.sql'):
+                schema_filepath = os.path.join(directory,filename)
+                if not schema_filepath in schema_files:
+                    schema_files.append(schema_filepath)
 
     if os.path.isdir('/usr/share/roundcubemail'):
         rcpath = '/usr/share/roundcubemail/'

Ending notes

I am not a Python developer. I wrote about this issue as it was an interesting way to learn about directory traversal and file filtering implemented in Python language.

It is up to you to decide which solution is appropriate for you - for more recent information please read the official bug report.

Milosz Galazka's Picture

About Milosz Galazka

Milosz is a Linux Foundation Certified Engineer working for a successful Polish company as a system administrator and a long time supporter of Free Software Foundation and Debian operating system.

Gdansk, Poland https://sleeplessbeastie.eu