[2016-09-03] A process pool in python

When you need to spawn a lot of processes, so many that you don't want to spawn them at the same time this PopenPool can help you.

    def run(self,):
        while self.running_tasks or self.upcomming_tasks:
            while self.upcomming_tasks and len(self.running_tasks) < self.pool_size:
            still_running_tasks = deque()
            while self.running_tasks:
                task = self.running_tasks.popleft()
                if task.poll() is None:
                    # task is not finished yet
            self.running_tasks = still_running_tasks

This code will spawn the maximum number of processes and as soon one of them finishes the next one is spawned.

The full source can be found here

[2016-08-25] Access shell script variables from python

In case you want to read bash variables from python, I have an example for you. The method is not safe, it will execute the shell script, so you have to trust the input! But the nice part is, that even calculated variables are returned.

Here the interesting part:

def load_config(config_filename):
    # to get the same set of env variables, we need to execute also multiple statements in one line
    default_env = check_output_shell("true;set").decode('utf8')

    config_data_list = open(config_filename).read().splitlines()
    # we join the lines with ; so the the BASH_EXECUTION_STRING will not contain newlines
    config_env_list = check_output_shell(";".join(config_data_list)).decode('utf8').splitlines()
    return dict(_get_dict_tuples(l) for l in config_env_list
                if l not in default_env and not l.startswith('BASH_EXECUTION_STRING'))

The key is, to append a set to the shell file and execute it and compare the output to an empty shell script with a set in it. Here is an example with string concatination:

$ cat input/string_concatenation.conf 
foo+=' baba'
$ python read-shell-vars.py input/string_concatenation.conf 
{'foo': 'bar baba'}

And one with a list concatination:

$ cat input/list_concatenation.conf 
lista=(a b c)
listb=(c d e)
listc=("${lista[@]}" "${listb[@]}")
$ python read-shell-vars.py input/list_concatenation.conf 
{'lista': ['a', 'b', 'c'],
 'listb': ['c', 'd', 'e'],
 'listc': ['a', 'b', 'c', 'c', 'd', 'e']}

This will only work for config like shell scripts that will not print to stdout. If you need to support this you need to redirect the output of both set commands to a file and read them.

[2016-08-13] Hello Lektor

I switched my blog from jekyll to lektor. The transition was smooth and things like the tag cloud didn't need a hack in the first place. You can check out the sources at gitlab

[2016-08-05] How to distribute an executable python module without pip

This will only work on linux, Free BSD, and probably Mac OS. Here is a simplified version of the file tree.

$ tree
├── boerewors
│   ├── command_line.py
│   ├── errors.py
│   ├── helper.py
│   ├── __init__.py
│   └── warmup.py
├── __init__.py
├── __main__.py
├── setup.py
└── tests
    ├── helpers.py
    ├── __init__.py
    └── test_warmup.py

boerewors is a python module that we want to distribute. In case we want to distribute this package to a server without pip and without root privileges we can run this little shell script:

mkdir -p dist
zip -r dist/tmp.zip boerewors -x \*.pyc
zip dist/tmp.zip __main__.py
echo "#!/usr/bin/env python" > dist/warmup
cat dist/tmp.zip >> dist/warmup
chmod +x dist/warmup
rm dist/tmp.zip

This script zips the python module, creates file with a python shebang appends the zip file to it and marks it executable. The entry point of this executable is __main__.py which contains only this lines:

from boerewors.command_line import main

This file can now be copied to the server where we want to execute it. We are not bound to write our script in one file and even have test for development.

[2016-05-17] jenkins and arch linux or systemd

first install jenkins

sudo pacman -S jenkins

then start it (or enable it first)

systemctl start jenkins

lookup the initial password. (show the systemd log files for user jenkins)

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

open the dashboard view http://localhost:8090 in your browser. Do not create a new user as suggested, it will not work. But search for the login button and use the long password you found in the previous logs.

Now you can create a new admin user.

[update: 2016-12-01]

The admin key is not printed in the logs at startup anymore but i keep this as a reference how to retrieve the systemd logs for a process that runs as a specific user.

journalctl -u jenkins

here you find the similar lines

May 17 11:15:46 hwm-arch jenkins[8295]: *************************************************************
May 17 11:15:46 hwm-arch jenkins[8295]: *************************************************************
May 17 11:15:46 hwm-arch jenkins[8295]: *************************************************************
May 17 11:15:46 hwm-arch jenkins[8295]: Jenkins initial setup is required. An admin user has been created and a password generated.
May 17 11:15:46 hwm-arch jenkins[8295]: Please use the following password to proceed to installation:
May 17 11:15:46 hwm-arch jenkins[8295]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
May 17 11:15:46 hwm-arch jenkins[8295]: This may also be found at: /var/lib/jenkins/secrets/initialAdminPassword
May 17 11:15:46 hwm-arch jenkins[8295]: *************************************************************
May 17 11:15:46 hwm-arch jenkins[8295]: *************************************************************
May 17 11:15:46 hwm-arch jenkins[8295]: *************************************************************

[2015-06-30] unlock jolla phone, from permanently locked mode

Put your phone into the Recovery Mode:

  1. Power off your phone, disconnect from all cables
  2. Remove the battery
  3. Hold down the Volume-Down-Button
  4. Insert the battery again
  5. Push the Power-Button
  6. Release all buttons as soon the phone vibrates
  7. Connect your phone with the PC via USB

Now you should have a second network interface, and now you can use telnet to access the recovery menu.


Just select number 4 and enter your device lock code.

     Jolla Recovery v0.3.1      
Welcome to the recovery tool!
The available options are:
1) Reset device to factory state
2) Reboot device
3) Bootloader unlock [Current state: locked]
4) Shell
5) Try btrfs recovery if your device is in bootloop
6) Exit
Type the number of the desired action and press [Enter]: 

If you continue, this may void your warranty. Are you really SURE? [y/N] y
Type your devicelock code and press [ENTER] key:
(please note that the typed numbers won't be shown for security reasons)
[OK] Code accepted.

modify the devicelock settings file

mount /dev/mmcblk0p28 /mnt
sed -i "/nemo\\\devicelock\\\maximum_attempts=/c\nemo\\\devicelock\\\maximum_attempts=-1" /mnt/usr/share/lipstick/devicelock/devicelock_settings.conf

disconnect the phone and reboot it.

I found the info on this zendesk page

[2015-03-24] emerge vlc player in funtoo

When I tried to simply emerge vlc via sudo emerge -a vlc i got the following error:

  The following REQUIRED_USE flag constraints are unsatisfied:
    httpd? ( lua )

  The above constraints are a subset of the following complete expression:
    aalib? ( X ) bidi? ( truetype ) cddb? ( cdda ) dvb? ( dvbpsi ) dxva2? ( avcodec ) ffmpeg? ( avcodec avformat swscale ) fontconfig? ( truetype ) gnutls? ( gcrypt ) httpd? ( lua ) libcaca? ( X ) libtar? ( skins ) libtiger? ( kate ) qt4? ( X !qt5 ) qt5? ( X !qt4 ) sdl? ( X ) skins? ( truetype X exactly-one-of ( qt4 qt5 ) ) vaapi? ( avcodec X ) vlm? ( encode ) xv? ( xcb )

First i had no idea what this lines meant, but it seams that if you want to use httpd you need also lua.

On videolan.org they give an advice for the minimal set of useflags for the vlc ebuild.

I ended up adding two lines to /etc/portage/package.use:

media-video/vlc dvd ffmpeg mpeg mad wxwindows aac dts a52 ogg flac theora oggvorbis matroska freetype bidi xv svga gnutls stream vlm httpd cdda vcd cdio live vaapi -qt4 qt5 avahi lua directfb opus samba sdl sdl-image
media-video/ffmpeg vaapi opus samba sdl -ieee1394

I had a problem to build the lib for firewire which was a depency for ffmpeg. Since I have no firewire device I just droped it and removed the default useflag ieee1394.

[2015-03-12] emerge python3.4 in Funtoo

insert the following line into /etc/make.conf

PYTHON_ABIS="2.7 3.4"

and unmask python 3.4, either with

emerge -auDN @world --autounmask-write

or just edit the /etc/portage/package.unmask and insert the line your self

then update your world

emerge -auDN @world

[Update:] added PYTHON_SINGLE_TARGET to make.conf

[2015-02-20] Delete remote tags with git

I recently needed to do this.

git tag -d 12345

But i realized, that these tags where just deletet locally. Even a push --tags didn't help. But luckily I found this blog post. You need to do this push to force a deletion.

git push origin :refs/tags/12345

[2014-11-28] use at to schedule future tasks

You can use this simple form to schedule future tasks.

echo "git push" | at 12:00

or even on another day.

echo "/etc/init.d/nginx restart" | at 03:00 Nov 25

To view task queue use atq. And atrm <jobnumber> to rm jobs again.