When you need to spawn a lot of processes, so many that you don't want to spawn them at the same time this PopenPool can help you.
def run(self,): while self.running_tasks or self.upcomming_tasks: while self.upcomming_tasks and len(self.running_tasks) < self.pool_size: self.consume_task() still_running_tasks = deque() while self.running_tasks: task = self.running_tasks.popleft() if task.poll() is None: # task is not finished yet still_running_tasks.append(task) else: self.finished_tasks.append(task) self.running_tasks = still_running_tasks
This code will spawn the maximum number of processes and as soon one of them finishes the next one is spawned.
The full source can be found here
In case you want to read bash variables from python, I have an example for you. The method is not safe, it will execute the shell script, so you have to trust the input! But the nice part is, that even calculated variables are returned.
Here the interesting part:
def load_config(config_filename): # to get the same set of env variables, we need to execute also multiple statements in one line default_env = check_output_shell("true;set").decode('utf8') config_data_list = open(config_filename).read().splitlines() config_data_list.append("set") # we join the lines with ; so the the BASH_EXECUTION_STRING will not contain newlines config_env_list = check_output_shell(";".join(config_data_list)).decode('utf8').splitlines() return dict(_get_dict_tuples(l) for l in config_env_list if l not in default_env and not l.startswith('BASH_EXECUTION_STRING'))
The key is, to append a set to the shell file and execute it and compare the output to an empty shell script with a set in it. Here is an example with string concatination:
$ cat input/string_concatenation.conf
foo=bar
foo+=' baba'
$ python read-shell-vars.py input/string_concatenation.conf
{'foo': 'bar baba'}
And one with a list concatination:
$ cat input/list_concatenation.conf
lista=(a b c)
listb=(c d e)
listc=("${lista[@]}" "${listb[@]}")
$ python read-shell-vars.py input/list_concatenation.conf
{'lista': ['a', 'b', 'c'],
'listb': ['c', 'd', 'e'],
'listc': ['a', 'b', 'c', 'c', 'd', 'e']}
This will only work for config like shell scripts that will not print to stdout. If you need to support this you need to redirect the output of both set commands to a file and read them.
I switched my blog from jekyll to lektor. The transition was smooth and things like the tag cloud didn't need a hack in the first place. You can check out the sources at gitlab
This will only work on linux, Free BSD, and probably Mac OS. Here is a simplified version of the file tree.
$ tree
.
├── boerewors
│ ├── command_line.py
│ ├── errors.py
│ ├── helper.py
│ ├── __init__.py
│ └── warmup.py
├── __init__.py
├── __main__.py
├── setup.py
└── tests
├── helpers.py
├── __init__.py
└── test_warmup.py
boerewors is a python module that we want to distribute. In case we want to distribute this package to a server without pip and without root privileges we can run this little shell script:
mkdir -p dist
zip -r dist/tmp.zip boerewors -x \*.pyc
zip dist/tmp.zip __main__.py
echo "#!/usr/bin/env python" > dist/warmup
cat dist/tmp.zip >> dist/warmup
chmod +x dist/warmup
rm dist/tmp.zip
This script zips the python module, creates file with a python shebang appends the zip file to it and marks it executable. The entry point of this executable is __main__.py which contains only this lines:
from boerewors.command_line import main
main()
This file can now be copied to the server where we want to execute it. We are not bound to write our script in one file and even have test for development.
first install jenkins
sudo pacman -S jenkins
then start it (or enable it first)
systemctl start jenkins
lookup the initial password. (show the systemd log files for user jenkins)
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
open the dashboard view http://localhost:8090 in your browser. Do not create a new user as suggested, it will not work. But search for the login button and use the long password you found in the previous logs.
Now you can create a new admin user.
[update: 2016-12-01]
The admin key is not printed in the logs at startup anymore but i keep this as a reference how to retrieve the systemd logs for a process that runs as a specific user.
journalctl -u jenkins
here you find the similar lines
...
May 17 11:15:46 hwm-arch jenkins[8295]: *************************************************************
May 17 11:15:46 hwm-arch jenkins[8295]: *************************************************************
May 17 11:15:46 hwm-arch jenkins[8295]: *************************************************************
May 17 11:15:46 hwm-arch jenkins[8295]: Jenkins initial setup is required. An admin user has been created and a password generated.
May 17 11:15:46 hwm-arch jenkins[8295]: Please use the following password to proceed to installation:
May 17 11:15:46 hwm-arch jenkins[8295]: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
May 17 11:15:46 hwm-arch jenkins[8295]: This may also be found at: /var/lib/jenkins/secrets/initialAdminPassword
May 17 11:15:46 hwm-arch jenkins[8295]: *************************************************************
May 17 11:15:46 hwm-arch jenkins[8295]: *************************************************************
May 17 11:15:46 hwm-arch jenkins[8295]: *************************************************************
...
Put your phone into the Recovery Mode:
Now you should have a second network interface, and now you can use telnet to access the recovery menu.
telnet 10.42.66.66
Just select number 4 and enter your device lock code.
----------------------------
Jolla Recovery v0.3.1
----------------------------
Welcome to the recovery tool!
The available options are:
1) Reset device to factory state
2) Reboot device
3) Bootloader unlock [Current state: locked]
4) Shell
5) Try btrfs recovery if your device is in bootloop
6) Exit
Type the number of the desired action and press [Enter]:
4
If you continue, this may void your warranty. Are you really SURE? [y/N] y
Type your devicelock code and press [ENTER] key:
(please note that the typed numbers won't be shown for security reasons)
[OK] Code accepted.
modify the devicelock settings file
mount /dev/mmcblk0p28 /mnt
sed -i "/nemo\\\devicelock\\\maximum_attempts=/c\nemo\\\devicelock\\\maximum_attempts=-1" /mnt/usr/share/lipstick/devicelock/devicelock_settings.conf
disconnect the phone and reboot it.
I found the info on this zendesk page
When I tried to simply emerge vlc via sudo emerge -a vlc i got the following error:
The following REQUIRED_USE flag constraints are unsatisfied:
httpd? ( lua )
The above constraints are a subset of the following complete expression:
aalib? ( X ) bidi? ( truetype ) cddb? ( cdda ) dvb? ( dvbpsi ) dxva2? ( avcodec ) ffmpeg? ( avcodec avformat swscale ) fontconfig? ( truetype ) gnutls? ( gcrypt ) httpd? ( lua ) libcaca? ( X ) libtar? ( skins ) libtiger? ( kate ) qt4? ( X !qt5 ) qt5? ( X !qt4 ) sdl? ( X ) skins? ( truetype X exactly-one-of ( qt4 qt5 ) ) vaapi? ( avcodec X ) vlm? ( encode ) xv? ( xcb )
First i had no idea what this lines meant, but it seams that if you want to use httpd you need also lua.
On videolan.org they give an advice for the minimal set of useflags for the vlc ebuild.
I ended up adding two lines to /etc/portage/package.use:
media-video/vlc dvd ffmpeg mpeg mad wxwindows aac dts a52 ogg flac theora oggvorbis matroska freetype bidi xv svga gnutls stream vlm httpd cdda vcd cdio live vaapi -qt4 qt5 avahi lua directfb opus samba sdl sdl-image
media-video/ffmpeg vaapi opus samba sdl -ieee1394
I had a problem to build the lib for firewire which was a depency for ffmpeg. Since I have no firewire device I just droped it and removed the default useflag ieee1394.
insert the following line into /etc/make.conf
PYTHON_ABIS="2.7 3.4" PYTHON_SINGLE_TARGET="python3_4"
and unmask python 3.4, either with
emerge -auDN @world --autounmask-write
etc-update
or just edit the /etc/portage/package.unmask and insert the line your self
then update your world
emerge -auDN @world
[Update:] added PYTHON_SINGLE_TARGET to make.conf
I recently needed to do this.
git tag -d 12345
But i realized, that these tags where just deletet locally. Even a push --tags didn't help. But luckily I found this blog post. You need to do this push to force a deletion.
git push origin :refs/tags/12345
You can use this simple form to schedule future tasks.
echo "git push" | at 12:00
or even on another day.
echo "/etc/init.d/nginx restart" | at 03:00 Nov 25
To view task queue use atq. And atrm <jobnumber> to rm jobs again.