duckpond.ch

Mit Eidechsen war alles besser

2026-01-25T00:00:00+01:00

Disclaimer: I am publishing the following article on behalf of a friend who recently wrote it. I found the piece engaging and worth preserving, which is why I decided to share it here. The article reflects the author’s personal views and style; all opinions, interpretations, and conclusions expressed are solely those of the original author and do not represent my own.

Ein in der Antike verbreitetes Navigationshilfmittel war neben der Sonne und den restlichen Sternen auch die gemeine Eidechse. Ein leicht zu übersehendes, aber nützliches Reptil. Auch vor Gericht fanden Sie weiträumig Verwendung, wenn jemand einen Eid ablegen musste. Wenn man log, kackte die Eidechse einem sofort auf die Hand. Das war damals genügend Abschreckung, denn Kleenex war noch nicht erfunden.

Auch als Politbarometer fanden Eidechsen schon Verwendung, denn häufig kommt die Süd- und Osteidechse eher von rechts. Also rechts vor links, wegen der STVO, insbesondere auch in Italien, seit Meloni und in Deutschland wegen der AFD.

Kommt eine Eidechse dann von links geht man offensichtlich in die entgegengesetzte Richtung, ist aber selten, dann muss die Eidechse ja warten, also zumindest die Deutsche. Die Viecher warten aber nicht gerne. Die haben viel zu tun und haben’s eilig, könnte man denken, aber weit gefehlt. Sie sind arbeitslos. Warum? Nun das ist ‘ne lange Geschichte. Also wo fangen wir an? Mit Navigation eben.

Navigation mittels Eidechsen ist heute nicht mehr sehr populär, seitdem es Kompasse und GPS gibt. Bis vor 65 Millionen Jahren war das allerdings anders, denn die Eidechsen waren grösser, sehr viel grösser. Nicht zu übersehen, gab es da nur eine Richtung:

Weg von der Eidechse!

Der Versuch der Integration dieser monströsen Tiere in tragbare Navigationsinstrumente ging natürlich wegen des unvorteilhaften Formfaktors drastisch schief. Also musste ein Meteor her, damit die Viecher endlich kleiner werden. Das führte dann schliesslich zu McDonalds, Microschrott, Trump, Musk und einer Menge brauner Scheisse. Einer von denen musste derart oft gelogen haben, dass alle Eidechsen der Welt spontan kacken mussten. Aus olfaktorischen Gründen wurden sie von der Jurisdiktion dann ausgeschlossen. Das war ein schwerer Schlag für die Eidechseninnung und natürlich auch ein grosser Einkommensverlust. Ihre Häuser auf Sylt mussten sie schliesslich aufgeben und in den billigeren Süden und Osten ziehen. Seitdem wohnen da nur noch neureiche Skandal-Pupköppe.

Die ersten Versuche die nun kleineren Eidechsen in antiken Smartphones einzubauen, schlugen jedoch drastisch fehl. Einerseits sind Eidechsen nicht SMD tauglich und andererseits haben sie mit dem Reflowlöten ein Akzeptanzproblem. Zudem sind die Viecher völlig unkooperativ und rennen immer weg, damit ist die Ausbeute sehr niedrig. Also nicht gerade interessant für börsennotierte Unternehmen.

Der Vorteil der Verwendung von Eidechsen wäre allerdings die Lebensdauer, die wesentlich länger ist als heute produzierte Mobiltelefone. Man sollte vielleicht doch nochmal darüber nachdenken.

Dennoch, Eidechsen haben einen nicht mehr wegzudenkenden Nischenplatz in unserer Gesellschaft eingenommen, insbesondere als Navigationshilfen auf dem Weg zum Klo in Italien, genauer Alzate, noch genauer in der Nähe des Hangars. Eidechse von rechts, man geht zum Klo, andersherum weg vom Klo. Das kann im Druckstress durchaus hilfreich sein.

Nur was tun, wenn gerade keine Eidechse zugegen ist? In Deutschland würde man sogleich ein Eidechsenpräsenzgesetz verabschieden und in der EU ein Gremium gründen, was sich mit der Regulierung der Eidechsendichte befasst. Und die Schweiz würde das Gesetz als Erste kritiklos übernehmen. In der Zwischenzeit hätten sich alle in Italien in der Nähe von Hangars in die Hose geschissen.

Man sieht also gewisse Navigationsprobleme können mittels Eidechseneinsatz vereinfacht werden. Nein, ich bin kein Lobbyist der Eidechsenvereinigung, aber ich denke, man sollte in weltwirtschaftlich schwierigen Zeiten auch mal lateral nach Lösungen suchen.

Aus diesem Grund hat man sich Anfang des 20. Jahrhunderts redlich bemüht, die Eidechse in unsere technologisch getriebene Gesellschaft zu integrieren. Das zeigte insbesondere der Versuch, sie als wesentliche Komponente eines innovativen Kompassystems einzubinden. Die Entlöhnung war überdurchschnittlich, auch für die Unterkunft und Krankenversicherung wurde gesorgt. Das ganze scheiterte leider am mangelnden Magnetisierungswillen seitens dieser Tiere. Damit waren sie endgültig aus dem Geschäft, keiner wollte mehr mit Eidechsen arbeiten. Die dann folgende Arbeits- und Perspektivlosigkeit legte später die Grundlage für die NSDAP. Der Rest ist bekannt.

Es gestaltete sich schliesslich einfacher Satelliten in die Erdumlaufbahn zu schiessen und ein bisschen Relativitätstheorie zu treiben, ohne Eidechsen. Denn Letztere erreichen sehr selten relativistisch relevante Geschwindigkeiten und grosse Masse haben sie auch nicht.

Darum führen Eidechsen heute ein eher zurückgezogenes Leben und verstecken sich, wenn jemand in ihrer Nähe erscheint. Das sollte nun ausreichen, um die gesamte Geschichte der letzten 245 Millionen Jahre zu verstehen. Wenn noch Fragen sind, wende man sich vertrauensvoll an tranalyzer.com. Dort sitzt ein Ameisenbär, der kennt sich mit der Weltgeschichte bestens aus, zumindest aus Ameisenbärsicht.

Die Abscheu der Eidechsen gegenüber dem Menschen zwang sie schliesslich in den Untergrund. Darum kann man sie nur erkennen, wenn man sich nahe am Boden befindet und weiss wo sie liegen. Z.B. wie schon erwähnt, hinterm Hangar in Alzate. Also aus der Luft wird man sie jedenfalls nicht entdecken. Ausser bis vor 65 Millionen Jahren, da waren Sie ja grösser. Dann war da aber auch kein Hangar und Landen wäre dann nicht gerade die beste Option gewesen. Darum ist damals auch keiner aussengelandet. Und wenn doch, war man Matsch oder Frühstück. Darum findet man auch keine intakten Segelflugzeuge in den einschlägigen geologischen Schichten.

Heute kann man aber Landen ohne gefressen zu werden, zumindest nicht von Eidechsen. Das tut man zweckmässigerweise auf einem Flugplatz oder irgendeinem angemessenen Feld; Und das alles trotz heutiger hoher Eidechsendichte im Süden. Aber manchmal kann man sich das nicht immer aussuchen, wo man landet oder von wem man gefressen wird.

Also nehmen wir mal an, man landet heutzutage, also letztes Jahr, aussen. Natürlich landet man immer aussen. Drinnen zu landen macht ja keinen Sinn, ausser es handelt sich um eine Kamikaze- drohne. Die Erörterung dieser autonomen Flugkörper und deren Interaktion mit Eidechsen im Urkrainekrieg würde den Rahmen dieses Artikels sicherlich sprengen und ist zudem klassifiziert. Interessierte wenden sich bitte an das eidgenössische Departement für Verteidigung.

Nun zum Thema, was war das nochmal? Eidechsenunterstütztes Kacken? Nein, Aussenlanden. Ah ja, das passiert, wenn man nicht da hinkommt, wo man hin will, meistens zurück zum Flugplatz wo es Futter und Bier gibt. Letzteres ist ein unwiderstehlicher Anreiz es doch noch zurückzuschaffen. Aber manchmal spielt das Wetter nicht mehr mit, eine verdammte Sauerei. Und irgendwann, nach heroischem Kampf, muss man eben die Eidechsen besuchen gehen, was auch unterhaltsam sein kann.

Porlezza ist so ein sagenumwobener Ort, passende Eidechsendichte, aber grosses Oma mit Hund Vorkommen. Das hätte ich mal vorher wissen sollen. Den Eidechsendichteplan hatte ich ja mit. Jeder Pilot in Italien erhält da so eine Karte, oder man kann die lokale Dichte auch bei Milano Info erfragen. Aber wer rechnet schon mit ‘ner Oma, und dann noch mit Hund während man landen will. Unabhängigkeit angenommen wäre das Produkt der Wahrscheinlichkeiten dieser drei Ereignisse sicherlich kleiner, als ein flötenspielender Braunbär in einer Pizzeria.

Also, jetzt reichts aber! Ich muss den Leser an dieser Stelle ernsthaft ermahnen, dass er sich mal konzentriere. Diese andauernd herumschweifenden Gedankengänge bringen mich als Schreiber völlig aus dem Konzept und machen den vorliegenden Text inkohärent und nur länger. Also den Teil, der noch kommt, der davor ist ja schon geschrieben und damit im Moment nicht mehr relevant. So fokussiere er sich endlich. Vielleicht trinke er ein Bier, esse einen Biber, um den Wald zu schützen, und lese dann weiter. ¹

Habe er sich jetzt wieder gesammelt? Gut. Also weiter. Man stelle sich nun vor, man komme in ca. 600m Grund in Porlezza an, versucht sich noch 10 Minuten am Hang zu halten und schaut sich währenddessen die Felder an. Sieht er das jetzt vor seinem geistigen Auge? Fühlt er die innere Anspannung ansteigen? Gut, das ist wichtig. So sei er nun hervorragend vorbereitet das Bild 1 unten zwecks Landefeldauswahl zu betrachten und der Echtzeitbeschreibung zu lauschen; Also soweit man mit den Ohren lesen kann.

Porelzza Landefelder: 1: OK für 15m Flieger Man beachte die Stahlstange und die Holzpflöcke, 2: Besseres, aber Baumreihe im Anflug, 3: noch besser, aber Weg mit Oma und Hund im Anflug, 4: am Besten, aber kostet 400 Euro wegen der Kühe. Blaue Anflugpfeile gemäss damals vorherschender Windrichtung, rot die Holzpflöcke/Eisenstange

Nun, welches nimmt man jetzt? Bei 400m fliege ich über die Felder und schau mir alles genau an. Zum Glück hab ich ‘ne neue Brille.

Feld 4, ja das wäre gut. Nur was sind das da für Tiere an der Seite? Kühe! Ne, nix mit vache. Eidechsen sind angesagt.

Ah ja, ein Windsack bei Feld 3 und ein kleines Häuschen. Das muss der Gleitschirmclub sein. Feld 1 wäre eine Option für einen Standardflieger, sieht gut vom Bewuchs aus, hat allerdings am Anfang rechts ein paar Holzpflöcke und irgendwas Orangenes im Anflug, aber nicht hoch. Wäre lang genug und das Offizielle halt. Und hat links einen Windsack, das muss ja für was Aeronautisches sein.

Oh, Nr 2 wäre auch gut, breiter, aber die Struktur des Feldes sieht komisch aus und es scheint mir etwas unduliert zu sein, aber gut landbar. Die Bäume im Anflug sind kein Problem, da pass ich durch, oder drüber mit ‘nem Slip. Nein, keine Unterhose, wir sind hier nicht im Porno! Nr 3 wäre die beste Wahl, langer hindernisfreier Anflug, gut im Wind, und am Anfang gemäht, das isses. Da kriegt man sogar ‘ne ASH runter.

Also Rad raus, abkreisen und nochmal alles anschauen. Bei ca 300m die Katastrophe. WAS IST DAS DA??? Oma mit Hund und beide bewegen sich synchron und langsam auf den Anflug von Feld 3 zu. Langsam aber sicher. Und was jetzt? Die Oma bleibt stehen, der Hund streift ziellos um sie herum. Und wahrscheinlich guckt die jetzt zu mir hoch und denkt:

“Oh ein Flugzeug! Wuffi, lass uns mal hier bleiben und schaun was der so macht.”

Hau ab mit deiner Dreckstöle! Rufen bringt nix, die hört dich eh nicht. Wahrscheinlich Hörgerät vergessen. Wat nu? Anfliegen und darauf hoffen, dass sie abhaut? Vielleicht kriegt sie noch einen Herzinfarkt und ich muss auf dem Hund landen. Auf ‘ner Eidechse ging ja, aber ein Hund und ‘ne Oma ist ‘ne ganz andere Nummer. Da stehste tagsdrauf in der Bildzeitung als islamistischer Terrorist. Hätte vorher meinen Bart absäbeln sollen, kommt auf die Checkliste.

200m, hau endlich ab!!!!!!!!!!!! Alte Leute mit Hund sollte man sofort verbieten, zumindest bei landbaren Feldern.

Die Alte bewegt sich nicht, der Hund schnüffelt unbeeindruckt herum und überlegt sich, ob er pinkeln soll. Also Alternativplan, den muss man immer haben. Entscheidung: Feld 1, 150m, im Queranflug schon Klappen etwas raus, keine Turbulenzen, also stabiler Endanflug mit 95, dann short final mit 90. Die orangene Stange! Da muss ich rüber und die AK setzt sich ca 30m hinter der Pflockreihe ins Gras (s.Bild 2 unten).

Also mit Minimalfahrt ins Feld und sofort bremsen. Bremsen? Zum Glück habe ich die am Tag vorher noch korrigiert. Die AK Gruppe hat die im Winter vorher so ziemlich verhunzt. Ansonsten hätte ich nicht bremsen können. Verdammte Hacke, Bremsen stellt man bei AUFGERÜSTETEN FLIEGER ein, ihr elenden Saftsäcke!

In so einer Situation funktioniert man einfach, da wird nix mehr nachgedacht. Alles läuft automatisch, Muskelgedächnis. Darum ist es so wichtig: Kleine Spannweite, niedrige Anfluggeschwindigkeit und gutes Training auf dem Typ. Mit ‘ner motorhavarierten AS34 wäre das wegen der Spannweite und der höheren Landegeschwindigkeit schwieriger gewesen.

Feld 1, man beachte die Holzpflöcke hinten links. Bei höherem Bewuchs Vorsicht!

Also Haube auf, erstmal Aussteigen, Photo machen, die Oma vermöbeln, den Hund in die Umlaufbahn schmeissen und dann den Eidechsen danken. Die Oma ist aber mittlerweile weitergezogen. Ja klar, es gibt nix mehr zu glotzen. Stattdessen kommt ein Auto mit ‘ner Frau drin: Tamara. Ihres Zeichens Gleitschirmfliegerin. Klasse!

Seil an die Anhängerkupplung, AK drehen, Seil in die Klinke würgen und das Ding rausziehen auf ihr Gelände, genau auf Feld 3. Und dann die Gretchenfrage von einem unbeteiligten deutschen Gleitschirmflieger, musste ja kommen:

“Warum biste nicht auf diesem Feld gelandet?”

Nun, der Oma Faktor. Gelächter…

Erstmal Max anrufen, zwecks Rückholung. Als nächstes, Kaffee und Kuchen fassen. Das scheint es grundsätzlich bei Gleitschirmfliegern zu geben, wenn sie abgesoffen sind. Also darüber lamentieren sie so lange, bis sich alle einig sind: Scheisstag! Dann kommt Max endlich an, schnell abrüsten und ab zum Restaurant, man hat ja kaum was im Magen.

In Bild 3 unten sieht man das Feld 3 vom Boden aus, wirklich schöner Anflug und relativ eben. Der Anflug in Gegenrichtung ist allerdings wegen des Weges Oma-Hund verseucht. Kann ich aber trotzdem empfehlen, das Landefeld, nicht das Oma-Hund Gespann.

Feld 3, gehört dem Paraclub, optimal, Blick Richtung Nord-Westen. freier Anflug von beiden Richtungen, quer dazu Feld 2

Also man merke: Wenn ‘ne Oma bei Feld 3 anschlappt, landet auf Feld 2. Wenn da auch ‘ne Oma auftaucht, auf Feld 1. Auf keinen Fall auf Feld 4 aufsetzen, das kostet 400 Euro, weil der grantige Bauer erstmal alle Kühe wegtreiben muss. Ein deutscher Segelflieger durfte das mal erleben, sagt Tamara. Achtet auf beide Windsäcke.

Es gibt noch ein Feld weiter nördlich, ist auch für ‘ne ASH geeignet. Warum ich da nicht gelandet bin. Nun, die Gleitschirmfliegerbutze, die transzendente Wahrnehmung von Kaffee und Kuchen und zwei Windsäcke waren entscheidende Faktoren. Der Windsack links von Feld 1 ist übrigens für einen privaten Helikopter gedacht. Der Sinn der orange angestrichenen Eisenstange in Feld 1 ist mir bis heute noch schleierhaft, vielleicht eine etwas eigenwillige Pistenbegrenzung.

Zusammenfassend kann man folgendes sagen:

Die Geschichte wird unser Zeitalter als völlig bescheuert klassifizieren. Nicht nur wegen der AS34 Kaufoption, sondern auch wegen Trump, Gaza- und Ukrainekrieg, Glühbirnen/Heizungen/Autos mit IP Stack, Katzenstreu, Kurbelwellenmontierte Ölpumpen, Facecrook, autonome nukleare Vorrichtungen, vierlagiges Klopapier, Ecoboost Motoren, soziale Medien, Frontantrieb, GPT4.5, Religionen, PTFE Dichtringe, usw.

Mit der technischen Integration der Eidechsen in unsere Gesellschaft wäre heute garantiert alles besser. Wir wären schon auf dem Mars, ich wäre nicht aussengelandet und wir hätten fliegende Autos und den Warp Drive. Man hat unsere Generation als Kinder in einschlägigen Science Fiction Serien schamlos belogen. Die Eidechsen, wären sie heute noch im Job, kämen nicht mehr aus dem Kacken raus. Viele Leute fühlen sich heutzutage unter anderem deswegen betrogen und abgehängt, darum wird auch alles brauner oder blauer. Das Resultat ist dasselbe: Scheisse.

Im Einklang mit dem Trumpschen Dekret zur spontanen Kausalitätsumkehr ↩

Reaping Poison Tasks In Celery

2025-08-04T00:00:00+02:00

Celery pipelines often work fine until one task brings everything to a halt. For us, the root cause was simple: when Kubernetes OOM-killed a Celery worker during execution, RabbitMQ would redeliver the unacknowledged task. In practice, this meant the same poison task (often memory-intensive) would bounce from worker to worker, gradually stalling the entire pipeline.

Task Queue Architecture

Our setup uses a one-queue-per-task-type model. Every Celery task is routed to its own RabbitMQ queue. This design ensures:

All tasks are treated with equal priority
Workers can subscribe to all tasks without any implicit weighting

The task-to-queue mapping is created dynamically at startup:

for task_name in app.tasks.keys():
    queue_name = f"celery:{task_name}"
    task_queues.append(Queue(queue_name, ...))
    task_routes[task_name] = {"queue": queue_name}

However, there’s a catch. Even with worker_prefetch_multiplier=1, Celery fetches one task per queue. If a worker listens to 100 queues, it may prefetch 100 tasks at once. When that worker is OOM-killed, all prefetched but unacknowledged tasks are redelivered, each with their delivery count incremented.

This results in two problems:

Poison tasks cause repeated worker crashes and restarts.
Healthy tasks also get redelivered unnecessarily, increasing delivery counts and wasting processing time.

Introducing Dead Letter Queues

To make task failure explicit and avoid stalling the pipeline, we introduced RabbitMQ Quorum Queues with x-delivery-limit and dead-lettering. The model now looks like this:

Each task-specific queue is a quorum queue with a delivery limit.
On reaching the delivery limit, tasks are routed to a graveyard queue.
The graveyard queue also has a delivery limit, after which tasks go to a final dead queue.

This setup ensures that poison tasks are automatically redirected out of the main processing flow.

Reaper Workers

To process the graveyard and dead queues, we introduced a dedicated class of workers:

They only subscribe to the graveyard and dead queues.
They run with worker_concurrency=1 and worker_prefetch_multiplier=1.
Graveyard tasks are re-executed in isolation - one at a time.
Dead queue tasks are never executed. They are skipped by immediately raising an exception.

This isolation is essential: the reaper worker only prefetches a single task from the graveyard, and because it subscribes to no other queues, there is no chance of delivery count pollution from co-scheduled tasks.

This design has several advantages:

Poison tasks no longer interfere with normal task execution.
Dead-lettering is now a first-class concept.
Tasks that exceed retry limits become visible failures, not silent stalls.

The task base class inspects RabbitMQ’s x-death headers to determine how many times a message has been dead-lettered. When the task sees multiple x-death entries, it is flagged as permanently dead and skipped.

class BaseTask(Task):
    def __call__(self, *args, **kwargs):
        x_death = self.request.headers.get("x-death", [])
        if len(x_death) > 1:
            raise DeadTask("This task has already been reaped.")
        return self.run(*args, **kwargs)

Why It Works

This architecture isolates failure from progress. Reaper workers handle the worst-case tasks without contaminating the main processing pool. Delivery limits and explicit dead letter queues prevent infinite retries. The one-queue-per-task model ensures fair scheduling, and now with dead-lettering, also ensures robustness under failure.

With this change, a single poison task no longer halts the system. Failures become visible, bounded, and isolated - the way they should be.

Convert Video For Chromecast

2022-04-20T00:00:00+02:00

Chromecast does only support a selection of audio and video codecs. I have used the following a lot to convert almost any video for streaming on the Chromecast:

$ ffmpeg \
  -i input.mp4 \
  -map 0:v:0 \
  -c:v copy \
  -map 0:a:0 \
  -map 0:a:0 \
  -c:a:0 aac \
  -ac:a:0 2 \
  -b:a:0 192k \
  -c:a:1 copy \
  output.mp4

This post is mostly just a copy&paste from stix.id.au. Just reposting this here to preserve the information. All credits to the original author.

Citation Needed

2021-01-25T00:00:00+01:00

Count Beta Release

2020-10-19T00:00:00+02:00

During COVID-19 times it is vital that we don’t gather in big groups. That puts the burden of tracking people in enclosed spaces onto the ones in charge. Counting is a lot of work with not much reward.

With Count I am beta Beta-releasing an app which wants to change that. Count helps you keep tack of things but then also provides visualizations of the data collected.

Yes, there are quite a few ¹ other counting apps on the Play Store already. Some other notable projects are:

But they are all not open source, lack important features (looking at the comment section) and don’t do much with the data they are collecting. This is where Count steps in.

Count is a simple tally counter with advanced visualizations based on metadata collected during counting. Every time the counter is changed the app collects time, location and position of the event. Using this data we can then provide insight about all different aspects such as frequency and locality.

Would you like to know when people visit which one of your branches? Count can tell you that.

With this blog post I am starting the open beta of the app. You can join now if you have an Android phone.

If want to contribute to Count, you can do this using the following channels:

Tweet:
Star: Star
Resolve: Issue
Donate: Sponsor

Thank you.

And by a few I mean a lot ↩

Disable reMarkable Touchscreen with evkill

2020-08-10T00:00:00+02:00

evkill is a silencer for evdev input devices. Run a single command and make your /dev/input/ devices go “psst!”. In this post we will use evkill on a reMarkable e-ink writing tablet to disable the capacitive display while writing.

The Problem

Since reMarkable has introduced page flips using swipe gestures, it happens a lot that I unintentionally change page while writing. They must have put some detection for this into their software, but for whatever reason this does not work for me on my device ¹. This is why I would like to be able to disable the capcitive sensor behind the screen by a button press. In a previous post I already showed how we can toggle flight mode with the hardware buttons. This time we will reuse the same script. But instead of switching network devices on and off, we will use evkill to disable input devices.

The Hack

First, we download the evkill armv7l build and upload the executable to the reMarkable.

$ curl https://raw.githubusercontent.com/Enteee/evkill/master/install.sh | ARCH="armv7l" sh
$ scp evkill root@10.11.99.1:.

Next, we cross compile and upload evtest.

$ git clone https://github.com/freedesktop-unofficial-mirror/evtest.git
$ cd evtest
$ docker \
  run --rm \
  dockcross/linux-armv7a > dockercross
$ chmod +x dockercross
$ ./dockercross bash -c "autoreconf -iv && ./configure --host=arm-linux-gnueabi && make"
$ scp evtest root@10.11.99.1:.

Given the device mapping from the table below, we adapt the script from the previous post.

Device	Name	Description
`/dev/input/event0`	Wacom I2C Digitizer	The electromagnetic resonance device for the pen
`/dev/input/event1`	cyttsp5_mt	The capacitive touchscreen
`/dev/input/event2`	gpio-keys	The hardware buttons

The idea is that the script detects button presses by listening on /dev/input/event2 using evtest. Once it registers a button press from the left and right hardware button it will start an evkill process in the background and disable /dev/input/event1. If we then again press both buttons, the script will terminate all running evkill instances with the effect of enabling the touchscreen again.

#!/usr/bin/env bash
set -euo pipefail

DEVICE_BUTTONS='/dev/input/event2'
DEVICE_TO_KILL='/dev/input/event1'

# commands
EVTEST=$(command -v evtest || echo './evtest')
EVKILL=$(command -v evkill || echo './evkill')

LEFT_DOWN=false
RIGHT_DOWN=false

toggle_evkill(){
  local evkill_pid
  evkill_pid="$(pidof evkill || true)"
  if [ -z "${evkill_pid}" ]; then
    echo "=> Disable touchscreen: ${EVKILL} ${DEVICE_TO_KILL}"
    "${EVKILL}" "${DEVICE_TO_KILL}" &
  else
    echo "=> Enable touchscreen: kill ${evkill_pid}"
    kill "${evkill_pid}" &>/dev/null
  fi
}

handle_events(){
  local key_left_down
  key_left_down='*type 1 (EV_KEY), code 105 (KEY_LEFT), value 1*'

  local key_left_up
  key_left_up='*type 1 (EV_KEY), code 105 (KEY_LEFT), value 0*'

  local key_right_down
  key_right_down='*type 1 (EV_KEY), code 106 (KEY_RIGHT), value 1*'

  local key_right_up
  key_right_up='*type 1 (EV_KEY), code 106 (KEY_RIGHT), value 0*'


  "$EVTEST" "$DEVICE_BUTTONS" | while read -r line; do
      case $line in
          ($key_left_down) LEFT_DOWN=true ;;
          ($key_left_up) LEFT_DOWN=false ;;
          ($key_right_down) RIGHT_DOWN=true ;;
          ($key_right_up) RIGHT_DOWN=false ;;
          (*) continue ;;
      esac
      echo "$line -> LEFT ${LEFT_DOWN} , RIGHT: ${RIGHT_DOWN}"

      if [ "${LEFT_DOWN}" = true ] && [ "${RIGHT_DOWN}" = true ]; then
        toggle_evkill
      fi
  done
}

handle_events

If we now save the script to an executable file called disable-touchscreen.sh and run it on the device, we have achieved what we wanted.

$ chmod +x disable-touchscreen.sh
$ scp disable-touchscreen.sh root@10.11.99.1:.
$ ssh -t -t root@10.11.99.1 ./disable-touchscreen.sh

Important: In case you omit the -t -t options to ssh, the script will keep running on the reMarkable even after you exit ssh with CRTL+C. This might cause some problems when the devices activates the lock screen.

Below a demonstration of how this looks like on an actual reMarkable:

Version 2.2.0.48 ↩

Support Binary File Objects with pandas.DataFrame.to_csv

2020-05-07T00:00:00+02:00

pandas.DataFrame.to_csv does not support writing to binary file objects ¹. This causes confusion ²³⁴⁵ and makes the function difficult to work with. In this article I will first illustrate the problem with an example. Then, I will present a monkey patch for pandas.DataFrame.to_csv which mitigates the known pitfall.

Let us write two files containing the names of three beautiful Swiss cities (in descending order).

data = ["Bern", "Genève", "Zürich"]

The files we will write are cities-utf-8.csv which is in utf-8 encoding an cities-latin.csv in ISO/IEC 8859-1 (latin).

DataFrame(data).to_csv("cities-utf-8.csv", index=False)
DataFrame(data).to_csv("cities-latin.csv", encoding="latin", index=False)

Everything works as expected and the files are written in utf-8 and latin encoding respectively.

$ file cities-*.csv
cities-latin.csv: ISO-8859 text
cities-utf-8.csv: UTF-8 Unicode text
$ hexdump -C cities-latin.csv
00000000  30 0a 42 65 72 6e 0a 47  65 6e e8 76 65 0a 5a fc  |0.Bern.Gen.ve.Z.|
00000010  72 69 63 68 0a                                    |rich.|
00000015
$ hexdump -C cities-utf-8.csv
00000000  30 0a 42 65 72 6e 0a 47  65 6e c3 a8 76 65 0a 5a  |0.Bern.Gen..ve.Z|
00000010  c3 bc 72 69 63 68 0a                              |..rich.|
00000017

So far so good. But what if we want to write those to a file object instead?

with open("cities-utf-8.csv", mode="w") as fd:
  DataFrame(data).to_csv(fd, index=False)

with open("cities-latin.csv", mode="w") as fd:
  DataFrame(data).to_csv(fd, encoding="latin", index=False)

Both files look suddenly very similar:

$ file cities-*.csv
cities-latin.csv: UTF-8 Unicode text
cities-utf-8.csv: UTF-8 Unicode text
$ hexdump -C cities-latin.csv
00000000  30 0a 42 65 72 6e 0a 47  65 6e c3 a8 76 65 0a 5a  |0.Bern.Gen..ve.Z|
00000010  c3 bc 72 69 63 68 0a                              |..rich.|
00000017
$ hexdump -C cities-utf-8.csv
00000000  30 0a 42 65 72 6e 0a 47  65 6e c3 a8 76 65 0a 5a  |0.Bern.Gen..ve.Z|
00000010  c3 bc 72 69 63 68 0a                              |..rich.|
00000017

In fact, they are exactly the same.

$ sha256sum cities-*.csv
ab401dff37c00f4d22e4ab2aa70fa2d67d89bd042787cb9e643bea7eeb5ee577  cities-latin.csv
ab401dff37c00f4d22e4ab2aa70fa2d67d89bd042787cb9e643bea7eeb5ee577  cities-utf-8.csv

Well, obviously. In order to be able to write a different encoding than utf-8 we have to open the files in binary mode:

with open("cities-utf-8.csv", mode="wb") as fd:
  DataFrame(data).to_csv(fd, index=False)

with open("cities-latin.csv", mode="wb") as fd:
  DataFrame(data).to_csv(fd, encoding="latin", index=False)

Which gives us a lovely: TypeError: a bytes-like object is required, not 'str'. Turns out that pandas.DataFrame.to_csv can not write to a binary file object. As a work around people have suggest just wrapping the file object in a io.TextIOWrapper:

with open("cities-utf-8.csv", mode="wb") as fd:
  DataFrame(data).to_csv(TextIOWrapper(fd), index=False)

with open("cities-latin.csv", mode="wb") as fd:
  DataFrame(data).to_csv(TextIOWrapper(fd), encoding="latin", index=False)

Which runs, but does not work because it also discards the encoding. On top of that, TextIOWrapper closes fd if we don’t call TextIOWrapper.detach before the wrapper is freed. Which brings us back to square one.

The best solution to this bug (or documentation issue) would be to create a pull request which implements support for binary file objects. Sadly, updating pandas is just not an viable option for me right now. Therefore I had to implement the following hackish solution which monkey patches pandas.DataFrame.to_csv so that the function supports binary file objects.

from contextlib import contextmanager
from threading import Semaphore

MONKEY_PATCH_DATAFRAME_TO_CSV = Semaphore()

@contextmanager
def monkey_patch__DataFrame_to_csv():
    """Monkey patch pandas.DataFrame.to_csv to make the function work with binary file objects.

    This is here because to work around the following issues:
      - https://github.com/pandas-dev/pandas/issues/9712
      - https://github.com/pandas-dev/pandas/issues/19827
      - https://github.com/pandas-dev/pandas/issues/13068
      - https://github.com/pandas-dev/pandas/issues/23854
    """

    import pandas as pd
    _DataFrame_to_csv_orig = pd.DataFrame.to_csv

    def _DataFrame_to_csv(self, path_or_buf, *args, **kwargs):
        from io import RawIOBase, BufferedIOBase, StringIO

        # Test if binary: https://stackoverflow.com/a/44584871
        if isinstance(path_or_buf, (RawIOBase, BufferedIOBase)):
            # Set line terminator of StringIO objec to DataFrame.to_csv's
            # default for line_termnator
            if int(pd.__version__.split(".")[1]) < 24:
                # the default was changed in pandas 0.24 version
                default_line_termnator = "\n"
            else:
                from os import linesep

                default_line_termnator = linesep

            # Note: Other than to_csv, StringIO will reject uncommon line terminators
            # by throwing an exception. But I think that's a good thing.
            sio = StringIO(
                newline=kwargs.get("line_terminator", default_line_termnator)
            )

            ret = _DataFrame_to_csv_orig(
                self,
                sio,
                *args,
                **{
                    **kwargs,
                    **{
                        # enforce utf-8, this is currently ignored because of:
                        # https://github.com/pandas-dev/pandas/issues/13068
                        # but if that issue ever gets fixed we prevent
                        # double-encoding with this.
                        "encoding": "utf-8",

                        # enforce mode, this is currently ignored because of:
                        # https://github.com/pandas-dev/pandas/issues/19827
                        # but if that issue ever gets fixed this patch
                        # should still work.
                        "mode": "w",

                        # always use Unix-style line_terminators. The conversion
                        # line_teminator will happen on StringIO.write
                        "line_terminator": "\n",
                    },
                }
            )

            path_or_buf.write(
                sio.getvalue().encode(
                    # encode to the specified encoding or fall back
                    # to pandas documented default
                    kwargs.get("encoding", "utf-8")
                )
            )
            return ret

        return _DataFrame_to_csv_orig(self, path_or_buf, *args, **kwargs)

    # apply monkey patch
    pd.DataFrame.to_csv = _DataFrame_to_csv

    # Since we are patching in global scope, we have to
    # ensure here that only one thread runs at the time.
    # Otherwise monkey patching and reverting the patch
    # can go really wrong...
    with MONKEY_PATCH_DATAFRAME_TO_CSV:
      try:
        yield _DataFrame_to_csv
      finally:
        # restore original to_csv
        pd.DataFrame.to_csv = _DataFrame_to_csv_orig

If we now use this monkey patch, we can finally generate the files

with monkey_patch__DataFrame_to_csv():

  with open("cities-utf-8.csv", mode="wb") as fd:
    DataFrame(data).to_csv(fd, index=False)

  with open("cities-latin.csv", mode="wb") as fd:
    DataFrame(data).to_csv(fd, encoding="latin", index=False)

… and get the expected content:

$ file cities-*.csv
cities-latin.csv: ISO-8859 text
cities-utf-8.csv: UTF-8 Unicode text
$ hexdump -C cities-latin.csv
00000000  30 0a 42 65 72 6e 0a 47  65 6e e8 76 65 0a 5a fc  |0.Bern.Gen.ve.Z.|
00000010  72 69 63 68 0a                                    |rich.|
00000015
$ hexdump -C cities-utf-8.csv
00000000  30 0a 42 65 72 6e 0a 47  65 6e c3 a8 76 65 0a 5a  |0.Bern.Gen..ve.Z|
00000010  c3 bc 72 69 63 68 0a                              |..rich.|
00000017

Citation Needed

2020-05-02T00:00:00+02:00

Types for PlantUML Parser

2020-02-25T00:00:00+01:00

plantuml-parser 0.0.12 introduces TypeScript declarations. Now you can parse PlantUML and get a fully typed result. PlantUML diagrams are awesome! In this post I first give a brief introduction into PlantUML. Then I will show how you can use plantuml-parser with TypeScript to make the most out of your diagrams.

She wouldn’t be my type either

Show source

@startuml
actor Alice
actor Bob

Alice -> Bob: Tells Bob that\n she does not like\ntypes
Bob --> Alice: Leaves her
@enduml

Show parser result

[
  {
    "elements": [
      {
        "left": "Alice",
        "right": "Bob",
        "leftType": "Unknown",
        "rightType": "Unknown",
        "leftArrowHead": "",
        "rightArrowHead": ">",
        "leftArrowBody": "-",
        "rightArrowBody": "-",
        "leftCardinality": "",
        "rightCardinality": "",
        "label": "Tells Bob that\\n she does not like\\ntypes",
        "hidden": false
      },
      {
        "left": "Bob",
        "right": "Alice",
        "leftType": "Unknown",
        "rightType": "Unknown",
        "leftArrowHead": "",
        "rightArrowHead": ">",
        "leftArrowBody": "-",
        "rightArrowBody": "-",
        "leftCardinality": "",
        "rightCardinality": "",
        "label": "Leaves her",
        "hidden": false
      }
    ]
  }
]

Why you should use PlantUML
Parsing PlantUML with TypeScript
Contribute

Why you should use PlantUML

PlantUML is an open-source tool allowing users to create UML diagrams from a plain text language. The language of PlantUML is an example of a Domain-specific language. It uses Graphviz software to lay out its diagrams.

– Source: wikipedia/PlantUML

I use PlantUML daily. A textual description of design diagrams alongside the source code is my definition of a living document. You know the pain of out of date documentation. Did you ever ask yourself why this happens? As a software developer I do reject a tool if it does not make my life easier. Writing sound and solid source code is already hard enough. Having to deal with a ton of different documentation formats makes it even harder. Belive me, if design changes can be documented with a few simple modifications of a text file, I will start doing it.

This has the effect that documentation evolves together with the code. Add version control to the mix and pull requests suddenly become self-documenting. Furthermore, switching to Graphviz means no longer spending hours layouting documents. Time better spent doing actual design work.

With PlantUML you can document without proprietary file mongering software or an overpriced vector graphic editor. In case you ask yourself what all that fuzz is about. You create your design documents in PowerPoint and you are happy with it - Then you should probably ask yourself when things started to go wrong in you life.

You will find another Alice

Show source

@startuml
(*) --> If "Do you know PlantUML?" then
  --> [No] "Learn PlantUML"
  If "" then
    --> [I don't\nhave time] "Leave Alice"
    --> "Learn PlantUML"
  else
    --> [Ok] "Good"
  EndIf
else
--> [Yes] "Good"
EndIf
--> "Parse it"
@enduml

You are now completely convinced and/or you already created a lot of documentation in PlantUML. Good. But at some point you might like to get even more from that the language. So why not start parsing it?

Parsing PlantUML with TypeScript

Parsing the syntax to something machine-processable is easy with the command line plantuml-parser.

Show example

$ npm install -g plantuml-parser
$ plantuml-parser <<EOF
> @startuml
> package "Wonderful world without Alice" {
>   file Doc
>   file JSON
>   interface PlantUML
>   component "plantuml-parser" as pp
>
>   Doc -right- PlantUML
>   PlantUML ..> pp
>   pp -left-> JSON
> }
> @enduml
> EOF
[
  {
    "elements": [
      {
        "name": "Wonderful world without Alice",
        "title": "Wonderful world without Alice",
        "type": "package",
        "elements": [
          {
            "name": "PlantUML",
            "title": "PlantUML",
            "members": []
          },
          {
            "name": "pp",
            "title": "plantuml-parser"
          },
          {
            "left": "Doc",
            "right": "PlantUML",
            "leftType": "Unknown",
            "rightType": "Unknown",
            "leftArrowHead": "",
            "rightArrowHead": "",
            "leftArrowBody": "-",
            "rightArrowBody": "-",
            "leftCardinality": "",
            "rightCardinality": "",
            "label": "",
            "hidden": false
          },
          {
            "left": "PlantUML",
            "right": "pp",
            "leftType": "Unknown",
            "rightType": "Unknown",
            "leftArrowHead": "",
            "rightArrowHead": ">",
            "leftArrowBody": ".",
            "rightArrowBody": ".",
            "leftCardinality": "",
            "rightCardinality": "",
            "label": "",
            "hidden": false
          },
          {
            "left": "pp",
            "right": "JSON",
            "leftType": "Unknown",
            "rightType": "Unknown",
            "leftArrowHead": "",
            "rightArrowHead": ">",
            "leftArrowBody": "-",
            "rightArrowBody": "-",
            "leftCardinality": "",
            "rightCardinality": "",
            "label": "",
            "hidden": false
          }
        ]
      }
    ]
  }
]

But today we want to have a look at the programmatic use of the parser. Therefore I created the following demonstration which shows how easy it is to parse PlantUML in TypeScript. The demonstration also contains an example on how type guards can leverage processing of diagrams.

Show source

let diag = `
@startuml
class Alice <<single>> {
  +cry(): any
}
note left of Alice::cry
Untyped. She
does not like types
end note
@enduml
`

// importing the parser
import {parse, Class, Note} from 'plantuml-parser';

// for each diagram parsed
for ( let d of parse(diag) ){
  // extract all elements
  let elems = d.elements;
  for ( let e of elems ) {
    // we can use type guards to
    // distinguish between the different
    // elements in the diagram
    if ( e instanceof Class) {
      console.log('Class: '+ e.name);
    }
    if ( e instanceof Note ){
      console.log('Note on '+e.of);
    }
  }
}

Contribute

If you do like the project and you would like to contribute, there are numerous ways how you can do so. Even if you do not write source code. Every contribution counts.

Tweet:
Star: Star
Resolve: Issue
Donate: Sponsor

Thank you.

reMarkable

2020-01-08T00:00:00+01:00

I got my reMarkable recently and writing a few words about it will possibly not hurt anyone. This is not a product review, I much rather try to focus on technical aspects. Such as packaging the software and analyzing some of its features.

Sketching and writing on the device

Feature Overview
Software for the reMarkable and Some NixOS Derivations
Hacking Hardware Button Flight Mode Toggling
Conclusion

Feature Overview

reMarkable is an electronic ink tablet designed for writing. E Ink writing tablets promise excellent writing experience and a long battery lifetime. Which should make them a good replacement for paper. There are a few competitors on that market. For example the Ratta SuperNote, heavily featured on goodereader¹.

My main use-cases for the tablet are to-do lists, meeting notes, mind maps, user interface mock-ups and ugly sketches. I bought an e ink tablet because I was fed up with manually digitalizing paper. I finally chose the reMarkable because all of the developers seem to be European cats and the ecosystem is hackable to at least some degree. Also they release frequently which is a big plus.

IANAA - I am not an artist

The device connects to the reMarkable cloud which has the main focus on document sharing and backup. There is an App for Android / iOS as well as a client for Windows and Mac OS. Sadly, the Linux client was discontinued in late 2017 ². For cross operating system compatibility the device can serve its files using a built in web server accessible via USB. When connected to the cloud, the device has some optical character recognition (OCR) capabilities and can convert documents to scalable vector graphics (SVG). The live view feature would be amazing but also requires the native QT app on the receiving end.

Software for the reMarkable and Some NixOS Derivations

In this section I am looking at software written for the reMarkable and because NixOS is awesome I also tried to create some derivations.

The Official Linux Client

Using the NixOS packaging guideline for QT, and the following script:

#!/usr/bin/env bash
set -euo pipefail

binary="${1?Missing binary}"

while IFS= read -r lib
do
  if ldd "${binary}" | grep "not found" | grep -q "$lib";  then
    echo "=> ${lib}"
    nix-locate -1 -w "lib/${lib}"
    echo
  fi
done < <(patchelf --print-needed "${binary}")

I was able to quickly pin down all the dependencies and patch the distributed executable with NixOS specific rpaths ³. With the resulting derivation remarkable-linux-client.nix I can now run the client inside a nix-shell:

$ nix-shell \
  --command reMarkable \
  -p $( \
    nix-build \
    --quiet \
    -E '(import (builtins.fetchGit {
        url = "https://github.com/nixos/nixpkgs/";
        ref = "master";
        rev = "471869c9185fb610e67940a701eb13b1cfb335a4";
    }) {}).qt5.callPackage ./remarkable-linux-client.nix {}' \
  )

The application starts and connects to the cloud and all other features ⁴ work just fine. But I ran into big problems when displaying any of the drawings. All notebooks are completely empty. Without success I have checked the whole log for any hints which might indicate issues related to the packaging.

My best guess is that the reMarkable developers have changed the proprietary file format and the extremely outdated Linux client is no longer able to read that format properly. I got to this conclusion because the app logs the following when displaying a a drawing:

Redrawing lines...
0 Lines starting at 0 of total 0
Redrawing lines completed in 0 ms
xochitl.documentworker: Storing page... 0

But this is just a guess. If you have a better idea what might be going wrong, I am curious to hear about them in the comment section. Having spent quite a few hours on this issue, I finally gave up getting the Linux client to work. Therefore I started looking for open source alternatives:

Accessing the reMarkable API With rMAPI

From the README.md:

rMAPI is a Go app that allows you to access the ReMarkable Cloud API programmatically.

You can interact with the different API end-points through a shell. However, you can also run commands non-interactively. This may come in handy to script certain workflows such as taking automatic backups or uploading documents programmatically.

In short a great tool! Creating a derivation and using it under NixOS was easy. I opened a pull request to share my work with the rMAPI project. Then I created another pull request which adds this derivation to nixpkgs.

Lessons Learned: My initial Idea was to keep the full derivation in the rMAPI repository and just use that repository in the nix-expression added to nixpkgs. This approach did not work because it requires import from derivations (IFD), which are currently disabled in hydra ⁵.

With rMAPI I can now easily replicate the file sharing aspects of the native Linux client. But how can we get the live view to work?

Streaming the Framebuffer with srvfb

The idea is simple. We grab the framebuffer from the reMarkable and send it back to the computer where we then render an image. In its most basic shape this can be a ⁶:

$ ssh root@10.11.99.1 "cat /dev/fb0" | \
  ffmpeg -vcodec rawvideo \
         -f rawvideo \
         -pix_fmt gray16le \
         -s 1408,1872 \
         -i - \
         -vframes 1 \
         -f image2 \
         -vcodec mjpeg /tmp/frame.jpg

The amazing project srvfb took this idea to the next level. From the README.md:

This repository contains a small webserver that can serve the contents of a Linux framebuffer device as video over HTTP. The video is encoded as a series of PNGs, which are served in a multipart/x-mixed-replace stream. The primary use case is to stream a reMarkable screen to a computer and share it from there via video-conferencing or capturing it. For that reason, there is also a proxy-mode, which streams the frames as raw, uncompressed data from the remarkable and can then do the png-encoding on a more powerful machine. Whithout that, the framerate is one or two frames per second, which might not be acceptable (it might be, though).

Running this on the reMarkable is super easy. The result is great and even the documented lag in non-proxy mode is acceptable. Problem solved. Thank you Merovius, keep up the good work!

My last pain point with the reMarkable is the missing hardware button for flight mode toggling. Because of battery lifetime I keep flight mode always on. But when I occasionally do need internet, the flight mode option in the settings is always a million clicks away.

Hacking Hardware Button Flight Mode Toggling

I could not find a tool to get the flight mode working via button press, so I built something on my own. But before doing so, I familiarized myself with the internals of the device. For this I had to become root first. Good news! It is possible and super easy:

Connect the tablet to your computer via USB.
Get the IP address as well as the root password from the about page.
ssh, done!

$ ssh root@10.11.99.1
ｒｅＭａｒｋａｂｌｅ
╺━┓┏━╸┏━┓┏━┓   ┏━╸┏━┓┏━┓╻ ╻╻╺┳╸┏━┓┏━┓
┏━┛┣╸ ┣┳┛┃ ┃   ┃╺┓┣┳┛┣━┫┃┏┛┃ ┃ ┣━┫┗━┓
┗━╸┗━╸╹┗╸┗━┛   ┗━┛╹┗╸╹ ╹┗┛ ╹ ╹ ╹ ╹┗━┛

remarkable: ~/ uname -a
Linux remarkable 4.9.84-zero-gravitas #1 Thu Jun 27 14:19:15 UTC 2019 armv7l GNU/Linux

remarkable: ~/ cat /proc/cpuinfo
processor         : 0
model name        : ARMv7 Processor rev 10 (v7l)
BogoMIPS          : 24.00
Features          : half thumb fastmult vfp edsp neon vfpv3 tls vfpd32
CPU implementer   : 0x41
CPU architecture  : 7
CPU variant       : 0x2
CPU part          : 0xc09
CPU revision      : 10

Hardware          : Freescale i.MX6 SoloLite (Device Tree)
Revision          : 0000
Serial            : ****************

The remarkable File system structure is partially documented on the remarkable wiki. Some other locations I found interesting:

/home/root/.local/share/remarkable/xochitl/: Your documents and metadata.

/etc/remarkable.conf: Passwords and keys.

remarkable: ~/ grep -r Password /etc/remarkable.conf
DeveloperPassword=*** ROOT PASSWORD (masked) ***
Password=*** Screen lock password (masked) ***

remarkable: ~/ sed -n '/wifinetworks/,$p' /etc/remarkable.conf
[wifinetworks]
SSID1=@Variant(*** Key for Wifi network (masked) ***)
SSID2=@Variant(*** Key for Wifi network (masked) ***)

Enough of the playtime, let’s get cracking. Again, my goal is to toggle flight mode when the left and right button are both pressed at the same time. For this I have to listen on evdev events and once both buttons are pressed run rfkill block all to enable flight mode or rfkill unblock all to disable it again.

In my simple approach I am using evtest to parse the binary events from /dev/input/event*. First I cross compiled evtest with dockercross for the 32 bit ARMV7-A and copied the resulting binary to the reMarkable.

$ git clone https://github.com/freedesktop-unofficial-mirror/evtest.git
$ cd evtest
$ docker \
  run --rm \
  dockcross/linux-armv7a > dockercross
$ chmod +x dockercross
$ ./dockercross bash -c "autoreconf -iv && ./configure --host=arm-linux-gnueabi && make"
$ scp evtest root@10.11.99.1:.

evdev exposes three different event sources on the reMarkable:

Device	Name	Description
`/dev/input/event0`	Wacom I2C Digitizer	The electromagnetic resonance device for the pen
`/dev/input/event1`	cyttsp5_mt	The capacitive touchscreen
`/dev/input/event2`	gpio-keys	The hardware buttons

The following bash script uses the just compiled evtest binary to fetch and handle button press events on /dev/input/event2. With rfkill list | grep -q "Soft blocked: no" it then detects if flight mode is disabled and calls out to rfkill accordingly.

#!/usr/bin/env bash
set -euo pipefail

DEVICE='/dev/input/event2'
EVTEST=$(command -v evtest || echo './evtest')
RFKILL=$(command -v rfkill)

LEFT_DOWN=false
RIGHT_DOWN=false

toggle_flight_mode(){
  if "$RFKILL" list | grep -q "Soft blocked: no"; then
    "$RFKILL" block all
  else
    "$RFKILL" unblock all
  fi
}

handle_events(){
  local key_left_down
  key_left_down='*type 1 (EV_KEY), code 105 (KEY_LEFT), value 1*'

  local key_left_up
  key_left_up='*type 1 (EV_KEY), code 105 (KEY_LEFT), value 0*'

  local key_right_down
  key_right_down='*type 1 (EV_KEY), code 106 (KEY_RIGHT), value 1*'

  local key_right_up
  key_right_up='*type 1 (EV_KEY), code 106 (KEY_RIGHT), value 0*'


  "$EVTEST" "$DEVICE" | while read -r line; do
      case $line in
          ($key_left_down) LEFT_DOWN=true ;;
          ($key_left_up) LEFT_DOWN=false ;;
          ($key_right_down) RIGHT_DOWN=true ;;
          ($key_right_up) RIGHT_DOWN=false ;;
          (*) continue ;;
      esac
      echo "$line -> LEFT ${LEFT_DOWN} , RIGHT: ${RIGHT_DOWN}"

      if [ "${LEFT_DOWN}" = true ] && [ "${RIGHT_DOWN}" = true ]; then
        toggle_flight_mode
      fi
  done
}

handle_events

done!.. One downside coming from directly using rfkill, is that the user interface does not properly detect this state change. Which means we can get the reMarkable into weird states where the device is connected to a network with the flight mode icon on. Rebooting the device recovers the clean state again. But let us call this a feature, shall we?

Conclusion

After having used the reMarkable for a few weeks now I am still very satisfied with the product. The overall user experience is very good. And the graphite tips don’t wear out too fast. For the future I would like to see the following features implemented:

toggle flight mode with a hardware switch
mixed PDF and sheet notebooks
basic shapes such as circles, squares and lines

Maybe we get all this with the reMarkable2. It seems that a new device is on its way. reMarkable has filed a request for certification by the FCC (ID: 2AMK2-RM110) which contains a wealth of pictures and specs. But in a letter correspondence from the 22. November 2019 they requested the dismissal of that FCC ID. Whatever that means…

Wrapping up, the reMarkable is a perfect example how open devices can encourage a community to make so much more out of a already great product. I sincerely hope that the company stays as open as it is right now ⁷ and no money hungry manager or acquiring company destroys what they built so far.

The platform is so unnaturally biased towards that alternative, leading me into questioning their independence. We all need money to support our life. But a bit more transparency, especially when your main focus are product reviews, would be nice. ↩
This needs to change! ↩
20.03pre199897.471869c9185 ↩
Except the most important one… ↩
I also recommend reading this excellent article by Domen Kožar ↩
Shamelessly stolen from canselcik/libremarkable Wiki ↩
I can haz xochitl github repo? ↩

Citation Needed

2019-11-08T00:00:00+01:00

udev Rule Script Template

2019-10-25T00:00:00+02:00

When writing shell scripts invoked by udev rules, I find the following template particularly useful:

#!/usr/bin/env bash
set -euo pipefail

CMD="`basename "${0:-udevscript}"`"
exec 1> >(tee >(logger -t "${CMD}"))
exec 2> >(ts '[stderr]' | tee >(logger -t "${CMD}"))

DEBUG="${1:-false}"
if [ "${DEBUG}" = true ]; then set -x; fi

echo "running: ${0}"
env

## Actual script starts here

We can then use such a script in a udev rule like we always would:

SUBSYSTEM=="usb", \
  ACTION=="remove", \
  ENV{ID_VENDOR_ID}=="17e9", \
  ENV{ID_MODEL_ID}=="6015", \
  RUN+="/usr/lib/udev/scripts/undock.sh"

And when the rule runs the script, everything printed is sent to the system log. Stderr messages will be prefixed with [stderr]. Which makes checking what the script does as simple as:

$ journalctl -e -t undock.sh
Oct 25 22:58:35 puddle undock.sh[32366]: running: /usr/lib/udev/scripts/undock.sh
Oct 25 22:58:35 puddle undock.sh[32366]: ID_SERIAL=DisplayLink_ThinkPad_Hybrid_USB-C_with_USB-A_Dock_10391787
Oct 25 22:58:35 puddle undock.sh[32366]: ID_MODEL_ID=6015
Oct 25 22:58:35 puddle undock.sh[32366]: ACTION=remove
...

The template also supports debug output. Setting the first argument to true will print commands and their arguments as they are executed to stderr.

Bonus: A Nix Expression

The script above is derived from the following nix expression.

{ pkgs ? import <nixpkgs> {}, ... }:

let

  logger = "${pkgs.utillinux}/bin/logger";
  ts = "${pkgs.moreutils}/bin/ts";
  tee = "${pkgs.coreutils}/bin/tee";

in name : script: {
    strict ? true,
    log ? true,
    debug ? false,
    printenv ? true,
  }: let

    strictCmd = if strict then
      ''
      set -euo pipefail
      ''
    else "";

    logCmd = if log then
      ''
      exec 1> >(${tee} >(${logger} -t "${name}"))
      exec 2> >(${ts} '[stderr]' | ${tee} >(${logger} -t "${name}"))
      ''
    else "";

    debugCmd = if debug then
      ''
      set -x
      ''
    else "";

    printenvCmd = if printenv then
      ''
      echo "running: ${name}"
      env
      ''
    else "";

  in pkgs.writeShellScript name
    ''
      ${strictCmd}
      ${logCmd}
      ${debugCmd}
      ${printenvCmd}

      ${script}
    ''

If invoked, the expression writes such a script to the nix store.

nix-repl> writeLoggedScript = import ./writeLoggedScript.nix {}

nix-repl> :b writeLoggedScript "test.sh" "echo this is just a test" {}

this derivation produced the following outputs:
  out -> /nix/store/bzjllwyhy0zwm2f352a0gya8fp13qc7q-test.sh

Migrate a git Repository with Submodules

2019-07-08T00:00:00+02:00

With git-sync-mirror migrating or mirroring a git repository is a piece of cake. But when the migrated repository contains submodules additional steps are required.

The Problem

Submodule references are not updated when mirroring git repositories. This is because those references are tracked in a file called .gitmodules inside the repository. This file is not changed during migration.

As an example imagine you want to mirror the githubtraining/example-dependency repository to Enteee/example-dependency. Using git-sync-mirror this is simple:

$ GITHUB_USER="Enteee"
$ GITHUB_ACCESS_TOKEN="<hidden>"
$ docker run \
  --rm \
  --env ONCE=true \
  --env SRC_REPO="https://github.com/githubtraining/example-dependency.git" \
  --env DST_REPO=https://${GITHUB_USER}:${GITHUB_ACCESS_TOKEN}@github.com/Enteee/example-dependency.git \
  enteee/git-sync-mirror

The just mirrored githubtraining/example-dependency repository contains one submodule at js. This submodule references the githubtraining/example-submodule repository.

$ git clone https://github.com/githubtraining/example-dependency.git
Cloning into 'example-dependency'...
remote: Enumerating objects: 39, done.
remote: Counting objects: 100% (39/39), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 39 (delta 5), reused 39 (delta 5), pack-reused 0
Unpacking objects: 100% (39/39), done.
$ cat example-dependency/.gitmodules
[submodule "js"]
  path = js
  url = https://github.com/githubtraining/example-submodule.git

For a full migration we now also have to mirror githubtraining/example-submodule to Enteee/example-submodule. We can do this exactly the same way as we did it before with githubtraining/example-dependency. But since we were creating 1:1 mirrors, the first repository Enteee/example-dependency still points to githubtraining/example-submodule. This is probably not what we want. The repository Enteee/example-dependency should point to Enteee/example-submodule instead.

The Solution: git-submodule-url-rewrite

git-submodule-url-rewrite is a git command that lets you rewrite submodule urls. Installing the command is as simple as copying the script somewhere to your ${PATH} and making it executable.

$ cd /usr/local/bin
$ curl \
  --output git-submodule-url-rewrite \
  https://raw.githubusercontent.com/Enteee/git-submodule-url-rewrite/master/git-submodule-url-rewrite
$ chmod a+x git-submodule-url-rewrite

Now, git-submodule-url-rewrite should be available as a git command.

$ git submodule-url-rewrite -h
usage: git submodule-url-rewrite [-h|--help] [-v|--verbose] [-q|--quiet] [-r|--recursive] [-s|--no-stage] [-u|--no-update] sed-command

Rewrites all submodule urls using the given sed-script

options:
  -h|--help       Print this help
  -v|--verbose    Make this script verbose
  -q|--quiet      Don't print anything
  -r|--recursive  Also rewrite submodules of submodules
  -s|--no-stage   Don't stage changed .gitmodule files for commit
  -u|--no-update  Don't run 'git submodule --quiet update --init' in each submodule

sed-command: A sed command used to transform urls.

Using this command we can now rewrite all urls in Enteee/example-dependency. We just have to use a substituting sed command (s) which replaces all githubtraining strings with Enteee. The following command does this: 's|githubtraining|Enteee|'. For more information about sed commands I recommend this tutorial on computerhope.com.

$ cd example-dependency/
$ git submodule-url-rewrite 's|githubtraining|Enteee|'
rewrite url for submodule 'js' in '/tmp/example-dependency' from 'https://github.com/githubtraining/example-submodule.git' to 'https://github.com/Enteee/example-submodule.git'

git commit && git push, done!

Why should I use this?

In this section I try to answer a few common questions. Keep in mind that git-submodule-url-rewrite is a very simple git command which I found useful in the past. Hence, I decided to open source it. Nobody forces you to use it. If you find good reasons not to, then don’t. In order to improve git-submodule-url-rewrite, I would still be interested in those reasons. Why not share them here?

Rewriting a a submodule url is a simple as:
$ git config --file .gitmodules submodule.js.url 'https://github.com/Enteee/example-submodule.git'
$ git submodule sync
why should I use git-submodule-url-rewrite?

– unknown git user

This is exactly what git-submodule-url-rewrite does. No magic involved. But in addition, git-submodule-url-rewrite provides the convenience of regex and a recursive ([-r|--recursive]) switch. Using recursion you can simply rewrite submodules of submodules of submodules of … You get the idea.

But I can just implement the same recursion with git submodule foreach --recursive 'git config ...'.

– the same unknown git user

Yes. Almost. Please note that git submodule foreach evaluates an arbitrary shell command in each checked out submodule ¹. This means you have to run git submodule update --init --recursive first. Which will connect and clone to the originally referenced repository. This was not possible in my environment. Also, looping over all submodules in a shell script, without git submodule foreach, is not trivial ²³. Hence, I had to implement a looping mechanism which does not rely on git submodule foreach.

By open sourcing git-submodule-url-rewrite, I do hope I can provide functionality which maybe does help others. If you find a bug or have a feature request, please open an issue on GitHub. All comments and thoughts are welcome here on this page.

From the manpage ↩
See: this answer on StackOverflow ↩
Maybe I should implement a command that does just that at some point. ↩

Citation Needed

2019-05-27T00:00:00+02:00

TOFU for Containers

2019-05-07T00:00:00+02:00

Running containers behind a HTTPS scanning proxy can be tricky. The proxy will send a certificate which is not trusted by the container with the effect of breaking the internet.

There are three possible ways to make the internet work again:

Disable SSL/TLS certificate chain verification
Forward and install the scanning certificate
Implement Trust On First Use (TOFU)

In case you decide to disable certificate chain verification, I hope your code reviewing fellow just silently gets up from his desk and punches you in the face. Because you deserve it. NEVER EVER DO THIS!

On the other hand, doing the right thing means forwarding and installing the certificate inside the container. In order to achieve this there are steps needed by the image maintainer as well as the poor soul running the container. Wouldn’t it be nice if you didn’t need to worry about certificates when deploying a container?

TOFU can be a good trade-off between security and usability. The idea is simple: Early on we try to reach out to the internet and simply trust every certificate that we get in response. TOFU then caches those certificate and ensures that subsequent connections are secure.

Well, yes, this is not perfectly secure as well. An attacker could make you trust a certificate if they are able to intercept the very first connection attempt made by TOFU. But in practice this deemed to be quite difficult. Especially for long running containers. Also there are other protocols which implement TOFU successful.

$ ssh duckpond.ch
The authenticity of host '[duckpond.ch]:7410 ([71.19.149.209]:7410)' can't be established.
ECDSA key fingerprint is SHA256:H28klEV+VMEwPst7POeHYzAGUrvfb15SRwZ/RnqhtDI.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[duckpond.ch]:7410,[71.19.149.209]:7410' (ECDSA) to the list of known hosts.

TOFU Inside a Container

What we are trying to do is:

TLS handshake with the server (or a proxy) and download all certificates.
Install the certificates.

The first step is easy, openssl s_client does most of the heavy TLS lifting for us. I am not a massive fan of having OpenSSL installed in containers. But in this case this is probably the right approach. Most applications build on top of that library anyways. Using OpenSSL we implement tls-tofu.sh:

#!/usr/bin/env sh
set -exuo pipefail
openssl s_client -showcerts ${@} 2>/dev/null < /dev/null \
| sed -n '/-----BEGIN/,/-----END/p'

Now we can easily print all the certificates sent to us either by the server or a transparent proxy:

$ ./tls-tofu.sh -connect duckpond.ch:443 -servername duckpond.ch
+ openssl s_client -showcerts -connect duckpond.ch:443 -servername duckpond.ch
+ sed -n /-----BEGIN/,/-----END/p
-----BEGIN CERTIFICATE-----
MIIGcjCCBVqgAwIBAgISA5P3qUe9JqxFqESJKhzHY/neMA0GCSqGSIb3DQEBCwUA
...
QraNjGOLb8+mDxQItPL5EymbmdPrdA==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
...
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----

In order to make the system trust those certificates, we need to store them in /etc/ssl/certs/ca-certificates.crt. But this file is only writable by root and we hopefully don’t have those permissions when running scripts inside a container.

This problem is solvable with kamikaze. kamikaze is a simple setuid binary which allows us to run a command as root once. Using the power of kamikaze we can now add trusted certificates.

#!/usr/bin/env sh
set -exuo pipefail
openssl s_client -showcerts ${@} 2>/dev/null < /dev/null \
| sed -n '/-----BEGIN/,/-----END/p' \
| /kamikaze tee -a /etc/ssl/certs/ca-certificates.crt > /dev/null

enteee/tls-tofu Container Image

Based on the tls-tofu.sh-idea, I did create the tls-tofu GitHub project and published enteee/tls-tofu container images. Building and running your own TLS-TOFU enabled image is as simple as:

$ docker build -t tls-tofu-enabled-image - <<EOF
  FROM enteee/tls-tofu
  # IMPORTANT: Drop privileges
  USER nobody

  # Run the application
  CMD ["echo", "Hello World!"]
EOF
$ docker run -ti tls-tofu-enabled-image
Hello World!

In a more elaborate example, we can use the just built image to run a container which trusts the self-signed BadSSL certificate.

$ docker run \
  -ti \
  -e TLS_TOFU_HOST="self-signed.badssl.com" \
  tls-tofu-enabled-image \
  curl https://self-signed.badssl.com/

This should first print certificate information:

CONNECTED(00000003)
---
Certificate chain
 0 s:C = US, ST = California, L = San Francisco, O = BadSSL, CN = *.badssl.com
   i:C = US, ST = California, L = San Francisco, O = BadSSL, CN = *.badssl.com
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPlgiuOcJ/T1MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
BAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNp
c2NvMQ8wDQYDVQQKDAZCYWRTU0wxFTATBgNVBAMMDCouYmFkc3NsLmNvbTAeFw0x
ODA4MTUxNTIxNTNaFw0yMDA4MTQxNTIxNTNaMGIxCzAJBgNVBAYTAlVTMRMwEQYD
VQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNpc2NvMQ8wDQYDVQQK
DAZCYWRTU0wxFTATBgNVBAMMDCouYmFkc3NsLmNvbTCCASIwDQYJKoZIhvcNAQEB
BQADggEPADCCAQoCggEBAMIE7PiM7gTCs9hQ1XBYzJMY61yoaEmwIrX5lZ6xKyx2
PmzAS2BMTOqytMAPgLaw+XLJhgL5XEFdEyt/ccRLvOmULlA3pmccYYz2QULFRtMW
hyefdOsKnRFSJiFzbIRMeVXk0WvoBj1IFVKtsyjbqv9u/2CVSndrOfEk0TG23U3A
xPxTuW1CrbV8/q71FdIzSOciccfCFHpsKOo3St/qbLVytH5aohbcabFXRNsKEqve
ww9HdFxBIuGa+RuT5q0iBikusbpJHAwnnqP7i/dAcgCskgjZjFeEU4EFy+b+a1SY
QCeFxxC7c3DvaRhBB0VVfPlkPz0sw6l865MaTIbRyoUCAwEAAaMyMDAwCQYDVR0T
BAIwADAjBgNVHREEHDAaggwqLmJhZHNzbC5jb22CCmJhZHNzbC5jb20wDQYJKoZI
hvcNAQELBQADggEBAKr7JtZHTDuYs8/vGDFrtXb+dkjdNsZEIgyVh4vWZtLOANtO
39wM/LwGXUSjonEsYJabJgYpRdRSex41f78QfnARJona7fkcc1aHci7jdrzsxaNJ
iCc4G49ahgJ1NEIFmRNeEYlKYNNFeyGT6wxkLaV9AnC45MHlaumQyrRJwuXCQH/i
16Wk/qDtsu2nw6t+13OqwGfxR9krxDikVFO0YqgSMhqPmufz/6nY6uaXuOqzGv+P
rjJZDqCoRmVMqrISIUALWGCF3yasrViM6owIEhtN71UwrFZYYOeZ9nw2wvRK210z
c8LlWjgG56wRkLrq/mSINsQ3xmChO1PsBAeSHDU=
-----END CERTIFICATE-----
---
Server certificate
subject=C = US, ST = California, L = San Francisco, O = BadSSL, CN = *.badssl.com

issuer=C = US, ST = California, L = San Francisco, O = BadSSL, CN = *.badssl.com

---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1599 bytes and written 450 bytes
Verification error: self signed certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 315114D851A9EE6B159B59BE5CF639C36FD4F6F40F8F53B8601FE9305E3DF8F1
    Session-ID-ctx: 
    Master-Key: A5C589C7A5739E39046574BA5AEC4D130F44E489DF77C7D627F35DF91A75E37FBFC3C5F20907A816692F8E46A765A775
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 8a a2 1a 77 48 87 f2 35-55 9c a0 3f 71 4d 37 a7   ...wH..5U..?qM7.
    0010 - 64 49 98 47 22 64 dc 00-99 89 2d ed 98 41 49 09   dI.G"d....-..AI.
    0020 - fc b8 1c f2 6e 08 5e 0d-28 9e 44 2f 3e df 34 9e   ....n.^.(.D/>.4.
    0030 - fa a5 f0 ca 1e dc b5 af-c0 54 cc 63 e8 1a a2 6d   .........T.c...m
    0040 - 41 61 e5 21 f9 82 68 39-4e 2f d6 9f 5e 21 7b 8f   Aa.!..h9N/..^!{.
    0050 - 06 9e 16 76 38 b6 08 12-ab ed ee 5b e5 e7 a5 eb   ...v8......[....
    0060 - 66 97 8b a8 fa fc d0 1f-aa 4d 53 6a 6f d6 07 df   f........MSjo...
    0070 - 5e 70 fe 72 18 82 38 c8-c8 c4 10 e5 05 b5 5f c6   ^p.r..8......._.
    0080 - ff cb d0 01 18 a8 66 9f-01 1c bd 2c 99 1c cc 14   ......f....,....
    0090 - 3d 37 c9 bb 4b 1d 1b 1a-ba 7d fd 15 19 e2 3a b5   =7..K....}....:.
    00a0 - 33 e9 68 d9 fa 20 55 ff-2f f1 10 a1 1c 88 e7 8f   3.h.. U./.......
    00b0 - 6c 72 e9 b6 bc 12 45 36-a7 d4 80 92 ea 82 34 b6   lr....E6......4.
    00c0 - fc bd ab 91 b8 8f aa 5a-a6 55 95 ae 23 5b b5 7b   .......Z.U..#[.{

    Start Time: 1557231372
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: no
---

And then the self-signed BadSSL page.

<!DOCTYPE html>
<html>
<head>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="shortcut icon" href="/icons/favicon-red.ico"/>
  <link rel="apple-touch-icon" href="/icons/icon-red.png"/>
  <title>self-signed.badssl.com</title>
  <link rel="stylesheet" href="/style.css">
  <style>body { background: red; }</style>
</head>
<body>
<div id="content">
  <h1 style="font-size: 12vw;">
    self-signed.<br>badssl.com
  </h1>
</div>

</body>
</html>

Let us make the last step, and simulate the HTTPS scanning proxy scenario. For this we first start mitmproxy/mitmproxy:

$ docker run \
  -ti \
  --name mitmproxy \
  mitmproxy/mitmproxy mitmdump
Proxy server listening at http://*:8080

And then connect to duckpond.ch through the proxy:

$ docker run \
  -ti \
  --link mitmproxy \
  -e TLS_TOFU_HOST="mitmproxy" \
  -e TLS_TOFU_PORT="8080" \
  -e TLS_TOFU_S_CLIENT_ARGS="-servername duckpond.ch" \
  tls-tofu-enabled-image \
  curl -x http://mitmproxy:8080 https://duckpond.ch
CONNECTED(00000003)
---
Certificate chain
 0 s:
   i:CN = mitmproxy, O = mitmproxy
-----BEGIN CERTIFICATE-----
MIICwjCCAaqgAwIBAgIGDinF743TMA0GCSqGSIb3DQEBCwUAMCgxEjAQBgNVBAMM
CW1pdG1wcm94eTESMBAGA1UECgwJbWl0bXByb3h5MB4XDTE5MDUwNTE5MzUyNVoX
DTIxMDUwNjE5MzUyNVowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AJh4sql+hJEvAstUd+gxcWIxy+vN9MC78WbT5Ox2oB7YO0r+3CCkeki3SehEBtHj
5MolxsspJipxF3gHN9el36vcN9dbj1sIGGukT7az7u8M0al4cVnP3Dyt+Fhb1Y2K
7UEiA2WibWEzeAlNbxASdHt7xqHTQfHMDBr+l1odg7eRe2QquFuLI8PhhSw1XSXw
yxY+A6/xxwkMcqU9s9pgQhP51qzAlY86SgZqHtgzCn7K0p3gCO+H2jhQu5R45h3X
1mJY6iRHf5WnEAPe/kdyjkme+8TsgeiZ13y7o+K9a6XwHQD1jTZFE9GW27cyVst/
0RHt6R+50DRSDJ1onPHUCGcCAwEAAaMaMBgwFgYDVR0RBA8wDYILZHVja3BvbmQu
Y2gwDQYJKoZIhvcNAQELBQADggEBAC7XwulSPQPHmESiQWiWjuDtyHu85lsGBvxS
C31zJRva8NuPlzIi4w3wQXXkuH+skNvTZ81WJxfxw+WjwcglYfmKG6yQVtZ0tdVk
uYUVq2Okn8oBSAI57XNaKqCGTioQhk7Mk/P93Y49UuNF/AC47IvxOWkj9QFm/wtu
+zRfko5VC7W7f1ji5tgnJduD46lR2hen12Px3tWyRWc1ECFjXZ4lBT1xUZIloPjQ
5PGDEIrg0zTmr3ZqUBGBKEjF1zrzXC3myMStn5pn0mBHgH0Fgrooswad0UzLAoDL
ECDdplotyGL9Wc1/CzLHsvsw8lvGCimi0ul0QOn9VQtnLKF/w/0=
-----END CERTIFICATE-----
 1 s:CN = mitmproxy, O = mitmproxy
   i:CN = mitmproxy, O = mitmproxy
-----BEGIN CERTIFICATE-----
MIIDoTCCAomgAwIBAgIGDinFhnrGMA0GCSqGSIb3DQEBCwUAMCgxEjAQBgNVBAMM
CW1pdG1wcm94eTESMBAGA1UECgwJbWl0bXByb3h5MB4XDTE5MDUwNTE5MjM1N1oX
DTIyMDUwNjE5MjM1N1owKDESMBAGA1UEAwwJbWl0bXByb3h5MRIwEAYDVQQKDAlt
aXRtcHJveHkwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCYeLKpfoSR
LwLLVHfoMXFiMcvrzfTAu/Fm0+TsdqAe2DtK/twgpHpIt0noRAbR4+TKJcbLKSYq
cRd4BzfXpd+r3DfXW49bCBhrpE+2s+7vDNGpeHFZz9w8rfhYW9WNiu1BIgNlom1h
M3gJTW8QEnR7e8ah00HxzAwa/pdaHYO3kXtkKrhbiyPD4YUsNV0l8MsWPgOv8ccJ
DHKlPbPaYEIT+daswJWPOkoGah7YMwp+ytKd4Ajvh9o4ULuUeOYd19ZiWOokR3+V
pxAD3v5Hco5JnvvE7IHomdd8u6PivWul8B0A9Y02RRPRltu3MlbLf9ER7ekfudA0
UgydaJzx1AhnAgMBAAGjgdAwgc0wDwYDVR0TAQH/BAUwAwEB/zARBglghkgBhvhC
AQEEBAMCAgQweAYDVR0lBHEwbwYIKwYBBQUHAwEGCCsGAQUFBwMCBggrBgEFBQcD
BAYIKwYBBQUHAwgGCisGAQQBgjcCARUGCisGAQQBgjcCARYGCisGAQQBgjcKAwEG
CisGAQQBgjcKAwMGCisGAQQBgjcKAwQGCWCGSAGG+EIEATAOBgNVHQ8BAf8EBAMC
AQYwHQYDVR0OBBYEFAHmNa8e++bQ4Nr1XzdqPek/tp4eMA0GCSqGSIb3DQEBCwUA
A4IBAQAxC9SNuBLjVhSY2ilJRQc21bv/WoJAcmGtxLxhXn43RwnYsNxKDmS3bRwj
CbKOX2mhV7zqKRDvrA0iRoWndGwfQodnc9eairo3LSLCqg8+vFkwgaRQICyCkv18
6ElxxHVQinNrd4XyaStqrweqK+gbB1NymR/87nOiRXzK9utGjESifaUNl97fymTg
LL8BwQH5iHHlU5ud14AKkwr14QWrTzTbyP/McxLo/KfTjVCl30YO2onzMpwu2oW5
cRRfx96ajPoKwtVFBJTX/hdBoqkovNFvRSITMU3VHEKRfoIG2OJRsA1dl7ezb6Ao
Uf30567z+pXa2Dp8YOnUA3ARWBCu
-----END CERTIFICATE-----
---
Server certificate
subject=

issuer=CN = mitmproxy, O = mitmproxy

---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2316 bytes and written 439 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: D15B53E637A92CF86580E51B8053626F50F3256138E5F5A55BAD5B37A903CE67
    Session-ID-ctx: 
    Master-Key: 72C569F0C2C5A4EF5EB666DE0CF7B908411854010BA303F53DB9199E7872F68EE0E524E780DDF003819CA8DFBB3A1C3E
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 24 fe cb 42 a8 12 eb b3-35 29 6a 12 11 79 af ba   $..B....5)j..y..
    0010 - 63 50 85 58 bc 6f a7 5a-c5 5f 9b 11 21 97 23 ed   cP.X.o.Z._..!.#.
    0020 - 54 9c 99 0c 5a 26 a8 75-21 06 b8 26 e1 57 f7 f0   T...Z&.u!..&.W..
    0030 - f1 a0 c1 5f a3 d8 25 36-4f de cc 6d 76 c2 d9 89   ..._..%6O..mv...
    0040 - 99 46 ec 64 8d d8 c1 41-04 58 4c 7a bf 8f 1c a8   .F.d...A.XLz....
    0050 - 8e b4 42 bd 2b 73 03 07-05 26 36 66 66 53 ac 63   ..B.+s...&6ffS.c
    0060 - 52 98 c9 31 cb ea 5d c4-b6 76 4a d7 c4 79 1c 4c   R..1..]..vJ..y.L
    0070 - f1 3b 76 04 ed 15 07 ff-d0 2d c7 92 6c d8 56 f9   .;v......-..l.V.
    0080 - 94 19 5a 61 9b 58 db 68-1d 2e 0c 87 fb f9 64 38   ..Za.X.h......d8
    0090 - 56 5a 8f 4e 2e a9 31 f6-ac db c9 30 51 5e 84 00   VZ.N..1....0Q^..
    00a0 - 29 fc 48 d9 70 f3 87 3f-07 b9 11 9b 2a 7a 72 ce   ).H.p..?....*zr.

    Start Time: 1557257748
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no
---
<!DOCTYPE html>
<html>
    <head>
...

From the mitmproxy/mitmproxy output we get that the request did actually go through the proxy. All this without curl complaining about certificate verification issues.

17.0.3:47120: clientconnect
17.0.3:47120: Client Handshake failed. The client may not trust the proxy's certificate for mitmproxy.
17.0.3:47120: clientdisconnect
17.0.3:47122: clientconnect
17.0.3:47122: Client Handshake failed. The client may not trust the proxy's certificate for mitmproxy.
17.0.3:47122: clientdisconnect
17.0.3:47124: clientconnect
17.0.3:47124: GET https://duckpond.ch/
               << 200 OK 20.86k
17.0.3:47124: clientdisconnect

Mission accomplished.

A Real World Example: enteee/git-sync-mirror

enteee/git-sync-mirror is a simple container image for synchronizing a git mirror. Inside the Dockerfile it installs a run script (/run.sh) and overwrites the default command. The ENTRYPOINT is still provided by enteee/tls-tofu which does all the TOFU magic.

FROM enteee/tls-tofu:alpine-latest

## Disable TLS-TOFU by default
ENV TLS_TOFU false

RUN set -exuo pipefail \
  && apk add \
    git \
  && addgroup -g 1000 -S git \
  && adduser -u 1000 -S git -G git

USER git:git

COPY run.sh /run.sh
CMD ["/run.sh"]

When running this container with -e TLS_TOFU=true enteee/git-sync-mirror silently does TLS-TOFU. And if we additionally specify -e TLS_TOFU_DEBUG=true, we can see what is happening under to hood.

$ docker run \
  -e TLS_TOFU=true \
  -e TLS_TOFU_DEBUG=true \
  git-sync-mirror
+ '[' true '=' true ]
+ openssl s_client -verify_return_error -connect google.com:443 -servername google.com
+ destroy_kamikaze
+ '[' -x /kamikaze ]
+ /kamikaze true
+ exec sh -c /run.sh
/run.sh: line 5: SRC_REPO: Missing source repository

This fails because we didn’t specify the mandatory SRC_REPO for enteee/git-sync-mirror. Nevertheless, we can still see enteee/tls-tofu connecting to google.com. But since all certificates are valid it does not add any new trusted ones. After this kamikaze is destroyed and control is being handed over to enteee/git-sync-mirror.

Caveat: Restart Policies

If an attacker is in control of your network, it is very likely that they can also crash applications you are running in containers. If you restart the container in this case, your application will re-TOFU. This means an attacker can make the container trust every certificate they want. This is very, very bad. Always disable automatic container restart with TLS-TOFU exactly for this reason.

Final Thoughts

enteee/tls-tofu implements a simple, yet powerful base image which allows containers to run in environments where something is tampering with the Internet’s public key infrastructure. I generally disagree that HTTPS scanning proxies are leveraging security in a network. They create a single point of failure and considerably weaken a secure protocol design to protect privacy and confidentiality.

JavaScript: this

2019-03-20T00:00:00+01:00

Fuck-this.js: A reddit post in r/ProgrammerHumor

The snippet below tests if node.js ¹ forwards the this-object in various scenarios. I wrote it to solve this issue when refactoring FluentFlow. If you can think of any unintuitive handling of the this-object in JavaScript please share a snippet in the comment section.

/**
 * Helper to require a module from a string
 */
function requireFromString(src) {
  var Module = module.constructor;
  var m = new Module();
  m._compile(src, __filename);
  return m.exports;
}

(function(){

  console.log('Body:');

  console.log(this.a === 1);                          // true
  (()=>console.log(this.a === 1))();                  // true
  (()=>console.log(this.a === 1)).call(this);         // true
  (function(){console.log(this.a === 1)})();          // false
  (function(){console.log(this.a === 1)}).call(this); // true

  console.log('\nFunctions:');

  var f1 = (() => {console.log(this.a === 1)});
  f1();                                               // true
  f1.call(this);                                      // true

  var f2 = function(){ console.log(this.a === 1) };
  f2();                                               // false
  f2.call(this);                                      // true

  var f3 = requireFromString('module.exports = (() => {console.log(this.a === 1)});');
  f3();                                               // false [1]
  f3.call(this);                                      // false [2]

  var f4 = requireFromString('module.exports = function(){ console.log(this.a === 1) };');
  f4();                                               // false
  f4.call(this);                                      // true

  console.log('\nClasses:');

  class c1 { log(){ console.log(this.a === 1) } };
  (new c1).log();                                     // false
  (new c1).log.call(this);                            // true

  c2 = requireFromString('module.exports = class c2 { log(){ console.log(this.a === 1) } };');
  (new c2).log();                                     // false
  (new c2).log.call(this);                            // true

  class c3 { constructor(log){ this.log = log } }
  (new c3(f1)).log();                                 // true
  (new c3(f1)).log.call(this);                        // true
  (new c3(f2)).log();                                 // false
  (new c3(f2)).log.call(this);                        // true
  (new c3(f3)).log();                                 // false
  (new c3(f3)).log.call(this);                        // false
  (new c3(f4)).log();                                 // false
  (new c3(f4)).log.call(this);                        // true

  c4 = requireFromString('module.exports = class c4 { constructor(log){ this.log = log } };');
  (new c4(f1)).log();                                 // true
  (new c4(f1)).log.call(this);                        // true
  (new c4(f2)).log();                                 // false
  (new c4(f2)).log.call(this);                        // true
  (new c4(f3)).log();                                 // false
  (new c4(f3)).log.call(this);                        // false
  (new c4(f4)).log();                                 // false
  (new c4(f4)).log.call(this);                        // true
}).apply({a:1});

With the documentation for this and arrow functions I can explain all cases except the ones labelled with [1] and [2]. But thanks to the excellent answer by aaronofleonard I now understand why:

The reason is because “fat arrow functions” always take their this lexically, from the surrounding code. They cannot have their this changed with call, bind, etc. Run this code as an example:

var object = {
  stuff: 'face',

  append: function() {
    return (suffix) => {
      console.log(this.stuff + ' '+suffix);
    }
  }
}
var object2 = {
  stuff: 'foot'
};

object.append()(' and such');
object.append().call(object2, ' and such');

You will only see face and such. So, as far as why that doesn’t work in the case of f3, it’s because it’s a self-contained module being required. Therefore, it’s base-level arrow functions will only use the this in the module, they cannot be bound with bind, call, etc etc as discussed. In order to use call on them, they must be regular functions, not arrow functions. What does “lexical this” mean? It basically works the same as a closure. Take this code for example:

// fileA.js
(function () {
    var info = 'im here!';

    var infoPrintingFunctionA = function() {
        console.log(info);
    };

    var infoPrintingFunctionB = require('./fileB');

    infoPrintingFunctionA();
    infoPrintingFunctionB();
})();

// fileB.js
module.exports = function() {
    console.log(info);
};

What will be the result? An error, info is not defined. Why? Because the accessible variables (the scope) of a function only includes the variables that are available where the function is defined. Therefore, infoPrintingFunctionA has access to info because info exists in the scope where infoPrintingFunctionA is defined. However, even though infoPrintingFunctionB is being called in the same scope, it was not defined in the same scope. Therefore, it cannot access variables from the calling scope. But this all has to do with the variables and closures; what about this and arrow functions? The this of arrow functions works the same as the closure of other variables in functions. Basically, an arrow function is just saying to include this in the closure that is created. And in the same way you couldn’t expect the variables of fileA to be accessible to the functions of fileB, you can’t expect the this of the calling module (fileA) to be able to be referenced from the body of the called module (fileB).

TLDR: How do we define “surrounding code”, in the expression “lexical this is taken from the surrounding code?” The surrounding code is the scope where the function is defined, not necessarily where it is called.

Tested for v6.8.1 and v8.15.0 ↩

Pdml2flow 5.0

2018-11-12T00:00:00+01:00

pdml2flow version 5.0 was released. New key features are:

Plugin interface version 2: cli integration
Replaced pdml2xml and pdml2json with pdml2frame
98% test coverage
Python 3.6 support

Upgrade using pip is as simple as:

$ sudo pip install --upgrade pdml2flow

Plugin Interface Version 2

Before 5.0 plugins were configure with environment variables. Now, plugins are loaded and configured from the command line. In order to support multiple output sink all the flow writing logic was moved into plugin as well. Using pdml2flow with the JSON output plugin:

$ pdml2flow +json '-h'

The argument after the plugin invocation is passed straight to the plugin. Which means that the -h from the example above is passed to the JSON plugin. To support this, the plugin interface was changed. Now __init__ will be passed all arguments (*args) from the command line. In this case [ '-h' ].

diff --git a/pdml2flow/plugin.py b/pdml2flow/plugin.py
index db016a3..06f7425 100644
--- a/pdml2flow/plugin.py
+++ b/pdml2flow/plugin.py
@@ -1,7 +1,20 @@
 # vim: set fenc=utf8 ts=4 sw=4 et :
 
-class Plugin1(object):
-    """Version 1 plugin interface."""
+class Plugin2(object): # pragma: no cover
+    """Version 2 plugin interface."""
+
+    @staticmethod
+    def help():
+        """Return a help string."""
+        pass
+
+    def __init__(self, *args):
+        """Called once during startup."""
+        pass
+
+    def __deinit__(self):
+        """Called once during shutdown."""
+        pass
 
     def flow_new(self, flow, frame):
         """Called every time a new flow is opened."""

This change alone is not the reason for the interface bump. The true interface break comes from a change in the flow class. Before 5.0 frames of the flow were accessible using a getter called get_frames(). This was replaced with a more pythonic getter frames. Example usage:

flow.frames['frame']['time_relative']['raw']

or as an alternative using a list:

flow.frames[['frame', 'time_relative', 'raw']]

Replaced `pdml2xml` and `pdml2json` with `pdml2frame`

Since all the output was moved into plugins, pdml2xml and pdml2json no longer made sense. This is why I replaced those tools with pml2frame which supports the same plugin interface as pdml2flow. This means pdml2xml becomes pml2frame +xml. And for pdml2json use pdml2frame +json instead.

Citation Needed

2018-08-10T00:00:00+02:00

Maintenance

2018-03-13T00:00:00+01:00

I did some blog maintenance and released a new duckpond.ch.

Migrated to Bootstrap v4.0.
The list numbers overflowed. list-style-position: outside; is not a god fit for long roman numerals. I am now using css counters and liquid templating instead.
The twitter button now tweets 280 characters.
The logo is now responsive.
Start nginx using dockerize.
Request certificates using dehydrated.
jekyll watches now for changes and rebuilds automatically.
jekyll-responsive-images with this layout automatically creates responsive versions of images.
jekyll-minifier minimizes .html and .css files.
MathJax CDN shutting down on 30. April 2017. Switched to Cloudflare.
Removed annotations from the duck in a toilet 404.

:for

2018-03-08T00:00:00+01:00

Syntactical sugar for your bash parallelism. :for calls a function for each argument in parallel. If at lest one call exits with a non zero exit code, :for retains this code and kills all running instances.

Usage

$ ./usage

duckpond.ch

Mit Eidechsen war alles besser

Weg von der Eidechse!

Zusammenfassend kann man folgendes sagen:

Reaping Poison Tasks In Celery

Task Queue Architecture

Introducing Dead Letter Queues

Reaper Workers

Why It Works

Convert Video For Chromecast

Citation Needed

Count Beta Release

Disable reMarkable Touchscreen with evkill

The Problem

The Hack

Support Binary File Objects with pandas.DataFrame.to_csv

Citation Needed

Types for PlantUML Parser

Table of Contents

Why you should use PlantUML

Parsing PlantUML with TypeScript

Contribute

reMarkable

Table of Contents

Feature Overview

Software for the reMarkable and Some NixOS Derivations

The Official Linux Client

Accessing the reMarkable API With rMAPI

Streaming the Framebuffer with srvfb

Hacking Hardware Button Flight Mode Toggling

Conclusion

Citation Needed

udev Rule Script Template

Bonus: A Nix Expression

Migrate a git Repository with Submodules

The Problem

The Solution: git-submodule-url-rewrite

Why should I use this?

Citation Needed

TOFU for Containers

TOFU Inside a Container

enteee/tls-tofu Container Image

A Real World Example: enteee/git-sync-mirror

Caveat: Restart Policies

Final Thoughts

JavaScript: this

Pdml2flow 5.0

Plugin Interface Version 2

Replaced pdml2xml and pdml2json with pdml2frame

Citation Needed

Maintenance

:for

Usage

Source

Replaced `pdml2xml` and `pdml2json` with `pdml2frame`