Ode to professionals

Tags: English, technology, essays
Created on Thu, 19 Apr 2018

A professional doesn't dress well to impress and fake his professionalism. He dresses well to feel professional.

A professional is not hack-y, quick or dirty. He is a thorough and precise.

A professional is not arrogant or know-it-all. He works together with his colleagues, learns from them and helps them.

A professional understands planned work, unplanned work and plans for the unplanned work.

A professional takes responsibility but never blames.

A professional always learns.

A professional uses the right tool for the job. Not the one that is most hyped at the moment, nor the one he is just most familiar with but too complex or too simple for this job.

A professional understands his human limitations. Thus he never overworks to show off or mistreats the rest of his life for his craft.

A professional always tells the truth, voices his opinions where they are relevant to the work and protects his craft and his colleagues.

A professional understands or tries to understand to the best of his abilities the risks and tradeoffs of choosing one solution over another in the broader context of the work being done.

Auto adapting signal to day-night scheme

Tags: English, hacks
Created on Mon, 02 Apr 2018

So I wanted to automatically change the day/night theme of my brand new Signal desktop app. The way that it is manually done is through the File -> Preferences... menu. Now that's not convinient for programatic person like me. Let's investigate!

.config folder

My first stab is looking into the mostly standard ~/.config folder in Linux for Signal if one actually exists. It turns out it does. Let's see. It even has a ~/.config/Signal/config.json file. Unfortunately, that file doesn't contain the settings for the theme, just window positions.

grep -ring in the folder for any of android or theme just returns the logfile that Signal is writing to.

...
./logs/log.log.1:{..."theme-setting changed to android-dark",...}
...

No luck.

Another stab is at /opt/Signal directory. Again - no luck. And the app is a binary ELF executable

$ file signal-desktop 
signal-desktop: ELF 64-bit LSB executable,...

strace-ing

My next attempt is to attach to attach to the Signal process via strace. Find the process by ps aux|grep signal, attach by strace -p <pid> and try to change the . Boy, this does a lot of stuff! Let's filter by writing:

strace -e write -p 12345

Still, too much but I saw something hapenning amid the writing to 40 and 71 file descriptors (fd).

#...
write(40, "!", 1)                       = 1
write(71, "\0", 1)                      = 1
write(40, "!", 1)                       = 1
write(1, "{\"name\":\"log\",\"hostname\":\"pi2-ho"..., 147) = 147
write(12, "\1\0\0\0\0\0\0\0", 8)        = 8
write(71, "\0", 1)                      = 1
write(40, "!", 1)                       = 1
#...

All right, the 1 fd is the log that is being written that I already saw. How about that 12 fd? Let's explore within the proc fs:

$ cd /proc/1234/fd
$ ls -la
# ...
lrwx------ 1 pi2 pi2 64 Mar 29 23:23 12 -> anon_inode:[eventfd]
# ...

Oh boy, some wierd anonymous inode. So we are definately not dealing with a good old filesystem file but rather some on the fly created fd. Let's explore a bit more...

Signal is an Electron app

Taking a look at the github repo of the project. Searching for dark and theme reveals css, sass files. Okay

Looking at the menus there is View -> Toggle Developer Tools and BAM - a good old Chrome developer tools console pops up! Inspecting the elements, it's pretty clear what is happening:

<body class="android">
<!-- changes to -->
<body class="android-dark">

Brilliant, the change of theme just changes the class on the body element. Let's see if we have jquery or I have to go to the dark ages of javascript's findElementByTagName:

> $
< function ( selector, context ) {
        // The jQuery object is actually just the init constructor 'enhanced' ...

Good, we have. Now I "just" have to find a way to connect to chrome console via a good old terminal in some way...

Chrome remote debugging

Searching the Internets, I find a post in Chromium blog from way back in 2011 that I can run Chrome with --remote-debugging-port flag which enables some sort of a remote console.

Would the electron app do the same?

signal-desktop --remote-debugging-port=9222

Indeed it does. This console however is not just a simple tcp connection to the javascript console. Running echo "alert('echo'); | netcat localhost 9222 does nothing. More reading to go...

Turns out, Chrome DevTools has its own protocol that among other things exposes websockets. Let's see what clients exist for that

Quite a lot.

Any pythonic ones? pychrome seems simple enough.

Let's be modern, do python3:

pip3 install pychrome

And I will use my favourite repl, ipython and follow the Getting started tutorial:

import pychrome

browser = pychrome.Browser(url="http://127.0.0.1:9222")

Awesome, I have a connection. The tutorial goes into opening a tab. I don't want that, I assume I should alread have a tab. Exploring browser object (via my favourite ipython repl):

In [3]: dir(browser)
Out[3]: [... _private_stuff
'activate_tab',
 'close_tab',
 'dev_url',
 'list_tab',
 'new_tab',
 'version']

In [4]: browser.list_tab()
Out[4]: [<Tab [38473d8e-ad8d-4325-8cb6-81aaec36c124]>]

In [5]: tab = browser.list_tab()[0]

In [6]: dir(tab)
Out[6]: [... _private_stuff
 'call_method',
 'debug',
 'del_all_listeners',
 'event_handlers',
 'event_queue',
 'get_listener',
 'id',
 'method_results',
 'set_listener',
 'start',
 'status',
 'status_initial',
 'status_started',
 'status_stopped',
 'stop',
 'type',
 'wait']

The README goes into navigating pages and the examples folder is also not what I am looking for. But they do tell me call_method is probably what I want, just need to find the proper string and parameters in the protocl. I want to execute a javascript call.

Back to the Chrome DevTools Protocol Viewer. I look through a few of the Domains listed and finally arrive at Runtime.evaluate method.

In [7]: tab.start()
In [8]: tab.call_method('Runtime.evaluate', expression='alert("hi");')
# I GET AN ALERT IN THE BROWSER! Dismiss it.
Out[8]: {'result': {'type': 'undefined'}}

Finally! I have access to my javascript console in the browser! Let's write the jquery change of theme:

tab.call_method('Runtime.evaluate', expression='$("body").removeClass("android").addClass("android-dark");')

That worked!

Wrapping it up

Let's write a propper python script now:

#!/usr/bin/env python3

import sys
import pychrome

sys_argv = sys.argv

if len(sys_argv) < 2:
    print("first argument needs to be day|night")  
    exit(1)

command = sys_argv[1]

if command == "night":
    expression = js_dark_to_light = """
$("body").removeClass("android").addClass("android-dark");
"""
elif command == "day":
    expression = js_light_to_dark = """
$("body").removeClass("android-dark").addClass("android");
""" 
else:
    print("first argument needs to be day|night")
    exit(1)

browser = pychrome.Browser(url="http://127.0.0.1:9222")
tab = browser.list_tab()[0]
tab.start()

tab.call_method('Runtime.evaluate', expression=expression)

tab.stop()

Running ./change_signal.py day changes to white theme and ./change_signal.py night goes to dark theme.

I also want to change the signal-desktop app to always start with the remote-debug flag enabled, so I do that:

rm /usr/local/bin/signal-desktop
cat > /usr/local/bin/signal-desktop << EOF
#!/bin/bash
/opt/Signal/signal-desktop --remote-debugging-port=9222
EOF
chmod +x /usr/local/bin/signal-desktop

And that's it. Code can be found in my sun repo which also deals with other day-night related dark-light themes including a script to detect whether the sun has set or not:

$ ./calc_sun.py 
usage: calc --loc=location_name [--date-format] 
       calc --lat=latitude --lon=longitude [--tz|--date-format]
returns 
       current_time sunrise sunset day|night

$ ./calc_sun.py --lat=51.21 --lon=4.40 --tz=Europe/Brussels
23:57 07:23 20:10 night

So now I can chain that:

state=`./current.sh | cut -f4 -d' '`
./change_signal.py $state

Voala! That was fun 🙂😊

Here's how I got my music from Google Music (as of March 2018)

Tags: English, technology, hacks
Created on Sat, 31 Mar 2018

At some point I uploaded all my music on Google Music - they provided 20,000 songs upload for free. Then of course, they started pushing their paid service - fair enought. I tried the trial several times, I paid subscriptions several times over the years, but I never enjoyed the suggestions it was giving me and somehow the music I wanted, wasn't there. So eventually I always just went to YouTube to listen to music.

But now that I am trying to detach a little bit from the cloud and get my data back I wanted to try to download my music from Google Music.

Google Checkout

My first attempt was the assumption that I can get my music as a part of the standard Google Get Back My Data or whatever. It was fairly easy to get there - MyAccount from my profile image, then Takeout then click on Create Archive.

google checkout

All seems good, right? Do you notice the small print?

google checkout small print - you can't download your music

Ah, I need to use Google Play Music Manager to actually get my music. Otherwise, I just get a metadata. Fine. Following the link doesn't really lead me to downloading the Music manager, just a help page. But I am Smart (or stubborn), I navigated around, found the link to actually download it.

Google Music Manager

google music manager

I install my music manager, sign in and select to Download all my music, specify a folder...

and nothing happens.

I am slightly annoyed but very stubborn so I push it. So let's see what's hapenning.

The .config folder in my home directory contains google-musicmanager and a log. Let's tail it:

pi2@pi2-home-laptop ~/.config/google-musicmanager $ tail -F google-musicmanager.log

[../Shared/HttpClients/CurlHttpClientImpl.cpp:57 ::CurlSslVerifyCertificate()]
2018-03-31 11:10:08,371 +0200 ERROR TId 0x7f5756183700 curl_easy_perform failed with 60 [../Shared/HttpClients/CurlHttpClientImpl.cpp:402 CurlHttpClientImpl::performPrivate()]
2018-03-31 11:10:08,372 +0200 ERROR TId 0x7f5756183700 Certificate error [../Shared/HttpClients/CurlHttpClientImpl.cpp:406 CurlHttpClientImpl::performPrivate()]
2018-03-31 11:10:08,372 +0200 ERROR TId 0x7f5756183700 Call to https://sj.googleusercontent.com/download?id=[REDACTED]&itag=25&source=skyjam_user_upload&o=[REDACTED]3&uits=1&tid=[REDACTED]&ip=0.0.0.0&ipbits=0&expire=1522487498&sparams=id,itag,source,o,uits,tid,ip,ipbits,expire&signature=[REDACTED]&key=sj3&targetkbps=768 failed with result 3 [../Shared/HttpClients/CommonHttpClientImpl.cpp:50 CommonHttpClientImpl::perform()]
2018-03-31 11:10:08,393 +0200 ERROR TId 0x7f5756183700 Failed to Send Request Result - 3 [../Shared/HttpClients/DownloadFileClient.cpp:220 DownloadFileClient::Execute()]
2018-03-31 11:10:08,397 +0200 ERROR TId 0x7f5756183700 Failed to download song id=[REDACTED] [Core/DownloadProcessor.cpp:128 DownloadProcessor::processDownloadRequest()]
2018-03-31 11:10:08,825 +0200 INFO TId 0x7f5756183700 errno 2 calling stat /home/pi2/.config/google-musicmanager/00 Brett Domino Trio- Ultimate Ch.mp3 [../../Shared/TpnBase/TpnPortC.c:150 ::_Port_GetFileInfo()]
2018-03-31 11:10:08,883 +0200 ERROR TId 0x7f5756183700 Failed to Verify Certificate: /C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.googleusercontent.com (unable to get local issuer certificate:0x00000000) [../Shared/HttpClients/CurlHttpClientImpl.cpp:44 ::CurlSslVerifyCertificate()]
2018-03-31 11:10:08,883 +0200 ERROR TId 0x7f5756183700 Certificate /C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.googleusercontent.com:
-----BEGIN CERTIFICATE-----
...

(I redacted some parts of the output, not sure if any of them are sensitive but better safe than sorry).

So it's an expired certificate from what I gather. I guess there might be some vulnerability with this level of unsupported tool but I don't want to do this now. I want my MUSIIIIC.

I am getting a bit more annoyed and a bit more stubborn. Time to fire the hard guns.

Windows Music manager client

Google must be supporting Windows client better. Obtained a completely legitimate copy of Windows 10, spin up a virtual machine on Virtualbox, download the manager, try to log in and...

google music manager - 2 step verification

Huh? Nothing happened. Okay, let's try the Google Authenticator...

google music manager - authenticator

Okay, I am quite angry now.

Manual to the rescue

My only left option seems to be to either try to dual boot and hope Google Music manager detects that it's on a virtual machine and for some /security?/ reason decides not to authenticate me.

Or manually download the songs from the web interface.

I decide manual is going to be quicker - I've got about 2000 songs.

google music web - 100 limit

Arghhhh!!! So in a batch of 100 I will need around 20 selections... Should I automate? Let's go to xkcd:

xkcd - is it worth the time

I think I will be better off manually. Besides, inspecting the code seems that for optimization reasons, the song list is not loaded in its completeness but it keeps around 20 songs in the DOM and the rest it dynamically fetches on scroll events.

The good news is, the table keeps a data-index field with consequitive numbers so that I can at least find where I am and download in batch of 100

dev tools - inspect elements of the table

Oh, just one more thing - you can download a certain song just 2 times from web and apparently I have already downloaded some before...

maximum two downloads

I am not too happy with this. But I got my Music back.

C'mon, Google... Just to put oil in the fire, today Google announced that it's discontinuing yet another service - goo.gl.

Getting back control of my digital life one bit at a time

Tags: English, technology, hacks, personal
Created on Thu, 29 Mar 2018

I have permanently deleted my Facebook account this week after the latest "scandals" around the "social" network. Of course I knew on some level what is happening with my data but I think it finally hit home. I've always hated Facebook, ever since I first made my account (post about it in Bulgarian) - and that hasn't changed for the 8 years that I used the service, it only got worse I think. It was not so much about the data that I have shared, it has been more about the mindset that it installed in me and in people around me. Being zombies, constantly scrolling useless content by people you barely know or not even know with pages you liked at some point, with no easy way to get out of it, a network hitting all weak spots in human psychology. So the latest revelations of which we were all well aware (in the hacker/techie community at least) got to me and crossed a threshold that I don't want to be part of it anymore.

I wanted to quit but I wanted to try to make it smartly - no emotional, spontaneous decisions. I have done this before and it lead to nowhere - a week or a month later I was back because I was missing the people I talk to.

Quitting Facebook

I decided to collect as many current ways to contact people as I can from my network about the people I most cared about (sorry 650 other "friends", maybe we have been friends at some point in life, but that's life - people move on). I also posted on my wall that I will be permanently quitting in 72 hours. So if someone saw it and wanted to contact me to exchange contacts - feel free. My contacts were email and phone number - things I assessed are least likely to change as a point of contact for me and if they do, it would be relatively simple to update my contacts of my new number/email. In my last 24 hours I also changed my profile picture and that I will be moving to Signal, which is tied to my phone number thus reducing the pieces of information I need to propagate to my contacts (no username there). More on the phone number below. I didn't want to try too hard to convince people to move to Signal, not least because I still saw some problems with it, but because I also understand my friends - messenger is where everybody is and pragmatically.. they are probably right. I hate installing just the next messaging app. But I didn't want to move to Whatsapp (facebook's), Viber was a close second (most my Bulgarian friends seem to have an account there and I might still consider it). But I wanted to give something I personally like a chance. Besides, this philosophy is relatively new thing for me and I am not ready to defend it too fiercely.

I liked Signal because of their cross-platform solution, open source end-to-end encryption protocol that seems to be used by many other messaging apps and apps, but mostly - due to the fact that they are a (pending) non-profit organization (Open Whisper Systems), not a company and hopefully have a more ideal intensive to keep working on a good messaging system than profit.

There were problems I found with Signal as well which could be seen as pros and cons: * Wanting my phone number and a smart phone - I would prefer the app to be web/desktop first with good smart phone solution and tied to a username/password even though I said above that it's one less piece of contact detail (pro and probably how they decided they would get a faster traction) but on the con side - it ties me to mobile operators. Good news is that it's now relatively easy to change mobile operators and keep the number in EU. * Long time syncing messages - due to the way protocol works as far as I understand (I haven't read the whole thing yet) * No history (or hard to get) of messaging upon changing devices/reinstalling OS - both a pro and a con. On the one side I rarely need history of messages so I am okay, on the other it makes other people less likely to join. Which defeats the purpose of a messaging app. * Non-federated protocol - the Signal server(s) still seem like Single point of failure * But for now, I can live with this cons until either they are fixed, I find workarounds (backup history service? background syncing of messages? it's open source after all, maybe I can look into something...) or I find another service which has more complete set of features and am more sure of the organization/company values.

How about the rest of digital life

I made some aggregations of my web history of domains and I found what i expected. I have the other probably 70% of what I do online in Google services - online search is in their service of course, Google Chrome, music and video entertainment on YouTube, my email is in Gmail, this blog is in Blogger, Google Music has my old mp3 music, Drive has pretty much everything document and project wise, Keep for notes, Photos has all my photos.

Yup.

Other than that, my code lives in GitLab/GitHub, I use HN, reddit for news.

Is Google just as evil as Facebook?

Probably not but I still have put a tremendous trust in one company.

I am not convinced that Google is just as bad yet. Maybe. But what I feel is that being a fanboy of Google for a non-trivial amount of time and putting almost all of my digital life there was (probably) a dumb move. And I want to work towards decentralizing this data from Google even if I mostly trust them so far, it's convenient to have it backed up automatically and accessible from everywhere. I traded convenience for personal data. Common thing I guess.

This project is going to take months, if not years. The bottleneck of the solution I believe are my own habits. Technological solutions do exist and I can pretty quickly download and upload it at different places. But this will just move the problem, not really solve it. Or even probably complicate it more.

What can I do?

The Martian taught me to think in a certain way of "what's going to kill me first", then Chris Hadfield's An Astronaut's Guide to Life on Earth confirmed that this is indeed how astronauts think. And who doesn't admire astronauts?!

Start with the biggest offenders by impact, one at a time and move to the next one. * I have started switching slowly to Firefox for the past year or so. I don't have exact metrics yet but I think I maybe have around 70:30 Chrome:Firefox relationship. I want try to balance this a bit more, Firefox 57+ (post-Quantum) is really awesome! * Search (I think) is the hardest to get rid of and used the most day-by-day. But most of my searches are around programming documentation and land on either the project documentation website or stackoverflow questions. My first stab will be to try to use site-specific searches instead of the general Google search silo. So for example, go on stackoverflow and use its search whenever it's a programming problem first. It's an idea I haven't tried and am curious how good it would be. I thought about DuckDuckGo but I couldn't make it work in the past. Let's see if this can work. * Move the blog to my own thing. I've wanted that for a long time and I know there are off-the-shelf solutions but I want my own solution for fun. * The rest of my data will take more time and I need to focus on these first and figure out the rest later. Without focus, doing 50 things won't help. And as I said, the biggest bottleneck is my own habits, will, discipline and focus.

Will it work

I am definitely not an astronaut. In fact, I think I am quite bad at following these cool on paper plans. I am a bit of a control freak and this might just be a phase that I want to get back control. Not sure. This post is as much for me to remind myself that I have wanted this in the past if I go with the flow or screw up this project in some way.

Let's see what happens.

On windows and software development

Tags: English, technology, opinions
Created on Mon, 26 Mar 2018

There are worker guys now in the office just changing a window. And I am thinking... I so wish it was this way with software development - they have a task, there is a due time, it is clear and there are standard tools to do the job. And even if their clothing is dirty, because that's what their line of work is, they look oh so professional. They know what they do, how much time it is going to take them, what to use and the job is done perfectly and even integration tested it by opening closing the window and then stress tested it by hanging on it to see it holds.

If he was a software developer... You never know... They may have a very new incompatible window, so they will have to stick with duck tape where there are some holes, do a unit test of the handle and the glass and say it has 99% coverage but actually opening and closing it crashes the window next to it and the stability of the whole building is under question. Then they would explain that it worked when they tried it on their hut in the mountains and worked perfectly. Then they would argue that philosophically, there must be only one standard window and it should work everywhere so they ask you to change the building.