Author: Joe McMahon

Necessary steps to get the GitLab Rails template app running on OS X
The GitLab template is a great way to get started on a new Rails app – it sets up most of what you need simply by checking out the code, but there are a few things you need to do if you’re starting fresh on a
new Mac.

The template app assumes you’ll be using Postgres for your database. I recommend sticking with this; if you grow your app past the proof-of-concept step, you’re going to want it configured for a robust and responsive database. I love SQlite, but it’s not that much harder to go ahead and get Postgres running, so let’s do that instead of downgrading the template.

If you have Homebrew installed, skip down to the brew install postgres section. If you don’t, run the following command:
```
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
```
Wait for a while for that to finish, and Homebrew will be installed. Once that’s done you can move on to installing and configuring Postgres:
```
brew update
brew install postgres
initdb /usr/local/var/postgres -E utf8
```
Now you need to configure your databases. The initdb above sets up your user ID as a root user to start, so you can now add the application’s user and databases. (Substitute an appropriate username for yourappdev).
```
psql postgres
create role yourappdev with login password '...some password here...';
alter user yourappdev createdb;
```
Now exit psql (^D will do) and log back in again as the yourappdev user to create the development and test databases for your Rails app. If you set a password, psql will prompt you for it. (If you forgot it, your root-level user created when you installed Postgres can still log in without a password at the moment.)
```
psql postgres -U yourappdev
create database yourapp_development;
create database yourapp_test;
```
Securing the production database

You now need to create the database that you’ll run in production. Remember this is the one with the important user data in it that you don’t want to lose, so we want to create a whole separate username that will be the creator of the database and give it a good strong password that you record in your password manager. (If you don’t have a password manager, get one! It’s way safer than writing it down on a sticky and putting it in an envelope.)
```
psql postgres
create role yourapp with login password 'a very strong password please';
alter user yourapp createdb;
^D
psql postgres -U yourapp
create database yourapp
```
You’re now ready to work on your Rails app. When you want to run the production version, you’ll need to set the DATABASE_URL environment variable like this:

DATABASE_URL=”postgres://yourapp:strongpassword@localhost/yourapp”

Further deployment issues are beyond the scope of this post.
August 21, 2018
HTTPS upgrade completed

That was actually pretty painless.

Hostgator (love y’all!) now provides a per-site HTTPS cert for free, so I didn’t have to use Let’sEncrypt for it; I just needed to install the Really Simple SSL plugin, back up my database, and turn it on to get SSL working.

Highly recommended if your site isn’t a complicated one.

August 21, 2018
VCVRack NotStraightLines plugin for Rack 0.6.0

This zip file (NotStraightLines-0.6.0-mac) is a Mac version of the NotStraightLines plugin. To make it super clear, this is all Andrew Belt’s work; I just built a binary!

June 15, 2018
Not black and white

This photo was taken on a heavily overcast day, and is straight out of camera. A wonderful interaction of the day and the sensor in my iPhone.

March 27, 2018
Followup on Go dependency Jenga
I was finally able to build a working version of the glide.yaml file for my project and convert it to dep. The items of note:
- To get openzipkin to work, I needed to specify
```
[[constraint]]
  branch = "master"
  name = "github.com/opentracing/opentracing-go"

[[override]]
  name = "github.com/openzipkin/zipkin-go-opentracing"
```
- To get logrus to work, I had to change it to github.com/sirupsen/logrus in all my code and specify
```
[[constraint]]
  name = "github.com/sirupsen/logrus"
  version = "^1.0.5"
```
March 20, 2018
Desperate times, desperate measures: dealing with a Go dependency Jenga tower
TL;DR

If you absolutely have to manually update your glide.lock file to add a specific SHA1 for a dependency and can’t do it right with glide update, edit glide.lock as needed, then:
go get mattfarina/glide-hash glide hash
This gets the correct checksum for your glide.lock file; update the hash: line at the top. You can now glide install without warnings.

The detailed explanation

Our microservices have a number of dependencies, one of which is logrus. Logrus is a great logging package, but was the trigger of a lot of issues last year when the repository was renamed from github.com/Sirupsen/logrus to github.com/sirupsen/logrus.

That one capitalization change caused havoc in the Go community. If you don’t understand why, let’s talk a little about dependency management in Go. (If you do, skip down to “The detailed fix”).

Go doesn’t have an official dependency management mechanism; when you build Go code, you pretty much expect to compile all the code it will need at once. Go goes have a linker, but generally we really do just build a single static binary from source files, including the source of libraries too. The Go maintainers decided that it’s simpler to store one set of source code to be pulled in and complied rather than store compiled libraries for multiple architectures and figure out which one needs to be pulled in. The Go compiler is pretty fast, and maintaining multiple native binary versions of libraries is hard.

Originally, all source management was done with go get, which would fetch code from a VCS endpoint and put it in the appropriate place in the GOPATH (essentially the location where “stuff related to but not part of this Go program” lives) so that it could be picked up during a compile. This is super simple, but fails in a number of ways: a set of go get commands are a set of commands, and have to be run before the program can be built. This may not be reproducible (if someone makes a new commit to the library, the HEAD changes). Telling go get to fetch a specific version of a library is harder to do. go get is great at pulling a specific isolated library, but not good at managing transitive dependencies: e.g., we’ve installed library foo, but it needs library bar to perform some functions, and bar needs baz to do some of its work. We’d really like to see all of these figured out and installed at once, and to not have to remember what all the dependencies are, or to have to have a script to run to fetch them. We’re potentially running on multiple architectures, and we don’t want to have to maintain multiple executable scripts just to fetch our dependencies.

Go’s first cut at solving this was the vendor directory. This directory lives in the same tree as the Go source and can be committed to the VCS, so one could get the required sources into the vendor library, then commit the “known-good” version. This works for the versioning problem, mostly, but means that it’s easy for many slightly different versions of those libraries to end up spread across multiple source code repositories, and keeping them synced up for fixes is difficult, and it doesn’t address the transitive issues at all. To fix this, the Go community built unofficial source management tools to handle versioned access to the vendor directory plus automated detection and resolution of transitive dependencies.

The problem is that because the Go community is large, inventive, and active, we have a lot of them. We’ve already used two different tools: Godep and, currently, glide, and are probably going to switch to dep, which looks to eventually be the standard dependency management tool blessed by the Go core team. [Update: wrong again. go mod is the current official winner.]

glide (our current tool, as noted) manages dependencies with two files: glide.yaml, which describes enough of the direct dependencies and their versions that all of the dependencies and their own transitive dependencies can be figured out. The glide.lock file stores the results of this dependency resolution as specific VCS commits (SHA1 hashes in the case of Git), allowing us to quickly fetch exactly what we want when getting ready to compile the code.

Like any other piece of software, the glide files have to be kept up to date, especially if there are dependencies on outside libraries (from Github and the like) by periodically doing a glide update to update dependencies in the glide.lock file that aren’t locked to a specific version (or range of versions) by glide.yaml. If one falls behind on this, or a change such as the Sirupsen/logrus to sirupsen/logrus one happens, or you simply need to upgrade something to a new version, these files can end up in a state where a glide install still works, because this simply downloads the revisions dictated by glide.lock without attempting dependency resolution again, but glide update doesn’t, because the glide.yaml didn’t limit the possibilities enough, and attempting resolution of the dependencies fails.

To fix this, we can do it one of two ways:
1. The right way, which entails plodding through all the revisions until we’ve found a new set that works, fixed the glide.yaml file so that it defines that new set, and then used glide updateto download them and rewrite glide.lock. This can be excruciatingly difficult, as it’s possible that the updated glide.yaml will no longer resolve, or will resolve the dependencies in ways that won’t actually build, and there will have to be many update/download/compile cycles to actually fix the issue.
2. The wrong way, which is to muck around with glide.lock directly, adding or changing something without making sure that glide.yaml “compiles” to the updated glide.lock. This gets us back on track with code that builds and runs, but leaves us in the dangerous situation that glide update is now broken.
The detailed fix

If you näively go the wrong way and just make changes to the glide.lock file, glide tries to be a good citizen and warn you that you’ve done something you ought not to:
[WARN] Lock file may be out of date. Hash check of YAML failed. You may need to run 'update'
appears when you glide install.

As noted, the problem is that if you run glide update, you’ll break everything because you didn’t fix glide.yaml first. And maybe you just don’t have time to find the right incantation to get glide.yaml fixed just now.

So, you lie to glide, as follows.
1. Add the dependency to glide.yaml.
  - Edit glide.yaml and add the dependency plus its version if it has one. (Use master if you want to track HEAD or a specific SHA1 if you want to pin it to that commit.)
    - package: github.com/jszwec/csvutil version: 1.0.0
2. Add the dependency to glide.lock.
  - This one must be the SHA1; the easiest way to get this is to go to the repository where it lives and copy it down. I won’t go into detail here, but however it works in your VCS, you’ll need the full SHA1 or revision marker.
    - name: github.com/jszwec/csvutil version: a9cea83f97294039c58703c4fe1937e57ea5eefc
3. If we stopped at this point, we’d get a warning from glide install that would recommend that we use glide update instead to install the required libraries. In our case, with a delicate web of dependencies between local libraries and Echo, openzipkin and Apache Thrift, and the two different versions of logrus, a glide update breaks one or more of these dependencies when we try it. To prevent someone else from spending way too much time trying to resolve the problem by juggling versions in the glide.yaml in the hope of creating a stable glide.lock, we need to fix the computed file hash at the top of the glide.lock file so that the warning is suppressed.
  
  This is a hack! The best option is probably to import all the SHA1’s into the glide.yaml file as versions, ensure glide update works, and then gradually relax the constraints until glide update fails again, then back up one step.
  
  To calculate the hash, we can go get mattfarina/glide-hash, which creates a new glide hash subcommand that does exactly that and prints it on the console.
  
  We install the subcommand plugin as noted, then cd to the codebase where we need to fix the glide.lockfile. Once there, we simply issue glide hash, and the command prints the hash we need. Copy that, edit glide.lock, and replace the old hash on the first line with this new one.
  
  Warning!
  
  This is absolutely a stopgap solution. Sooner or later you’re going to need to update one or more of the libraries involved, and you really will want to do a glide update. Yes, you could keep updating this way, but it would be a lot better to solve the problem properly: go through all the dependencies, update the ones you need, and then make the necessary fixes so that your code and the library code are compatible again.
March 14, 2018

Postgres array_to_string() and array_agg() to decouple your interface

Let’s say you’ve got a collection of job data for a set of users that looks like this, and you want to create a nice summary of it to be displayed, with a count of how many jobs are in each category for each user, all on one line.

 │id │ user_id │ job │ status    │
 ├───┼─────────┼─────┼───────────┤
 │ 1 │ 12      │ 1   │ Completed │
 │ 2 │ 12      │ 2   │ Cancelled │
 │ 3 │ 14      │ 3   │ Ready     │
 │ 5 │ 14      │ 4   │ Completed │
 │ 6 │ 14      │ 4   │ Completed │
 │ 7 │ 14      │ 4   │ Cancelled │
 ...

Here’s the report of summarized statuses of jobs for each user that you want.

│ user_id │ summary                           │
├─────────┼───────────────────────────────────┤
│ 12      | 1 Cancelled, 1 Completed          │
│ 14      | 1 Cancelled, 2 Completed, 1 Ready │

I’ll show you how it’s possible to provide this solely with a Postgres SELECT.

(more…)

February 15, 2018

Reducing Google access for Pokemon GO
Pokemon GO players on iOS: the new release today (7/12/16, in the App Store now) reduces the information it wants from your Google account from “full access” to your email and “know who you are on Google”. If you were already signed up, do this:
- Go to accounts.google.com; log in if you’re not already logged in
- Go to https://security.google.com/settings/security/permissions
- Click on “Pokemon GO release”
- Revoke privileges
- Go to your iOS device
- Download the updated app, wait for it to reinstall
- Kill the app; if you don’t know how to do this, just power your phone off and back on again
- Launch Pokemon GO; it’ll fail to get access to your account. THIS IS OK.
- Tap “try another account”
- Log back in with your Google username and password.
- This time it should ask for only “know your email” and “know who you are”.
At the time I write this, it looks like many people are doing this, as the Pokemon GO servers are rendering the server overload screen:

For the paranoid: It sounds like the iOS programmers just screwed up and released without reducing the account permissions request; this is not a nefarious scheme to steal all your email and Google+ naked selfies. From Niantic (via Kotaku):

We recently discovered that the Pokémon GO account creation process on iOS erroneously requests full access permission for the user’s Google account. However, Pokémon GO only accesses basic Google profile information (specifically, your User ID and email address) and no other Google account information is or has been accessed or collected. [Emphasis mine – JM] Once we became aware of this error, we began working on a client-side fix to request permission for only basic Google profile information, in line with the data that we actually access. Google has verified that no other information has been received or accessed by Pokémon GO or Niantic. Google will soon reduce Pokémon GO’s permission to only the basic profile data that Pokémon GO needs, and users do not need to take any actions themselves.
July 12, 2016
A quick note on Mac malware
The most recent bit of OS X malware making the rounds is a fake file converter program that installs a PHP backdoor accessible via Tor, allowing some rando to rummage around on your machine. As is usual, you, the victim, have to download the software, and run it, and allow the program to install its backdoor code.

If you want to protect yourself from malware on your Mac, there are three principal things you can do.
1. Don’t download software if you’re not 100% sure of its provenance, function, and dependability. Just because it’s on MacUpdate doesn’t mean it’s okay.
2. If you do accidentally download something – say you’re using one of those dodgy file-sharing sites with fifteen ads masquerading as the real download button – don’t run it. Just delete it.
3. If, despite all this, you did download something and run it, under no circumstances enter your administrator password when it asks to install “support software” or the like unless you know exactly what’s going to happen if you do.
You still have the backstop that nothing is going to get installed as a persistent backdoor if you don’t enter your administrator password when prompted, but it’s trivially easy to build a backdoor that runs only when the software is running. Don’t run random programs you aren’t sure you can trust. Find a trusted third party that knows for certain that the program is safe.

If you insist on living dangerously, there are a couple utilities I’ll mention below that will allow you to try to prevent the damage, and I emphasize try. They are no guarantee of safety; if you download pirated software or random things to “try them out” or “see if they work”, you’re sooner or later going to mess yourself up. These monitors are counting on your personal software sophistication to protect yourself from harm. They are only useful if you understand what they are telling you.

I run the programs I mention below not because they magically keep me safe from bad programs, but because I like to know what’s going on behind the scenes on my Mac. If you aren’t certain you know what ~/Library/LaunchAgents does, or what a connection over port 443 means, you may not want to try using these programs, because they will confuse you; if you try to use them by simply blocking everything, you’ll find that things that are actually supposed to make outgoing connections (like Mail) will stop working, and that software that really needs to install agents, like Dropbox, will break. Conversely, if you just say “yes” to everything, things like the fake file converter mentioned above will get to install their hooks and they will allow who knows who to read your mail and download your naked selfies.

If I haven’t lost you at this point – you understand OS x/Unix well enough to understand what connections are good and what ones are bad, and you know what a file in ~/Library/LaunchAgents is for:
- Little Snitch is a program that sits in the background and alerts you whenever your machine tries to make a network connection, whether incoming our outgoing. If you don’t respond, the connection is automatically blocked. You can add rules to automatically allow or automatically block connections. This utility will let you know if someone is actively trying to connect to your machine, or if your machine is trying to make an unexpected outgoing connection.
- BlockBlock is a utility that monitors attempts to install long-running processes like the one that constitutes the Tor/PHP backdoor and reports them to you with the option to block them. In the case of EasyDoc Converter, it’d be pretty easy to spot that the software was up to no good, as it’s attempting to install stuff named “dropbox” in an attempt to hide their nasty software as part of good software.
As helpful and useful as these monitors are – I run them, and I like them – they’re still not going to 100% protect you from what happens if you run random things you download from the Internet, especially if you say “sure, why not?” when they ask for your administrator password.

Just avoid the off-the-wall random links and wait until someone reputable has said, “yeah, that’s good” before trying it.
July 8, 2016
Squashing commits in the same branch

Okay, I’m sure this is documented lots of places, but I just now figured out that git rebase -i is perfectly happy to rebase from a commit on the same branch, so if you’ve got a branch that you’d like to smoosh some commits on, do a git log to get the SHA1 of the initial commit, then

git rebase -i <SHA1>

It will happily show you all the commits and you can then twiddle to your heart’s content to clean up.

March 18, 2016

Author: Joe McMahon

TL;DR

The detailed explanation

The detailed fix

Warning!