I have found a couple of howtos on how to set up WiKID to use SSH, but I had a horrible, horrible time using them.  This is partially because of my version of Linux (RHEL 5 and 6) and partially because the howtos don’t do a good job of explaining why you are taking various steps.   I want to remedy that issue, so that others can learn from my pain.

Background information

Most of you probably know that on Linux, at least, ssh uses PAM to authenticate users.  PAM is a pluggable tool that allows you to control how different applications authenticate users when they try to log into a machine.   By default, typically, PAM will provide instructions and rules that ensure that users are authenticated against the /etc/passwd database.

But you can include other tools in the authentication system (and in fact, you can write you own).   One of these PAM rule libraries is focused on connecting PAM to RADIUS – an authentication protocol that was very popular back in the dial-up modem days.     It continues to live on in various implementations, including, most popularly, FreeRadius.

FreeRadius

FreeRadius was built many years ago, and relatively recently, a new version was created: FreeRadius2.     Because of the relatively glacial pace of Linux software change (not necessarily a bad thing), many of the documents and howtos still reference FreeRadius, instead of FreeRadius2.   More frustratingly, FreeRadius2 is sometimes labelled as simply “FreeRadius”, which can be confusing.

FreeRadius and PAM

FreeRadius includes a module, specifically built to help integrate RADIUS with PAM.   That would be the aptly-named PAM Authentication and Accounting Module.  Now, as far as I can tell, it only comes as source, so be prepared to build it. (I hope you know how to use/install Make and cc/gcc, etc).

In addition, you can’t just download it and build it, you need to install the FreeRadius base RPM, as well as the developer and utility RPMs.   After that, you should have a pam_radius_auth.so which you can install into /lib/security or /lib64/security, depending on your OS.

pam_radius_auth.so

This module is fairly straightforward – it attempts to find a configuration file called /etc/raddb/server, and looks in the file, to determine which RADIUS server to authenticate against.   It then sends that server UDP packets on port 1812, and if there’s a RADIUS server listening there, the server can answer and validate the username/password provided.

Updating PAM’s sshd configuration

On my system, I have to update the /etc/pam.d/sshd file in order to include the pam_radius_auth.so .  I found three different examples of how to do this on the Internet, and all three were different, so I won’t pretend that I know the “right” answer here.  Let me just say that I put my entry fairly early in the config file, which seemed like the correct thing to do.  On my system:

auth	   sufficient	pam_radius_auth.so  debug

Is the very first line.   debug is there so I could get some logging output of what was happening, to help troubleshoot.  Once you’re live, remove it.

The situation so far

At this point, if you’ve followed the steps  described above, when you log in with ssh, it will attempt to find a RADIUS server and authenticate against it.

“Well, I have just installed FreeRadius, why don’t I use that” your instincts might say, at this point.  But your instincts are wrong.

In my odyssey to get this software working, I have discovered that, at the end of the day, once you have the pam_radius_auth.so compiled, and you have configuration rules specified in /etc/raddb/server, you could (if you wanted) safely delete FreeRadius.  You don’t need it anymore.  Keep the utility and dev libraries though.  You’ll need those later.

Setting up /etc/raddb/server

Again, as I mentioned earlier, pam_radius_auth.so is preconfigured to look for a configuration file named /etc/raddb/server.  This file contains the following line:

127.0.0.1	your_shared_secret       3

What does this do?   Well, it tells that library (pam_radius_auth.so) that the RADIUS server it wants to talk to is running on localhost, on port 1812 (by default), and it “knows” the shared secret key “your_shared_secret“.

Completely unnecessary test

If you had a wild hare, or you just like billing by the hour, at this point, you could set up FreeRadius, add a fake user to /etc/raddb/users, and see if you could log in as that user, with the username and cleartext password specified in the /etc/raddb/users file, and the shared secret “your_shared_secret” specified in the /etc/raddb/client.conf  . There are already guides for doing this, if you feel compelled.

Will it work for ssh at this point?

No, not until you add that user to /etc/passwd.  LFMF!

Ok, done.  Now will it work for ssh?

Probably.  But that’s not what you want.  You want to use WiKID, right?  (otherwise, why are you here?).   That’s a bunch of additional steps along the path.

Ok, so how do I set it up with WiKID?

Well, first off, install WiKID.   Set up the server using their documentation.   Note that if you try to move it off of ports 80 and 443, you will be miserable.   LFMF!

When it comes time to set up a Network Client, set up a RADIUS client, using “your_shared_secret“.

IMPORTANT!

When you do this, make sure that FreeRadius isn’t running.  Also, make sure that you restart WiKID after you add the new Network Client.

Basically, when you add a RADIUS network client to WiKID, WiKID becomes a RADIUS server!. I cannot stress this enough – it took me almost two weeks to realize that I was trying to connect a server to another server.  Yes, I am dumb.   LFMF!

Ok, I have a network client

Well, now you need to add a WiKID user.  This was relatively straightforward from the documentation, except for one little detail – when you install the wikid software token on a laptop, PC, phone, whatever, you have to:

a) Register it back with the WiKID server

b) Make sure that the user exists in /etc/passwd

c) You can’t ever re-use a registration with a given username.  So if you, say, end up testing the client on two different computers, with two different registrations, you have two different users. If you attempt to log in with the client token registered for user A, but with the username for user B, it will silently fail.

Testing

You could try ssh at this point, by following these steps:

1) get your wikid temporary passcode

2) ssh with the username associated with the passcode, and use the passcode as the password

3) If you have a shell, congrats!

If you have problems, there are some pretty good logs built in to the WiKID web ui.   Set various things to debug, and you’ll see more info.     Note that these logs disappear after restarting WiKID, so stay vigilant about turning them on.  note also that after a few failures, your users are silently disabled, which will cause logins to silently fail, even if everything else is correct.  Stay vigilant!

Note also that you can use the Freeradius test tool (radtest) to test your configuration, if ssh isn’t cooperating (or if you don’t want to change ssh until the very end).

Testing with radtest

radtest <username> <wikid_passcode> 127.0.0.1 0 <your_shared_secret>

username: the username in WiKID and also in /etc/passwd

wikid_passcode: the temporary 1 minute passcode you get from using the WiKID token client.

your_shared_secret :  the secret you used to configure the RADIUS Network Client in WiKID.

I can say that the logs you get from using radtest are easier to follow than using ssh directly.

Fin

At this point, you should be able to use ssh to log in, using the WiKID username, and the temporary passcode.   Good luck!

There are really two subsets of programmers here.   First, this is the catchall for everyone who doesn’t fit into any of the other categories.  A bit of a cop-out, but then, it’s been six years since I started this list, so perhaps you want to just cut me some slack maybe

The second subset is real – it is the programmers who change things at random when something isn’t working.

Good Things:

• There are situations, like debugging or when you’re trying to learn something new, where changing things at random and finding out what happens next can be incredibly illuminating.
• This technique helps you learn how the software works – it helps you build your mental model (assuming the software is consistent), or it angers and enrages you when it (the software) is inconsistent.

Bad Things

• Unfortunately, in my experience most Random programmers are people who don’t understand what they are doing, and change things at random in production systems, to see what works.
• And when they find anything that works, they stop.
• Which leads to nasty, horrible, confusing, mismatched code – pretty much the worst of all possible “successful” worlds.
• Big Ball of Mud (the famous anti-pattern) is almost certainly a by-product of Random programming
• Which implies that there are a _lot_ of them out there.

How to identify them

• The Good ones say things like:
  • “I picked up this new library/framework/app/etc – and I’m trying to build something with it to learn how to use it”
  • “I discovered, through trial and error, that if you do X, Y happens.  After figuring out why, I was able to solve the problem in a consistent way”
• The Bad ones say things like:
  • “I found some code that seemed to solve a similar problem, so I copied it.”
  • “I discovered, through trial and error, that if you do X, Y happens.  So it’s fixed.”

just a quick rant while i build up steam on this particular issue.

Why oh why is this so ridiculously hard? I can find lots of advice on how to get freeradius1 integrated with pam, but the naming conventions and the shared libraries are different in FR2, and for the life of me, I can’t find any indication of which FR2 library I need to add to PAM so it (PAM) can use FR2 to authenticate a user. BLEAH!

This blog post discusses the philosophy of not trying to catch every exception – let the app crash when you don’t understand or anticipate the problem

I agree that you shouldn’t just catch Exceptions, politely log them and then swallow them.    You then place the burden on yourself (or the operations guys) to actually realize that there’s a problem – the customer may not know – they may just think the app is horrible.   Without operational feedback, you’ll never know that the app is misbehaving.

Here’s what I do – in the situations where an unexpected Exception is thrown, I capture the exception, log it, send a polite error message to the user explaining that something went wrong, and then I email the stack trace to myself.

That way, I know when things have gone wrong.   This has saved me a dozen times – errors I didn’t know about, issues with disk space, issues with an edge condition that I thought was practically impossible.

So, my advice: don’t ignore the exceptions, don’t “just let it crash”.

instead: Recover, and behind the scenes, sound the alarm.

After my struggles with OpenLDAP, I have to admit – OpenVPN’s commercially packaged version (which is free to set up and use for the first 2 simultaneous users) is incredibly easy to get up and running.   And additional users are $50/10, which is chump change.

Well, not really, but I bet I got your attention.

Having just gone through two days of pain involved in getting OpenLDAP to work in place of /etc/passwd, I have such a set of troubleshooting tips unlike anything I’ve experienced before.   I am going to write these down as quickly as I can, before the horrifying experience is mercifully driven from my mind.

Note that this was using RHEL 6.0 and yum

• Recent (as in 2012, perhaps 2011) versions of OpenLdap use a different config structure than the older version.  Dramatically different.   Many of the examples I found on the ‘net are based on the old model, and can be very, very confusing.
• Lots of examples discuss creating your own database of stuff.  But you dont have to do that!  There’s already a database ready and waiting for you in: /etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif
  • Update that DB, instead of creating your own bdb, which is what most of the examples do.
• I had several problems getting syslog to properly capture the OpenLdap logging events.   I had to add an olcLogFile= directive to actually get things to log.
• This was pretty clear, but just for the sake of thoroughness, I had to add a ‘top level’ user who could do most management functions.  This was very helpful with creating users, setting passwords, etc.  I created:
  • olcRootDN: cn=Manager,dc=example,dc=com
  • olcRootPW: {SSHA}Cw888Xm4M77777Zb8Zhut7LLgwHfb3y8b

• • in the bdb database (described above).
    • use slappasswd to generate the {SSHA} password from clear text.
• The examples of rules to set up ACLs so that users can access their own data, and change their own password were woefully unclear.   I kept getting this error in the logs: “RESULT tag=103 err=50 text=” .  That’s because the ACLs weren’t set up right.
• To set up the ACLs so your users can access their own data, and change their own password (but no one else can even see it), here were the magical lines I had to include in: /etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif
  • olcAccess: {0}to attrs=userPassword by self write by anonymous auth by dn.base=”cn=Manager,dc=example,dc=com” write by * none
  • olcAccess: {1}to * by self write by dn.base=”cn=Manager,dc=example,dc=com” write by users read by * none
• I had to install all of the following packages to successfully get ssh to connect to OpenLdap
  • pam_ldap
  • openssh-server
  • openssh-clients
  • openssh-ldap
  • openssh
  • openldap-clients
  • openldap-servers
  • openldap
  • nss-tools
  • nss-sysinit
  • nss
  • nss-util
  • nspr
• I had to modify the following config files:

  • /etc/openldap/ldap.conf

  • /etc/openldap/slapd.d/cn=config.ldif

  • (the aformentioned bdb database, which is where the various users are stored.  more about that later)
  • /etc/pam_ldap.conf
  • /etc/nsswitch.conf
  • /etc/nslcd.conf
  • /etc/sysconfig/authconfig
  • /etc/sysconfig/ldap
  • /etc/ssh/sshd_config
• I used authconfig to enable ldap in PAM.  Wait, that sounds really dirty.
• I had to copy /usr/share/openldap-servers/DB_CONFIG.example into /var/lib/ldap/DB_CONFIG to get the database to stop complaining about not having a configuration
• I had to define the People and Group organizational units, and use them consistently.
• Learn how to use ldapadd to add data to the database.  Clunky, but easier than adding it by hand.

Here’s what I used to create my People and Group organizational units:

dn: dc=example,dc=com

objectclass: dcObject

objectclass: organization

o: example.com

dc: example

dn: ou=People,dc=example,dc=com

objectClass: organizationalUnit

objectClass: top

ou: People

dn: ou=Groups,dc=example,dc=com

objectClass: organizationalUnit

objectClass: top

ou: Groups

After that, I could add users to both the People and Group domains:

Adding a user to People:

dn: uid=jbsw,ou=People,dc=example,dc=com

objectClass: top

objectClass: account

objectClass: posixAccount

objectClass: shadowAccount

cn: John SW

uid: jbsw

uidNumber: 1001

gidNumber: 1001

homeDirectory: /home/jbsw

loginShell: /bin/bash

userPassword: {crypt}x

shadowLastChange: 0

shadowMax: 0

shadowWarning: 0

I took that data, wrote it to a file called user_jbsw.ldif, and then imported it using ldapadd:

ldapadd -x -D "cn=Manager,dc=example,dc=com" -f /etc/openldap/user_jbsw.ldif -W

and to group:

dn: cn=jbsw,ou=Groups,dc=example,dc=com

objectClass: top

objectClass: posixGroup

cn: jbsw

userPassword: {crypt}x

gidNumber: 3001

I wrote that data to a file called group_jbsw.ldif

ldapadd -x -D "cn=Manager,dc=example,dc=com" -f /etc/openldap/group_jbsw.ldif -W

And last but not least, (for that user) I used ldappasswd to change that password to something that ldap and ssh would accept:

ldappasswd -s blarneystone -D "cn=Manager,dc=example,dc=com"  -W -x uid=bsw,ou=People,dc=example,dc=com

Then what happened?

I had everything set up properly – I had pam_ldap.conf pointing to the LDAP server, the LDAP server was running, sshd was configured to use PAM, but I still couldn’t log in.  It kept complaining about unauthorized access (error 49).   Eventually, I realized that I hadn’t installed the nss package(s), which created yet another way to describe how to talk to ldap.

And then all was well?

Well, no, after I figured that out, and got the password working, it demanded a new password.  I tried to enter it, but it wouldn’t accept it, saying I didn’t have the rights to modify the user object in LDAP.  That was when I figured out the ACL issue (already described above).

And then you were done?

Well, no, in the process of monkeying around with /etc/pam.d/system-auth, I inadvertantly set up the ‘change password’ functionaity so it asked for and verified the new password twice.   But that was just a matter of editing.

Anything else?

• modifications to  /etc/ssh/sshd_config require a restart of the sshd daemon
• modifications to /etc/nslcd.conf require a restart of the nslcd daemon
• modifications to openldap config require a restart of slapd
  • I also deleted all the data in /var/lib/ldap/ each time I modified /etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif
• modifications to PAM were instantaneous.

Hope all my pain helps someone, somewhere.

Bryan Caplan has a lot of thoughts on signaling as the motivator for education.   The idea makes a lot of sense, up to a point.   But let’s  go back and re-assemble my college mental model, as best I can.

So I sign up for a class, and I pay my class’s $1000 tuition up front.  It’s now a $1000 sunk cost.   Let’s say there will be 50 hours of classroom lecture between now and the end of the semester.    Let’s also say, for argument’s sake, that the class is something topical to my expected career path.

And due to illness, the first class is cancelled.  Woo hoo.  Yes, I’ve lost 2% of my class time, but I am quite sure that I can still learn 100% of what I need to learn (because a lot of class is fluff or bookkeeping) in 98% of the time.

The second class is cancelled.  Still happy about it.

Third class is cancelled.  Okay, first week, nothing.  I’m probably still happy, but starting to get a little concerned.

Fourth class is cancelled.  Arg.  Fifth. Arg.   Sixth?  Um, really?   Seventh, Eight, Ninth – ok, at this point, I’m annoyed, and I start complaining.

So, this seems useful – after about 20% of the class has been cancelled, I start to feel a little cheated.   I paid to learn something useful and potentially interesting, and I’m not getting my money’s worth.   If it was all about signaling, I wouldn’t have cared – I would have happily gotten an ‘A’ and never gone to class at all – no one would really be the wiser.

What makes more sense to me is that I accept that the class isn’t a perfectly efficient mechanism for skill transfer, and that a modest amount of “slack” is not going to meaningfully impact what I learn.

Another way to think of it is that the knowledge I gain from a class is not homogenous – not every skill, not every fact is of equal value.  And it’s reasonable to assume that if the class is cancelled, that what “falls off the table” in terms of skills learned are the least valuable skills and the least useful facts – the most critical skills and facts are still taught, just a day later.

Or, to wrap it up in economese:  the marginal cost of a single cancelled class is low, based on the rational expectation that the professor will adjust her schedule and ensure that the most valuable skills are still taught.

Interesting – if it works and deploys well, HPCC is a good thing for the scalable processing space.

Either:

Okay, look. We both said a lot of things that you are going to regret. But I think we can put our differences behind us. For science… you monster

or

Do you know the biggest lesson I learned from what you did? We’re a lot alike, you and I. You tested me, I tested you. You killed me, I – oh no, wait. I guess I haven’t killed you yet. Well … food for thought

I have to give great props to the Google Android team. I was able to go from “I want to build an app for my phone” to “My app is now published on the Android market” in approximately 90 minutes of effort.

1) Download the SDK – that was easy.

2) Add the plugin to Eclipse – also easy

3) Set up the various platforms in the SDK “system”. This was confusing at first. Eventually I realized that one wants (typically) to go for ‘lowest common denominator’ – so Android API 1.5

4) Find a skeletal app to rework. I was writing a soundboard, so it turned out to be incredibly easy to repurpose.

5) Make my app changes – easy, because of the template.

6) Load the emulator and try out my changes. This took a little while to get right, because I didn’t have the target set up properly in my app.
7) Run the emulator and test my code – straightforward
Create a package-able version, and load it directly onto my phone

9) Use Astro file manager to install the app from the phone itself – easy

10) Verify that the app worked on my phone – easy. surprisingly so.

11) Create a signed package for publishing (this took a little while because the wizard was a little confusing, but not horrible)

12) Sign up as an android developer. This cost me $25, but was very easy.

13) Upload my app. This took a little while because I had to rename it (there was a conflict), create screenshots, etc.

14) Publish.

Within minutes, the app was live on the Android Marketplace.

I have spent a long time working in software startups of various stripes, and I have also spent a long time working with Agile teams delivering software of various stripes.

It is  difficult to sufficiently emphasize how similar those two models are.    Think about it:

Read this fascinating article about Dropbox, and how it succeeded where its competitors did not. It should help you down this path.

If I could save Time in a Timesheet
if Jobs could make iPods call you
Google would cache every page til Eternity passes
and then it would trend them for you….

at least for me:

find . -name \*.jar -exec jar -tf \{\} \; -print > ../alljars.txt

What this does:

1. find every file in or below this directory with an extension of .jar
2. create a listing of every file in that .jar file
3. outputs that listing, including the name of the originating jar file at the bottom into one big text file:  alljars.txt

You can now search for classes/resources you need in the alljars.txt file – and when you find it, simply scroll down to find the name of the jar file the class you are looking for is in.

But not for the faint of heart:

http://db.cs.berkeley.edu/jmh/calm-cidr-short.pdf

разтегателни диваниA physicist may be described
(to first approximation)
As a simple prolate spheroid
Of infectious obfuscation.
Attempts to oversimplify
Reveal their odd propensity
To speak of spheroid cattle
Which are uniform in density—
Their perfect planes are frictionless;
Collisions are elastic;
They’re rarely seen acknowledging
The random or stochastic.
The chaos of the world outside
May leave them full of fears;
Such terra incognita
Might be filled with… Engineers!

I read this, and I am somewhat idignant:

Big Ball of Mud still seems to be the most popular way to design and architect software.

Just because something is ‘common’ doesn’t make it popular.   Your standard everyday cold is pretty common, but it is not popular.   Traffic jams are common, but I doubt anyone involved in them thinks that they are popular.  Wading through bureaucratic red tape is common, not popular.

Primarily, BBoM happens because cost-benefit analysis is time-consuming and difficult.   If programmers, architects and managers could measure and understand the longer-term cost of short-term poor decision making, we would get better decisions.    Remedy?  There’s no one magic bullet, but I suspect that a focus on code coverage and limited code complexity is a great place to start – I don’t know that I’ve ever seen a BBoM project with high code coverage and low complexity.     This may be correlation and not causation, but I think there’s a legitimate story for how forcing unit testing discipline and simplicity pays dividends in terms of architectural strength.

I’ve been using Bloglines for a long time, since  2004 if I’m not mistaken.  It’s been a constant and welcome part of my online experience.

Alas, apparently, they could not find a way to make money off of it.

Which is unfortunate, because I always thought that it would have been a fabulous corporate knowledge-sharing tool – a “private” Bloglines, within a company, that you could add subscriptions that others could use to stay up on important events and thought-leaders in your industry.   Saving and ranking specific posts so that they would potentially become more widespread – identify competitive threats and potential strategic opportunities.

I realize that this model is not without challenges (“We don’t want our employees visiting the web!”), but I’m sure there would have been some organizations with enough strategic vision to see the opportunity inherent in such “corp-sourcing”.    Enough, I would imagine, that they could have made enough money to keep the public site going.  Alas

Also, I see various mentions in the Blogline obituaries that suggest that the day of the RSS reader is done – that we’re replacing it with social link-sharing like Twitter or Facebook.  As someone who generally produces more of these links than I consume, I am puzzled – RSS Readers allowed me to review a wide assortment of feeds at my leisure – Twitter and Facebook are far more ephemeral and constrained to the strategies I use to follow people and to be followed in return.

Programmer who cares deeply about performance disagrees with claim that ‘Premature Optimization is the root of all evil’

Glenn Alleman, who is often critical of some of the less structured aspects of Agile  (not in a nasty or spiteful way) mentions a project he is working on:

I’m working a moderate ($300M) Army program through January

Moderate?  That’s a jaw-droppingly large amount of money, and IMO, it explains a lot of the friction between his perspective and a more classical “agile” perspective – agile projects are (in my experience) 100-1000x less expensive, with a corresponding lack of scrutiny/accountability/oversight from management.  Many of the problems/issues that agile was designed to resolve would never happen on a project that large, because that’s way too much money to be sloshing around without high levels of management accountability.

Anyways, I am endlessly fascinated by all the different ways that people can build things.