What I learned about screening coders

A lot of you know that I have been coding since I was eleven. When I graduated from Caltech and started my first startup, I began interviewing other coders. How to interview properly and what to ask during interview questions were always a bit of a challenge. Sometimes I was confident in how I screened applicants, sometimes I walked away thinking whether I asked the right questions to properly filter candidates who applied. After all, I believed myself to be a good coder, so I felt confident in my ability to interview other coders.

Overall, I thought I was doing a good job! I built my own way of interviewing engineers and built series of interview questions to ask coders during interviews. These interview questions became more complex as my requirements became more specific. Other people even asked me to interview coders on their behalf, as a lot of them didn’t know how to code and wanted help in their interview process. I gladly helped.

Then, 14 years later, after having screened tens of thousands of resumes and personally interviewing thousands/hundreds of coders, I realized that I was doing this all wrong all along…

Let me explain.

Permanent Knowledge vs Transient Knowledge

It was the beautiful morning in the fall of 2018. I was teaching students in our San Jose campus in Silicon Valley, one of the campuses for a school I founded (Coding Dojo).

I just got done teaching a group of 30+ students how to approach advanced algorithms and data structure questions. I had just taught them the concept of a linked list, binary search tree, and a hash table; some of the tough interview questions that were asked during interviews, and how to approach these problems. The students were attentive and I was excited to help them along.

After the lecture, as I was reflecting back on how the class went and what I could have done better, I couldn’t help but to think about why it is that some of the students do really well on these data structure concepts consistently and how some students really struggled. I wondered what I could do better to better help my students who were working so hard to learn how to code.

Many topics crossed my mind as I sat in my campus in San Jose, CA, deep in thought. A big question weighed on me also, which was “was there a way where I could more quickly and accurately assess which of my students would end up doing really well in data structures, say within weeks after they just began to learn how to code?” Then a thought entered into my mind… It’s hard to describe what this thought exactly was but in a way, it felt like someone telling me this:

“Michael. In life, there are permanent knowledge and transient knowledge. Permanent knowledge are things that one never forgets. One learns and this knowledge becomes permanently engrained in them. There are also transient knowledge, which if you don’t put in the time every day, you will soon forget… When you are thinking about assessments, also consider which bucket of knowledge you’re testing.”

It’s hard to describe in words how this insight came about. It seemed to come out of nowhere and it was more of a feeling that a thought really. This was interesting, because I never thought of knowledge this way and never really thought about breaking ‘knowledge’ into two different buckets.

Then, I thought about this more… What if it was like this?

Imagine that you learned how to ride a bicycle. Initially you fall, you struggle, but eventually you get it. Once you learned how to ride a bicycle, even after years of not riding a bicycle, you know you can ride a bicycle. It’s not like just because you have not ridden your bicycle for a year, you’ll forget how to ride a bicycle. So, the knowledge on how to ride a bicycle, is a ‘permanent knowledge’. Once you learn, you will never forget.

I then thought, “Okay. This makes sense but what are some examples of transient knowledge?” As I asked myself this question, I then thought about data structures questions, which we as a class just reviewed.

“Would data structure questions belong to transient knowledge or permanent knowledge?”, I thought.

I think an overwhelming percentage of coders would agree that advanced data structure questions and how to solve them eloquently on the whiteboard is something we will have to brush up again whenever we prepare for technical interviews. If you haven’t done a complex problem in binary search tree, linked list, tries, or graphs, and if I had asked you to do it on the whiteboard right now and do it within 10 minutes, most people would struggle unless they have studied these topics again just recently.

“Okay. So this means data structure questions are actually testing transient knowledge…”

“Let’s take this a step further”, I thought. “What about when I asked questions about a specific Ruby question, or questions related to Angular, React, or Node.js, or Python, or Django… Would these be questions testing permanent knowledge or transient knowledge?” The rule was simple. If it is something one forgets if they haven’t applied that knowledge recently (weeks or months), then it is a transient knowledge.

Majority of developers would agree that any specific programming syntax questions or any questions related to a specific framework or libraries, unless they have been actively coding in those languages or frameworks, it’s hard to remember the syntaxes and tricky questions specific to that language or framework, especially when developers have to jump back and forth between different programming languages and tools so often. As different versions of the same language or tools also change syntaxes and how things are implemented, staying on top of this for all the major programming languages and frameworks is just not possible…

“Okay. So even asking questions related to a specific programming language or any particular framework or a library are really questions testing one’s transient knowledge.” This was interesting as I had never really thought of these types of questions to be ones that are testing “transient knowledge” of developers.

“So then, what is a test where I can test people’s ability to code that’s on the permanent knowledge category?”, I asked myself.

This also made me think about my journey as a coder. I learned to code when I was eleven. I taught myself Quick Basic, C++, and a bit of assembly, to create computer games (back before we had Windows). Then later I picked up other programming languages such as Visual C++, Perl, Fortran, PHP, Ruby, Python, Java, C#, Objective C, Swift, etc.

A thought entered into my mind:

“Michael. If someone was to interview you when you were twenty five, if they had asked you advanced data structure questions, would you have been able to answer these questions?”

The answer was a strong NO, I learned how to code myself. I didn’t pursue a CS degree so I never learned about binary search tree, linked list, hash table, tries, graphs, etc. In fact, I have never used any of these concepts in my 25+ years as coder, except to interview candidates. None of the professional applications I built over the years required me any knowledge related to these areas anyway. It was after I was 25 years old that I picked up some of these data structure knowledge, mainly really to interview coders.

“Okay. So if someone asked you advanced data structure questions when you were 25, you wouldn’t have passed their interview?”

Grudgingly I answered, “yes…”

“Okay. Now…. are you as good of a developer now as when you were 13?”

13 is a significant age for me as that’s when my brain was re-wired to think like computers. Before that, I struggled on creating what I imagined (for about two years). After the brain was re-wired somehow, when I was 13, I could then create anything I could set my mind to. I could construct all problems in the world into simple building blocks of code and have learned how to cleverly arrange these building blocks to create whatever I was thinking about (whether that was a computer game I was creating back then, or later shifted into a web application or a mobile app later as I aged).

Another question popped into my mind: “If someone asked you advanced Ruby or Javascript questions, early in your tech career, would you have been able to pass those assessments?”

I answered, “No.” I didn’t pick up Ruby or Advanced Javascript until about 10 years ago, so if someone asked me trick question related to that or any advanced React or Angular questions, I would have failed miserably.

Taking this a step further

Let’s take these concepts a step further. The events during that morning in San Jose has shifted my view quite a bit. It took me a year or so to more deeply internalize insights from that simple insight: “when assessing, think about whether you’re assessing permanent knowledge or transient knowledge.”

The way that I reflect back on that account and the way I wrote it, it may seem weird and portray me as a weirdo who talks to oneself. Some of the accounts where conversation was going back and forth was true or at least felt to me like I was conversing with another part of myself.

Bike Analogy (taking it a step further)

Knowing how to rider a bicycle is a permanent knowledge. Knowing about a specific new model of a bike that came out is a transient knowledge.

Knowing how to code at the fundamental level (where you’re using the core building blocks of software, which I consider to be variables, if/else statements, loops, functions, and OOP) is a permanent knowledge. Knowing about a specific implementation of those core building blocks in a specific programming language or how specific framework operates (whether that is front-end framework such as React version 16 or whether that’s. a back-end framework such as Rails, Django, Express, Laravel, etc) is a transient knowledge.

Also, as I mentioned earlier, even “data structure questions” are transient knowledge.

This made sense as I know various people who know so much about data structures (transient knowledge) but yet when asked to code and build something, they really struggle (permanent knowledge).

People who “know” the difference between a breadth-depth-search, a depth-first-search, what a binary search tree is, its pros/cons, what a hash table is, the best way to construct them and when to use it, not to use it, etc, have “transient knowledge” in that domain but whether they can go to the whiteboard and solve these challenges or whether they can jump into their computer and produce quality codes quickly is a separate matter (this requires good permanent knowledge of coding).

What does this mean?

The danger of interviewing coders and asking questions related to transient knowledge (say a developer who knows X, Y, and Z) has a huge downfall, of which the hiring managers could fall into several dangerous scenarios.

For example, say you were looking for a developer who knows Python, Django, Flask, Node.js, React, Amazon AWS, Docker, Restful API, etc. Because the job description by nature focuses on ‘transient knowledge’, it’s very easy to fall into the trap of asking interview questions for these “transient knowledge” as well as screen candidates with these specific keywords. For example, you may ask specific questions related to one of these technologies to gauge their understanding and when they meet a lot of checkboxes, you may think this person is a perfect hire for you!

Falling into that conclusion can be dangerous for a few reasons:

1. Think back to the bicycle example. You could have finally found someone who knew about bicycle model X, all the latest gadget about Y, and particular performance improvements introduced by accessory Z. You finally feel like you found someone who met a lot of the job requirements (check off list in the must have section of your job ad). The resume has all these keywords that you were looking for and when hiring managers asked questions about X, Y, and Z, candidate answered the questions brilliantly. All hiring managers love the candidate and excitedly give an offer, only to find out through 3–6 months after hire, that this person is actually not that good at riding that bike! Knowing about model X, gadget Y, and accessory Z is all good but remember that someone who has not even rode a bicycle can ‘study’ about all of these topics and give you the appearance as if they are really good! This can be extremely dangerous and costly for you.
2. The top developers in the world are language agnostic and framework agnostic. In fact, they can create new frameworks and tools of very which majority of the job requirements list. For example, give them a few days, they can create a library/framework like React, Express, Django, Rails, Spring, Angular, Vue.js, Ember, Backbone, jQuery, CodeIgniter, Laravel, Zend, Symphony, and the list goes on. If you ask questions related to a specific language or specific framework, you may over filter and entirely miss out this category of highly intelligent programmers (with great permanently knowledge) but who just haven’t had time to pick up particular list of tools that you’re looking for or they used to but forgot because it’s been months since they’ve used them. As these people won’t naturally get as much attention as the first category (those with lots of keywords), it is easy to filter out these candidates. This is especially challenging for HR/recruiters as the resume has no reference to one’s innate ability to code (read below for more information on this).
3. Related to above, looking for a coder with a lot of checkoff list items that relate to transient knowledge, will give you very small number of developers to target. There are so many technologies out there and coders have limited time learning them. Therefore, longer your checkoff list items are (in terms of what language they know, what frameworks they should know, what other experiences they should have had), your pool of potential candidates become smaller and smaller. In this scenario, you may also find out that all your competitors are also looking for the same developers that you are looking for and the cost of hiring these developers could be very costly. These developers often get spammed by so many recruiters (especially if they have the keywords that a lot o people are looking for) that they may also not be interested in talking to you.

Now, this does not mean that transient knowledge is not important. What I am saying is that we need to test coder’s permanent knowledge as well as their transient knowledge, with the understanding that if they are a great coder that they can pick up any transient knowledge within days.

We must also remember that Data structure questions or any questions that a developer would naturally forget if they haven’t practiced that domain for several months, are questions we should really be careful when we’re doing interviews. Otherwise, we may find someone who looks glamorous but inside they weren’t really that good or we may miss out great developers who could have easily created these frameworks that the role is for, but where when you ask specific question about these frameworks, they give a very shallow answer (and give you a false negative that they aren’t good).

For me, this insight has helped me to put a higher focus of my interview questions on testing developer’s permanent knowledge (their ability to use if/else statement, loops, functions, variables, OOP to create solutions to problems). It also caused me to entirely stop asking any data structure questions during interviews, and where maybe only 10% of my interview deals with questions specific to a particular framework or language that I am hiring for. 90% of my interview is spent on testing their permanent knowledge and how they are wired to think in fundamental coding blocks that applies across all programming languages/frameworks.

My journey afterwards

This insight that came from that morning in the fall of 2018 had a big profound impact on me as a teacher and also as an entrepreneur. I have since then moved on to start another company where we specialize in creating assessments. This time though, we are trying to be more clear on which test is testing permanent knowledge vs transient knowledge however.

After about a year of R&D, I built out an assessment that measures a developer’s core permanent knowledge. This test, we named it a “Programming IQ test”, and it only deals with the core building blocks. of software (if/else statements, loops, functions, variables, and OOP). We then compute, based on all the developers who were asked the same or similar questions, what percentile they came out with, in terms of accuracy of their code and speed of getting to the right answer.

If you are interested in learning more about this programming IQ test, this test is freely available for you or any other company to use. To learn more, please join Data Compass portal for Companies and we’ll send you more information.

Questions/Comments?

I am cautious about statements I made in this article. I hope that you won’t take offense on anything I stated above, but I do welcome feedback/thoughts/challenges.