How to design a good technical interview test (take home or live)

Good advice here: https://jacobian.org/series/work-sample-tests/ (also take a look at the series on interview questions and what makes them more or less predictive of performance)

I don't think that take-at-home tests are fair. Most of times I've been given one the 3-4 hours have been actually a weekend. In fact, I have the policy of only accepting take-at-home tests that are interesting problems that help me learn something along the process of developing them.

On the other hand, I also don't like on-site interviews. A random question not-related at all with the position you're interviewing for is given, and have to solve it while you're beholded by a bunch of strangers... Being tired/nervous after a 3/4 hour in-site interview, scenic panic... To sum up, tThe recipe for disaster.

Whe I was the interviewer long ago, I gave them a questionary that could be solved in 1 hour. I made sure to tell the candidates that don't get stressed, look for whatever info you want in internet, and that there's no good/bad answers. In fact, I gave them the questionary and once the time has passed I would get them to check the answers.

There was several level of questions, to prune candidates at different levels. That also helped to determine the level of the candidate (I was working for a 200 employee non-tech company at the time).

I suppose that the questionary could have been asked to be completed in-place, in an empty room and without internet... But, what's the point of that? In the real world a developer is allowed to use internet, in fact, knowing to search is a skill I'd like to check.

Besides, if they had some open-source projects I would take a look at that, and at the end, there was a technical conversation between them and some senior developers about their career.

That sounds like you want to keep people away from your organization. Why not ask them these questions during the interview? Usually ask questions around the issues that your organization is facing. See how they would solve that problem. Often times, teams get intimidated by strong talent so they often pass on these scouts, but your team should be the one that is providing you with the problems for these candidates to solve their riddles =p

Always present real time challenges, so that you are helping the teams pass obstacles and not providing tests that make no sense in teams therefore resulting in steep learning curves.

None of this at all. People already take time for multiple interview stages. Ask targeted questions, but don't trouble them with more 'work'. Ask for references.

The principles below are specifically for take-home tests, though some will still apply to technical interviews conducted live:

1. Design tests to be open-ended and demonstrate how a candidate thinks, rather than ones that drive toward a binary pass/fail outcome. Asking why a candidate worked on the pieces they did and where they would focus if given more time gets you high signal without requiring as much time from the candidate.

2. Provide clarity on the grading rubric. When candidates understand how they'll be evaluated, they'll spend less time on work that doesn't result in higher signal to the company. For example, having a candidate write some tests or talk through how they would approach testing is great! Expecting full test coverage isn't as useful.

3. Scoping is the toughest part about test design. Try to choose topics that closely mirror the day-to-day work in the role itself but that can be boiled down to a 1-2 hour task. You want to make sure that candidates who perform well are more likely to succeed if hired, but you also want to keep it short. From interviewing 50 hiring managers who designed the hiring processes at Basecamp/Medium/Mailchimp/etc, they revealed that the candidates who were ultimately hired consistently spent 2x the recommended time, which means that designing a 3-hour take-home is asking top candidates to commit 6 hours. It's a tough ask in the current hiring climate.

Here are 3 examples for inspiration:

- Full-stack role for Flow Club (YC S21) that runs virtual WeWork sessions: https://app.mightyacorn.io/flow-club/founding-front-end-engi...

- Front-end role for Playhouse (YC S21), building TikTok for real estate: https://app.mightyacorn.io/demo/playhouse-hiring-challenge

- Back-end role for zero5, a parking automation startup: https://app.mightyacorn.io/zero5/challenge-fullstack-softwar...

Happy to share more details if you have specific questions!

Source: my team spent the past 6 months designing personalized take-home tests for 20 VC-backed startups hiring founding engineers. We evaluated ~500 candidate submissions.

1. Focus on a simplified version of a real problem you had to solve at your current job (MIN(arbitrary))

2. Needs to be explainable in 1 page with example data (MIN(completely))

3. Needs to not require any complex interaction. No web request, no complex libraries, just a unit testable piece of code. (MAX(impl speed))

4. Make it fun and something you and all your coworkers could easily do.

This has been my approach and it's been a good filter. If it's done before the interview have them come in and add a feature. Make sure they bring a machine with their prefered IDE / editor to maximize similarity to their real programming flow. Offer to do something else while they code so they don't feel "watched"

I don't like take homes, because you are able to cheat... (ask a friend, hire a fiverr, etc.) - so a brilliant solution is not very helpful to find out, if you really met an expert.

I prefer a 2 - 6 hour training day (depending on whats possible) for the best applicant, where he / she meets the team, sees how they are working and most of the people I met, were interested, because they benefit from this, too.

When you MUST provide a take home one, you could take a look at adventofcode problems as inspiration. They are fun to do, not too complicated and do not depend on a specific technology.

I encourage you to see what happens when you leave testing out of the interview.

Learn about them, ask them to participate in a code review if you have to, pretend you work with them already.

Don't. My best interview where ones where we talk about my open source contributions. One would be enough, but I have a few because I can work on my employer own open source project.

For jobs that I am excited about I wouldn't mind doing a take home test as long as the project was interesting.

no take homes pls