No matter how you are planning to judge your hackathon submissions, being fair and efficient are key. When you are planning your judging period out, you should define the below:
- Who is judging?
- How much time will they need?
- What are the criteria you want your judges to care about?
- How will they judge the projects?
Who is judging?
The prizes and the theme of your hackathon should be an influence on who is judging the submissions. You can set up multiple Judging Panels or stick to a single panel of Judges and then you can also choose to let the public vote on their favorite projects.
Public Voting. Community Choice prizes awarded through public voting are a great way to amplify publicity for your hackathon and engage the larger community. Participants will have an incentive to promote their submissions to their personal networks and grow your hackathon’s reach. However, there are drawbacks to releasing your control over judging. Due to fraudulent votes and attempts to game the system, we recommend making the prize value small and not showing the results live until you review your votes.
One Judging Panel. Most commonly, hackathons set up one panel of judges made up of subject matter professionals. These reliable individuals are asked to judge all of the eligible projects based on the same judging criteria. Make sure your panel is diverse and co-ed to ensure different backgrounds and opinions throughout the field are considered while judging projects.
Multiple Judging Panels. Category-based prizes are getting more common and sometimes involve different groups of subject matter experts judging specific projects based on their project category. This complicates things for hackathon managers, but it works. You still want to make sure each panel is made up of at least three diverse and co-ed individuals. We would recommend limiting yourself to 2 - 4 panels, purely to avoid logistic headaches and confusion. Here is an article on Creating categories for judging on Devpost.
How much time will they need?
This question depends on how many submission projects your judges need to review. We suggest narrowing down your submissions before sending them all off to the judges and setting times on their calendars to ensure they budget some time for judging.
Narrowing down submissions. When there are numerous eligible submissions, hold an internal review with a mix of individuals from your organization to narrow down the submissions to be reviewed by the judges. For example, we often send a max of 30 projects to judges with the expectation that scoring will take about 5 hours.
Managing Judges’ Time. Respect your judges’ time and try to give them a window of 1-2 weeks. If a shorter window is needed, block time on their calendars to be sure they leave time to complete judging. Remember to check on their status throughout the judging period and send them reminders if needed. The last thing you want is to delay your winner announcement, so stay on schedule with your judging, since participants are eagerly awaiting the results.
What are the criteria you want your judges to care about?
Before you launch your hackathon, you should define the judging criteria and their weight. Both are important for participants to know how their projects will be judged.
Criteria. Judging criteria should mirror what you would like the projects to focus on. There is a section on your Devpost hackathon site committed to outlining your criteria and their descriptions. Most often judging criteria include:
- Quality of the idea (How creative and unique is the project?)
- Implementation of the idea (Does the end result demonstrate quality software development and design?)
- Potential Impact (How big of an impact could the project have on customers?)
Weight. Judging criteria is usually weighted equally and this is how the Devpost judging platform is set up. If you want to shift the focus of your criteria, you will need to do judging offline, determine the % weights, and communicate this on your hackathon site.
Note: The Devpost online judging platform does not currently support varying weights in criteria. This method of judging would be taken offline.
How will they judge the projects?
Devpost enables you to do Judging on the platform or off of it. We call this Online and Offline, but this doesn’t necessarily mean that Offline is not using your computer and Excel sheets. TIP: Toggle off the Send email notifications to Judges and Managers under the Edit Hackathon > Judging area within your manager dashboard.
Online judging process. The Devpost Judging Platform is considered Online Judging on Devpost. The platform makes judging easy, but there are limitations. Online judging only works when all judges use a single set of equally weighted criteria. This means you cannot divide up the criteria per category. Here is an article on How to judge an online hackathon and Starting the judging period.
Offline judging process. If your prizes and judging requirements are a little more complicated, you should bypass the Devpost judging platform. TIP: Typically, offline judging is used when judges will review projects via live demos or presentations at in-person hackathons, but you can also enable your judges to judge using Excel or Google Sheets. To do this, just request Devpost Support to pull your hackathon’s submission review project data. Once judging is complete and you have your winners, you can end the judging period and select your winners. Here is an article on Offline judging: Using printed score sheets and Ending the judging period and selecting winners.