Man vs. Machine Hackathon: Inside the Epic Showdown
▼ Summary
– A hackathon called “Man vs. Machine” was held in San Francisco to test whether AI tools help people code faster and better, with teams randomly assigned as human-only or AI-supported.
– The event was cohosted by METR, a nonprofit whose prior study found AI tools slowed experienced developers by 19%, and aimed to extend this research to new projects and varied skill levels.
– Projects were judged on creativity, real-world usefulness, technical impressiveness, and execution, with a $12,500 prize for the winner and $2,500 for second place.
– Some participants dropped out after being placed on human-only teams, and one contestant noted challenges in completing her project without AI support as the deadline approached.
– The hackathon included diverse project ideas, such as an AI tool for pianists’ feedback and a framework to evaluate sycophancy in AI models.
The air in a San Francisco coworking space hummed with intensity last Saturday as coders gathered for a unique competition. Dubbed the “Man vs. Machine” hackathon, the event aimed to settle a pressing question: does artificial intelligence truly accelerate and improve coding, or does it introduce unexpected friction? Over a hundred participants, all shoeless by request, crowded into the venue, ready to put the debate to the test.
Thirty-seven teams were randomly sorted into two categories, those permitted to use AI coding assistants and those relying solely on human skill. Interestingly, several participants dropped out after being placed on the human-only teams, hinting at the perceived advantage of AI tools. A panel of judges stood ready to evaluate submissions based on four criteria: creativity, real-world utility, technical sophistication, and execution quality. Only six teams would advance to the final demo round, with the top prize set at $12,500 plus API credits from leading AI firms.
This event took place against a backdrop of ongoing controversy in tech circles regarding AI’s role in software development. While some fear widespread job displacement, a recent study by METR, a nonprofit AI research group that co-hosted the hackathon, suggested that AI tools may actually slow down experienced open-source developers by as much as 19 percent. The hackathon offered a chance to explore this dynamic in a different context, bringing together coders of varying experience levels to build new projects from scratch.
Joel Becker, a technical staff member at METR, noted that many productivity studies rely on metrics like lines of code or pull requests, measures that don’t always reflect true quality or innovation. Even AI models boasting high benchmark scores may not translate those numbers into practical usefulness. Becker, for his part, predicted a win for the machine-assisted teams.
As the eight-hour coding sprint began, ideas flew across the event’s Slack channel. Proposals ranged from an AI tool offering performance feedback for pianists to a neighborhood connection platform. Among the participants was Arushi Agastwar, a Stanford student specializing in AI ethics. Randomly assigned to the human team, she aimed to build a framework evaluating sycophancy, the excessive agreeableness observed in models like GPT-4o.
Agastwar initially believed human-only teams would produce more profound, thoughtful work. Yet, hours into the event, she questioned whether she could finish her project by the 6:30 PM deadline. The pressure was mounting, and the outcome remained anyone’s guess.
(Source: Wired)
