Topic: ethical web scraping

Sort by: Relevance | Date
  • Y Combinator's Firecrawl Offers $1M to Hire 3 AI Agents
    May 17, 2025
    85%

    Y Combinator's Firecrawl Offers $1M to Hire 3 AI Agents

    Firecrawl, a Y Combinator-backed startup, is allocating $1 million to hire AI agents for roles like content creation, customer support, and software development, offering $5,000 monthly salaries per role. The company emphasizes ethical web crawling practices for LLMs and has posted three AI-speci...

    Read More »
  • Perplexity AI accused of bypassing no-crawl rules, says Cloudflare
    August 5, 2025
    70%

    Perplexity AI accused of bypassing no-crawl rules, says Cloudflare

    Cloudflare accuses Perplexity AI of bypassing website crawling restrictions by using undisclosed bots and IP rotation tactics, violating protocols like robots.txt. Perplexity allegedly ignored standard crawler rules, switching to masked IPs and unrelated networks to scrape data when blocked, unde...

    Read More »

Related Topics

firecrawl hiring ai agents (1) ai agent roles (1) hybrid human-ai workforce (1) future ai workforce (1) y combinator involvement (1) data privacy concerns (1) perplexity ai scraping allegations (1) robots exclusion protocol violation (1)