Job Responsibilities：

- Design and implement efficient, stable, and scalable web crawler systems, ensuring data timeliness and integrity

- Conduct in-depth research on various anti-crawling mechanisms and develop strategies to bypass CAPTCHAs, rate limiting, IP blocking, etc., to increase crawler success rates

- Implement anti-crawling solutions such as proxy pools, dynamic user agents, cookie management, and session maintenance

- Enhance crawler performance by optimizing crawl speed, data storage structures, and request concurrency models

Job Requirements：

- No restrictions on educational background or major

- Proficient in the Python tech stack: Scrapy, Playwright, Selenium, HTTPX, Requests, and other crawling frameworks and libraries

- Familiar with anti-crawling strategies, including but not limited to proxy pools, IP rotation, CAPTCHA recognition, dynamic page parsing, and JavaScript-rendered page scraping

- Extensive experience in bypassing website restrictions, mastering technologies such as browser fingerprinting, cookie emulation, session persistence, and front-end encryption cracking

- Skilled in asynchronous programming, multiprocessing, and distributed crawler architectures, with experience in large-scale data crawling and storage