Comment scraping script

BradyB's Avatar

BradyB

05 Feb, 2018 08:18 PM

It is very hard to find comments of mine that were made more than a year ago since the comment search engine bricks if the search date is more than a year old.

If I wrote a script to crawl my comment pages and aggregate everything into a local database, would that be okay? I am worried that your servers will see a long series of very quick requests from my PC and block me due to spam concerns.

Thanks.

  1. Support Staff 1 Posted by Daily Kos Staff on 05 Feb, 2018 09:42 PM

    Daily Kos Staff's Avatar

    Hello, and thanks for asking this good question.

    If you search on your own user comment pages (the Perl pages, https://www.dailykos.com/user/BradyB/comments) you shouldn't encounter any throttle.

    We do have a new search function in the works, but it will still be a while before it's implemented.

  2. Daily Kos Staff closed this discussion on 05 Feb, 2018 09:42 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac