Comment scraping script

BradyB's Avatar


05 Feb, 2018 08:18 PM

It is very hard to find comments of mine that were made more than a year ago since the comment search engine bricks if the search date is more than a year old.

If I wrote a script to crawl my comment pages and aggregate everything into a local database, would that be okay? I am worried that your servers will see a long series of very quick requests from my PC and block me due to spam concerns.


  1. Support Staff 1 Posted by Daily Kos Staff on 05 Feb, 2018 09:42 PM

    Daily Kos Staff's Avatar

    Hello, and thanks for asking this good question.

    If you search on your own user comment pages (the Perl pages, you shouldn't encounter any throttle.

    We do have a new search function in the works, but it will still be a while before it's implemented.

  2. Daily Kos Staff closed this discussion on 05 Feb, 2018 09:42 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac