HACKER Q&A
📣 anigbrowl

Why aren't tweets picked up by the duplicate filter?


I'm not sure Elon Musk shitposts constitute news at any time, but why are people able to post the same tweet over and over without being flagged by HN's otherwise-efficient duplicate filter? HN strips query data from URLs before they are posted, maybe it should do so prior to the duplicate check too since most Twitter URLs come with a large helping of cruft.

And please, check the New page before submitting. There are 4 links to the same evidence-free tweet in the last 10 minutes.

  👤 r721 Accepted Answer ✓
Possibly related - looks like recent posts are missing in Algolia search index:

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

👤 PaulHoule
There's a button for that and you have enough karma to use it.

But really I can't decide if links to Twitter are better or worse than links to Medium. 1 time in 20 or so somebody posts a highly intelligent series of tweets that are worth reading, but most of the time a link to twitter points to a poster that doesn't really have anything to say, a bunch of comments that say at best "me too!" and a lot of irrelevant junk. (Almost as bad a link to a "good" discussion on reddit that is spammed up with irrelevant discussions like a porn site where you have the hardest time telling your wife, 'ignore the 15 animated gifs of pistoning action and look at that picture which I think is really hot')

👤 r721
Dang's reply: https://news.ycombinator.com/item?id=33790811

>they've found the problem and are currently backfilling the missing data

👤 Trouble_007
I have no affiliation with HN.

The duplicate filter, along with all of HN code is a work in constant review, rewrite, test, and publish.

Setting the parameters for such filtering requires time, https://news.ycombinator.com/from?site=twitter.com/elonmusk

to adjust window of operation, frequency, etc. https://news.ycombinator.com/from?site=twitter.com