Why are there so many duplicate articles?

Duplicate of https://news.ycombinator.com/item?id=27098446

We treat a submission as a duplicate if the story has had significant attention in the last year or so. This is in the FAQ: https://news.ycombinator.com/newsfaq.html.

If a story hasn't*had significant attention in the last year or so, then we don't treat it as a dupe, because it's important for good articles to get multiple chances at getting attention. Otherwise the randomness of what gets noticed on /newest would be even more dominant than it already is.

In favour of duplicates, not everyone is on HN frequently, and can miss on some excellent posts if they're not reposted.

It does spot duplicates. I have submitted something and been immediately directed to an existing submission of the same story.

Presumably an exact URL match, and maybe within a timescale?

I like it the way it is currently. Now suppose you add some smartness to website , on what basis you decide which duplicate to remove? HN is the last website I want to add AI to it's recommendation system. We are already being fed so much, on every other platform.

Let community moderate the itself. It's old school and maybe dumb but not everything needs smart ass AI.

You are right.

I propose automatic de-dupe: whatever the title says, if the exact same URL has already been submitted, just count it as an upvote on the existing story... optional: if the submitted caption is different, then add a comment that says "also submitted by X with the caption `Y`."

People want upvotes so they submit it. Setting up scanning for recent articles is probably harder than it seems. Can’t think of any other forum that does it.

Who marks something as a duplicate? Moderators or is there some kind of a bot?