> Why do you think Bing cares or has to know that "q=" means a Google query term?
If that's true, then they should also be associating the sites linked with all the other weird parameter values in a search query, which would spam them to heck. Here are all the params from a search I just did on google:
They'd start making a lot of strange associations between random sites and "utf-8" if that were true, because that parameter shows up in just about every Google search done in English.
It's also a perfectly normal thing for programmers to search for, so they'd clutter up their index with millions of sites that had nothing whatsoever to do with utf-8.
So to make any real use out of that, they had to understand what the parameters in there actually mean, rather than associating ALL of them with whatever site was next in the clickstream.
Though I grant you, that does not disprove the alternate hypothesis that they were dumb and polluted their index with loads of irrelevant crap.
And I admit that I found weak support for that hypothesis by trying to see if utf-8 was linked to rim.com (one of the tests sites, if memory serves):
If that's true, then they should also be associating the sites linked with all the other weird parameter values in a search query
If it's a parameter value people will ever actually use Bing to search for ('utf-8'), there's probably plenty of other signal to help them figure out which results to return. If it's not ('kecgxjpgqoe'), we already know they sometimes return crap, thanks to Google's little experiment.
They'd start making a lot of strange associations between random sites and "utf-8" if that were true, because that parameter shows up in just about every Google search done in English.
If it shows up as a parameter for every search, why do you think Bing's algorithm would decide it was a good indicator for a particular url? P(foo.com|utf-8) wouldn't be any different from P(bar.com|utf-8), making 'utf-8' basically worthless as a discriminator. I'm pretty sure the folks at Bing understand the concept of conditional probability.
It's also a perfectly normal thing for programmers to search for, so they'd clutter up their index with millions of sites that had nothing whatsoever to do with utf-8
I see no justification for the idea that url parameters are somehow a more difficult challenge in this regard than the mass of crap that is content on the web, which we know they crawl and index at a massive scale.
It would only show up as a parameter "relevant" to whatever sites were next in the clickstream. Remember, not every page transition is Google -> other site.
They'd also be gobbling up tons of random things from forums and whatnot (which should appear on long tail searches, if we knew where to look), most of which spam the heck out of you with random parameters, forum names, and whatnot.
> I see no justification for the idea that url parameters are somehow a more difficult challenge in this regard than the mass of crap that is content on the web, which we know they crawl and index at a massive scale.
Which is why I see no reason to assume that they don't understand or attempt to understand the actual meaning of parameters passed to one of the biggest sites on the internet.
You said 'utf-8' shows up as a parameter in almost every English Google search, and suggested that this would cause weird associations between 'utf-8' and random pages.
I pointed out that there's no reason to expect this to be true, because Bing engineers are likely smart enough to realize that if the probability of clicking on to foo.com is not significantly different than the probability of clicking on to bar.com given the presence of the 'utf-8' parameter, then 'utf-8' is a pretty poor discriminator between foo.com and bar.com, and probably shouldn't be used to determine search results.
It doesn't matter that not every page transition is Google -> another site. You still wouldn't need to special-case Google to determine an association with a parameter is useful or useless - the same code could build that model for any site with params.
They'd also be gobbling up tons of random things from forums and whatnot (which should appear on long tail searches, if we knew where to look), most of which spam the heck out of you with random parameters, forum names, and whatnot.
How do you know that they don't? Google pointed out some longtail results that look bad, and you yourself pointed some out in a previous comment.
Which is why I see no reason to assume that they don't understand or attempt to understand the actual meaning of parameters passed to one of the biggest sites on the internet.
You're misunderstanding what I'm saying. I don't assume that they don't; I'm just saying there's no evidence that they do, and that assertions of wrongdoing based on the belief that they do are just irresponsible speculation. I have no knowledge of what Bing actually does, but neither do the vast majority of the people on the internet who are talking about this, many of whom assume the worst based on a mistaken notion of what's technically necessary to see the results that Google demonstrated.
> the presence of the 'utf-8' parameter, then 'utf-8' is a pretty poor discriminator between foo.com and bar.com, and probably shouldn't be used to determine search results.
And yet, it will link random text to websites even if they appear only in Google's URLs. I realize you're talking about discrimination (as in, "what's the better result for utf-8?"), but if the code is generic, it ought to be generic in this respect as well. After all, it linked up random nonsense to random sites given nothing more than Google's say-so, even though there's plenty of information out there about, say, rim.com that would tend to indicate that nobody except Google thinks that random text is relevant to an otherwise well-known site.
> How do you know that they don't? Google pointed out some longtail results that look bad, and you yourself pointed some out in a previous comment.
Indeed, I do not know. I know that it would be dumb to link those things to random sites, but you are correct that I do not know if they're doing things that dumb.
> I don't assume that they don't; I'm just saying there's no evidence that they do, and that assertions of wrongdoing based on the belief that they do are just irresponsible speculation.
Well, for one, I'm not really asserting "wrongdoing" here. That is, I don't particularly think that it's wrong of them to do things this way. My interest is mainly technical, so I'm more interested in figuring out exactly what they're doing rather than blaming them for it. As such, I'm going for the most likely explanations I can find, rather than worrying about whether it's been proven to such an extent that they can be blamed for it (as I'm not really going to blame them anyhow).
You may have seen where I pointed out that I don't think it will "ruin search" in the end because they should expect a crapflood from spammers now that it's clear that they use clickstream data to rank sites. After all, there's a large spam attack right now on a tiny wiki for a game I play. I have to think Bing is more of a target than that. I can't prove that, true, I'm just playing the odds here.
If that's true, then they should also be associating the sites linked with all the other weird parameter values in a search query, which would spam them to heck. Here are all the params from a search I just did on google:
q=test&ie=utf-8&oe=utf-8&aq=t&rls={moz:distributionID}:{moz:locale}:{moz:official}&client=firefox
They'd start making a lot of strange associations between random sites and "utf-8" if that were true, because that parameter shows up in just about every Google search done in English.
It's also a perfectly normal thing for programmers to search for, so they'd clutter up their index with millions of sites that had nothing whatsoever to do with utf-8.
So to make any real use out of that, they had to understand what the parameters in there actually mean, rather than associating ALL of them with whatever site was next in the clickstream.
Though I grant you, that does not disprove the alternate hypothesis that they were dumb and polluted their index with loads of irrelevant crap.
And I admit that I found weak support for that hypothesis by trying to see if utf-8 was linked to rim.com (one of the tests sites, if memory serves):
http://www.bing.com/search?q=utf-8+rim&go=&form=QBRE
Those results appear to be crap, though I'm not sure that any sensible results exist that they could return.