What the White House doesn't want you to see

[Nowhere]

Or rather find easily...

http://www.whitehouse.gov/robots.txt

[Dr Mordrid]

And you're suprised that they limit searches

EVERY govt. or commercial site does that to some degree, and political ones even more so.

Dr. Mordrid

[Nowhere]

Look at those sites.

They don't appear to be semi-randomly chosen...not even close. (nevermind that White House site is definatelly not your typical gov agency/corporate site)

[Dr Mordrid]

Much of the material covered would be public domain; first lady news items, photo galleries etc., meaning available on news sites. Ever think they're just trying to reduce their bandwidth because it's a heavly mined site?

Dr. Mordrid

[Nowhere]

Oh, like they don't have money for the bandwith/server bill?

And you seems you don't want to see very clear pattern there...

[Jessterw]

Yes, yes we know. The US government is evil and bent on world domination. Contained within those pages are hidden details to the plan to make this happen.

Truth is, a good portion of those exclusions are probably to maintain a central page, for each of the topics those excluded areas relate to, for search engines to enter. So instead of just any old page being searchable, there is an index page of sorts that is.

Doc is right as well that it is also likely used to limit bandwidth usage. Believe it or not, but most government websites aren't backed by the sort of server hardware/software and funding that one would think/hope they would be - even the one for the White House.

[schmosef]

Oh, like they don't have money for the bandwith/server bill?

And you seems you don't want to see very clear pattern there...

I have no idea what you're talking about?

Just what secret do you think lays buried in their directories anyway?

[Nowhere]

You don't see the point...just look at broader picture.

Because either way, in the case of "least disturbing" version, this means that enormous part of news/pages there are about...?

[Jessterw]

Pretty sure one could take their pick. There's not one single topic that is overwhelmingly present, unless of course you mean Iraq, because well, it is a pretty big topic at the moment... go figure.

If you pick some random lines and plug them in, many are just link pages or just don't exist to begin with. Others are plain text versions of HTML versions that are present elsewhere or in upper directories.

[Taz]

Surely if they really didn't want people to see those pages they'd just stick them in a secure/passworded area

[schmosef]

I think that it's more an issue of them prefering for search engine to link/cache their main pages so that visitors see those pages first.

I really don't see that as much of an issue.

[Dr Mordrid]

I really don't see that as much of an issue.

Me either. More tin-foil hat material

Dr. Mordrid

[Nowhere]

Ehhh...why do I have impression that if anyone doubts something about US gov he is immediatelly called "conspiracy theorist"?

Why look so deep? OK, point after point:
Why block it?
What's blocked?

which translates to:

What this cadency is about in large part?
What is in the center of all "issues" for average citizen?

[Sasq]

Whats blocked = nothing.
Why block it = save bandwidth.

Simple fact of the 10gig or so bandwidth we use a month, 7 gig of that is google, yahoo, msn bots.

Considering the amount of external sites that link to various pages on whitehouse.gov, its possible the search bots can bring the server to its knees.

That's why it's blocked.
Please collect your tin foil hat at the door.
Thankyou and goodnight.

[Nowhere]

...
Why block it = save bandwidth.
...
Considering the amount of external sites that link to various pages on whitehouse.gov, its possible the search bots can bring the server to its knees.

That's why it's blocked.
...

And can't you see this is the point?

Ehh...whatever...