Mystery with Googlebot and IT-Austria

I placed some quite private page on irrlicht3d.org. I placed it in a subdirectory which didn't exist before. Nothing links to that page. At least nothing I know of. Two minutes after I had uploaded that page, Googlebot came along and indexed it.
WTF? How did it do that? I mean, I'm no web expert, but how did googleBot find there?
I immediately added a no-archive meta tag to that page and moved it into another directory.
The even more scary thing: Just two hours ago, someone from it-austria.net (what is that, btw?) came along to the page again and read it. How did he get there? The referer log doesn't tell anything. So now I added a .htaccess password. This will help, but I still would like to know how this was possible. Hm.

six comments, already:

Did you surf the web from that page?
If you did.. the referer of your browser left some trace in some logs..
Anyway, that’s quite strange, and scary..
mightypanda () (link) - 01 06 05 - 18:27

That’s a good point, but no, I didn’t.
niko - 02 06 05 - 06:18

Niko, I came across the same problem some years ago. Here is something you will find useful:

http://en.wikipedia.org/wiki/Robots.txt

By putting a file called robots.txt in your root web folder, you can tell bots to ignore some of your folders and files.
Zaelsius () (link) - 02 06 05 - 12:45

That’s why I added the meta-tags I mentioned in the text, they should do the job too. :)
niko - 02 06 05 - 14:17

Hm. Same happened to me. Google indexed a page which it can’t know because nothing linkes to this page.
I thought that this was my mistake so I forgot it. Now I will discover the HTTP Header of my Apache…
blackbird () (link) - 06 06 05 - 21:40

Back in the days I used a program called GetRight. It had a feature that allowed you to browse an entire site and see all of the files and such. Im not sure if it was just some sort of spider that linked off of the main page, but it impressed me at the time.
[Declassified] - 09 06 05 - 06:38


Name:  
Remember personal info?
yes
no
Email (optional):
URL (optional):
Enter "layered" (antispam):
Comment:Emoticons / Textile

  ( Register your username / Log in )

Notify: Yes, send me email when someone replies.  

Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.
Note: If you type in your email adress above, it will be visible to other visitors, although it will be hidden for bots using javaScript.