Web Publishing - Disabling Indexing

[RPalmer]

As referenced here:

http://www.reuniontalk.com/showthread.php?t=6882&page=3

From Eastman's Online Genealogy Newsletter:

http://blog.eogn.com/eastmans_online_genealogy/2011/07/publish-your-genealogy-database-on-the-web-with-dropbox.html?cid=6a00d8341c767353ef01543380901a970c

"Depending upon what you are sharing, you may or may not want to have your files found and indexed by Google and other search engines. The default is to add your files to the search engine's index. If you prefer to NOT index the files and add them to the search engines, create a short text file called ROBOTS.TXT and place it in the same folder with your index.html file. To create a ROBOTS.TXT file, use any simple text editor (Windows Notepad or Macintosh TextEdit or something similar, not a word processor) and enter the following two lines of text:

User-agent: *
Disallow: /
That's it! Save the file as ROBOTS.TXT. This will tell the search engines to

[Mark]

Re: Web Publishing - Disabling Indexing

Hi Richard,

You should put the robots.txt file in the same folder as the wc_toc.html file.

Since it's not replacing a file that Reunion created as part of the web project, it won't cause any problems.

HTH

[theKiwi]

Re: Web Publishing - Disabling Indexing

You should put the robots.txt file in the same folder as the wc_toc.html file.

Only thing is though that I'm not sure it will be respected by the indexing bots since it's not actually at the root level of the site overall, but in a sub-folder. The site is at

dl.dropbox.com/u/24330138/Genealogy/Web%20Project/Report%20000,%20Web%20Project/wc_toc.htm

and the wc_toc.htm file is many layers deep from the root of dl.dropbox.com. Even if it's put in the top most folder that the user has access to - presumably the "24330138" folder it's still not in the site's root.

(And note that URL demonstrates what happens with folder and file names that have spaces in them, and end up un-necessarily nested in folders they don't need to be in - the whole project probably could have gone in the Genealogy folder in DropBox I suspect)

Roger

[Mark]

Re: Web Publishing - Disabling Indexing

He doesn't have access to the root level (dl.dropbox.com), so putting it in the same folder as wc_toc.html would be his best bet.

However, dl.dropbox.com does have it's own robots.txt file (see here) which instructs bots not to index anything at that subdomain (aside from the twitter bot having access to the /t directory). That should keep bots from indexing his site.