From: Kir Kolyshkin (no email)
Date: Mon Aug 12 2002 - 09:22:43 EDT
Wim Godden wrote:
> Kir Kolyshkin wrote:
> > > I think he needs it for the same reason many of us would like that feature : one
> > > indexer is way too slow. If you want to index a whole ccTLD, it'll take you
> > > several months with aspseek.
> > Hmm have you tried higher number after -N together with
> > upgrading your server to have more RAM and higher disk I/O
> > throughput? Also, moving MySQL to separate box, and searchd
> > to another separate box helps a lot. Actually s.cgi can be
> > put on "yet another" box (I'm not sure if this will help),
> > so you will end up with four machines.
> Well, even that won't do... I tried running up to 500 threads, but that simply slowed
> everything down even more. MySQL is on a seperate box and s.cgi is not required yet,
Actually moving MySQL to the separate box makes sense, because it eats
some RAM and CPU time while indexing.
And if someone wants really distributed system, my advise would be
to take ASPseek (or other open source search engine) sources and
modify it to be distributed. This is not that complex as it seems
to be, and requires about one man-year of a good programmer to
be implemented. More to say, if somebody will spend $60.000 on it
and contract ASPseek team to develop distributed (or cluster)
version of ASPseek, we can make it in about half the year.
> because I'm still indexing and not providing search access yet.
> > Also, PageRanks will be computed separately
> > for two indexes, which is not a good thing.
> Indeed... might as well tell your visitors the search results aren't good.
Actually PageRanks is not the only algorythm that make results
better. And half of .com is a good source for PageRanks anyway.
> Anyway, not a problem for me anymore, since I've stopped using aspseek and built my
> own system in PHP, so I can spread the load over our webserver farm... works like a
> charm !
Can you give an URL so we can be amazed too? ;)
-- ICQ UIN 7551596 Phone +7 903 6722750 --
Guinness a Day Keeps a Doctor Away (people's wisdom)