The Little Search Engine That Could

[In2Kermit]

Maybe because I'm not a computer guru, but I think it would be cool if while you wait for your results, to see the acorn growing into the tree. You know those pictures, like a flip book, where the acorn is planted and grows into a mighty tree. Something like that. Anyway, there's my two cents.

[Hunter]

Wow, that would take some resources. I'm not sure you could do that effectively.

[facugaich]

I think a simple .gif would do, you can set them to play only once. On the other hand, I don't know if everyone will be able to see it, since search results tend to come up really fast.

[Hunter]

That's kinda my point. If you have an image playing all the time it would be fine to use a gif, but if you wanted an image to play only while you were loading results, you would have to incorporate a flash menu into the search process. I've seen it done, but it's never quick. Personally, I know that if I used a site, other than Acorn, that wasted time with anything, heck if it fetched to slowly, I'd move on to another search engine. I think the graphic idea could be used; it is a good idea, just not in this case.

Personally, I feel Acorn would be better off with resources dedicated to expanding on the features found in Googles search. I realize that they are a huge company now, but let's keep in mind that they started off small. I would most like to see specific searches incorporated into Acorn, such as picture, news, and (hopefully one day) map searches. I realize I'm looking at the big picture while we are still working small, but that'll always be a fault of mine.

[In2Kermit]

It was just a thought...

[Hunter]

It was a good thought, just not useable in that particular form. I do like the idea of using that idea elsewhere, though I'm not sure how just yet.

[Larry R]

The animated "flipbook" of an acorn growing into a tree would be an awesome animation to use for banner advertisements (assuming those get used) to advertise the new search engine once it's up and running.

[PK]

An animated acorn/tree would be very cool. Maybe we can use it to show the progess of the the project. And also a banner, like Larry suggested.

Do you think showing the step-by-step process of building a search engine would be interesting to some people? We'll have a blog with the latest updates and also one of our webcams will be pointed at the server racks 24/7 (Ginafish is doubting those last comments.) We will give updates of how many sites have been crawled and how many pages are in our index. Also, we'll ask for input on how search results should be

The first hardware will be purchased next week and delivered (for free) from Bangkok . Now we just need to find a Linux expert in this area.

[KitKat]

  This is very exciting news.   

[Ginafish]

We'll have a blog with the latest updates and also one of our webcams will be pointed at the server racks 24/7 (Ginafish is doubting those last comments.)

Oh yeah, baby. Watching your servers will be exciting. Like watching paint dry...

[Hunter]

Yeah well I'm a bit more (okay a lot more) geek. I'd love to watch the servers created! And I think Larry's idea is an awesome way to use the acorn animation.

[Stasven]

Acorn is an awesome name. 

Always
Cooking
Oat
Recipes
Now

I like it!   

[thartley]

And now, freshly plagiarized from a forwarded e-mail just today.....

"Today's mighty oak is just yesterday's nut that held its ground."

Not sure that's the best inspirational message to send to PK, but well, seems kinda true.   

Awesome
Charity
Organization
Requests
Newbies

[ZeroG]

I thought

A
Click
Others
Really
Need

was good

It's fun thinking about names and acronyms and animated gifs, but we need an actual search engine.

I have been working on Unix boxes for the last 15 years, with Linux RedHat being the most recent version. I am glad to consult and give advice in this respect, but I can barely find time to work on adding the pulldown menus to the Favorites on Isara. And that is EASY! I wish I had the time to spend on the search engine, but I can review plans or coding designs when someone tries to tackle this.

BTW: I am almost ready to add the pulldown menus. Glad PK isn't holding me to any deadlines .

 

[thartley]

I thought

A
Click
Others
Really
Need

was good

It's fun thinking about names and acronyms and animated gifs, but we need an actual search engine.

Your nutty acronym IS good, I just know nothing about the mechanics of building a search engine, so I preoccupy myself with other ways of thinking ACORN, names, acronyms and animated gifs.  People like me let people like you do what we know you do best. 

[PK]

ZeroG, there's definitely no deadline. I know you're a busy man, father, husband, worker, and poker player. Just finish it by tomorrow, ok?

You're right, we need an actual search engine.

Had a small detour last week because of passport/visa issues, but I'm back and in the process of scouting the best and cheapest hardware. Which is not easy in Thailand. We might buy some hard-to-find items on eBay and have them shipped here (maybe you can help with that - wink, wink). Either that or Mom will be packing some CPUs and harddisks in her luggage when she visits in March.

I've drawn up the plans for the data mining cluster of machines (just 10 nodes to start).  That's the easy part because speed is not necessarily important. It's the actual search engine setup that has me confused. I tried asking questions on a hardware forum but that went nowhere. Maybe someone here will know.

Is having all the data spread across a Linux cluster of 10 machines (with fast 250Gb HDD, 1Gb RAM) better than having all the data in a 3-4Tb disk array and attached to a highend server? Seems like having the data in a cluster would be better since a disk array is limited to the speed of the drives and RAM on the server. Whereas the individual nodes of a cluster can access the data quicker (has much less data to read on each drive) and together they have a lot more RAM than the server (especially if we go up to 20 or 30 nodes).

Any ideas?

[ZeroG]

I have never set up a cluster, but one of our applications at work uses a cluster to store seismic data. I have worked on installation scripts for this software so I know a little bit.

I would suspect the crawler is the biggest consumer of computing resources. So that is the process you would want to spread out amoung the machines on the cluster. Your crawler software would need to be parallelized. This is accomplished in Linux boxes with a library of routines called MP, which comes installed.

It would be good if you can find some public domain software which is ready to run on a cluster (parallelized).

As for the disk configuration, just make sure all the machines can see all the disks. This will allow for parallelization.

[Hunter]

Uh yeah, what he said.

Sorry, PK, I'm out on this one. Way past my league.

Edit: But that doesn't mean I'm not good for research!

[PK]

Acorn now has its own blog, where people can go and get updates. Comments have been disabled so that all questions and suggestions can be done here, in the forum.

http://www.isarablog.com/acorn/

Edit: Added the link. D'oh!

[victaniac]

NO revenue from web traffic (your visits and clicks) will be used for this project. Equipment will be donated or purchased using donations specifically for this project.

Great to hear the equipment will be donated and you nor anyone else has to purchase them. We probably wouldn't want our next Isara project being a debt consolidation one, anyway!


Has anyone seen the PP link by any chance? I can't find it. Maybe we could have it linked at the bottom of the homepage to make it better accessible for anyone who may want to donate!

[thartley]

Acorn now has its own blog, where people can go and get updates.

Linky please? 

[PK]

D'oh! Forgot the link. Thanks, thartley. The original post has been fixed.

UPDATE:
The first set of equipment should arrive any day now. To prepare for the computers arrival we've been busy building a rack (see photo). Nong Khai has a limited amount of resources for highend computer equipment so we had to use our imaginations to make a rack that would work. Powerstrips, fans, and another shelf will be added. We might decide to keep the sides open since it gets really hot here in the summers and airflow is very important.



If this first set of computers work well (i.e., we can cluster then and they can crawl the Internet) then we'll be getting another batch and another rack will need to be built.

[KitKat]

 

HA! I found it really cool that Amazon has this particular book for sale on their homepage.

http://isaraurl.com/apNmf

[ZeroG]

Most of the stuff on the Amazon homepage is based on what you have
purchased or looked at on Amazon. So, I don't think anyone
else is going to see what book you are talking about.

Which one? "At my Wits End?"

[KitKat]

Thanks ZeroG, I didn't know that.  The book title is "The Little Engine That Could".