That kinda defeats the goal of feeding AI as much garbage as possible. They will just fetch a page from a different site in that time, instead of spending cycles on this page. It’s not like the crawler works strictly serially.
The idea is to protect own server from unnecessary loads. You’re welcome to provide a faster AI tar pit, just mind that ultimately this is a waste of resources.
I’m guessing that Markov chains are pretty efficient computationally compared to AI training. Don’t have a site currently, but I’d love to see a bot rip through hundreds of pages a minute.
Here is a demo for anyone interested. It’s deliberately slow to load.
https://zadzmo.org/nepenthes-demo/
That kinda defeats the goal of feeding AI as much garbage as possible. They will just fetch a page from a different site in that time, instead of spending cycles on this page. It’s not like the crawler works strictly serially.
The idea is to protect own server from unnecessary loads. You’re welcome to provide a faster AI tar pit, just mind that ultimately this is a waste of resources.
I’m guessing that Markov chains are pretty efficient computationally compared to AI training. Don’t have a site currently, but I’d love to see a bot rip through hundreds of pages a minute.