[REQUEST] Sources of downloading files...

[old_death]

There are several proposals I wold like to make for the source management of downloading files.

First, sources of downloading files should always be sorted by their entry in the status column. This means that sources showing a countdown until next query should be at a defined place (either before those without text in the status column or behind them, but not mixed with them).

My second request would be to sort the unqueried sources without text in the status should be sorted accordingly whether they are gnutella/G2 sources or whether they are eD2k sources. This would make keeping the overview better as all 'H' sources and all 'donkey' sources were separated. (This makes manually querying sources more easy.)

The last request I'd like to make is to make sure that in a list of sources, gnutella/G2 sources are queried with a higher probability than eD2k sources, as they are more probable to start a transfer sooner - which speeds up the total download as it starts more quickly downloading.
The problem with the current way of querying sources is that there are much more eD2k sources than other sources and therefore the slow sources with the huge queues (= most eD2k sources) have a higher probability of being queried than the sources with short queues (G2/gnutella). This is very bad for people having the eD2k network activated in Shareaza when downloading smaller files which have by chance also sources in eD2k (In fact, it is generally bad for people using eD2k in Shareaza, but for those downloading medium sized files (around 40MB), it's especially annoying.) In order to solve this, I'd like to suggest, that if there are both, gnutella/G2 and eD2k sources in a download, Shareaza should query 2 gnutella/G2 sources, followed by 1eD2k source, then 2gnutella/G2 ones and again 1eD2k one and so on.

BTW, this last thing is especially useful on restarting paused downloads with many eD2k sources...

[old_death]

*bump*

[ailurophobe]

The way sources are queried is defined in the G2 and ED2K protocols. ie When you query a source and it is busy there is a time that you are supposed to wait before re-querying the same source again. To query G2 sources more often would require querying them before that re-query timer runs out. This, I believe, is called hammering and if done regularly will get you banned. Also, the timers exist for a reason: to reduce the number of useless queries, when a minority of clients does it they get an "unfair benefit" at the expense of other clients, but the overall performance of the network goes down.

The actual problem you have observed, is that if Shareaza has ED2K sources for a file it stops searching for new G2 sources, which would be faster, for it. Combined with the greater stability of ED2K sources, Shareaza (IIRC, this might have been fixed) not connecting to sources it already has over ED2K over G2 even if it found them there, and the greater source number of ED2K making source exchange work better these make it usual for Shareaza to "run out" on known G2 source on long running downloads. I suppose the fix would be counting ED2K and G2 sources separately, but on the other hand the current way does reduce the load on G2 significantly... So I doubt ryo is in a hurry to fix this until hub performance improves.

[old_death]

I'm sorry I have been explaining myself poorly for that last part of the post. I was talking about querying sources that have not yet been queried this session (so either when starting downlaods after Shareaza starting up or after a new download has been added to the list).

I was not speaking about breaking the build-in source re-query timers.

Also I think, part of the actual problem, as you call it, is that Shareaza manages to saturate the number of sources it has per download by eD2k sources, even if there are unqueried sources from other networks available. In order to resolve this, it would be sufficient in most cases to rotate threw the networks when selecting the next (yet unqueried) source to connect to. Just make Shareaza first look for 1 BT source, 1 eD2k source, 1 HTTP (=G2, gnutella, HTTP) source and 1 DC++ source and then start looking again for the next random BT source and so on. Like this, there is an equal distribution of load between the networks (at least as equal as there are sources available for one download) and Shareaza shouldn't run out of G2 sources, too, as via source exchange on querying the G2 sources, it were able to get enough new ones without having to use the G2 Hub overlay network to search for G2 sources for the file (so there wouldn't be more stress on the Hubs).

As for optimizations on the Hub mode, have a look at Quazaa: it manages to run at less then 1% CPU usage in G2 Hub mode with 600+ connected leaves at my PC. So once the first stable of Quazaa is out, it is likely we will see the Hub overlay network being improved a lot as Quazaa clients can operate as long time stable Hubs on servers. However, also the Quazaa leaf clients can bring further improvement: they use some cool techniques to choose which Hub to use to route their queries (in case they are firewalled), which should distribute that part of the network load more equally and thereby improve overall response times across the network.

[ailurophobe]

Yes, that is a different, although related, problem.

But are you sure ED2K sources really crowd out G2 sources? I was under the impression that only simultaneous active connections are limited, with ED2K connecting sources in passive queued mode having a separate, much larger, limit. I think that is the way it is supposed to work, anyway. Could be a bug, of course.

I agree there seems to be problems in getting G2 sources when ED2K is active. Just not sure that you (or me for that matter) have the right idea about the cause.

[old_death]

[cyko_01]

shorten the max queue length to around 1000. This frees up space for more g2 peers by eliminating the ed2k peers with long queues. works great!

[old_death]

Yeah, but that's in fact a lazy work-around, not a real fix... And it eliminates potentially good sources...