Constant G2 "lack of traffic" errors - can't stay connected

Get answers to your Shareaza related problems.
Forum rules
Home | Wiki | Rules

Constant G2 "lack of traffic" errors - can't stay connected

Postby Winterkeeper » 08 Feb 2014 00:25

I have been using Shareaza since last August with no problems whatsoever. I only use it to connect to G2.

However, during the last week or so I find myself constantly unable to stay connected to G2 neighbors. The connection is established fine but within a few minutes, the bandwidth on a neighbor will drop to 0 and then the connection is dropped with the error: Closing connection to neighbour x.x.x.x due to lack of traffic. or Neighbour x.x.x.x dropped the connection unexpectedly. Though, most often it is the "lack of traffic" error.

This happens over and over, and the most I am able to stay connected to a neighbor is around 10 minutes. Some drop after 1-2 minutes. Before, I would connect to neighbors with no problems and stay connected to them for hours.

I haven't made any recent changes to the computer that is running Shareaza and I'm experiencing no other network problems. I started with Shareaza 2.6.0 and upgraded to 2.7.1 in December but it was working fine. Again, this is only happening during the last week.

Any ideas on what might be causing this?
Winterkeeper
 
Posts: 8
Joined: 08 Feb 2014 00:12

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby raspopov » 08 Feb 2014 05:10

90% of network problems caused by connection/hardware errors - check network card connectors, try to ping some hosts for example "ping -t google.com", you can also test bandwidth by http://www.speedtest.net/ site etc.
User avatar
raspopov
Project Admin
 
Posts: 811
Joined: 13 Jun 2009 12:30
Location: Russian Federation

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby Winterkeeper » 09 Feb 2014 01:20

raspopov wrote:90% of network problems caused by connection/hardware errors - check network card connectors, try to ping some hosts for example "ping -t google.com", you can also test bandwidth by http://www.speedtest.net/ site etc.

That's what is strange about this. I'm not seeing any other network problems. Pings and speeds are totally normal. Torrents work perfectly as always. I keep my network here very well optimized at all times and have good quality equipment. Shareaza is the only thing I'm having a problem with, after it was already working fine for months. These G2 connections keep dropping unexpectedly. I do still connect and people can still download from me, it's just very rare that they are able to, because I'm getting dropped so much.

Is there anything that would cause Shareaza specifically to behave this way when the connection is otherwise fine?
Winterkeeper
 
Posts: 8
Joined: 08 Feb 2014 00:12

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby raspopov » 09 Feb 2014 07:36

1. Too short connection timeouts.
2. Too high speed settings.
User avatar
raspopov
Project Admin
 
Posts: 811
Joined: 13 Jun 2009 12:30
Location: Russian Federation

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby Winterkeeper » 10 Feb 2014 02:24

I've tried increasing the timeout settings, and now I've actually had one G2 neighbor stay connected for more than 90 minutes so far. However, the second neighbor connection still constantly drops every few minutes. I guess I will keep experimenting with it.

I should note that when someone downloads from me, there are no problems at all. They stay connected for the entire time and get the complete file, even in the case of large files. It's only the G2 neighbors that are dropping, and usually with that "lack of traffic" error.
Winterkeeper
 
Posts: 8
Joined: 08 Feb 2014 00:12

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby dew » 10 Feb 2014 05:11

I'm experiencing the same issue.

If you look at the network crawler http://crawler.doxu.org/history.html there was definitely something weird that happened 9 days ago.
dew
 
Posts: 9
Joined: 10 Feb 2014 05:05

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby Lanigiro » 10 Feb 2014 14:31

Add me in as a third user who's been seeing this.

Those graphs are alarming, for three reasons.

1. The strong diurnal oscillation suggests that G2 has weak uptake outside of one or a few time zones, instead of strong global support. (This is odd in light of the geographic diversity of hubs I routinely see, but maybe hubs are more evenly distributed than leaves.)

2. There appears to have been a huge and recent drop in the total population of hubs alright. It's as if millions of peers suddenly cried out in terror, and were suddenly silenced. But by which evil empire? The RIAA, the MPAA, or someone else? The hubs also have fewer leaf connections, which suggests the hubs that survive are having problems staying online and keep crashing for some reason. Was an update pushed out for a popular client around that time? I know it's been longer since the last Shareaza version bump (2.7.0.0 -> 2.7.1.0).

3. The year-long graph at lower right has perhaps the worst news of all. There seems to be an overall exponential decay in the size of G2 over that time period, with a half-life of actually just about one year. G2 should be growing, not shrinking. What is going on here?

In the meantime I can report that whatever the recent change is has also impacted G1, as I use it routinely. G1 is much harder to connect to than it used to be, and worse a lot of the failed attempts time out rather than failing quickly. The amount of timeouts and refuseds suggests that the G1 GWebCaches are heavily laden with stale entries for IPs not presently running G1 clients (and, given the prevalence of timeouts, mainly IPs that aren't even presently in use). Either a popular G1 client is also experiencing problems sustaining uptime, or the GWebCaches themselves are acting up (or being poisoned by some bad actor). This may have started at the same time as the G2 anomaly.

Once established G1 connections seem maybe a bit more unstable than in the past, but not nearly as much so as G2. But it's now both G1 and G2 that Shareaza connects to very slowly, with G1 spending a long time on stale webcache entries before finding a good ultrapeer and G2 banging its head against the Great Firewall of China, by the looks of it, for as much as half an hour before finally connecting to seemingly one single small population of hubs in France.

At this point I'm finding most of my stuff via eMule, which used to be a distant second to G2.
Lanigiro
 
Posts: 132
Joined: 10 Feb 2014 14:19

Ten minute CPU saturations every hour

Postby Lanigiro » 10 Feb 2014 17:15

Is anyone else seeing this? For the past few days, my install of 2.7.1.0 keeps saturating my CPU for almost precisely 11-minute-long blocks almost exactly once every 52 minutes, so there's a just over 40 minute gap from the end of one to the start of the next incident.

The CPU saturations are accompanied by numerous "network core overloaded" messages in the network tab, with lower than normal (NOT higher than normal) traffic. And when the CPU saturation ends, Shareaza instantly drops every single hub connection it has save for the ED2K server.

Putting the network tab in "debug" shows something even more remarkable. Usually there's a steady flow of yellow, green, and blue background traffic, particularly if G1 is connected (tons of "Multicast querying Gnutella1 neighbors"). When the CPU surge starts, all of this stuff just stops, abruptly. Not a single event that is not shown at "info" or above apparently happens, or if it does it is no longer reported to the network tab. Far from "network core overloaded", it seems the network core is virtually idle during these events.

I mention this because it only started happening recently and is NOT the result of an update I made (I switched to 2.7.1.0 much earlier than this started). Not only that, I think its onset may coincide with the anomalous G2 "lack of traffic" problem being discussed in another thread here, which suggests a possible cause: that Shareaza G2 hubs are afflicted by this same phenomenon, and drop their hub-to-leaf connections or simply starve them of traffic when their CPU surges occur. So the unstable G2 connections we're all seeing are caused by these surges hitting the hubs we're connected to.

As for the cause, the obvious culprit would be malicious traffic. It's a DOS attack, in all likelihood.

Here is my network log (in "debug") of the onset of a surge I saw here. I have redacted an uploader IP but not any others, as each of them may be the source of the suspected malicious traffic, and most of the other IPs are hubs:

[10:41:21] Querying 180.177.218.52
[10:41:21] Processing query acknowledge from 218.250.73.14 (time adjust -53 seconds): 30 hubs,
2848 leaves, 5 suggested hubs, retry after 300 seconds.
[10:41:21] Multicast querying Gnutella1 neighbours
[10:41:21] Requesting query key from 114.43.112.157
[10:41:21] Received a malformatted query packet from 125.196.163.189, ignoring.
[10:41:22] Rejecting upload connection from [redacted], network core overloaded.

So, it queried a particular G2 hub, processed a query acknowledge, multicast queried G1 neighbors, requested a G2 query key, and received a malformatted query packet from G1. The queries are outbound and so is the G2 query key request, so they're less likely to be the triggering event. G1 malformatted query packets are a dime a dozen, on the one hand, but on the other one happened at almost the exact instant the surge began. If someone is sending out floods of malicious G1 query packets that cycle back to a particular target IP every 52 minutes it could explain what we're seeing. The other strong candidate trigger is the last query acknowledge processed before all hell broke loose, although I don't see anything hugely anomalous about it.

I might try to packet capture the G2 traffic peripheral to a surge onset to see if there's a query acknowledge right beforehand that stands out as abnormal in some way.
Lanigiro
 
Posts: 132
Joined: 10 Feb 2014 14:19

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby Winterkeeper » 11 Feb 2014 05:21

Those graphs are very interesting. So there's definitely something weird going on.

Lanigiro wrote:2. There appears to have been a huge and recent drop in the total population of hubs alright. It's as if millions of peers suddenly cried out in terror, and were suddenly silenced. But by which evil empire? The RIAA, the MPAA, or someone else? The hubs also have fewer leaf connections, which suggests the hubs that survive are having problems staying online and keep crashing for some reason. Was an update pushed out for a popular client around that time? I know it's been longer since the last Shareaza version bump (2.7.0.0 -> 2.7.1.0).

The graphs from that site show there is a huge increase in hubs, not drop. Which would explain the average leaves per hub also dropping drastically. And, you can see this now when connecting to hubs. The hubs that can actually maintain the connection are usually close to max capacity like 299/300 or 1023/1024. The hubs that drop all the time are way below capacity, because they are not maintaining connections (I even see many close to zero - then they get lots of new leaves - then they start dropping them). But why? Initially I had the same thought as you - that someone might be trying to poison the network.
Winterkeeper
 
Posts: 8
Joined: 08 Feb 2014 00:12

Re: Ten minute CPU saturations every hour

Postby raspopov » 11 Feb 2014 16:14

The "network core overloaded" message means that Shareaza has been (dead)locked.

You need to run for example ProcessExplorer and find with its help a most CPU consuming thread inside Shareaza process, then press "Stack" button and find out guilty dll.

Also you can open "About" dialog box and force Shareaza crash by Shift + Right Clicking blue URL in it, then paste generated crash log here.
User avatar
raspopov
Project Admin
 
Posts: 811
Joined: 13 Jun 2009 12:30
Location: Russian Federation

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby Lanigiro » 11 Feb 2014 18:00

Winterkeeper wrote:Those graphs are very interesting. So there's definitely something weird going on.

Lanigiro wrote:2. There appears to have been a huge and recent drop in the total population of hubs alright. It's as if millions of peers suddenly cried out in terror, and were suddenly silenced. But by which evil empire? The RIAA, the MPAA, or someone else? The hubs also have fewer leaf connections, which suggests the hubs that survive are having problems staying online and keep crashing for some reason. Was an update pushed out for a popular client around that time? I know it's been longer since the last Shareaza version bump (2.7.0.0 -> 2.7.1.0).

The graphs from that site show there is a huge increase in hubs, not drop. Which would explain the average leaves per hub also dropping drastically. And, you can see this now when connecting to hubs. The hubs that can actually maintain the connection are usually close to max capacity like 299/300 or 1023/1024. The hubs that drop all the time are way below capacity, because they are not maintaining connections (I even see many close to zero - then they get lots of new leaves - then they start dropping them). But why? Initially I had the same thought as you - that someone might be trying to poison the network.


I'm not sure if that really represents a growth in (possibly bogus) hubs, though, or if a hub dying and restarting several times in a short enough period (a day?) gets counted as more than one. So the seeming increase in hub count could be an artifact of hub lifespan shortening. I'd have to know much more about how that site generates its data to know what it's really counting there. Even if it's unique hub IP addresses if something is making hubs lose their DHCP leases frequently that would both boost the apparent hub population (if stale entries hang around a long time) and make the hubs lose all their connections each time (as the existing leaves are suddenly using the wrong IP). Depending on how long a hub is counted in those statistics after its IP has actually gone stale before the dud entry is removed from their database, a hub could be getting double- or even triple-counted. For example, if stale entries are weeded out hourly but active hubs are (re)discovered much more frequently and average DHCP lease lifetime has dropped to 30 minutes, an affected hub would typically have 2-3 entries at a time under the last 2-3 IPs it's had.

This suggests one possible explanation for what happened ten days ago. For some reason a large number of hubs (more than half) have tended to be in France (perhaps it's the most populous country with cheap really high speed internet, and so a lot more leaves are elected to hub status there?) and France seems to especially dominate the short-lived-hub phenomenon. This suggests that perhaps a major French ISP has been having problems for the past ten days, and the primary symptom of those problems is frequent DHCP IP reassignments. Hubs connected via this ISP would then be having frequent changes of IP addresses, and might be numerous enough for this to be having a noticeably disruptive effect on G2 as a whole.

If so, it points to the need for greater diversity of G2 hubs. Whereas I've seen G2 hubs with nearly every national flag, they're not even close to evenly distributed, with over half being in France and the next biggest contributor being the United States. Judging by the French hub instability problem there might be a single huge French ISP almost everyone there uses (perhaps a national telco monopoly?) and there is a small oligopoly of US ISPs as well (Comcast, Verizon, AT&T, and Time Warner Cable probably cover 80% or more of all US high speed subscribers), so we might be looking at over 3/4 of G2's eggs being in as few as five ISP baskets, and fully half in just one of them. If that is the case, it's surprising we've gone until now before encountering a significant problem.

All of this suggests that Shareaza might benefit from adding a nation-preference feature, where one can add a country to a list and give it either of two statuses, Low Priority or Banned, as well as remove a country from the list. One could then give France and the US Low Priority and China and its satellite nations Banned status, with the following effects:

1. G2 hubs in "Banned" nations would be auto-dropped as soon as they would have been added to the host list, whether from querying discovery services or from peer-to-peer discovery. They never get added to the G2 host cache in the first place and no attempt to make a hub connection to them will occur. This doesn't affect G1, ed2K, other protocols, or leaf business such as downloads and uploads, so G2 leaves (and hubs) in nations on your "Banned" list can still download and upload to you. They just won't be considered for the G2 hub cache. This would mainly be useful for all those Chinese hubs that reject Westerners.

2. G2 hubs in "Low Priority" nations would be tried last; so if Shareaza wants to establish a new G2 hub connection, and the host cache contains a mix of hubs in "Low Priority" nations and hubs not in "Low Priority" (or "Banned"), it will attempt all of the latter before attempting any of the former. Also, if the G2 host cache contains no non-"Low Priority" hubs, Shareaza would also begin to query discovery services immediately, until either finding non-"Low Priority" hubs by that or another method or establishing a connection successfully. This would mainly be useful for putting countries with a high number of hubs but a low diversity and reliability of Internet providers, which seems to presently mean France and the US.

In the meantime, I'd suggest investigating this short-lived-hub-session problem further. If it is a b0rken ISP in France there's nothing Shareaza's developers can do about it, but it's possible there's a problem with a recent version that's being provoked by something new somewhere else, or otherwise that there's a change to Shareaza that could be made that would make it more tolerant of whatever the external factor is.

There is, definitely, an external factor though. Those graphs on that site make abundantly clear that the transition between G2-acting-normal and G2-acting-like-it-has-been-lately took a matter of hours, if even that. If the cause was a buggy 2.7.1.0, the problem would have developed gradually over weeks or months as people upgraded one by one. I still see a lot of 2.6.0.0 peers in various places (hubs, search results, uploaders) so uptake of new versions clearly does not saturate too too quickly.

I foresee two possibilities. One is that the external factor is wholly exogenous: malicious network traffic, malicious hubs masquerading as one or more Shareaza versions that aren't (or are heavily modified from Shareaza), malicious GWebCaches, broken ISPs in France, or some other cause, some of which may be more addressable (e.g., a Finger Poke of Doom DOS whereby a maliciously crafted network packet can crash Shareaza, and which someone is spraying at hubs, where a fix for the crash bug would fix things just as much as stopping the traffic) than others (there's nothing you guys can do about a broken ISP in France).

The other is that a subtle bug introduced to Shareaza recently caused the G2 network to become bistable, with two attractor states, one of them normal operation and the other the situation of the past few days, where some kind of nudge (enough hubs rebooting near simultaneously, a weird network partition, or something) can kick it from one state to the other. In both states, some sort of infrequent circumstance can cause a Shareaza in hub mode to restart, drop a lot of packets for a while, crash, or otherwise start losing at least temporarily, but in the normal state this happens sporadically and knocks out only one hub at a time. But, there's some way for a hub being affected by this or recovering from it to trigger the problem in other hubs, with a low likelihood that any of its neighbors will be affected. Usually a hub tripping over the problem affects only none, one, or a few neighbors and the whole thing peters out, but if enough hubs are affected at close to the same time, they would start constantly knocking each other offline and there'd be a self-perpetuating chain reaction. In this case, the problem is endogenous in that Shareaza is wholly at fault, and the external factor was just a trigger. The long term fix would be to fix the bug, but a short-term one to restore the network to the "normal behavior" attractor state could exist as well. Whatever peculiar network packets propagate the problem from an affected hub to another one could be filtered out at key routers, for example, or hubs immune to the problem placed strategically in the peer topology somehow. (The former approach won't work if the problem is triggered simply by a large enough number of simultaneous incoming hub to hub connection attempts, though, as it would block all new hub to hub connection attempts in that case.)
Lanigiro
 
Posts: 132
Joined: 10 Feb 2014 14:19

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby Lanigiro » 11 Feb 2014 19:00

A few more thoughts on the "bistable" hypothesis.

First, the graph looks an awful lot like the behavior of a bistable system nudged out of an attractor state. It wanders chaotically for a short time and then falls into an attractor again, in this instance the other attractor.

Second, one of the likelier ways for such a bug to exist is a race condition with a chance of wedging or crashing Shareaza if an inbound hub-to-hub connection enters a particular phase of the handshake while another inbound connection is still in that phase. Adding the hub to the global data structure representing the totality of established hub connections is the likely place where two threads handling separate network connections might step on each others' toes in this case.

I'd go over the hub-to-hub connection establishment code with a fine-toothed comb looking for a place where there's unsynchronized access to a shared data structure -- the established-connections list is a likely culprit, as is the host cache (newly connected hubs propagating info about hubs they know of prompting update). If a bug in that area is found, pushing out a 2.7.2.0 with the bug fixed could (eventually) fix this.

The other thing I'd look at is the system for promoting leaves to hubs -- or demoting hubs to leaves. I noticed that the onset of the problem happened while the leaf count as at its daily low, or very close to that time, and that the previous daily low had been the lowest daily low in a while. I'm wondering if a big enough daily swing in the number of leaves can trigger a destructive oscillation where a very low number of leaves per hub causes a lot of hubs to demote themselves to leaf, and then all the leaves losing these hub connections trying to make new connections trigger the remaining hubs to do whatever they do to try to elect some leaves to hubs, and the network gets stuck oscillating between "too many hubs so demote some" and "too few hubs so promote some leaves" and won't stabilize with a reasonable number of hubs. This would explain why there are surges and dips now in the number of hubs, much more than the nearly level number of hubs there used to be. But the smoking gun for this scenario, an oscillation of hub and leaf counts on a timescale of minutes, isn't visible at the crawler site. That may just be because the crawler's temporal resolution is too coarse to see it, though.

The fix in this case has an obvious short term as well as an easy long term solution. The short term fix is just to get enough people to nail their Shareazas into either hub-only or leaf-only behavior rather than let it be promoted/demoted. Preferably without ending up with too few hubs. The long term fix is to push out an update designed to damp out oscillations caused by hub promotion/demotion. Making promotion require a chronically high number of "rejected because leaf slots full" leaf rejections per minute over a longer span of time before a hub does something to try to get one or more of its leaves to become hubs (or however exactly promotion works) could do it. Hub count would climb more slowly and might level off rather than overshoot and start swinging wildly, even though the oscillations in leaf count are apparently wider than in the past for some reason, or at least reach lower low points. Another option is for hub to leaf demotion to have a refractory period: unless the node has been a hub more than X hours, it won't demote itself to leaf. Randomizing X at the time of promotion would be a good idea too. Otherwise, it will just broaden the peaks of hub count before the hub count crashes and rebounds again, slowing but not halting the oscillations. Randomizing X will make the hub count drop more slowly from its peak, and hopefully level off instead of undershoot. If there's already a refractory period programmed in, randomizing it or broadening the distribution would help. Each time a leaf becomes a hub, it should roll a random number between, say, 1 and 6 and be incapable of being demoted to leaf (other than by explicit user action) until that many hours after when it became a hub, even if it has a chronically low leaf count the whole while. The effect is that if there really are excessive hubs, the ones that rolled a 1 will drop, and then if there are now adequate hubs but not too many, the ones that rolled a 2 will have enough leaves by the 2-hour mark not to demote, and the oscillation is halted. (Making it minutes, random between 60 and 360, would be even better, as the hubs would just start dropping off slowly after the 1 hour mark until an optimum number was reached.)

There is a way to test each of the above bistability hypotheses.

Race condition: actually examine the code to see if the stuff involved in establishing hub connections touches a shared data structure without holding the mutex for that structure. Fix any instance where it does and deploy 2.7.2.0. Then wait a while to see if the problem goes away once 2.7.2.0 has broad enough uptake. If so, problem diagnosed and solved.

Deadlock and livelock: related to the above, the problem might instead be that the threading in Shareaza is prone to deadlock (wedges the hub) or livelock (temporarily jams at least some of its operations) trying to acquire mutexes. The cause would be if handling new inbound hub to hub connections acquires some particular two mutexes and holds them simultaneously and some other operation (maybe outbound hub to hub connection establishment) acquires the same mutexes in the reverse order and holds them simultaneously. Livelock instead of deadlock results if there's some slow timeout or something to eventually give up trying to get a mutex and abort the associated operation. The fix is essentially the same: fix the bug (this time by making both processes try to acquire the relevant mutexes in the same order) and push out 2.7.2.0, then see if the problem goes away.

Of course, it's possible that threading bugs like the above exist but aren't actually the cause of our present difficulties. In that case, the bugs will be found but fixing them won't fix G2. But it won't be a total waste, as bugs ought to be found and fixed regardless, and Shareaza and G2 will be more robust going forward.

Oscillatory hub election: the test for this is easy enough -- run some Shareazas (at least two) in "either hub or leaf" mode and see if some or all keep switching back and forth frequently and in something close to synchrony; say, most of the leaves become hubs within a few minutes, then after say half an hour of little change most of the hubs become leaves in only a few minutes, then after another longer period of little change most of the leaves become hubs, etc.

The fix in that case is to nail the test machines into hub mode to help stabilize the network, alert everyone here and wherever else you can to the situation and advise people with good long-lived broadband connections to go into hub-only mode until 2.7.2.0 is released and they've upgraded, and then make 2.7.2.0 with something like random-duration-set-on-promotion hub-to-leaf-demotion refractory period, test it for regressions, and release it.

If none of the above apply, then we're back to square 1 on this, though some sort of DOS attack or network poisoning then is the likeliest explanation.
Lanigiro
 
Posts: 132
Joined: 10 Feb 2014 14:19

Re: Ten minute CPU saturations every hour

Postby Lanigiro » 12 Feb 2014 01:26

The DLL is hal.dll. Thread using most CPU during listed that four times during but not at all after, while it showed ntdll.dll, kernel32.dll, MSVCR100.dll, and mfc100u.dll both during and after and USER32.dll only after.
Lanigiro
 
Posts: 132
Joined: 10 Feb 2014 14:19

Re: Ten minute CPU saturations every hour

Postby raspopov » 12 Feb 2014 03:57

Very bad for you, hal.dll is a hardware abstraction layer, so your computer is broken.
User avatar
raspopov
Project Admin
 
Posts: 811
Joined: 13 Jun 2009 12:30
Location: Russian Federation

Re: Ten minute CPU saturations every hour

Postby Lanigiro » 13 Feb 2014 19:38

raspopov wrote:Very bad for you, hal.dll is a hardware abstraction layer, so your computer is broken.


Everything else on it works fine. Most of the time so does Shareaza.

Here is full thread dump for misbehaving thread during CPU excursion:

ntkrnlpa.exe!KeWaitForMultipleObjects+0xabc
ntkrnlpa.exe!KeWaitForMutexObject+0x492
ntkrnlpa.exe!KeTestAlertThread+0x78
hal.dll!KfRaiseIrql+0xd1
hal.dll!KeRaiseIrqlToSynchLevel+0x70
hal.dll!HalEndSystemInterrupt+0x73
hal.dll!HalInitializeProcessor+0xcc1
mfc100u.dll!Ordinal2516+0x26
mfc100u.dll!Ordinal267+0x16
Shareaza.exe+0x92632
Shareaza.exe+0xef69e
Shareaza.exe+0xee7c9
Shareaza.exe+0x90cfa
Shareaza.exe+0x8e19b
Shareaza.exe+0x8df23
Shareaza.exe+0x8d31e
Shareaza.exe+0xd6b1e
Shareaza.exe+0xd93a8
Shareaza.exe+0xdd7d8
Shareaza.exe+0x8780
Shareaza.exe+0x128043
mfc100u.dll!Ordinal7493+0x16f
MSVCR100.dll!endthreadex+0x3a
MSVCR100.dll!endthreadex+0xe4
kernel32.dll!BaseThreadInitThunk+0x12
ntdll.dll!RtlInitializeExceptionChain+0x63
ntdll.dll!RtlInitializeExceptionChain+0x36
Lanigiro
 
Posts: 132
Joined: 10 Feb 2014 14:19

Re: Ten minute CPU saturations every hour

Postby raspopov » 13 Feb 2014 20:08

For getting function names instead of ordinal numbers you'll need to use corresponding debug symbol files (*.pdb) or daily build (with built-in .pdb-files).
User avatar
raspopov
Project Admin
 
Posts: 811
Joined: 13 Jun 2009 12:30
Location: Russian Federation

Re: Ten minute CPU saturations every hour

Postby Lanigiro » 13 Feb 2014 21:45

raspopov wrote:For getting function names instead of ordinal numbers you'll need to use corresponding debug symbol files (*.pdb) or daily build (with built-in .pdb-files).


I don't know what that means. If you mean translating the names and numbers into specific parts of Shareaza's code, can't you do that much more easily than I can, just by pasting them into something? I don't have these pdb files, at least I don't think I do, and I'm certainly not reinstalling Shareaza right now.
Lanigiro
 
Posts: 132
Joined: 10 Feb 2014 14:19

Re: Ten minute CPU saturations every hour

Postby raspopov » 14 Feb 2014 04:15

Anyway it's driver problem, btw antivirus and fierwall also has a driver.
User avatar
raspopov
Project Admin
 
Posts: 811
Joined: 13 Jun 2009 12:30
Location: Russian Federation

Re: Ten minute CPU saturations every hour

Postby Lanigiro » 14 Feb 2014 05:00

raspopov wrote:Anyway it's driver problem, btw antivirus and fierwall also has a driver.


I haven't changed anything about either recently, other than keeping the definitions up to date. The trigger for these things is some external change. Something happening on the network that didn't used to, probably related to the recent change in G2 behavior.

Which, by the way, is becoming extremely frustrating. I want someone to tell me how to force Shareaza to stay connected to G2, or at the very least how to make it so it will reconnect properly on its own, quickly and without the usual dance of turn off G2, wipe all those Chinese hubs from the hostcache, and turn on G2. So it will recover on its own even when it's left unattended for hours, instead of working for maybe half an hour and then staying offline until someone comes back to the machine and gives it the proverbial boot to the head.

Really, it's got to be rather embarrassing for Shareaza that it won't even do such a basic thing as stay connected or reconnect on its own without manual intervention. Why has this gone unfixed in several major versions now? It's got to be as simple as making the G2 host cache auto-drop Chinese IPs that get added to it, and that's probably as simple as adding one line of code to whatever function inserts new entries into the host cache, something like if (get_country(ip) == COUNTRY_CHINA) return; right at the start of it. Not too different from how it decides what flag to show in the UI when displaying an entry.
Lanigiro
 
Posts: 132
Joined: 10 Feb 2014 14:19

How do I force Shareaza to stay connected?

Postby Lanigiro » 14 Feb 2014 19:47

This is getting frustrating. The other threads related to connectivity issues all have drifted off onto tangents, or become moribund. So let's address this head-on.

1. How do I make my copy of Shareaza stay connected to G2? That means, how do I make individual connections last a "normal" amount of time (tens of hours) without interruption, AND how do I make Shareaza replace lost connections AUTOMATICALLY and IN A TIMELY FASHION? I want it to replace a lost connection, by itself, in a maximum of 60 seconds. How do I make this happen? Note: a valid solution must not compromise other functionality, for example it must not make any potential file sources unavailable for the purpose of downloading files from them.

2. How do I make my copy of Shareaza stay connected to a specific ED2K server? If I have pending downloads from "push" sources on, say, eMule Security #1, I can only potentially get the files if I stay connected to eMule Security #1. But if I connect to a particular ED2K server and go to sleep, I usually wake up to find that it's been connected to the wrong server for six or seven hours, uselessly spinning its wheels and unable to progress on any of the files I want that are from push sources on the first server. And then it often refuses to reconnect to the original server even if I explicitly tell it to using the hostcache. This behavior is incorrect. The major ED2K servers are up 24/7, never full, and long term stable, unlike ephemeral G1 and G2 hubs, so Shareaza should NEVER drop its connection to one without being instructed to by me and Shareaza should NEVER fail to connect to one on demand. Yet it keeps doing so. How do I prevent this undesired behavior? It's MY copy of the software, running on MY computer, so I have the absolute right to prevent this undesired behavior. I just lack the exact knowledge of how to enforce correct behavior here. (And am puzzled as to why Shareaza comes configured OOTB to behave incorrectly in this regard, instead of Just Working(tm).)

3. How do I make Shareaza not require constant nursemaiding and frequent restarts? As I've previously noted, a) it (usually) won't reconnect to G2 without manual assistance and b) it sometimes starts refusing to connect to particular ED2K servers. The latter behavior does not seem to go away without restarting Shareaza completely. I don't want to nursemaid it. I don't want to restart it every few hours. I want to "set it and forget it" and have it gradually acquire the files on its download list without my having to do anything but leave the machine switched on and hooked up to the Internet. Now someone please tell me how to configure it so that I CAN just "set it and forget it" as described.
Lanigiro
 
Posts: 132
Joined: 10 Feb 2014 14:19

Re: How do I force Shareaza to stay connected?

Postby Lanigiro » 14 Feb 2014 20:44

Lanigiro wrote:This is getting frustrating. The other threads related to connectivity issues all have drifted off onto tangents, or become moribund. So let's address this head-on.

1. How do I make my copy of Shareaza stay connected to G2? That means, how do I make individual connections last a "normal" amount of time (tens of hours) without interruption, AND how do I make Shareaza replace lost connections AUTOMATICALLY and IN A TIMELY FASHION? I want it to replace a lost connection, by itself, in a maximum of 60 seconds. How do I make this happen? Note: a valid solution must not compromise other functionality, for example it must not make any potential file sources unavailable for the purpose of downloading files from them.

2. How do I make my copy of Shareaza stay connected to a specific ED2K server? If I have pending downloads from "push" sources on, say, eMule Security #1, I can only potentially get the files if I stay connected to eMule Security #1. But if I connect to a particular ED2K server and go to sleep, I usually wake up to find that it's been connected to the wrong server for six or seven hours, uselessly spinning its wheels and unable to progress on any of the files I want that are from push sources on the first server. And then it often refuses to reconnect to the original server even if I explicitly tell it to using the hostcache. This behavior is incorrect. The major ED2K servers are up 24/7, never full, and long term stable, unlike ephemeral G1 and G2 hubs, so Shareaza should NEVER drop its connection to one without being instructed to by me and Shareaza should NEVER fail to connect to one on demand. Yet it keeps doing so. How do I prevent this undesired behavior? It's MY copy of the software, running on MY computer, so I have the absolute right to prevent this undesired behavior. I just lack the exact knowledge of how to enforce correct behavior here. (And am puzzled as to why Shareaza comes configured OOTB to behave incorrectly in this regard, instead of Just Working(tm).)

3. How do I make Shareaza not require constant nursemaiding and frequent restarts? As I've previously noted, a) it (usually) won't reconnect to G2 without manual assistance and b) it sometimes starts refusing to connect to particular ED2K servers. The latter behavior does not seem to go away without restarting Shareaza completely. I don't want to nursemaid it. I don't want to restart it every few hours. I want to "set it and forget it" and have it gradually acquire the files on its download list without my having to do anything but leave the machine switched on and hooked up to the Internet. Now someone please tell me how to configure it so that I CAN just "set it and forget it" as described.


What the hell? Why has this post become a part of an existing thread? I started a new thread because this is, to a significant extent, a new topic. Plus that other thread was being completely ignored. And I want a thread focused on how we Shareaza users can take full control over how and when our copies of Shareaza connect or (don't) disconnect.

Regardless of what's going on elsewhere with the network, I want MY COPY of Shareaza to auto-reconnect to G2 without having to manually turn off G2, delete everything from the hostcache, and turn on G2. I want it to reconnect automatically instead of getting stuck banging its head against the Great Firewall of China until the cows come home. And I want it to stay connected to ED2K properly, which it also isn't doing, and which has nothing to do with G2. Indeed, both G2 not reconnecting automatically and ED2K not staying connected are separate issues from the wonky behavior of G2 hubs the past two weeks. Both problems have been around longer, and G2 not reconnecting automatically has been around for YEARS, I think even since before Shareaza 2.6.0.0 came out.

Lastly, I find it disturbing that someone could be bothered to mess with my post but not to post a constructive reply to any of it, or to any of the other threads on related topics. Whoever you were, you were online, you read my post, now what do you have to say about it that might be of some use to me and any other users getting frustrated by this creeping bugfest? I mean, first G2 stops replacing lost connections without manual intervention, circa 2010. Then ED2K starts disconnecting randomly from time to time and requiring a full restart before it will reconnect (until then if you try Shareaza doesn't even send handshake packets, according to the packet monitor, and that's why it's not reconnecting). And now G2 won't stay connected for more than ten minutes at a time. It just seems to get worse and worse over time and nobody seems to be doing anything about it, since the G2 not reconnecting problem has gone unfixed for years and all it would take is adding one line to the add-to-G2-hostcache function to just immediately return if the IP is behind the GFC. One lousy line of code added to the G2 hostcache insert function, and nobody's found the time to add it in three years now, since the GFC (apparently) started blocking gnutella connects that cross it, and it has a major deleterious effect on the overall utility of Shareaza.

It's especially silly that when one is forced to disable G2 to clear the hostcache in order to fix this Shareaza bugs you with a nag window about how G2 is Shareaza's flagship network or something -- it sure isn't being treated that way by the actual developers, to judge by how a whole lotta nothing is being done to make the G2 hostcache insert function silently return without doing anything when the IP is behind the GFC and now a whole lotta nothing is being done about the new flaky G2 behavior. I myself have now done a lot of research to try to identify the problem, but reached the limit of what I can do with my level of technical knowledge and the tools at my own disposal, so I tossed the ball into your court three days ago, and what happened? You promptly dropped that ball, to judge by the lack of traffic in this thread by anyone other than me lately.
Lanigiro
 
Posts: 132
Joined: 10 Feb 2014 14:19

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby raspopov » 14 Feb 2014 21:00

Does your computer gets BSOD often?
User avatar
raspopov
Project Admin
 
Posts: 811
Joined: 13 Jun 2009 12:30
Location: Russian Federation

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby Lanigiro » 14 Feb 2014 21:12

raspopov wrote:Does your computer gets BSOD often?


No. What does that have to do with widespread problems with G2 that everyone is experiencing?
Lanigiro
 
Posts: 132
Joined: 10 Feb 2014 14:19

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby raspopov » 15 Feb 2014 06:46

Nope, It's about your problem driver.

I can see "10 minute"-hubs too (probably) but also a normal ones. Can you research more about this case? Be a hub, play with options etc.
User avatar
raspopov
Project Admin
 
Posts: 811
Joined: 13 Jun 2009 12:30
Location: Russian Federation

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby dew » 15 Feb 2014 07:01

I've not experienced the other problems Lanigiro mentioned, but something odd has happened to the network. The uptime graphs here http://crawler.doxu.org/uptimes.html show a drastic drop in uptime, and that does correlate with what I've seen. My neighbour connections drop within a few minutes (but reconnect with another hub after a few seconds automatically). As an experiment, I've tried several times to force my Shareaza into hub mode, but it always crashes after a few minutes.
dew
 
Posts: 9
Joined: 10 Feb 2014 05:05

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby raspopov » 15 Feb 2014 07:26

Can you send me crash report?
User avatar
raspopov
Project Admin
 
Posts: 811
Joined: 13 Jun 2009 12:30
Location: Russian Federation

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby dew » 15 Feb 2014 08:00

Okay, I've PM'd the error report to you. It took about 9 minutes for Shareaza to crash.

Incidentally, when this error occurred previously, I tried to submit the report through the automatic email mechanism but I received the error "There is no email program associated to perform the requested action. Please install an email program or, if one is already installed, create an association in the Default Programs control panel." I have Thunderbird installed, and in the Default Programs control panel it has all its defaults already.
dew
 
Posts: 9
Joined: 10 Feb 2014 05:05

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby raspopov » 15 Feb 2014 08:42

dew wrote:Okay, I've PM'd the error report to you. It took about 9 minutes for Shareaza to crash.


Stack trace are pretty confusing, looks like crash happens after G2 handshake completion code, can you also turn on "General.DebugLog" and send me last 50 lines of log before crash ( %APPDATA%\Shareaza\Data\Shareaza.log )?

Does Shareaza 32-bit crashes too?
User avatar
raspopov
Project Admin
 
Posts: 811
Joined: 13 Jun 2009 12:30
Location: Russian Federation

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby dew » 15 Feb 2014 09:23

Sent the log. I'll try 32 bit tomorrow. Going to bed now :)
dew
 
Posts: 9
Joined: 10 Feb 2014 05:05

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby Winterkeeper » 15 Feb 2014 10:12

I tried forcing hub mode on mine tonight. This is on a 32-bit dedicated server. Just to be conservative, I set it to only allow 100 leaves. I also only connected to G2 - no G1 connections. Everything was running fine until suddenly the CPU spiked to 100% just as Lanigiro talked about in the other thread, and every incoming connection was then rejected due to "network core overloaded". Then, just as Lanigiro said, after almost exactly 11 minutes the CPU spike ended and connections resumed. Of course, during that time all of my leaves had dropped. Afterwards, everything ran fine again for a short time, and then there was another 100% CPU spike with the same results. At that point I tried to disconnect from G2 and Shareaza became completely unresponsive.

I don't think it's a stretch to infer that this must be happening to a lot of hubs out there right now.
Winterkeeper
 
Posts: 8
Joined: 08 Feb 2014 00:12

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby raspopov » 15 Feb 2014 10:36

Crash Shareaza forcibly during high CPU utilization and send crash log to me please.
User avatar
raspopov
Project Admin
 
Posts: 811
Joined: 13 Jun 2009 12:30
Location: Russian Federation

Re: Ten minute CPU saturations every hour

Postby skinvista » 15 Feb 2014 15:25

FYI, I have been chasing a similar problem for the last week or so in a long-time fork of Shareaza.
Old builds (a year with no problems) exhibit the same behavior (at least when installed over current), debug seems worse.

By any chance are you recently running Windows 8.1?
The only recent local change was an upgrade from Win 8 a few weeks before the problem.
"Antimalware Service Executable" for Windows Defender seems to spike Disk usage at the onset,
but that could be a symptom or unrelated.

Otherwise, some external network trigger is exploiting shared code (lack of defense)
that I haven't successfully located/fixed yet.
User avatar
skinvista
 
Posts: 65
Joined: 13 Jun 2009 16:34
Location: Boston / New York City

Re: Ten minute CPU saturations every hour

Postby queuesclimber » 15 Feb 2014 19:13

Something happend. rev. 9357 works well. no crashes, connected to all networks
Attachments
something.PNG
Support European free vaping with your signature
monwiki.fr/efvi/
youtube -> watch?v=gDaqEe06CNA
User avatar
queuesclimber
 
Posts: 146
Joined: 29 Oct 2013 16:24

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby dew » 16 Feb 2014 00:29

I tried 32-bit Shareaza now. Same results as others. It doesn't crash (ran for 3 hours). I wasn't monitoring it while running, so I don't know if there were 11 minute spikes, but when looking at it after 3 hours there was high CPU (25%, i.e., one of my cores) and dropped most connections. I force-crashed it and PM'd the crash log.
dew
 
Posts: 9
Joined: 10 Feb 2014 05:05

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby dew » 16 Feb 2014 03:37

Earlier, I was testing hub mode. Now in leaf mode (32-bit Shareaza), I still see the CPU spikes. With 64-bit Shareaza, it ran fine in leaf mode (just the problem with neighbour connections lasting a short time).

With me, the CPU spikes are always precisely 4 minutes and 38 seconds long (I'm recording CPU usage with perfmon). They occur at somewhat random intervals between 10 - 20 minutes apart (from start of one spike to the start of the next).
dew
 
Posts: 9
Joined: 10 Feb 2014 05:05

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby raspopov » 16 Feb 2014 07:58

Looks like your Shareaza just overloaded by G2 packets, it spends many time in packet parsing code and incoming connections are dropped with "503 Busy" error.
User avatar
raspopov
Project Admin
 
Posts: 811
Joined: 13 Jun 2009 12:30
Location: Russian Federation

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby Lanigiro » 16 Feb 2014 08:18

raspopov wrote:Looks like your Shareaza just overloaded by G2 packets, it spends many time in packet parsing code and incoming connections are dropped with "503 Busy" error.


If so, then the data suggests that everyone is overloaded by G2 packets for ten minute bursts every hour or so. DoS attack on filesharers looking more and more likely the cause, though bistability isn't ruled out (if the packets are a symptom of that) and neither is a malfunctioning but not maliciously intended crawler or something.
Lanigiro
 
Posts: 132
Joined: 10 Feb 2014 14:19

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby raspopov » 16 Feb 2014 09:58

Try latest daily build r9358 please.
User avatar
raspopov
Project Admin
 
Posts: 811
Joined: 13 Jun 2009 12:30
Location: Russian Federation

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby Lanigiro » 17 Feb 2014 00:35

raspopov wrote:Try latest daily build r9358 please.


So far no ten-minute CPU surges, after a bit over half an hour. But it's much less CPU efficient than 2.7.1.0, tending to use double the amount of CPU used by 2.7.1.0 while functioning normally and when all other variables (library size, downloads size, connections, leaf vs. hub, and so forth) are held constant.

Avg. G2 connection lifetime seems to me to be unchanged. :(
Lanigiro
 
Posts: 132
Joined: 10 Feb 2014 14:19

Re: Constant G2 "lack of traffic" errors - can't stay connec

Postby Lanigiro » 17 Feb 2014 04:52

And now it seems to have locked up after running for about five hours. No change in CPU usage; it seems to be doing exactly what it was doing before "under the hood" to judge by the data from ProcessExplorer; but the UI is "spinning" and not updating itself all of a sudden. No apparent trigger, either. It was on the network tab when it happened and the tail of the log there looks perfectly typical.

Obviously, the changes you made didn't pan out. Something caused the CPU use during normal operation to be grossly increased, and though the CPU surges seem to be gone, it's now unstable and prone to hang outright after a while.

I think analysis of those excess G2 packets during one of the ten-minute-long "storms" will be needed to find out more about what is going on here and address the problem at its root. Simply adding some inbound packet-dropping throttle in front of the G2 subsystem, as it's implied you did, is a mere salve on the symptoms that, though it apparently works, seems to have deleterious and sometimes-lethal side effects.
Lanigiro
 
Posts: 132
Joined: 10 Feb 2014 14:19

Next

Return to Help and Support

Who is online

Users browsing this forum: No registered users and 1 guest