Very slow startup if search tabs left open

[blackflag100]

If search tabs with results (even just a couple of dozen results) are left open during shut-down, shareaza is very slow to start up again next session. It spends literally 10 minutes or more with the splash screen showing "Starting database" and barely any progress on the little green thingy. That thingy doesn't budge at all for the whole ten minutes, then races the rest of the way to the right afterwards -- not a very useful "progress" meter -- and in the meantime Shareaza chews up a core.

Examining the process with debugging tools reveals a single thread is responsible for the CPU usage and its stack looks like this:

ntkrnlpa.exe!KeWaitForMultipleObjects+0xabc
ntkrnlpa.exe!KeWaitForSingleObject+0x492
ntkrnlpa.exe!KeTestAlertThread+0x78
hal.dll!KfRaiseIrql+0xd1
hal.dll!KeRaiseIrqlToSynchLevel+0x70
hal.dll!HalEndSystemInterrupt+0x73
hal.dll!HalInitializeProcessor+0xcc1
ntdll.dll!RtlLeaveCriticalSection+0x33
RegExp.dll+0xced6
RegExp.dll+0xc251

So, startup is being consumed with doing a lot of regular expression matches somewhere, or perhaps with turning textual regexps into compiled representations.

This seems wrong. If it's compiling regexp patterns, the compiled patterns should be saved across sessions. If it's just doing a bunch of matches:

a) The only obvious candidate is applying the security filter rules to something. The saved search results? But they had already had the security filter rules applied to them, before the last shutdown. There is no need to apply them again; they all passed already. Also, if there are only a couple of dozen hits showing in those search tabs there is no FREAKING way that the regexp tests should be taking 10 minutes on a multi-GHz core. Not even when it's single-threaded, and not even if it were doing it in Java, BASIC, Python or even, God forbid, COBOL. (And I'm given to understand it's actually C++.)

b) To emphasize. It's spending several trillion instruction cycles in one single function in RegExp.dll. That probably represents hundreds of millions of tests of strings against patterns. Even if there were 1000 search results instead of under 100 and 1000 security rules and the algorithm was dumb enough to apply every rule to every result, even if another rule had already decided that result should be rejected, that would only be one million, unless it's stupidly doing each single pair of rule/result checks hundreds of times instead of just once!

Something is woefully inefficient here and it definitely involves the result objects themselves. Clearing all of the result tabs before restart results in it only taking 1 minute or so to start up, as does having no search tabs open at all.

[cyko_01]

[blackflag100]

[old_death]

blackflag100, I am currently working on an improvement of the security filter code. Once it is finished, it should be hundreds of times faster on very long security lists, such as you do have for example.

Note however that it is necessary to do the search filtering and security checking again while Shareaza is starting up, because it is possible to update the security filters while Shareaza is not running - this means that what has been filtered before might be allowed after a restart - and vis versa.


mfg,
Old

[cyko_01]

I highly recommend switching away from the xray filter to the AIOASF (and not just because I made it). The xray filter has over 1500 rule,s all of which are IP filters. The x-ray filter is about 3 years old now and most of the rules no longer apply since most spammers change IP addresses on a regular basis. Security filters need to be updated AT LEAST once a year, preferably more often. The xray filter is so old that it does not have support for regexp filters (which were introduced in 2.4.0.0). it also does not include any filters for file-size or keyword spam. If I recall correctly, the foxy filter simply blocks all of china (by IP address), and is therefore also a very large filter as well.

[blackflag100]

[blackflag100]

[cyko_01]

[blackflag100]

I'm now using PeerBlock instead of Shareaza's security rules to block China and Taiwan to prevent Foxy host-cache pollution.

I'll look into the link you posted. Thanks.