Timestamps are in GMT/BST.
[1:09] * jay (~jcl@ool-18bf6dac.dyn.optonline.net) Quit ("Leaving")
[1:51] * FallingBuzzard (~Administr@c-24-13-19-35.client.comcast.net) has joined #freenet
[1:52] * FallingBuzzard (~Administr@c-24-13-19-35.client.comcast.net) has left #freenet
[1:57] * FallingBuzzard (~Administr@c-24-13-19-35.client.comcast.net) has joined #freenet
[2:04] * FallingBuzzard (~Administr@c-24-13-19-35.client.comcast.net) has left #freenet
[2:04] * FallingBuzzard (~Administr@c-24-13-19-35.client.comcast.net) has joined #freenet
[2:05] * FallingBuzzard (~Administr@c-24-13-19-35.client.comcast.net) has left #freenet
[3:57] <Ribs> IRCing from an admin account
[3:57] <Ribs> that's smart(!)
[4:32] <hobx> I never understood the whole don't IRC as root thing
[4:33] <hobx> if the bloody IRC client is full of exploits we shouldn't be running it at all.
[5:08] <Ribs> hobx: Same reason you don't do normal day-to-day things as root
[5:08] <mazzanet> vital information for mac g5 owners! --> http://docs.info.apple.com/article.html?artnum=86816
[5:09] <Ribs> lol
[5:13] <Ash-Fox> hmm http://pdos.lcs.mit.edu/tarzan/
[5:14] * d-ArkAngel (~Rob@213-131-104-86.onyx.net) has joined #freenet
[5:17] * pittaman (~mistery_b@d51A410CA.kabel.telenet.be) has joined #freenet
[5:21] * tyesinclair (~tyesincla@170.252.64.1) has joined #freenet
[5:21] * tyesinclair (~tyesincla@170.252.64.1) has left #freenet
[5:26] * Ribs (~ribs@riblet.plus.com) Quit ("I'm no quitter.")
[5:46] * spaetz (~spaetz@217-162-197-136.dclient.hispeed.ch) has joined #freenet
[5:53] * pittaman (~mistery_b@d51A410CA.kabel.telenet.be) Quit (Client Quit)
[6:02] * kevloral2 (~kevloral@CZ1-RAS-8-u-0179.du.onolab.com) has joined #freenet
[6:03] * kevloral (~kevloral@CZ1-RAS-8-u-0179.du.onolab.com) Quit (Nick collision from services.)
[6:03] * kevloral2 is now known as kevloral
[6:49] * guido^pe (~unknown@dsl-082-082-158-227.arcor-ip.net) has joined #freenet
[6:49] * Ribs (~ribs@riblet.plus.com) has joined #freenet
[6:58] <spaetz> brb
[6:58] * spaetz (~spaetz@217-162-197-136.dclient.hispeed.ch) Quit ("Verlassend")
[6:58] * spaetz (~spaetz@217-162-197-136.dclient.hispeed.ch) has joined #freenet
[8:10] * guido^pe (~unknown@dsl-082-082-158-227.arcor-ip.net) Quit (Remote closed the connection)
[8:11] * cbreak (~cbreak@dhcp-32-094.via-eth.ch) has joined #freenet
[9:03] * tyesinclair (~tyesincla@170.252.64.1) has joined #freenet
[9:03] * tyesinclair (~tyesincla@170.252.64.1) has left #freenet
[9:23] * Hory (~Miranda@82.76.81.56) has joined #FreeNet
[9:42] * Hory (~Miranda@82.76.81.56) Quit (Read error: 54 (Connection reset by peer))
[9:47] * spaetz (~spaetz@217-162-197-136.dclient.hispeed.ch) Quit (Remote closed the connection)
[9:49] * spaetz (~spaetz@217-162-197-136.dclient.hispeed.ch) has joined #freenet
[9:51] * FallingBuzzard (~srademach@207.152.112.129) has joined #freenet
[10:16] * pupok (~r00t@81-178-76-180.dsl.pipex.com) Quit (Read error: 232 (Connection reset by peer))
[10:16] * sanity (~ian@81-178-76-180.dsl.pipex.com) Quit (Read error: 104 (Connection reset by peer))
[10:16] * pupok (~r00t@81-178-76-180.dsl.pipex.com) has joined #freenet
[10:21] * sanity (~ian@81-178-76-180.dsl.pipex.com) has joined #freenet
[10:23] * lostlogic (~lostlogic@node-4024215a.mdw.onnet.us.uu.net) Quit ("Going to the moon")
[10:32] * greycat (~wooledg@192.35.79.70) has joined #freenet
[10:55] * spaetz (~spaetz@217-162-197-136.dclient.hispeed.ch) Quit ("Verlassend")
[10:59] * lolo-laptop (~lostlogic@68.251.84.226) has joined #freenet
[11:17] * Para[LT] (~Paramount@ip68-230-126-43.ph.ph.cox.net) has joined #freenet
[12:04] * spaetz (~spaetz@217-162-197-136.dclient.hispeed.ch) has joined #freenet
[12:07] * toad (toad@82-32-18-233.cable.ubr03.azte.blueyonder.co.uk) has joined #freenet
[12:07] * sanity (~ian@81-178-76-180.dsl.pipex.com) Quit (Read error: 104 (Connection reset by peer))
[12:07] <toad> this is more than a little odd...
[12:08] <toad> if I run a simulation from scratch, the cycle times slowly increase to a little over 200 seconds
[12:08] <toad> if I then use the serialized data to start a simulation from the snapshot, then it starts off slow, and gets slower
[12:09] <toad> with kernel 2.6.8.1, it will actually hang the computer for about 5 minutes in the middle of reading the data in
[12:09] <toad> and trigger the OOM killer
[12:09] <toad> the JVM reported memory usage is only ~ 300MB on a 1536MB-RAM plus 2GB swap box
[12:09] <toad> on 2.6.9, it will hang the box solid, some hours after serialization
[12:10] <toad> i've just run over 24 hours of memory tests, and i'm pretty sure the ram is solid
[12:10] <toad> any ideas?
[12:13] * sanity (~ian@81-178-76-180.dsl.pipex.com) has joined #freenet
[12:13] <toad> naturally nobody is here...
[12:13] <d-ArkAngel> Sorry toad, I've been so involved in RL stuff that I've not had any chance to look at it
[12:14] <d-ArkAngel> the only thing I could think might be maybe writeing some code to store/initilise the network your self if serialise just isn't floating your boat.
[12:15] <d-ArkAngel> the only cause I could think for such a thing was that when loading the serailised data it's not un-serialiseing it, it's just keeping it in the serialised format and doing something CRAZY like serialising changes to the end of the data structures or something...
[12:15] <toad> this is my own code
[12:15] <d-ArkAngel> ahh
[12:15] <d-ArkAngel> ok
[12:15] <toad> i'm not using pickling
[12:16] <d-ArkAngel> sorry thought you were using a java construct to do it.
[12:16] <toad> although my coldstore etc background suggests maybe i should
[12:16] <toad> is that supported by regular jvms?
[12:17] <d-ArkAngel> well i know you can send objects through TCP streams, so I can't see why you wouldn't be able to "send and receive" them to file writer streams
[12:17] <d-ArkAngel> since java goes to such lengths to keep all the stream types compatible.
[12:18] <toad> how?
[12:21] <toad> okay, what is the next move?
[12:23] <d-ArkAngel> (I'm still at works, sorry if my replies are a little stop/start)
[12:25] * toad wonders why his dyndns-hosted website hasn't updated, whereas the .dyndns.org has updated quickly enough
[12:25] <toad> s/website/domain
[12:25] <spaetz> hi toad
[12:25] <toad> hi spaetz
[12:26] <toad> have you read my mail on tech and devl? it's hard to miss, it's the only thing anyone's posted in days :<
[12:26] <toad> Subject: [Tech] Wierd problems with simulations
[12:26] * Para[LT] (~Paramount@ip68-230-126-43.ph.ph.cox.net) Quit (Read error: 104 (Connection reset by peer))
[12:26] <spaetz> after the 2Ghz Celeron died which I promised you., I upgraded a little. You may use my new "Athlon 64" box now :-)
[12:26] * kevloral is now known as kevlAway
[12:27] <toad> cool
[12:27] <toad> not much use to me if I can't get these simulation serialization problems sorted out though :<
[12:28] <spaetz> ouch, sounds wierd, yes
[12:28] <spaetz> mmh, I wonder if its really kernel related
[12:28] <cbreak> so how is it going with freenet? Still only simulating?
[12:28] <cbreak> (Not that sims are not fun :)
[12:28] <spaetz> cbreak: I'd say simulating is way more important than doing day-to-day bug fixing :)
[12:30] <d-ArkAngel> it's probably not caused by the kernel, but the kernel just demonstrates different amounts of resiliance.
[12:30] <spaetz> If simulations indicate that something might be seriously wrong.... developing further as if nothing had happened might be a waste
[12:30] <toad> it might be a jvm bug...
[12:30] <cbreak> well... freenet simulations give the project more the look of a scientific project and not an OS Project :)
[12:30] <toad> heh
[12:31] <spaetz> d-ArkAngel: Sigh, that's why I hate a close source java. You never really know
[12:31] <toad> I like being OS, but most OS projects have clearly definable goals and technologies...
[12:31] <spaetz> toad: Well, many tinker around a lot. But then also many OS projects go into hibernation after a while
[12:31] <spaetz> ;-)
[12:32] <toad> hmmm
[12:32] <toad> interesting
[12:32] <toad> I run it in Kaffe, and it reports negative memory usage...
[12:33] <d-ArkAngel> 8-O
[12:33] <spaetz> Did you know that the median of the number of devs on sourceforge is one?
[12:33] <d-ArkAngel> wow, run it loads of times... infinite memory! :-)
[12:38] <d-ArkAngel> toad: check out the documentation for ObjectOutputStream (and ObjectInpputStream )
[12:39] <toad> d-ArkAngel: what package?
[12:39] <d-ArkAngel> java.io
[12:41] <d-ArkAngel> it probably won't deal with static variables tho... you'd probably have to check what's using static stuff. probably just the logging numbers, and you can probably just note those down seperately.
[12:58] * spaetz (~spaetz@217-162-197-136.dclient.hispeed.ch) Quit ("Verlassend")
[13:00] * d-ArkAngel (~Rob@213-131-104-86.onyx.net) Quit (Read error: 104 (Connection reset by peer))
[13:01] * spaetz (~spaetz@217-162-197-136.dclient.hispeed.ch) has joined #freenet
[13:04] * tyesinclair (~tyesincla@170.252.64.1) has joined #freenet
[13:05] * tyesinclair (~tyesincla@170.252.64.1) has left #freenet
[13:22] * Hory (~Miranda@82.76.81.56) has joined #FreeNet
[13:22] <toad> hrrm
[13:22] <toad> reproduced with smaller simulated network
[13:23] <toad> ~ 3 seconds per cycle running straight
[13:23] <toad> ~ 50 seconds per cycle post serialization
[13:31] <toad> hmmm
[13:31] <toad> welll
[13:32] <toad> using 100 nodes instead of 800, it loads much faster, of course...
[13:33] <toad> cycle is about 3 seconds from scratch
[13:33] <toad> and about 18 seconds from reading in the serialized data
[13:35] <KenMan> so toad, i guess i grabbed your sim code at just the right time ;)
[13:35] <toad> it works fine if you don't try to restart from a snapshot.. i think..
[13:35] <KenMan> don't spend much more time on it then, just keep going where you were
[13:36] <toad> nah, the ability to restart simulations is pretty essential
[13:36] <KenMan> well, rather than 'serialization' could you do 'write out a data file' and 'read in a data file' ? I realize they should be about the same thing.
[13:37] <KenMan> I mean, have you confirmed that it is working properly ? reading in the proper data ?
[13:37] <toad> KenMan for my purposes they are the same thing
[13:37] <toad> i'm not using ObjectOutputStream etc
[13:37] <toad> maybe I should
[13:38] <toad> if it was drastically wrong as far as data goes, it'd trip one of the numerous checks... and i dump pretty much everything to stderr while reading in
[13:38] <KenMan> okay. Well, try working smaller - can you serialize one , or ten objects, without trouble ?
[13:38] <toad> KenMan: i tried with 800 nodes and got all sorts of fun
[13:39] <toad> I am now working with 100 nodes
[13:39] <toad> and get exactly the same
[13:39] <toad> no obvious discernible differences
[13:39] <KenMan> maybe 1 node would be less 'fun' ...
[13:39] <toad> except it takes a LOT longer
[13:39] <toad> per cycle
[13:39] <KenMan> that's the broken part, heh
[13:39] * michaelkuijn (~michaelku@fia41-111.dsl.hccnet.nl) has joined #freenet
[13:39] <toad> 4-5 times more time per cycle
[13:39] <cbreak> when you export something, import it again and then export it a second time, are both exports equal?
[13:40] <michaelkuijn> 'evening guys
[13:40] <KenMan> good idea, cbreak
[13:40] <toad> huh?
[13:40] <KenMan> just a sanity check that you really are doing what you think you are doing. Validate the serialization.
[13:41] <toad> i don't think it's a problem with the content of the data
[13:41] <toad> it's something else...
[13:41] <toad> some flag that gets turned on because the serialization happened, or something similarly insane...
[13:41] <michaelkuijn> Just a question... how is the /. article going?
[13:42] <KenMan> yes, sounds believable, but if you can't resolve it quickly, get yourself to 'move on'
[13:42] * spaetz is now known as spaetzAway
[13:42] <toad> KenMan: that means long term simulations are impossible
[13:42] <KenMan> i know, but there are other approaches that you can take , and still keep learning
[13:42] <toad> like what?
[13:43] <toad> how can we prove that freenet scales without simulating large freenets?
[13:43] <KenMan> okay. You are right. Don't let the computer win. Spend a day a week or a month, whatever it takes to make this serialization work.
[13:44] * KenMan is trying to play hardball ;)
[13:45] <toad> heh
[13:45] <KenMan> you are a bright fellow, just don't get tunnel-vision focus on this one problem. Learn how to 'work around it' , in whatever creative way you can
[13:45] <toad> but it's a bug
[13:45] <toad> and i haven't spent that much time on it really
[13:45] <toad> the last 2 days don't count
[13:46] <toad> because of the crashing/kernel issues
[13:46] <KenMan> toad_s don't have to eat every bug, certainly not on this project! You can pick and choose to meet your dietary requirements
[13:46] <toad> yay, i found a difference...
[13:46] <toad> the key fetch limit is 357 on the restarted one
[13:46] <toad> and 250 on the other one
[13:46] <KenMan> hmm ?
[13:47] * pittaman (~mistery_b@d51A410CA.kabel.telenet.be) has joined #freenet
[13:48] <KenMan> oh, you mean there are a differing number of 'still in store' keys ?
[13:48] <KenMan> s/store/stores
[13:48] <toad> hmmm
[13:48] <toad> brb
[13:48] * toad (toad@82-32-18-233.cable.ubr03.azte.blueyonder.co.uk) Quit ("BitchX Official WWW Site -- http://www.bitchx.com")
[13:49] <KenMan> I think toad is just lonely, not that he is incapable of self-reliance...
[13:49] <KenMan> it is nice when someone else takes an interest and brings you new ideas though
[13:49] * toad_ (toad@82-32-18-233.cable.ubr03.azte.blueyonder.co.uk) has joined #freenet
[13:50] <KenMan> ribbit
[13:50] <toad_> ribbit
[13:50] <KenMan> so, after reading in a stored state, you have a different number of keys globally available ?
[13:50] <toad_> i'm tracking a different number of keys
[13:51] <toad_> i think...
[13:51] <cbreak> Serialisation is for pausing a sim?
[13:51] <toad_> maxFetchKeys is wrong...
[13:51] <toad_> because OVERRIDE_FETCH_FRACTION isn't saved, perhaps...
[13:52] <KenMan> ah, what about Node.Peer ? could references become copies or something ? no , i guess not.
[13:52] <KenMan> OVERRIDE_FETCH_FRAC doesn't explain excess run times, could it ?
[13:53] <KenMan> i still haven't read your code enough to know how you manage the global keyset :(
[13:53] <KenMan> could you keep a hashtable with reference counts ?
[13:54] <toad_> Setting fetch fraction to -1.0
[13:54] <toad_> aha
[13:54] * KenMan toad applies his miracle wart-removing cream ...
[13:55] <toad_> in the original, it's Setting fetch fraction to 0.025
[13:55] <toad_> ewww
[13:58] <KenMan> that means, of all still in store keys, only use 0.025 of them ?? ? or what ?
[13:58] <toad_> not exactly
[13:58] <KenMan> or, do 40 inserts for each request ?
[13:58] <toad_> we fetch the last N inserts
[13:59] <toad_> where N is fetchfrac*#nodes*storeSizeInKeys
[13:59] <KenMan> oh weak. You are not keeping a list of available keys ?
[13:59] <toad_> eh?
[14:00] <KenMan> sooner or later we will be playing with stores of different sizes... we should keep a ref-counted list of active keys
[14:00] <toad_> this is just to calculate psuccess
[14:00] <toad_> okay, now they both have the same...
[14:00] <toad_> and it still sucks...
[14:00] <KenMan> :p - murphy strikes you again !
[14:01] <toad_> the one that was already running takes very short cycle times
[14:01] <toad_> the restart still takes fscking ages
[14:01] <toad_> new:
[14:01] <toad_> Cycle took 35170ms
[14:01] <toad_> Cycle took 32287ms
[14:01] <toad_> Cycle took 36248ms
[14:01] <toad_> Cycle took 43402ms
[14:01] <cbreak> If I had such a problem, I would compare the memory before and after serialisation... (well... not that I am experienced :)
[14:01] <toad_> old:
[14:02] <toad_> Cycle took 4691ms
[14:02] <toad_> Cycle took 4508ms
[14:02] <toad_> Cycle took 4628ms
[14:02] <toad_> Cycle took 4649ms
[14:02] <toad_> Cycle took 4573ms
[14:02] <toad_> Cycle took 10665ms
[14:02] <toad_> cbreak: compare the memory?
[14:02] <KenMan> the 'data'
[14:02] <toad_> I don't see how that would cause this
[14:02] * d-ArkAngel (~robert@cpc2-midd2-5-0-cust16.midd.cable.ntl.com) has joined #freenet
[14:02] <toad_> I don't see how ANY POSSIBLE change could cause this
[14:02] <KenMan> validate that the serialize ops work correctly
[14:02] <toad_> it must be something in the globals or soemthing
[14:03] <toad_> but the fact that it serializes without running into any of the numerous corruption traps shows that it's not just a byte wrong somewhere, as that would corrupt the whole file
[14:03] <toad_> the summaries are sane enough
[14:03] <toad_> i can see individual connections, but that's not very informative
[14:03] <toad_> the HTL taken to find a datum is roughly the same
[14:03] <toad_> even the psuccess isn't that far off
[14:04] <KenMan> oh, you probably just forgot to free a pointer or something :o
[14:04] <toad_> Memory usage: 29634840 (13631208 free of 43266048) (new)
[14:04] <toad_> Memory usage: 18646472 (2939448 free of 21585920) (old)
[14:04] * Ash-Fox (Hal-9000@aao142.neoplus.adsl.tpnet.pl) Quit (Read error: 54 (Connection reset by peer))
[14:05] <toad_> a difference but not a dramatic one...
[14:05] <toad_> bigger diff on bigger networks i think
[14:05] * pittaman (~mistery_b@d51A410CA.kabel.telenet.be) Quit (Client Quit)
[14:06] <toad_> hmmm
[14:06] <toad_> does Long implement the methods needed for inclusion in a hashtable?
[14:06] <toad_> equals and hashCode...
[14:06] <d-ArkAngel> I would have thought so
[14:07] * pittaman (~mistery_b@d51A410CA.kabel.telenet.be) has joined #freenet
[14:08] <toad_> hmm, according to docs it does
[14:08] <toad_> sh*t
[14:08] <toad_> Routed: 0 newbie, 0 random, 8565 by estimators
[14:08] <toad_> Success ratio on last 10K reqs: 0.9615 (770 0s, 19230 1s, 20000 total)
[14:08] <toad_> Connections added: 32, tried 2821, force tried 0, passed estimators: 0
[14:08] <toad_> vs
[14:08] <toad_> Routed: 0 newbie, 0 random, 8200 by estimators
[14:08] <toad_> Success ratio on last 10K reqs: 0.95735 (853 0s, 19147 1s, 20000 total)
[14:09] <toad_> Connections added: 31, tried 3102, force tried 0, passed estimators: 0
[14:09] <toad_> there is very little difference
[14:09] <toad_> Memory usage: 21463936 (822400 free of 22286336)
[14:09] <toad_> vs
[14:09] <toad_> Memory usage: 23396328 (19869720 free of 43266048)
[14:09] <toad_> the latter is the post-serialization one
[14:09] <toad_> the ONLY difference is:
[14:09] <toad_> Cycle took 43299ms
[14:09] <toad_> vs
[14:09] <toad_> Cycle took 4401ms
[14:10] * toad_ tries with kaffe...
[14:10] <cbreak> maybe the deserialized objects are constant, or contain constants (or whatever), so the runtime has to cast them somehow...
[14:11] <cbreak> I had such a problem with an other language once...
[14:11] <toad_> cbreak: one small problem there
[14:11] <toad_> I'm not using the built-in serialization stuff
[14:11] <toad_> i'm just writing a data file
[14:12] * Ash-Fox (Hal-9000@aao142.neoplus.adsl.tpnet.pl) has joined #FreeNET
[14:14] <toad_> definitely doesn't work better on kaffe...
[14:14] <toad_> Cycle took 98491ms
[14:16] <toad_> i suppose i could rewrite it to use ObjectOutputStream etc...
[14:17] <cbreak> maybe the serialisation continues to use CPU time after it is completed? In an other Thread?...
[14:17] <toad_> cbreak: nothing obvious
[14:17] <KenMan> ObjectOutputStream sounds like a good path .
[14:17] <toad_> anyway it'd have to be a whole bunch of extra threads
[14:17] <toad_> sure, and discard all the existing serialization code
[14:17] <toad_> what fun
[14:17] * toad_ mumbles something about painting the grass green
[14:17] <KenMan> hey, you said you didn't spend much time on it yet...
[14:18] <toad_> i spent a day or two writing it and doing preliminary debugging just to get it to run
[14:18] <toad_> anyway bbiab, food..
[14:18] <KenMan> yum, chinese chicken for me... LUNCHTIME
[14:19] <d-ArkAngel> grass is evil and should be painted red accordingly
[14:47] <cbreak> well... good luck with your sims :)
[14:47] * cbreak (~cbreak@dhcp-32-094.via-eth.ch) Quit ("...")
[14:50] <toad_> heh
[14:50] <toad_> hi ppl
[14:50] * toad_ back
[14:51] <toad_> Routed: 0 newbie, 0 random, 8634 by estimators
[14:51] <toad_> Success ratio on last 10K reqs: 0.96515 (697 0s, 19303 1s, 20000 total)
[14:51] <toad_> Connections added: 33, tried 2998, force tried 0, passed estimators: 0
[14:51] <toad_> Cycle took 105083ms
[14:51] <toad_> Memory usage: -341823488 (402653184 free of 60829696)
[14:51] <toad_> vs
[14:51] <toad_> Routed: 0 newbie, 0 random, 8407 by estimators
[14:51] <toad_> Success ratio on last 10K reqs: 0.96265 (747 0s, 19253 1s, 20000 total)
[14:51] <toad_> Connections added: 41, tried 3088, force tried 0, passed estimators: 0
[14:51] <toad_> Cycle took 11108ms
[14:51] <toad_> Memory usage: -358600704 (402653184 free of 44052480)
[14:51] <toad_> they are producing eerily similar results
[14:52] <KenMan> so what, the problem must be shot !
[14:52] <toad_> what would make one so much slower, but produce almost identical results?
[14:52] <KenMan> what diff was the memory use on a large scale config ?
[14:53] <KenMan> thousands of unneeded memory references might slow it down ?
[14:53] <d-ArkAngel> ooo, Firefox 1.0 is out...
[14:53] <toad_> KenMan: it's only between 45MB and 60MB on small scale
[14:53] <toad_> not going to be much GCing at that level...
[14:53] <greycat> oh, maybe they'll build it for hp-ux 10.20! hahahahaha, right.
[14:54] <KenMan> what classes do you write out ? each of the nodes (which contain lots of Peers), what else ?
[14:54] <toad_> greycat: the question is, can you build it for hp-
[14:54] <toad_> ux?
[14:54] <KenMan> heh, right ;)
[14:54] <toad_> the estimators for each of these...
[14:54] <toad_> the options
[14:55] <greycat> toad_: I, personally, cannot. Mozilla has been built for hp-ux 10 in the past (I'm still using moz 1.4). But they used an arcane and twisted build process which required specific versions of non-free C++ compiler suites.
[14:55] <toad_> the stats...
[14:56] <KenMan> hmm, the estimators would be most likely to affect runtime this way, no ?
[14:56] <toad_> perhaps
[14:56] <toad_> but the serialization code for the estimators works
[14:57] <KenMan> could you instrument (or profile) the time spent in estimator update function(s) ?
[14:57] <KenMan> from the external calling point ? wrap with two get SystemTimeInMillis maybe
[14:58] <KenMan> i guess it would be useful anyway to get some time measurements on various places in this sim code...
[14:59] * KenMan must go do a pressure-washer job now - pay=chinese food :)
[14:59] <toad_> :)
[15:05] <toad_> if fast estimator wasn't working for some reason after reading in, that'd explain it..
[15:06] <toad_> found it
[15:10] <toad_> aha
[15:10] <toad_> fixed it
[15:10] * toad_ committing...
[15:11] <toad_> well soon
[15:11] <toad_> diffing at the moment
[15:16] <greycat> I'm still seeing firefox 1.0-PR everywhere I look on mozilla.org.
[15:16] <greycat> no sign of a 1.0 release on their pages
[15:23] <d-ArkAngel> they must be in the process os posting it.... seems none of the downloads are there.
[15:23] <d-ArkAngel> ahh well maybe tomorrow
[15:24] <toad_> shit, who modified the sim code? is there a clash...
[15:24] <toad_> most likely there is...
[15:26] * michaelkuijn (~michaelku@fia41-111.dsl.hccnet.nl) Quit ()
[15:27] <d-ArkAngel> I havn't commited in at least a week, and I've not made any changes to the main sim code far as I rememeber
[15:28] <toad_> it doesn't look like it was a big problem...
[15:28] <toad_> anyway
[15:28] <toad_> now that's dealt with
[15:28] <toad_> first, draw some graphs
[15:30] * toad_ draws some graphs...
[15:41] * Tril (tril@bespin.org) has joined #freenet
[15:42] <toad_> hi
[15:42] <Tril> hey toad
[15:58] <toad_> bbl
[16:00] * jay (~jcl@198.38.10.200) has joined #freenet
[16:00] <jay> lo toad
[16:16] * d-ArkAngel (~robert@cpc2-midd2-5-0-cust16.midd.cable.ntl.com) Quit (Read error: 113 (No route to host))
[16:26] * guido^pe (~unknown@dsl-213-023-243-254.arcor-ip.net) has joined #freenet
[16:44] * Ribs (~ribs@riblet.plus.com) Quit ("I'm no quitter.")
[16:45] * Usurp (~Usurp@trudaine-8-82-230-34-86.fbx.proxad.net) Quit (Remote closed the connection)
[16:51] * Ribs (~ribs@riblet.plus.com) has joined #freenet
[16:53] * kers (kers@m213-101-157-102.swipnet.se) has joined #freenet
[17:01] * MrNaughty (MrNaughty@d198-166-246-103.abhsia.telus.net) Quit ("\(^_^)/' No Soliciting!!! Unless you have legs way, way up and really, really big tits....")
[17:17] * kers (kers@m213-101-157-102.swipnet.se) Quit ("Leaving")
[17:21] * greycat (~wooledg@192.35.79.70) Quit ("Lick Bush in '04")
[18:03] * pittaman (~mistery_b@d51A410CA.kabel.telenet.be) Quit (Client Quit)
[18:07] * dalibor (~topic@p5480D10D.dip.t-dialin.net) has joined #freenet
[18:07] <dalibor> toad_, freeMemory / max Memory fixed now ;)
[18:25] * FallingBuzzard (~srademach@207.152.112.129) has left #freenet
[18:39] * Hadaka (naked@naked.iki.fi) Quit (Read error: 60 (Operation timed out))
[18:42] * yonkeltron (~yonkeltro@pcp04665066pcs.wilog501.pa.comcast.net) Quit ("Leaving")
[18:44] * MrNaughty (MrNaughty@d198-166-246-103.abhsia.telus.net) has joined #freenet
[18:45] <toad_> cool
[18:46] <dalibor> thanks for the bug report!
[18:46] <dalibor> and good night ;)
[18:46] * dalibor (~topic@p5480D10D.dip.t-dialin.net) has left #freenet
[18:49] * jay (~jcl@198.38.10.200) Quit ()
[19:01] * plixed_ (~plixed@pD9E259BD.dip.t-dialin.net) has joined #freenet
[19:13] <Redb3ard> hey guys, whats up?
[19:18] * plixed (~plixed@p50823D5B.dip.t-dialin.net) Quit (Connection timed out)
[19:23] * Hory (~Miranda@82.76.81.56) Quit (Read error: 54 (Connection reset by peer))
[20:19] * lolo-laptop (~lostlogic@68.251.84.226) Quit ("Client exiting")
[20:56] * lostlogic (~lostlogic@node-4024215a.mdw.onnet.us.uu.net) has joined #freenet
[21:05] <KenMan> hello lostlogic .
[21:05] * Naked (naked@naked.iki.fi) has joined #freenet
[21:05] * Naked is now known as Hadaka
[21:20] * bluephile (bluephile@69-160-215-63.clvdoh.adelphia.net) Quit (leguin.freenode.net irc.freenode.net)
[21:20] * pwk_ (~phweak@dsl-62-3-71-168.zen.co.uk) Quit (leguin.freenode.net irc.freenode.net)
[21:20] * vsalento (~vsalento@cs136227.pp.htv.fi) Quit (leguin.freenode.net irc.freenode.net)
[21:21] * bluephile (bluephile@69-160-215-63.clvdoh.adelphia.net) has joined #freenet
[21:21] * pwk_ (~phweak@dsl-62-3-71-168.zen.co.uk) has joined #freenet
[21:21] * vsalento (~vsalento@cs136227.pp.htv.fi) has joined #freenet
[21:22] * pwk__ (~phweak@dsl-62-3-71-168.zen.co.uk) has joined #freenet
[21:25] * pwk_ (~phweak@dsl-62-3-71-168.zen.co.uk) Quit (Read error: 104 (Connection reset by peer))
[21:27] * Ash-Fox (Hal-9000@aao142.neoplus.adsl.tpnet.pl) Quit (Nick collision from services.)
[21:27] * Ash-Fox (Hal-9000@aam192.neoplus.adsl.tpnet.pl) has joined #FreeNET
[21:40] * Tril (tril@bespin.org) has left #freenet
[21:44] * guido^pe (~unknown@dsl-213-023-243-254.arcor-ip.net) Quit (Remote closed the connection)
[22:09] * FallingBuzzard (~Administr@c-24-13-19-35.client.comcast.net) has joined #freenet
[22:11] * FallingBuzzard (~Administr@c-24-13-19-35.client.comcast.net) has left #freenet
[22:19] <KenMan> toad_ be still awake ?
[22:19] <toad_> KenMan: yes
[22:19] <toad_> temporarily
[22:20] <KenMan> howdy, how's the brain doing ?
[22:20] <toad_> hmm?
[22:20] <KenMan> mine is busy totally reworking your sim...
[22:20] <toad_> totally reworking? how so?
[22:20] <KenMan> i'm switching to using ints for CHKs, trimming down the code for the logic
[22:21] <toad_> I doubt that using ints instead of doubles will help much
[22:21] <toad_> in fact you should definitely use doubles, not ints
[22:21] <toad_> also I tried using doubles, and although there were some majorish problems exposed by related work that went in at the same time, the performance gain was pretty small
[22:21] <KenMan> who knows... but I'm certainly not competing with you. It could be useful to have two implementations to compare and contrast, for certain experiments, no ?
[22:22] * toad_ is working on d-ArkAngel's idea re pre-sorted buckets...
[22:22] <KenMan> cool, how does it help things ?
[22:23] <toad_> dramatically speeds up routing, especially on larger RTs, hopefully
[22:23] <toad_> anyway
[22:23] <toad_> whatever you do with the code, PLEASE do some sort of parallel test
[22:23] <toad_> to validate the results
[22:23] <toad_> if you change routing, that is
[22:23] <KenMan> pre-sorted at the time of (?) - sliding the buckets ?
[22:23] <toad_> ummm
[22:23] <KenMan> yes, i should keep a baseline comparator. I don't expect any extra work on your part...
[22:24] <toad_> you divide the keyspace up into fixed location sub-buckets
[22:24] <toad_> e.g.
[22:24] <toad_> 0x0000 to 0x0800, 0x0800 to 0x1000, etc
[22:24] <toad_> you get the min/max for each bucket for each node
[22:24] <toad_> and keep that up to date
[22:24] <toad_> and sort them
[22:24] <KenMan> ah, gotcha. I think that is more sound than the sliding that goes on...
[22:24] <toad_> hopefully you only need to do the actual calculation on the top few nodes, as a result
[22:24] <toad_> and you know which nodes to get
[22:25] <KenMan> neat
[22:25] <toad_> no, the sliding remains
[22:25] <toad_> this is an optimization on top of that
[22:25] <toad_> the sliding is critical, otherwise we can't have real specializations
[22:25] <KenMan> no need to explain, i'm already lost on the concept
[22:25] <KenMan> :(
[22:25] <KenMan> for epDNF, the widest bucket should center around the area where most keys exist, right ?
[22:26] <KenMan> you are counting DNF events, not DF events, yes ?
[22:26] <toad_> no
[22:26] <toad_> the narrowest bucket
[22:26] <toad_> specialization
[22:26] <toad_> more detail where most of the action is
[22:26] <KenMan> narrowest bucket for DFs would be a/the spec point
[22:27] <toad_> yes, and epDNF counts both
[22:27] <toad_> in order to produce a probability
[22:27] <KenMan> but for DNFs it would be the opposite. Both ? Two sets of buckets ?
[22:27] <toad_> no
[22:27] <KenMan> why can't we call it epDF ? I'm missing something obvious :(
[22:28] <toad_> you report an event on epDNF
[22:28] <toad_> it is either 1.0 or 0.0
[22:28] <toad_> either way the bucket gets compressed slightly
[22:28] <toad_> and the value gets reported to the running average for the bucket
[22:28] <KenMan> hit or a miss both compress the bucket ?
[22:29] <toad_> so if you have a big concentration of requests around 0xf0000, even if they all fail, it will have more detail i.e. smaller sectors there
[22:29] <KenMan> don't we want the bucket size to correlate to the ra for success/failure ?
[22:29] <toad_> however in practice if an area sucks it won't get requests
[22:29] <toad_> no, that's dealt with by the feedback from specialization
[22:30] <KenMan> hrrm, i know you guys are comfortable with that. Will have to ponder for awhile for my own comfort
[22:30] <KenMan> The more buckets we have, the more samples we need to make them 'accurate' right ?
[22:31] <toad_> somewhat yes
[22:31] <KenMan> Somehow the math doesn't work out, until you sim 1M requests over a single route (estimator)
[22:31] <toad_> eh?
[22:32] <KenMan> We want fast response, fast 'accuracy' of the estimator (not just a single bucket)
[22:32] <toad_> uh
[22:32] <toad_> what do you mean by fast?
[22:32] <KenMan> less than 1M samples, for example
[22:32] <toad_> ah
[22:33] <toad_> well it's not just the single estimator
[22:33] <toad_> it's the estimators right across the network that have to learn
[22:33] <KenMan> indeed
[22:33] <toad_> anyway my simulations don't come close to 1M requests per node
[22:33] <toad_> that's an absurd and unsimulatable scale
[22:34] <toad_> unless you have an RT of 5 and are prepared to accept 30% success as best case
[22:34] <KenMan> the required samples is rather large for 16 buckets - 20x16 = 320 for a single route. And they don't tend to be uniformly distributed...
[22:36] <KenMan> I mean, we still aren't sure at what point they are accurate "enough"... your graphs settle towards an asymptote after 1 million / 800 / 50RT = 1m/20,000 links = avg of 50 per route
[22:36] <toad_> i'm not sure about that
[22:36] <KenMan> so they exhibit learning and usefulness that soon !! wow.
[22:37] <toad_> i don't understand
[22:37] <toad_> http://amphibian.dyndns.org/simulations2-epass-0.025-800x25-smooth.png
[22:37] <toad_> 800 nodes, 50 RT
[22:37] <toad_> makes 40,000 connections
[22:37] <KenMan> I left out the HTL multiplier
[22:37] <KenMan> 40/000 over two (they cut both ways)
[22:38] <toad_> after about 10M requests, it falls a lot
[22:38] <toad_> and then it probably rises a bit and then falls again to an asymptote
[22:38] <toad_> i don't know what you mean about cutting both ways
[22:38] <KenMan> each connection represents 2 routes
[22:38] <toad_> ah ok
[22:38] <KenMan> or, you are counting them twice. yeah, u get it
[22:39] <toad_> so 10M divided by 20K is 500 requests per connection
[22:39] <KenMan> yeah, but your math is better in fact.
[22:39] <KenMan> because there ARE 40 000 routes, right ?
[22:39] <toad_> it'd be very nice if it learned everything needed in 500 requests...
[22:39] <toad_> but we certainly can't assume that
[22:40] <KenMan> every node IRL is at a different point along your curve. Using a fixed set of nodes with a common start time is misleading :(
[22:40] <toad_> no, it's a collective
[22:40] <KenMan> what is nice about your sim is the constraint of success from 75 - 85 %
[22:40] <toad_> one node's state is meaningless in isolation
[22:41] <KenMan> i know what you mean.
[22:41] <toad_> well, we'll see
[22:41] <toad_> the target is 95%
[22:41] <KenMan> users would appreciate 9.5 % :)
[22:41] * toad_ is tempted to kick KenMan for that plainly facetious remark ;)
[22:42] * toad_ resists the temptation :|
[22:42] <KenMan> sorry , can't make that leap of a relation !!
[22:42] <KenMan> yet !
[22:42] <toad_> we need to show that freenet can scale
[22:43] <KenMan> yes. I'm still lost in the simpler world of fixed connections. Or examining how conn flux affects samples per conn over the avg conn lifetime
[22:43] <toad_> the way i have defined this is such: for N nodes, we want O(log N) HTL for asymptotic performance above 95%, with O(log^2 N) connections at worst, and in at worst O(N*log^2 N) requests for the whole network
[22:43] <KenMan> What if, on average, they die after only 285 samples ? Then you can simulate into infinity and only get that level of accuracy :o
[22:44] <KenMan> let me digest your last statement
[22:44] <toad_> eh?
[22:44] <KenMan> The longer a route exists, the better its accuracy, right ?
[22:44] <toad_> estimator passing
[22:45] <KenMan> argh - i don't like that one either. Cuz i don't understand it enough.
[22:45] <toad_> without estimator passing, with non-fully-connected networks, everything messes up pretty badly
[22:45] <toad_> that's what the stats say
[22:45] <toad_> also we don't drop the conn until it's had at least 200 hits
[22:46] <toad_> this parameter can be varied
[22:46] <KenMan> estimator passing, huh ?
[22:46] <toad_> 100 sucks. 500 sucks.
[22:46] <KenMan> the option name, I meant.
[22:46] <KenMan> estimatorpassing vs noestimatorpassing
[22:46] <toad_> "estimatorpassing"
[22:47] <KenMan> I've not been doing it.
[22:47] <toad_> i could be wrong; if you want to try simulating with small RTs, huge HTLs, huge numbers of requests, no estimator passing, and very high newbiehits, that's fine
[22:48] <KenMan> I want to understand all the knobs - options are 'knobs' in my book
[22:48] <toad_> what happened to your psuccess?
[22:48] <toad_> last i heard it was 25% and climbing slowly?
[22:49] <KenMan> I'm still playing in the 1000 x (5-25) x (5-15)HTL x (100-1000)DS space
[22:49] <toad_> psuccess?
[22:49] <KenMan> few routes more hops = faster solid success. More routes fewer hops = slow rising success, sometimes it falls after time.
[22:49] <KenMan> I'm in the 6-25% range
[22:49] <toad_> you call 25% solid?
[22:50] <KenMan> well, it is something to compare, once I get running sets of configs...
[22:50] <KenMan> I was to run all of 5-25RT x 5-15HTL and plot the curves
[22:50] <toad_> i really ought to do something about inserts...
[22:50] <toad_> maybe i'll have enough hardware available not to need to run simulations actually on my box...
[22:51] * KenMan crosses his fingers for toad_
[22:51] <toad_> well i have had some additional offers...
[22:51] <KenMan> cool. My empty wallet can't buy you any nice opterons :(
[22:53] <toad_> well, i really need to go to bed now
[22:53] <toad_> bbl zzz
[22:53] <KenMan> the necessary #requests is a function of nodes*routes*htls*buckets*(1/DS) for a given config...
[22:53] <KenMan> night toad_
[22:54] <KenMan> necessary to feel you've reached a representative point of sim accuracy for a given config, anyway.
[23:11] * Mujin (~Mujin@adsl-67-36-189-84.dsl.chcgil.ameritech.net) has joined #freenet
[23:12] * Mujin (~Mujin@adsl-67-36-189-84.dsl.chcgil.ameritech.net) has left #freenet
These logs were automatically created by Jay Oliveri with his gimp hapi on irc.freenode.net.
