Timestamps are in GMT/BST.
[0:16] -lilo- [Global Notice] Hi all. Outages scheduled this morning, as well as on Sunday or Monday morning. Please consult the news page ( http://freenode.net/news.shtml ) for further information. Thanks, and thank you for using freenode!
[2:12] -lilo- [Global Notice] Hi all. Rehubbing beginning.
[2:15] * dystopia (~admin@gaia-wan0.ipv6.freeshell.bofx.net) Quit (zelazny.freenode.net irc.freenode.net)
[2:15] * sanity (~ian@81-178-106-165.dsl.pipex.com) Quit (zelazny.freenode.net irc.freenode.net)
[2:15] * verl (verl@h152n2fls33o877.telia.com) Quit (zelazny.freenode.net irc.freenode.net)
[2:15] * locksy (~locksy@mrtg.sisgroup.com.au) Quit (zelazny.freenode.net irc.freenode.net)
[2:15] * KenMan (~dogzilla@pcp403292pcs.mntcrm01.md.comcast.net) Quit (zelazny.freenode.net irc.freenode.net)
[2:15] * MrNaughty (MrNaughty@d199-126-25-30.abhsia.telus.net) Quit (zelazny.freenode.net irc.freenode.net)
[2:15] * yonkeltron (~yonkeltro@pcp04665066pcs.wilog501.pa.comcast.net) Quit (zelazny.freenode.net irc.freenode.net)
[2:16] * sanity (~ian@81-178-106-165.dsl.pipex.com) has joined #freenet
[2:16] * verl (verl@h152n2fls33o877.telia.com) has joined #freenet
[2:16] * locksy (~locksy@mrtg.sisgroup.com.au) has joined #freenet
[2:16] * KenMan (~dogzilla@pcp403292pcs.mntcrm01.md.comcast.net) has joined #freenet
[2:16] * MrNaughty (MrNaughty@d199-126-25-30.abhsia.telus.net) has joined #freenet
[2:16] * yonkeltron (~yonkeltro@pcp04665066pcs.wilog501.pa.comcast.net) has joined #freenet
[2:16] * dystopia (~admin@gaia-wan0.ipv6.freeshell.bofx.net) has joined #freenet
[2:17] * TheSeeker (Fridlekh@4.27.135.55) Quit (zelazny.freenode.net irc.freenode.net)
[2:17] * Overand (common@67.99.252.64.snet.net) Quit (zelazny.freenode.net irc.freenode.net)
[2:18] * TheSeeker (Fridlekh@4.27.135.55) has joined #freenet
[2:18] * Overand (common@67.99.252.64.snet.net) has joined #freenet
[2:22] * Abdel (marshal7@host-62-141-224-47.kalisz.mm.pl) has joined #freenet
[2:23] * Abdel (marshal7@host-62-141-224-47.kalisz.mm.pl) has left #freenet
[2:40] * Elly is now known as Elly|afk
[3:14] <TheSeeker> oh yeah, scheduled outages...
[3:19] * d-ArkAngel (~Rob@cpc2-midd2-5-0-cust16.midd.cable.ntl.com) has joined #freenet
[3:22] * TLF (~francisco@231.Red-81-40-116.pooles.rima-tde.net) has joined #freenet
[3:36] * d-ArkAngel (~Rob@cpc2-midd2-5-0-cust16.midd.cable.ntl.com) Quit (Read error: 110 (Connection timed out))
[3:38] * d-ArkAngel (~Rob@cpc2-midd2-5-0-cust16.midd.cable.ntl.com) has joined #freenet
[3:38] * d-ArkAngel (~Rob@cpc2-midd2-5-0-cust16.midd.cable.ntl.com) has left #freenet
[3:42] <TheSeeker> in making my freenet sim, should I assume all users are on broadband?
[4:16] * d-ArkAngel (~Rob@cpc2-midd2-5-0-cust16.midd.cable.ntl.com) has joined #freenet
[4:48] * moskau23 (~Miranda@dsl-213-023-254-135.arcor-ip.net) has joined #freenet
[4:48] * sdogi_ (~java@84-50-16-106-dsl.kjj.estpak.ee) Quit (Read error: 104 (Connection reset by peer))
[4:51] * sdogi (~java@84-50-20-163-dsl.kjj.estpak.ee) has joined #freenet
[4:56] * Hory (~Miranda@82.76.81.56) has joined #FreeNet
[5:39] * TLF (~francisco@231.Red-81-40-116.pooles.rima-tde.net) Quit ("Adi?s a todos y a todas, y que les vaya bien. || http://www.vitaliano.esp.cc")
[5:39] * TLF (~francisco@231.Red-81-40-116.pooles.rima-tde.net) has joined #freenet
[5:44] * Sugardude (~Sugadude@mordor.neutraldomain.org) has joined #freenet
[5:51] * Sugadude (~Sugadude@216.32.201.125) Quit (Remote closed the connection)
[6:00] * TLF (~francisco@231.Red-81-40-116.pooles.rima-tde.net) Quit ("Adi?s a todos y a todas, y que les vaya bien. || http://www.vitaliano.esp.cc")
[6:35] * guido^pe (~unknown@dsl-213-023-239-253.arcor-ip.net) has joined #freenet
[6:39] <toad_> TheSeeker: it's a different mapping so that each store directory has roughly the same number of files in it
[6:40] <toad_> TheSeeker: it is interesting in and of itself that you got consecutive same winner 3000+ times. your patch would wipe out this information.
[6:40] <toad_> TheSeeker: and yes, if you write a sim, assume broadband to start with
[6:40] <toad_> you can always complicate it later
[6:41] <toad_> bbiab
[6:45] <i2p_iip> <Michelle> *hugs toad*
[7:11] * guido^pe (~unknown@dsl-213-023-239-253.arcor-ip.net) Quit (Remote closed the connection)
[7:21] * guido^pe (~unknown@dsl-213-023-239-253.arcor-ip.net) has joined #freenet
[7:24] * Hory (~Miranda@82.76.81.56) Quit (Read error: 110 (Connection timed out))
[7:48] * sanity (~ian@81-178-106-165.dsl.pipex.com) Quit ()
[8:17] * Hory (~Miranda@82.76.81.56) has joined #FreeNet
[8:31] * sanity (~ian@81-178-106-165.dsl.pipex.com) has joined #freenet
[8:41] * asciiwhite (~scegs@ppp197-125.lns1.syd2.internode.on.net) has left #freenet
[9:54] * Elly|afk is now known as Elly
[10:01] * sanity (~ian@81-178-106-165.dsl.pipex.com) Quit ()
[10:12] * d-ArkAngel (~Rob@cpc2-midd2-5-0-cust16.midd.cable.ntl.com) Quit (Read error: 113 (No route to host))
[10:13] * moskau23 (~Miranda@dsl-213-023-254-135.arcor-ip.net) Quit (Read error: 104 (Connection reset by peer))
[10:13] * sanity (~ian@81-178-106-165.dsl.pipex.com) has joined #freenet
[10:16] * Elly is now known as Elly|afk
[10:17] * moskau23 (~Miranda@dsl-213-023-254-135.arcor-ip.net) has joined #freenet
[10:36] * linagee (~linagee@netblock-66-245-227-226.dslextreme.com) Quit (Read error: 104 (Connection reset by peer))
[10:36] * Elly|afk (~Elly@ool-182c3b26.dyn.optonline.net) Quit (Read error: 104 (Connection reset by peer))
[10:36] * linagee (~linagee@netblock-66-245-227-226.dslextreme.com) has joined #freenet
[11:06] <toad_> http://amphibian.dyndns.org/simulations3-100x99.png
[11:06] <toad_> hrrrrmmmm
[11:14] * sdogi_ (~java@84-50-21-16-dsl.kjj.estpak.ee) has joined #freenet
[11:26] * sdogi (~java@84-50-20-163-dsl.kjj.estpak.ee) Quit (Read error: 110 (Connection timed out))
[11:36] <toad_> i wonder what's wrong...
[11:36] <toad_> http://amphibian.dyndns.org/simulations3-100x99-avg25.png
[11:36] <toad_> (the first one was crazy-averaged)
[11:37] <toad_> the latter is averaged with a period of 25 reports i.e. 125000 requests
[11:38] <toad_> ah
[11:38] <toad_> formula bug
[11:38] * toad_ fixes...
[11:44] <toad_> now it's looking a lot healthier...
[11:53] <toad_> wobbling a lot though
[11:54] <toad_> but at least it's wobbling a full 10% higher than last time
[11:54] <i2p_iip> <Michelle> *hugs toad*
[11:54] <toad_> hi Michelle
[11:55] <toad_> or is it michael? or shall I call you gott? :)
[11:56] * sanity (~ian@81-178-106-165.dsl.pipex.com) Quit ()
[11:58] * Hory (~Miranda@82.76.81.56) Quit ("CyberLore.net - Recommendations on the best games, freeware and websites.")
[12:03] * moskau23 (~Miranda@dsl-213-023-254-135.arcor-ip.net) Quit (Read error: 104 (Connection reset by peer))
[12:11] * pittaman (~mistery_b@d51A410CA.kabel.telenet.be) has joined #freenet
[12:25] <toad_> hi
[12:34] * sanity (~ian@81-178-106-165.dsl.pipex.com) has joined #freenet
[12:37] <toad_> hi sanity
[12:37] <toad_> i just got the new simulator working, about to commit it.. not finished yet of course
[12:37] <sanity> hi toad
[12:37] <sanity> cool, what is new about it?
[12:37] <toad_> it's largely rewritten from scratch, it lives in freenet/node/simulator/newsim/
[12:38] <toad_> the intent is to compare it with the previous simulator, and the recorded output of earlier versions of it
[12:38] <toad_> to show that the crappy graphs of 400x50 are a fluke, hopefully
[12:38] <toad_> an earlier version of the current simulator got radically better results on 400x50, you see...
[12:38] <sanity> will it be faster / more efficient / more accurate?
[12:39] <toad_> hopefully more efficient and more accurate and better written
[12:39] <sanity> cool
[12:39] <toad_> not MUCH faster, although it may be easier to plug new optimisations into
[12:39] <toad_> the core goal is to deal with the uncertainty surrounding the 400x50 graphs
[12:40] <toad_> of course i haven't yet implemented enough to do that :)
[13:18] * TLF (~francisco@146.Red-81-40-113.pooles.rima-tde.net) has joined #freenet
[13:22] * Redb3ard (oylerjm@c-24-125-12-101.va.client2.attbi.com) has joined #freenet
[13:28] * Elly (~Elly@ool-182c3b26.dyn.optonline.net) has joined #freenet
[13:29] * Redb3ard (oylerjm@c-24-125-12-101.va.client2.attbi.com) has left #freenet
[13:29] <TheSeeker> toad: that patch would NOT wipe out that you got consicutive same winner 3000+ tiems, it would only show the LAST consecutive same winner instead of all of them between "4 times" and "N times"
[13:30] <toad_> okay, THAT's better... http://amphibian.dyndns.org/simulations3-100x99-supersmooth.png
[13:30] * TheSeeker wakes up
[13:30] <toad_> TheSeeker: huh?
[13:31] <toad_> TheSeeker: don't you reset the same counter you are showing?
[13:31] <toad_> which patch?
[13:33] <toad_> i may be thinking of the wrong patch
[13:33] <TheSeeker> as I said, I have no idea WHAT that error is put there for, all I know is that it puts a HUGE emount of redundant crap into the log file. The original code as is is on top, and the code that DOESN'T show every consecutive same winner, only the last one, is on the bottom. I dont' have toold for compiling or making diff files, so I had to send the patch to devl...
[13:34] <TheSeeker> I still have no idea what you're talking about in regards to "resetting the counter"
[13:34] <toad_> consecutiveSameWinner = 0;
[13:34] <toad_> resetting the counter !
[13:34] <TheSeeker> I didn't add that!
[13:34] <TheSeeker> it's doing that NOW
[13:35] <toad_> oh
[13:35] <toad_> okay, i see
[13:35] <TheSeeker> it resets the counter NOW whenever the same winner is no longer true, so it can start counting the next winner.
[13:35] <toad_> okay
[13:35] <TheSeeker> I just want to have it stop dumping the intermediate N-5 (since it starts at 4) reports about the 'error' to the log file :P
[13:36] <toad_> yes, your patch is okay
[13:36] <TheSeeker> thank you :P
[13:37] <toad_> http://amphibian.dyndns.org/simulations3-100x99-supersmooth.png
[13:37] <toad_> hmmmm
[13:38] <toad_> it would appear that it learns pretty fast!
[13:38] <TheSeeker> what is this an image of?
[13:38] <toad_> the new simulator, running 100x99 fully connected
[13:38] * Hory (~Miranda@82.76.81.56) has joined #FreeNet
[13:38] <toad_> hi Hory
[13:39] <TheSeeker> yes, but what does the graph represent?
[13:39] <toad_> success probability against time
[13:39] <Hory> hi
[13:40] <TheSeeker> hmm, 97% success probability for routing to known keys?
[13:40] <toad_> more or less known keys
[13:40] * MrNaughty (MrNaughty@d199-126-25-30.abhsia.telus.net) Quit ("\(^_^)/' No Soliciting!!! Unless you have legs way, way up and really, really big tits....")
[13:40] <toad_> the last N keys, where N is 2.5% of datastore size * node count
[13:42] <TheSeeker> hmm, I wonder what happens when over half of your requests are for keys that are known to not exist? Still only draw the graph for the known inserted (if not still avaliable) keys, but see how the added junk requests affect the chart?
[13:42] * sanity (~ian@81-178-106-165.dsl.pipex.com) has left #freenet
[13:44] <TheSeeker> If spiders like TFE were all that people used on Freenet, then the "only request keys that were inserted at some time" simulation would be grand. but, even with freesites only you have NIMs and activelinks... then you add Frost to the equasion... x.x
[13:44] * d-ArkAngel (~Rob@cpc2-midd2-5-0-cust16.midd.cable.ntl.com) has joined #freenet
[13:44] * d-ArkAngel (~Rob@cpc2-midd2-5-0-cust16.midd.cable.ntl.com) has left #freenet
[13:45] <TheSeeker> Frost goes over my head like freenet does, but I've gleaned enough off conversations that it pretty much runs entirely on a guessable key system...
[13:50] * Nightblade (~incognito@c-67-165-151-225.client.comcast.net) has joined #Freenet
[13:55] <i2p_iip> <Sonax> Toad: Good to see you back! I was begining to fear that you had left us all...
[13:55] <TheSeeker> otherwise, that's pretty good routing even for a small network I think... I wonder what the datastores of the nodes look like?
[14:01] <TheSeeker> My node has been up for 24 hours. My log file is a little over 30 MB. Without consecutive same winner it's less than 3 MB.
[14:05] <toad_> TheSeeker: spiders have to request nonexistant keys too to find new editions
[14:05] <toad_> these results are looking more than a little absurd...
[14:06] <toad_> I wonder if there are more bugs to be found in the newsim...
[14:07] <toad_> http://amphibian.dyndns.org/simulations3-100x25-supersmooth.png
[14:09] <TheSeeker> a not-fully connected network getting better results than a fully connected one?
[14:10] <toad_> that's possible..
[14:10] <toad_> i suppose
[14:11] <TheSeeker> you learn faster about your surrounding nodes when you have less of them to choose from for requests?
[14:11] <toad_> 100*100 = 10,000 ... * 0.025 = 10,000/40 = 1000/4 = 250
[14:11] <toad_> hmmm
[14:11] <toad_> so that's not it...
[14:13] <toad_> hmmm
[14:13] <toad_> i haven't implemented connections!
[14:13] <toad_> connect on storedata is not implemented...
[14:13] * TheSeeker watches that go over his head
[14:16] * Elly is now known as Elly|afk
[14:27] * TLF (~francisco@146.Red-81-40-113.pooles.rima-tde.net) Quit ("Adi?s a todos y a todas, y que les vaya bien. || http://www.vitaliano.esp.cc")
[14:28] <KenMan> toad - it may be useful to log the number of queries (and successes?) for each connection that gets dropped.
[14:29] <toad_> compare http://amphibian.dyndns.org/simulations3-100x25-supersmooth.png to http://amphibian.dyndns.org/simulations2-100x25-htl10.png
[14:29] <KenMan> Also, to log the same stats for connections that don't get dropped. Then you can observe how they compare.
[14:29] <toad_> something is seriously wrong with one or other simulation!
[14:29] <toad_> the former is the new code, which so far does not yet include connection churn
[14:30] <toad_> and it works amazingly well...
[14:32] <TheSeeker> so.. never drop a connection yourself, even if it's overloaded? just wait for it to get not-overloaded? (better to have nodes you know about that are slow to respond to requests than nodes you don't know about who are fast?)
[14:32] <KenMan> don't get too hung up on comparisons. You have a new code body. Compare it to itself, as you make changes and try things.
[14:33] <TheSeeker> how about .... "How is the current sim different from the actual network code, and how can we make the actual network code more like the sim" ;)
[14:34] * toad (toad@82-32-17-1.cable.ubr03.azte.blueyonder.co.uk) has joined #freenet
[14:34] <toad> grrr, my other irc client hung
[14:34] <toad> were you saying something kenman?
[14:35] * toad is simulating 200x25 with the new simulator
[14:35] <KenMan> not really. Just focus on what you have, and move forward. Pretend [old]sim never existed.
[14:35] <toad> and it's EXTREMELY promising
[14:36] <toad> well.. it's just that it's absurd
[14:37] <toad> http://amphibian.dyndns.org/simulations3-100x25-supersmooth.png
[14:37] <toad> observe this graph
[14:38] <toad> this is with 100x25, no connection churn (random connections), no pcaching, and no announcements, and HTL 10
[14:38] <toad> observe the Y-scale!
[14:38] <toad> the long term average has never been below 0.97
[14:38] <toad> but what is far more impressive is this:
[14:40] <toad> http://amphibian.dyndns.org/simulations3-200x25-supersmooth.png
[14:40] <toad> !!!!
[14:40] <KenMan> no pcaching. Hmm, I've been there. How did you derive 2.5% of global datastore ?
[14:40] <TheSeeker> are you sending back the 'data' through the network before declaring a key successful? (is bandwidth at all factored into your sim?)
[14:41] <toad> no, bandwidth is ignored for now
[14:41] <toad> KenMan: guess :)
[14:41] <toad> it's a fudge factor, i suppose
[14:41] <KenMan> same formula as before ?
[14:41] <toad> hmm?
[14:42] <KenMan> did you pluck 2.5% out of your backside ?
[14:43] <TheSeeker> hmm, so what you're saying, is that if we made routing of searches towards keys never get hampered due to flow control, all findable keys would probably be found... it's now just a matter of how do you deliver the data back to all those people requesting?
[14:43] <toad> KenMan: no, I derived it empirically :)
[14:43] <KenMan> okay. Just for education's sake, try using other values. 1%, and 5%... and, you need a count of the number of unique keys stored globally, to compare against.
[14:43] <toad> TheSeeker: distribution of load is about 3:1 from the most used node to the least used node
[14:45] <TheSeeker> Gnutella had a HUGE problem with connections being taken up primarily just with search and reply traffic with a broadcast model... I realize freenet is a more pointed search, but it has the huge overhead of a "found" key always resulting in an upload for that key... correct?
[14:45] <toad> hmmm
[14:45] <toad> TheSeeker: yes, over the whole node chain
[14:45] <toad> which is quite a lot of bandwidth
[14:46] <KenMan> toad - do you interleave inserts with requests (400:10000) ? so every 25 requests you do an insert ?
[14:46] <TheSeeker> which is why I was wanting to make a sim that thinks of bandwidth, and only having as many connections as I could have active connections to at some minimum average speed.
[14:46] <TheSeeker> err, active transfers between
[14:47] <toad> KenMan yes
[14:47] <toad> 100:1 though
[14:47] <toad> which i think is accurate to the real world
[14:48] <toad> brb
[14:49] <KenMan> i decided that pcaching really does what it is supposed to. It greatly boosted the number of unique keys that can be stored in the net, while not really affecting success. So redundancy goes down.
[14:49] * sanity (~ian@81-178-106-165.dsl.pipex.com) has joined #freenet
[14:50] <KenMan> even though success didn't change, i considered it to be a big win.
[14:50] <TheSeeker> the 200x25 is interesting.... I'm thinking of lowering my max connected nodes from 100 to something more along the lines of 30 ...
[14:50] <toad> http://amphibian.dyndns.org/simulations3-200x25-avg25.png
[14:51] <toad> sanity: essentially, the new simulations are giving absurdly good results, with just about everything turned off
[14:51] <toad> everything being announcements, pcaching, estimator passing, and connection churn
[14:51] <sanity> a bug?
[14:51] <toad> perhaps
[14:51] <toad> there is definitely some learning going on
[14:51] <sanity> are we seeing specialisation?
[14:51] <toad> good question...
[14:52] <toad> easiest way to find out is probably to dump all the datastores...
[14:52] * kevloral (~kevloral@CZ1-RAS-8-u-0179.du.onolab.com) has joined #freenet
[14:52] <toad> brb
[14:52] <TheSeeker> !?
[14:53] <TheSeeker> I've heard over and over again that specialization of routing has nothing to do with datastore contents ... :P (Was that the Ian viewpoint?)
[14:54] <TheSeeker> I've always thought that the two would be inexorably linked... how could you route well to an area of the keyspace... be recognized for routing well to an area of the keyspace, and thus have a lot of data from that area of the keyspace filter through your node ... without any of that data being cached?
[14:55] <sanity> it certainly wasnt my viewpoint
[14:56] <sanity> specialisation of connections would likely be directly proportional to specialisation of cached data
[14:56] <TheSeeker> I suppose if you have an uber-node that has virutally unlimited resources, then you can have massive numbers of open connections and be good at routing anything because you're connected to half the network... but most people don't have that luxury :P
[14:56] <sanity> bbiab
[14:57] <kevloral> g'evening all
[14:57] <toad> rehi
[14:58] <toad> REQUESTS: 1183647/1260000=0.9394023809523809 4906/5000=0.9812
[14:59] <toad> http://amphibian.dyndns.org/simulations3-200x25-avg25.png
[14:59] <TheSeeker> so I think that the specialization range of a node should be proportionate to the number of nodes it can connect to... which should be proportionate to upstream bandwidth. the question is if this would cause nodes to be too 'strung out' and if islands would appear of various specialization areas that can't see eachother because they're too far away... (I'm thinking that would require many more nodes than are running now though)
[14:59] <toad> that's more or less what i was looking for all along in the simulations, but it's far and away better than the last TWO DIFFERENT sims
[15:00] <toad> TheSeeker: # connections and # transfers are not closely related
[15:00] <TheSeeker> I've noticed that at any given time, about half of my conenctions are transferring.
[15:01] * sdogi (~java@84-50-16-20-dsl.kjj.estpak.ee) has joined #freenet
[15:01] <TheSeeker> currently, almost half of my open connections have outbound transfers (don't care about incoming, as I have enough bandwidth for that)
[15:01] <TheSeeker> 41/95
[15:02] <toad> there can be more than one transfer on a conn
[15:02] <TheSeeker> considering that I'm only giving 20 KB/s upsteam to Freenet, each transfer is only getting around 500 Bytes/s
[15:03] <TheSeeker> perhaps, but I'm looking for a minimum performance metric for any given trailer... and if I'm only uploading at 500 Bytes per second per trailer... that's a pretty low level of performance for the end user :P
[15:04] <toad> so you want a certain number of transfers
[15:04] <toad> not connections
[15:04] <KenMan> yes, but you should feel the love. Other nodes love you because you are providing -something- , even if it is mainly frustration.
[15:05] <toad> lol
[15:05] <TheSeeker> I also don't want to be backed off 90% of the time. I feel it messes up routing. :P
[15:05] <toad> :)
[15:05] <KenMan> heh
[15:05] <toad> it does, but i'm not sure what we can do about it
[15:06] <toad> i'm sure we can do something...
[15:06] <TheSeeker> when you connect to a node, does it tell you how many other nodes it's connected to?
[15:06] <toad> okay, every 100,000 requests, it will dump all the stores...
[15:07] <toad> TheSeeker: certainly not, none of its business
[15:07] <KenMan> woohoo ! what is it dumping ? 20 byte hex keys ? want a gnuplot script to plot them ?
[15:08] <toad> well i need it to get to at least 200,000 requests for them to be really interesting
[15:08] <toad> more like 600,000 would be even more interesting
[15:09] <KenMan> i set it up to dump each store after each J insertions, J=.5*storesize. But I didn't add a global clockstamp, so they log at different rates.
[15:09] <TheSeeker> well, it's rather difficult for a network to self-organize if it has to guess about everyhing... heh. you can choose not to trust your neighbor's claim, and drop nodes who's performance doesn't come close to their claims, but it makes things a LOT simpler if you just have some basic information exchange on connect. it doesn't hurt anonymity, and helps network organization.
[15:09] <KenMan> sometimes I wish my irc client chopped off long lines due to a limited buffer size :/
[15:09] <toad> lol
[15:10] <KenMan> he is touching on something i believe, though.
[15:10] <TheSeeker> my IRC client starts to beep at me if I type more than common buffers will clip at :P
[15:11] <toad> hmmm
[15:11] <toad> no obvious specialization
[15:11] <toad> but maybe i want to graph them?
[15:11] <KenMan> yup. What would be obvious ? you are eyeballing 100 values ?
[15:12] <KenMan> sort them and scroll through them. That helps bring something out, when the a's scroll by quickly, but the b's take a long time...
[15:13] * sdogi_ (~java@84-50-21-16-dsl.kjj.estpak.ee) Quit (Read error: 110 (Connection timed out))
[15:14] * toad nods, will do
[15:15] <KenMan> if you have a script to generate plots for each sampling, you can generate a series of images over time, and play them back as a movie. I use ImageMagick and it works pretty well.
[15:16] <toad> hmmm
[15:16] <toad> should i try to count the number of copies of each key?
[15:17] <KenMan> sure. it gets interesting, and we can compare count distributions. Did I ever show you one ? they were weird.
[15:18] <KenMan> It seemed that the insertion length was far and away the most common count.
[15:18] <toad> did?
[15:18] <KenMan> hold on, i go make one for you...
[15:20] <TheSeeker> hmmm, how any
[15:20] <TheSeeker> erk
[15:20] <TheSeeker> how many rate limiting violations have to occur before a node is dumped for flooding?
[15:21] <toad> i don't think we ever implemented sanctions against such nodes :)
[15:21] <toad> because the code to detect violations was buggy! ;)
[15:22] <TheSeeker> ah, I only saw one node repeatedly causing that error in my log, but it was happening a LOT :P
[15:22] <KenMan> http://mywebpages.comcast.net/jkcorson/FreeNet/redcnt.png
[15:22] <KenMan> this one is quite different from all the others I ever looked at, but it still has the insert-length-dominant property
[15:22] <TheSeeker> http://theseeker.bounceme.net/log-nowinner.txt
[15:23] <toad> hmmm
[15:23] <toad> looks like an average duplication of around 10.. hmmm
[15:23] <KenMan> this is an unusual shape for what I've seen previously, but it is easier to interpret
[15:24] <toad> well
[15:24] <toad> there have been 8 datastore dumps
[15:24] <toad> I pick a random key: da1d388f177cb62a6b6d73261ff2642fdb08d4d10f0302
[15:24] <toad> there are 57 mentions
[15:25] <toad> 60 of another one
[15:25] <toad> 35 of an early one
[15:25] <TheSeeker> seeing how well a key got spread?
[15:25] <toad> 33 of a late one.. hmmm
[15:26] <TheSeeker> when you do an insert, how many nodes tend to cache the data right away?
[15:26] <toad> 55 again...
[15:26] <toad> TheSeeker: all of them, in this simulation
[15:26] <KenMan> yeah, plot the global store too. With our low number of nodes, you can still make out specializations .
[15:26] <toad> so if we assume median is around 57, then that is about 6 nodes per cycle...
[15:27] <toad> well there seems to be SOME specialization...
[15:27] <KenMan> be nice to your brain. give it pictures.
[15:27] <toad> how?
[15:27] <KenMan> gnuplot, of course!
[15:28] <TheSeeker> toad: in regards to Jusa Saari's coment on the consecutive same winner thing, my reply would be: "I assume whoever was interested in the error was interested in how many times it happened, not just that it was > 3..."
[15:28] <KenMan> one script to bucketize the key counts, and one to make the images.
[15:29] * sanity (~ian@81-178-106-165.dsl.pipex.com) has left #freenet
[15:29] <KenMan> given a file with a single (bucket) count on each line: plot 'buckets.txt' with impulse
[15:31] <KenMan> this will generate a plot like the one i just shared
[15:31] <KenMan> you can plot 512 or 1024 values on a graph of the same dimension (width) and get something much like a barcode.
[15:32] <toad> it would probably take as long to bucketize my existing data as to rerun it with some internal bucketizing code
[15:32] <KenMan> yeah, that was a toss-up. It's more efficient to do it internally ;)
[15:32] <toad> not the way i do it :)
[15:32] <KenMan> http://mywebpages.comcast.net/jkcorson/FreeNet/N594DS.png
[15:33] <KenMan> i mean, it eats a whole lot less diskspace :)
[15:33] <KenMan> that's why i expected you would prefer to gobble the platters !
[15:33] <toad> well i want the raw data as well :)
[15:34] <TheSeeker> oops, I misread that patch suggestion *sends a different reply than the one I stated here*
[15:34] <KenMan> there is a sample datastore plot with impulses , it is useful for most brains .
[15:37] <KenMan> i haven't done it yet for incoming requests... but that will be useful too!
[15:40] <toad> hmmm
[15:40] <toad> i'm not sure you can prove it so easily with only 100 items per store...
[15:43] <toad> there isn't massive specialization
[15:43] <toad> there appears to be SOME specialization
[15:43] <toad> usually with 3 or so peaks...
[15:43] <toad> but they're only twice as high as the surrounding area
[15:44] <toad> 0: 5 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 1: 3 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 2: 3 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 3: 10 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 4: 7 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 5: 0 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 6: 3 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 7: 2 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 8: 6 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 9: 6 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 10: 3 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 11: 3 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 12: 6 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 13: 8 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 14: 16 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> 15: 19 on freenet.node.simulator.newsim.Node@186c6b2 (5)
[15:44] <toad> that's an example
[15:44] <toad> that just happens to be the last node
[15:44] <toad> i wonder if i just got kicked
[15:45] <TheSeeker> nope
[15:45] <toad> hmmm
[15:45] * MrNaughty (MrNaughty@d199-126-25-30.abhsia.telus.net) has joined #freenet
[15:45] <toad> well SOME of them have clear specialization
[15:46] <toad> there's one where buckets 4 and 5 have 37 and 26
[15:46] <toad> the rest have no more than 6 each
[15:46] <TheSeeker> I lowered my number of connections to 30, with 30 KB/s oubound limit. I've sent 44 MiB in the first 30 minutes of running.
[15:48] <KenMan> yes, larger datastores will improve the fidelity of the picture
[15:50] <KenMan> i can do datastores of 10k keys with the C++ sim, because I am using unsigned ints. :)
[15:51] <KenMan> oh, and the speed overhead of using full CHKs is negligible with java. Memory use... well, that's another matter.
[15:53] <toad> i dunno
[15:53] <toad> do you think it's a bug?
[15:53] <KenMan> what, that you saw some spec ?
[15:54] <toad> no, that it gets to 95% on 200x25@10 in under a million requests
[15:54] <KenMan> no, that is quite reasonable. Try using something other than 2.5% to characterize the behaviour.
[15:54] <toad> you think so? it sounds reasonable, but it's much much better than prior sims
[15:55] <toad> and it's pretty much not using any of our recent work either
[15:55] <toad> no estimator passing, no connection churn even!!
[15:55] <KenMan> what estimator(s) are you using ?
[15:55] <KenMan> i don't think it is a bug. I think when you begin wiggling the parameters you will better understand it.
[15:56] <KenMan> you are only drawing from a pool of 250 search keys, as you stated earlier.
[15:56] * toad tries 5%
[15:56] <KenMan> I got a big success drop simply by drawing from the full set of existing keys.
[15:56] <KenMan> But I understood it. Nothing really changed, except for the key selection.
[15:56] <toad> well of course you can't expect to find ALL of them
[15:57] <toad> anyway, i started it with 5%
[15:57] <KenMan> no, but i want to consider all aspects that affect success, redundancy factor, whatever.
[15:57] <toad> as well as bugginess :)
[15:58] <KenMan> right now your model is so simple, you will find any structural bugs with a day.
[15:58] <toad> implementation bugs are perhaps more likely
[15:58] <toad> i suppose i ought to single step it a bit to see if i can find any clear bugs
[15:59] <toad> okay, with 5%, it goes down a LOT further...
[15:59] <KenMan> yes, and analyze as many aspects as you can. Make sure they make sense.
[15:59] <toad> down to 60% or so, maybe less
[15:59] <KenMan> what estimator(s) are you using ?
[15:59] <toad> it seems to be turning...
[15:59] <toad> KenMan: keyspace estimators with the RL formula
[16:00] <KenMan> remember, you don't know the real % of retained keys... but you just got a sense of its impact. What is RL formula ?
[16:00] <toad> estimate = pSuccess * tSuccess + tFailure / pSuccess
[16:00] <toad> roughly
[16:00] <toad> okay, it looks a lot like it's turning
[16:00] <KenMan> would you consider using a single, pure pSuccess SBKE ?
[16:01] <toad> there is an option for that
[16:01] <KenMan> awesome ;)
[16:02] <KenMan> i'm not betting any chickens on this one, but i bet you top out somewhere near 80% ...
[16:02] <KenMan> anyway, quite a distance from 100% !!
[16:02] <TheSeeker> I found a problem testing my "lower number of connections" idea... since most other people probably still have a large number of conenctions, they are more likely to be overloaded, leaving me RNFing a lot :P
[16:03] <toad> hmm, this is odd
[16:03] <toad> there may be some wierd reporting bug...
[16:04] <toad> ah i see
[16:04] <toad> it doesn't introduce a systematic bias
[16:09] <TheSeeker> hmm, is there any way to tell how many succesful requests I have that were not a result of having the data in my store already? or would I need to wipe my datastore for that? (all initial request successes must be routing sucesses due to having no keys in datastore)
[16:09] <toad> TheSeeker: yes
[16:09] <toad> look around on the diagnostics page
[16:09] <toad> occurrences on routingSuccessRatio is one way to see it
[16:13] <TheSeeker> thanks
[16:14] <toad> KenMan: is it possible that connection churn makes things a lot worse?
[16:14] <toad> bbiab
[16:15] <toad> the thing is, I'm pretty sure for theoretical reasons that connection churn is essential for logarithmic scalability in htl etc
[16:15] <toad> if we can get that without major|any connection churn, that would open up a whole range of applications otherwise inaccessible to us
[16:15] <toad> but i have no idea what the theory behind it would be
[16:16] <toad> except that.. hrrm
[16:16] <toad> well we know that LRU with greedy routing works.. maybe changes in routing simulate the mesh connections changing in the right way.. or something..
[16:17] <TheSeeker> hmm, I only have 200 succesful request keys listed in the nodestatus... but this seems to say otherwise: http://theseeker.bounceme.net:8888/servlet/nodestatus/diagnostics/graphs?image-type=image/bmp&period=minute&type=1&var=routingSuccessRatio ??
[16:18] <KenMan> you need a knob to control the rate of churn, then you can study it. Too high is no good.
[16:18] <toad> hmmm
[16:18] <toad> it's an open question whether this will in fact turn...
[16:19] <KenMan> so try at 8% and confirm :o
[16:19] <toad> http://amphibian.dyndns.org/simulations3-200x25-0.05-avg25.png
[16:20] <TheSeeker> if this doesn't come down in teh next 12 hours I'll restart with 50 connections and see if that helps... http://theseeker.bounceme.net:8888/servlet/nodestatus/diagnostics/graphs?image-type=image/bmp&period=hour&type=6&var=routingFailRNFRatio x.x
[16:21] <toad> hmmm
[16:21] <toad> better leave it for a few hours
[16:21] <toad> bbiab
[16:21] <KenMan> you see , one of the reasons i like working at 50% (or, something other than 99+) is that I can quantify the resulting change when I alter something.
[16:22] <KenMan> pcaching, good. DS-self trimming, bad. Less routes, very good. More store, very bad.
[16:22] <KenMan> things like that . I just made those up.
[16:23] <TheSeeker> ok, was about to say... more store bad??
[16:23] <KenMan> i suspect it very much can be. If one node has 224GB of store, and all the rest have 3GB, it tends to influence things in a major way.
[16:26] * toad tries with 5% and htl 14
[16:26] <toad> bbiab
[16:27] <KenMan> keep an eye on average htl for successes. that is a good one to plot. Changes there are highly relevant.
[16:35] <toad> hmmm
[16:35] <toad> okay
[16:35] <toad> i think i'll call that a week
[16:35] <toad> seeya on monday
[16:35] * toad (toad@82-32-17-1.cable.ubr03.azte.blueyonder.co.uk) has left #freenet
[16:36] * toad_ (toad@toad-with-underline.active.supporter.pdpc) Quit (Read error: 104 (Connection reset by peer))
[16:36] <TheSeeker> hrm
[16:37] <TheSeeker> I think I want to test an idea for self-ogranization before trying to simulate a freenet node (way over my head) ...
[16:37] * Elly|afk (~Elly@ool-182c3b26.dyn.optonline.net) Quit (Read error: 104 (Connection reset by peer))
[16:38] * KenMan (~dogzilla@pcp403292pcs.mntcrm01.md.comcast.net) Quit ("Leaving")
[16:41] * MrNaughty (MrNaughty@d199-126-25-30.abhsia.telus.net) Quit (Read error: 104 (Connection reset by peer))
[16:41] * MrNaught (MrNaughty@d199-126-25-30.abhsia.telus.net) has joined #freenet
[16:41] * MrNaughty (MrNaughty@d199-126-25-30.abhsia.telus.net) has joined #freenet
[16:44] * kevloral is now known as kevlAway
[16:49] * MrNaught (MrNaughty@d199-126-25-30.abhsia.telus.net) Quit ("\(^_^)/' No Soliciting!!! Unless you have legs way, way up and really, really big tits....")
[16:49] * MrNaughty (MrNaughty@d199-126-25-30.abhsia.telus.net) Quit ("\(^_^)/' No Soliciting!!! Unless you have legs way, way up and really, really big tits....")
[16:49] * MrNaughty (MrNaughty@d199-126-25-30.abhsia.telus.net) has joined #freenet
[17:25] * Hirvox (~hirvox@cs181124095.pp.htv.fi) has joined #freenet
[17:25] * Hirvox changes topic to 'Stable: Upgrade to 5100 | Unstable: 60260 | Channel logs: http://newton.matcmp.ncc.edu/~lockej/freenet/chanlog/ | #freenet-politics and #freenet-chat are available for offtopic discussions'
[17:26] * Elly (~Elly@ool-182c3b26.dyn.optonline.net) has joined #freenet
[17:41] * guido^pe (~unknown@dsl-213-023-239-253.arcor-ip.net) Quit (Read error: 110 (Connection timed out))
[17:41] * guido^pe (~unknown@dsl-213-023-240-181.arcor-ip.net) has joined #freenet
[17:44] * pittaman (~mistery_b@d51A410CA.kabel.telenet.be) Quit (Client Quit)
[18:23] * Elly is now known as Elly|afk
[18:40] * kevlAway is now known as kevloral
[18:47] * guido^pe (~unknown@dsl-213-023-240-181.arcor-ip.net) Quit (Remote closed the connection)
[19:04] * Elly|afk (~Elly@ool-182c3b26.dyn.optonline.net) Quit (Read error: 110 (Connection timed out))
[19:26] * Hory (~Miranda@82.76.81.56) Quit ("CyberLore.net - Recommendations on the best games, freeware and websites.")
[19:44] * pupok (~r00t@81-178-106-165.dsl.pipex.com) Quit (Read error: 110 (Connection timed out))
[19:58] * kevloral (~kevloral@CZ1-RAS-8-u-0179.du.onolab.com) Quit ("Peace and Protection 4.22")
[20:00] * moskau23 (~Miranda@dsl-082-082-236-248.arcor-ip.net) has joined #freenet
[20:08] * sanity (~ian@81-178-106-165.dsl.pipex.com) has joined #freenet
[20:17] * pupok (~r00t@81-178-106-165.dsl.pipex.com) has joined #freenet
[20:41] * Elly (~Elly@ool-182c3b26.dyn.optonline.net) has joined #freenet
[20:43] * moskau23 (~Miranda@dsl-082-082-236-248.arcor-ip.net) Quit ("Miranda IM! Smaller, Faster, Easier. http://miranda-im.org")
[21:20] * TheSeeker resets his DS and RT then starts his node and frost
[21:22] <TheSeeker> actually, I think I'm gonna stay off freenet for a while to let other nodes forget about me... (I wonder if a day or two will be enough?)
[21:28] <Elly> whoops
[21:28] <Elly> I just tried to open a 600MB ISO with notepad.
[21:29] <Elly> $ram_free--;
[21:30] <TheSeeker> heh
[21:30] <Elly> stupid context menus
[21:30] <TheSeeker> wordpad will try and do a better job of it, but for files that big I usually have to use a hex editor.
[21:31] <Elly> Well, I don't actually have 600MB of free memory, which resulted in my disk swapping a lot.
[21:33] <TheSeeker> even if you do, notepad and wordpad seem to allocate a LOT more memory than the size of the file (for formatting I think)
[21:34] <Elly> about double it.
[21:35] <TheSeeker> hmm, for a freenet sim, a 46 character long hex string representation of a key will suffice?
[21:38] * Elly is now known as Elly|afk
[21:46] * Sugardude (~Sugadude@mordor.neutraldomain.org) Quit (Remote closed the connection)
[21:48] * Sugadude (foobar@outpost.zedz.net) has joined #freenet
[21:54] * Nightblade (~incognito@c-67-165-151-225.client.comcast.net) Quit ("Farewell")
[22:05] * pupok (~r00t@81-178-106-165.dsl.pipex.com) Quit (Read error: 110 (Connection timed out))
[22:15] * sanity (~ian@81-178-106-165.dsl.pipex.com) Quit ()
[22:22] * bluephile (~bluephile@69-160-215-63.clvdoh.adelphia.net) has joined #freenet
[22:24] * Elly|afk is now known as Elly
[22:34] * bluephile (~bluephile@69-160-215-63.clvdoh.adelphia.net) Quit (Client Quit)
[22:59] * Elly is now known as Elly|afk
[23:16] * Sugadude (foobar@outpost.zedz.net) Quit (Remote closed the connection)
[23:17] * Sugadude (ygf6lgc11q@swamp.fox-den.com) has joined #freenet
These logs were automatically created by Jay Oliveri with his gimp hapi on irc.freenode.net.
