Timestamps are in GMT/BST.
[1:21] * Zorix (Brandon@fl-198-70-222-58.dyn.sprint-hsd.net) has joined #freenet
[2:20] * Zorix (Brandon@fl-198-70-222-58.dyn.sprint-hsd.net) Quit (Read error: 110 (Connection timed out))
[2:43] * mmllaabb (~r00t@ool-18be3a49.dyn.optonline.net) has joined #freenet
[2:43] <mmllaabb> ho
[2:43] <mmllaabb> *hi
[2:43] <mmllaabb> what is freenet like now
[2:43] <mmllaabb> i havent been here in like a year
[3:27] * mburns (~maburns@c-24-22-50-182.client.comcast.net) has joined #freenet
[3:27] <mburns> freenet is MUCH more usable on broadband than it is on dialup, i have discovered
[3:41] * Zorix (Brandon@fl-198-70-222-58.dyn.sprint-hsd.net) has joined #freenet
[3:55] <mburns> is there anything big on the horizon for freenet?
[4:07] * mburns (~maburns@c-24-22-50-182.client.comcast.net) Quit ()
[4:52] * guido^pe (~unknown@dsl-082-082-158-143.arcor-ip.net) has joined #freenet
[5:04] * d-ArkAngel (~Rob@213-131-104-86.onyx.net) has joined #freenet
[5:43] * spaetz (~spaetz@80-218-147-20.dclient.hispeed.ch) has joined #freenet
[5:57] * nextime (~nextime@danex.i-m-c.it) Quit ("nextime has no reason")
[6:06] * unknown_ (~unknown@dsl-213-023-246-188.arcor-ip.net) has joined #freenet
[6:06] * guido^pe (~unknown@dsl-082-082-158-143.arcor-ip.net) Quit (Nick collision from services.)
[6:06] * unknown_ is now known as guido^pe
[6:11] * Hory (~Miranda@82.76.81.56) has joined #FreeNet
[6:12] * nextime (~nextime@ns0.nexlab.net) has joined #freenet
[7:39] * Ash-Fox (Hal-9000@abl10.neoplus.adsl.tpnet.pl) has joined #FreeNET
[8:08] * robilad (~topic@mpiat2320.ag2.mpi-sb.mpg.de) has joined #freenet
[8:29] * guido^pe (~unknown@dsl-213-023-246-188.arcor-ip.net) Quit (Read error: 104 (Connection reset by peer))
[8:56] * greycat (~wooledg@192.35.79.70) has joined #freenet
[9:15] * spaetz_ (~spaetz@80-218-147-20.dclient.hispeed.ch) has joined #freenet
[9:15] * spaetz (~spaetz@80-218-147-20.dclient.hispeed.ch) Quit (Read error: 104 (Connection reset by peer))
[9:34] * Hory (~Miranda@82.76.81.56) Quit ("Visit http://CyberLore.net for recommendations on the best games, websites and freeware. Don't settle for anything else!")
[9:53] * FallingBuzzard (~srademach@207.152.112.129) has joined #freenet
[10:01] * lostlogic (~lostlogic@node-4024215a.mdw.onnet.us.uu.net) Quit ("Going to the moon")
[10:10] * Zorix (Brandon@fl-198-70-222-58.dyn.sprint-hsd.net) Quit (Read error: 110 (Connection timed out))
[10:50] * yonkel-cluster (~yonkeltro@pcp04665066pcs.wilog501.pa.comcast.net) Quit (Client Quit)
[10:51] <KenMan> http://mywebpages.comcast.net/jkcorson/FreeNet/1000x20x8x50.png , 1000x20x8x100.png
[10:58] * lolo-laptop (~lostlogic@68.251.84.226) has joined #freenet
[11:27] <toad_> mmllaabb: still here?
[11:29] <d-ArkAngel> hey toad_
[11:29] <toad_> hi d-ArkAngel
[11:29] <d-ArkAngel> did kers ever show up?
[11:29] <toad_> nope
[11:29] <d-ArkAngel> that's a little worrying :-)
[11:30] <d-ArkAngel> ahh well. he probably got distracted by something else. I suppose that's life
[11:30] <toad_> since he hasn't been here either and doesn't seem to have emailed me, yeah..
[11:30] <d-ArkAngel> how longs he over here for? a week or something?
[11:30] <toad_> yeah
[11:30] <toad_> 1-6 nov
[11:30] <d-ArkAngel> how was he online over here? laptop? I suppose it could have been broken.
[11:31] <d-ArkAngel> or maybe his battry is flat :-)
[11:34] <d-ArkAngel> lol @ http://www.theregister.co.uk/2004/11/05/us_bomber_school_attack/
[11:34] * Zorix (Brandon@fl-198-70-222-58.dyn.sprint-hsd.net) has joined #freenet
[11:43] <mikeDOTd> nice
[11:43] <KenMan> prolly had access at his hotel ??
[11:44] <d-ArkAngel> In england? :-)
[11:45] <KenMan> yeah, why not ?
[11:46] <KenMan> oh, that's right, the english are extremely backwards, technically. I forgot about this.
[11:46] * d-ArkAngel has yet to stay in a UK hotel that provides anything better than a rip off phone line
[11:46] <KenMan> BestWestern chain of hotels, many of them should have some form of access.
[11:47] * d-ArkAngel doesn't recognise that name...
[11:47] <KenMan> it is an american chain, that has affiliated itself with many hotels internationally
[11:47] <KenMan> i travelled across sweden in 2000, and every place i stayed (BestWestern related) had access. Like 10 different places all over the map.
[11:48] <KenMan> And if that wasn't enough, I visited the local library, to avoid a usage fee.
[11:48] <KenMan> Can't england afford to put computers into some of their libraries ?
[11:49] <KenMan> for free public access, i mean
[11:49] <d-ArkAngel> yeah, I must say that there are lib with computers...
[11:49] <d-ArkAngel> not sure if you need to be a member somehow, or if you can just use one.
[11:49] <d-ArkAngel> tho I have noticed wifi hot spots in a few places recently, tho you have to pay to the access (quite expensive as I recall)
[11:50] <KenMan> I wasn't caught for being a non-member until after 2hrs use, and then they made me sign a schedule, so I got another 1/2 hour.
[11:50] <d-ArkAngel> :-)
[11:50] <KenMan> I mean, the concept is 'public access'
[11:50] <KenMan> I had to help an old man (who only spoke swedish) to retrieve the results of his son's wrestling match. That was my 'fee'
[11:51] <d-ArkAngel> for a given value of "public" and over here a hidden deffinition of "access"
[11:52] <KenMan> a clever person could easily sweet-talk any native with a less-than-pressing schedule to share their work computer , in almost any 1st world country
[11:53] <KenMan> computers are everywhere.
[11:54] <d-ArkAngel> thnking about it and the technical ability of most companies, it'd probably be easy enough to take a laptop and a wifi card, and a directional areial, and just tap someones connection.
[11:55] <KenMan> anyway, kers was here a few hours before toad went looking for him. And he wasn't trying to reach toad.
[11:59] <toad_> I'm sure many english hotels have inet access
[11:59] <toad_> the kind that cost $300+/night
[11:59] <d-ArkAngel> I've done a quick search and some seem to yes.
[11:59] <d-ArkAngel> tho I've stayed in some relativly posh ones, and never seen any (not wireless anyways)
[12:00] <d-ArkAngel> I'll do plenty of war walking with my 6000w and see how accurate my negative opinion of technological development arround here is :-)
[12:01] * toad_ wonders exactly what the legality of wardriving is
[12:01] <d-ArkAngel> if you don't steal anything, or leech their internet connection, then it's pretty legal.
[12:02] <toad_> in other words, you can check for the existance of a connection, but you can't actually connect to it
[12:02] <d-ArkAngel> yeah
[12:02] <KenMan> just don't get caught walking around GCHQ facilities with funny looking antennaes strapped across your body. Not a good idea.
[12:02] <toad_> since there's no way to easily tell whether you're supposed to have access to that connection or whether it's just somebody being careless
[12:02] <d-ArkAngel> so you could easily do a useage servay
[12:02] <toad_> KenMan: that's in chelsea...
[12:02] <KenMan> good enough.
[12:03] <KenMan> no sensitive secret installations around FishPonds, eh ?
[12:03] <d-ArkAngel> tape the zaurus to your back ala bush's audio prompter and they'll never know :-)
[12:04] <KenMan> so what is the current life-signs reading on FreeNet ?
[12:05] <toad_> actually, there were where i used to ;)
[12:05] <KenMan> how we gonna face your newly discovered prob ? the newbie/experienced turnover deal ?
[12:06] <KenMan> what if we only used references (storeData refs) obtained through our top 1,2,or 3 peers ?
[12:06] <d-ArkAngel> all experiences seems a valid solution. at least for the short term.
[12:07] <KenMan> does the sim do 5% -> newbie routing ? I think it does...
[12:07] <KenMan> But, without a similar effort to 'balance' or spread across those non-newb's , it will still be the same situation, just with lower turnover.
[12:08] <toad_> eh?
[12:08] <toad_> I don't see what the problem is
[12:08] <toad_> all we have to do is not drop anything until we have reasonable data on all nodes in the RT
[12:08] <KenMan> you can change the newbieness threshold, but the most frequently removed peer will typically be the last one to have crossed that line
[12:09] <KenMan> no ?
[12:09] <toad_> should we link to http://www.p2psicuro.it/freenet/index.htm ?
[12:09] <toad_> KenMan: why?
[12:09] <toad_> KenMan: no, we don't drop ANYTHING until we have data on ALL nodes
[12:09] <KenMan> the node at the front of LRU will be most used (most exp) and the one at the end will be the least used/experienced
[12:09] <KenMan> regardless of your threshold setting
[12:10] <toad_> yes, so?
[12:10] <toad_> what is your point?
[12:10] <KenMan> just that you better pick exactly the right value for threshold, and don't expect that to solve your problem
[12:11] <toad_> I still don't understand
[12:11] <toad_> what is the alleged problem?
[12:11] <KenMan> that the most likely node to get the boot will be the one with the least experience
[12:11] <KenMan> at least, in our pure sim environment. Real life is even more confusing
[12:12] <toad_> why?
[12:12] <d-ArkAngel> the one with the least chance of being selected.. not nesiarilly the one with the least experience...
[12:12] <KenMan> oh wait, no real life will just bow down to mRI effects.
[12:12] <toad_> the one booted is the Least Recently Used one
[12:12] <toad_> heh
[12:12] <toad_> as long as the threshold is good, that should be the least good one
[12:13] <d-ArkAngel> so the last one to cross the line is in fact automaticaly the most recently used...
[12:13] <KenMan> but that changes in very few steps
[12:14] <KenMan> typically he gets moved to the end of the line ASAP
[12:14] <toad_> huh?
[12:14] <toad_> no, it's the other way around
[12:14] <toad_> when we use a node, we move it to the front
[12:14] <toad_> so they gradually slide to the back
[12:14] <toad_> and jump instantly to the front
[12:14] <toad_> so it will take at least #RTnodes steps to get to the back
[12:15] <KenMan> yes, and i suggest that is almost the exact formula for how fast it will happen
[12:16] <KenMan> presumably, the next connection attempt will happen faster than the node can get pushed to the bottom, so we can ignore the whole issue.
[12:17] <d-ArkAngel> except that one node has to be at the end ;-) not matter how many accesses you have
[12:17] <toad_> KenMan: no, it won't
[12:18] <toad_> because it will occasionally get promoted
[12:18] <toad_> it must do, because we don't let it get dropped until we've used it N times
[12:18] <KenMan> are we absolutely certain that experienced peers with more experience aren't favored over experienced peers with less experience ?
[12:19] <KenMan> i mean, in a significant manner ?
[12:19] <toad_> eh?
[12:19] <d-ArkAngel> it depends, it's possible, but its more down to how the estimators of a given node match up with the incomming request trafic.
[12:20] <KenMan> i need to 'update' to the "all experienced" model and take some readings
[12:21] <KenMan> my concern is that the nodes that get dropped will tend to be those with the fewest samples, but there's only one way to find that out...
[12:22] <toad_> after a certain point, we have a pretty good idea of whether a node is any good
[12:22] <toad_> but it's possible that point is 200 samples, in which case it'll take ages
[12:22] <toad_> even with small keys
[12:23] <KenMan> this approach will limit new connections to a much lower rate, as toad already pointed out
[12:23] <toad_> that's not automatically a bad thing
[12:24] <KenMan> i think that each sample contributes to the knowledge linearly... the spec we are trying to guess is constantly shifting...
[12:25] <KenMan> we have no idea what the popularity/redundancy of each of those 200 samples , it is a random crapshot. I dunno, maybe 200 is enough.
[12:25] <toad_> <jepel_tailweaver> when will people stop saying MI6 and start saying SIS?
[12:25] <toad_> when the press stops calling them MI6...
[12:25] <toad_> KenMan: what do you mean?
[12:26] <toad_> i think that, for example, we'll probably get very few samples in buckets that appear to be bad for that node
[12:26] <toad_> although smoothing helps on this
[12:26] <toad_> you could for example require at least N hits in each bucket of each node
[12:26] <toad_> that would make it take even longer...
[12:27] <KenMan> but the buckets are supposed to be self balancing, no ?
[12:27] <toad_> <d-ArkAngel> I always worry about asking "Do I know you?", it either means that I've forgotten someone that I should remember, or leaves me open to someone playing silly buggers :-)
[12:27] <toad_> yeah, sometimes...
[12:27] <toad_> KenMan: what do you mean by self balancing?
[12:27] <KenMan> they are supposed to keep the same number of samples in each bucket
[12:27] <toad_> in theory, if one bucket gets lots of hits, it will shrink, and hopefully the hits will go into adjacent buckets
[12:27] <KenMan> roughly
[12:27] <toad_> right
[12:28] <toad_> but that takes some considerable time
[12:28] <KenMan> but if you have 8 buckets, and only one is magnetic (attractive), this might not happen, huh ?
[12:28] <KenMan> i mean, you don't resize ALL buckets on each event, right ? just center left and right ones
[12:29] * toad_ has a look at a typical nice node...
[12:29] <toad_> you resize the one that it hit
[12:29] <KenMan> which affects the two neighbors
[12:29] <toad_> this results in the two adjacent ones getting slightly larger
[12:29] <toad_> and yes, they are magnetic, because they're used for routign
[12:29] <toad_> if you keep getting hits at exactly one key, obviously it'll never get any hits in the adjacent buckets
[12:29] <toad_> okay, one good node
[12:30] <KenMan> i know we discussed this once before, but what if we scaled all buckets for each event ?
[12:30] <toad_> my pDNF bucket reports vary from 318 to 445
[12:30] <KenMan> that's not a bad distribution
[12:30] <toad_> of course this is IRL and therefore swamped with MRI effects.. but with queueing, I'm sure there's some routing going on
[12:30] <KenMan> not at all
[12:30] <KenMan> not a bad dist at all :)
[12:31] <d-ArkAngel> how much damage would be done to estimators if a node were to start generating requests for totally random keys?
[12:32] <toad_> <jepel_tailweaver> After leaving it on for about 20 hours, Freenet is already fast enough to be usable - cool...
[12:32] <toad_> d-ArkAngel: to random nodes?
[12:32] <d-ArkAngel> ones that could never be retrieved I mean.
[12:32] <d-ArkAngel> I mean, just making up requests and passing them on to all their peers
[12:32] <KenMan> good question... but how could we evaluate this ?
[12:33] <toad_> <michaelkuijn> Ha! 1670 songs, 4 days, 9 hours and 0 minutes, 8.8 GiB - on a Frost board!? I bet you can't get them all...
[12:33] <KenMan> oh, as in, how would the bucket boundaries be affected ?
[12:33] <d-ArkAngel> well I suppose we'd need a network running that becamse stable, and then turn one node to the dark side.
[12:33] <toad_> well, because of the pLegitDNF thingy, it shouldn't break things too badly...
[12:33] <d-ArkAngel> how's that work?
[12:34] <d-ArkAngel> that's the probability that a request from a given node is for a key that is really retrivable?
[12:34] <toad_> <cbreak> frost is a quite demanding program, it causes freenet to integrate fast. <cbreak> well... but maybe it just causes an overload :) - that is a subject of some debate...
[12:34] <toad_> yeah
[12:35] <KenMan> d-ArkAngel: i don't think it considers the "given node" part
[12:36] <d-ArkAngel> which is essentialy the % of requests that are passed to you from one peer as opposed to another which result in a DNF.
[12:36] <d-ArkAngel> KenMan so it's not peer wise?
[12:36] <KenMan> i'm not sure. Toad ?
[12:36] * toad_ LOL | <michaelkuijn> "...cigarettes are great, if you can get over the fact that they taste like hot snake piss." -- seen at a Slashdot posting
[12:36] <d-ArkAngel> :-)
[12:37] <toad_> tobacco is great, if you can get over the fact that it'll kill you and it's seriously addictive.. and yet legal
[12:37] <KenMan> i've never actually tasted hot snake piss, so how can i be sure ?
[12:37] <d-ArkAngel> it was psoted on slashdot, so it must be true ;-)
[12:37] <toad_> KenMan: what?
[12:37] <KenMan> i'm sick right now, so i can't smoke much anyway (1 per day)
[12:38] <toad_> <jepel_tailweaver> <michaelkuijn> jepel_tailweaver, do a full refresh in FROST and your connections will go through the ROOF
[12:38] <toad_> which connections? outgoing connections?
[12:38] <d-ArkAngel> one a day keeps good health at bay? j/k
[12:38] <KenMan> could be ;)
[12:42] <toad_> <Ribs> d-ArkAngel: Sadly, nothing is black and white - okay, my 2 pence: If ISPs block child porn, they will be compelled to block EVERYTHING ELSE that may be illegal. And unless this is done entirely by court order, they will end up blocking stuff pre-emptively that MIGHT be illegal, or that offends specific groups, such as for example the ADL, or the Church of Scientology, or Diebold, or... you get the picture
[12:43] <d-ArkAngel> LOL @ http://www.theregister.co.uk/2004/11/05/mod_oks_win2k_warships/
[12:43] <toad_> :)
[12:43] <toad_> to be fair it's just admin stuff, right?
[12:43] <toad_> Win2K is not capable of critical use, just as linux isn't
[12:44] <d-ArkAngel> indeed.
[12:44] <d-ArkAngel> mind you it'd be a good excuse for fighting back next time the US Airforce decides to indulge in a little frienly fire ;-)
[12:44] <toad_> <d-ArkAngel> and even if by some random chance my node happens to cache every single bit of child porn on freenet, I still don't see it as me contributing to the problem. I'm contributing to the network, and the people inserting and retrieving the files are the ones who are part of the problem. - the situation is analogous to having an IP router (i.e. being an ISP). If you are an ISP, do you have an obligation to block illegal c
[12:44] <toad_> above...
[12:45] <toad_> the moral problem is not "stuff might be cached on my node", it's "and the cops can't trace it".. THAT is the hard part
[12:46] <toad_> <Ribs> But I'm no martar (sp?) for the cause <Ribs> I'm the kinda guy who likes to jump on the bandwagan - so is it the moral issues, or the fear of getting caught?
[12:46] <d-ArkAngel> but they can look at what and individual is doing by tapping their internet connection.
[12:46] <toad_> <d-ArkAngel> what will probably happen is that freenet will develop to a point where people will be able to attain fast access to popular content, then the MP3/XVID people will migrate over (or to some other equivelent system) and the % usages will shift very quickly to wards more normal high volume usages.
[12:47] <toad_> well i rather hope it will migrate to mostly legal content actually in the long term
[12:47] <toad_> (free hosting!)
[12:47] <d-ArkAngel> toad, the internet is even mostly legal content :-)
[12:47] <d-ArkAngel> s/is/isn't/ :-)
[12:48] <toad_> <d-ArkAngel> also there's no reason why someone could run a service to stop nodes cacheing content that is centrally blacklisted somehow. - i've been through this.. no, you can't have whitelists without major changes to the architecture, and with eventually being compelled to use only state approved whitelists
[12:48] <toad_> d-ArkAngel: yes it is
[12:48] <toad_> it's mostly boring corporate marketing sites
[12:49] <toad_> admittedly 35% of traffic is bittorrent
[12:49] <toad_> but the rest of P2P is less than that
[12:49] <toad_> perhaps the inet is actually over 50% P2P
[12:49] <toad_> but even if that is true, the WEB is mostly legal content
[12:50] <toad_> <d-ArkAngel> human nature is open source and never gets patched. - LOL
[12:50] <toad_> human nature is open source?
[12:52] <toad_> <orange_> hi all. fred freezes for me all the time (web and fcp interface stop to respond). i have to restart it at least once a day. this started 3-4 stable builds back. i'm i the only one with this problem..? - he's not here? oh well...
[12:52] <d-ArkAngel> ever heard of psycologists getting suied for pattent infringment? it's all open and shared knowledge.
[12:53] <toad_> <cbreak> java has a gc. unless there is a bug, you should not have memory leaks. - umm, that's kinda circular.. :)
[12:53] <d-ArkAngel> and we've all got one, and it's easy enough to look at your own motivations, even if it's hard to understand of express them :-)
[12:53] <toad_> if there are no bugs in freenet then by definition there are no memory leaks in freenet...
[12:53] <toad_> d-ArkAngel: well, we don't have easy access to the source code; it's difficult to modify stuff, at least it's a lot more difficult than in software
[12:54] <d-ArkAngel> true. :-) it was a good sound byte tho :-)
[12:54] <toad_> <xbit_> my freenetnode has been up for almost two days and it's still unusable :/
[12:54] <toad_> hmmm
[12:54] <toad_> i don't suppose he's here either?
[12:54] <toad_> actually he is...
[12:55] <d-ArkAngel> well he's here, but he's not said anything for quite a while
[12:55] <d-ArkAngel> only says he's been idle for 24 seconds on a query tho.
[12:56] <toad_> [00:07:31] <KenMan> yes. I do not experience that situation. For every 10000 unique requests, only about 10-20 new connections are established. I played with that part of the code.
[12:56] <toad_> [00:07:58] <KenMan> well, actually, i don't know if those 10-20 are the most experienced nodes.
[12:56] <toad_> [00:08:08] <d-ArkAngel> :-)
[12:56] <toad_> [00:08:10] <KenMan> but the connection turnover rate is 'low' in my book
[12:56] <toad_> [00:08:31] <d-ArkAngel> how many inter connects do you have?
[12:56] <toad_> [00:08:41] <KenMan> the only way I can see that happening is if turnover was too high
[12:56] <toad_> KenMan: you're wrong
[12:56] <d-ArkAngel> anyway gents, I've gotta go. time to head home for the evening, and then go check out my new house, and try and resolve some of the damn legal issues.
[12:57] <d-ArkAngel> maybe catch you later on.
[12:57] <toad_> <KenMan> I can guarantee that I don't have the problem toad described, because I added an extra check - you must have more experienced nodes than inexp ones before a new connection can happen. - okay, that's a start
[12:57] <toad_> ttyl d-ArkAngel
[12:57] * d-ArkAngel (~Rob@213-131-104-86.onyx.net) Quit (Read error: 104 (Connection reset by peer))
[12:01] <toad_> <KenMan> he'll never get that in the real network, mRI makes things far too imbalanced. One holdout node with an mRI of 1Mms will screw the whole thing.
[12:01] <toad_> shit
[12:01] <toad_> good point
[12:02] * jay (jcl@ool-18bf6dac.dyn.optonline.net) has joined #freenet
[12:02] <toad_> <d-ArkAngel> plus we can use timeouts, whereas the sim is all about doing things as fast as possible. - well, sort of..
[12:02] <toad_> we can't really use times
[12:02] <toad_> KenMan: ideas?
[12:03] <KenMan> you will come up with some, i am sure ;) must go feed my empty stomach atm
[12:03] * Hory (~Miranda@82.76.81.56) has joined #FreeNet
[12:04] * Hory (~Miranda@82.76.81.56) Quit (Read error: 54 (Connection reset by peer))
[12:05] * Hory (~Miranda@82.76.81.56) has joined #FreeNet
[12:06] * pittaman (~mistery_b@d51A410CA.kabel.telenet.be) has joined #freenet
[12:17] <KenMan> toad - i don't know how we can apply the "all must be experienced" concept to real life
[12:18] <KenMan> we could allocate a portion of our RT to fit the rules i suppose, but ...
[12:18] <KenMan> i mean , that one node which has really bad mRI isn't going to be the normal case !!
[12:20] <KenMan> looks like we need to rethink the whole approach of connection management
[12:23] <KenMan> one the one hand, we want all nodes to be evaluated 'fairly' which would mean each having the same number of samples. On the other hand, we want to accomodate a system where requests are not perfectly distributed among the peers.
[12:23] <KenMan> So we don't want to force a single common rate of sampling onto all peers, but we want info that reflects such a model.
[12:24] <Ribs> toad_: no, it's not the fear of being caught at all
[12:24] <Ribs> toad_: It's the fear that my node might have child porn on it, even if I don't request the data myself
[12:24] <KenMan> Ribs: that is a rational fear, one we don't have a solution for, other than moral anti-censorship explanations...
[12:25] <toad_> yep
[12:25] * pittaman (~mistery_b@d51A410CA.kabel.telenet.be) Quit (Client Quit)
[12:25] <toad_> if you can't accept that, then don't run freenet; but don't run a postal system either, or an ISP
[12:25] <Ribs> oh yeah, I know that
[12:25] <Ribs> and as toad_ just said, I can't accept that, so I shut down the node.
[12:25] <toad_> the difference with freenet, as i explained, is that an explicit goal of freenet is to prevent tracing people
[12:25] <toad_> THAT is different
[12:26] <toad_> and that is the key concern
[12:26] <Ribs> toad_: It's great software, with a good cause
[12:26] <toad_> if you run an ISP, they can trace the people going to alt.sex.subteen
[12:26] <toad_> and go bust them
[12:26] <toad_> but if you run a freenet node, it's not so easy for them to trace them
[12:26] <KenMan> The question is... if tracking criminals becomes harder with FreeNet, does that make it more attractive to criminals ?
[12:27] <toad_> granted most of the people who visit that group limit their activities to downloading pictures.. at least at first..
[12:27] <toad_> but there is a problem here
[12:27] <KenMan> And, does the percentage of criminal use of FreeNet exceed the percent for non-anon systems ?
[12:27] <KenMan> If it does attract crime, then some might argue that FreeNet has a duty to confront that matter.
[12:27] <toad_> KenMan: it's more than that
[12:28] * KenMan keeps his ears open
[12:28] <toad_> do the potential and actual good uses of freenet outweigh the potential and actual bad uses of freenet?
[12:28] <toad_> that's the bottom line, and it's hard to answer it - especially if you have a vested interest!
[12:29] <plixed> KenMan: who defines what is a crime and what is not?
[12:29] <KenMan> my investment is the time spent trying to help this tech to work. Someone will get it done sooner or later, so that is my rationale for working on it, rather than avoiding it.
[12:30] <toad_> (and I plainly do have a vested interest..)
[12:30] <KenMan> well, there are some reasonably global ethics (murder is wrong in 90%+ of the countries of the world)
[12:30] <KenMan> but things become muddied very quickly
[12:30] <plixed> KenMan: not if murder is interest of the state
[12:30] <plixed> +in
[12:30] <toad_> right, it's muddy. But that doesn't mean you can abdicate responsibility.
[12:30] <KenMan> states generally frown upon random, non-interested murder
[12:31] <toad_> is it wrong to make kitchen knives? is it wrong to make guns? is it wrong to make freenet?
[12:31] <KenMan> only time will tell ;)
[12:31] <toad_> the line between tools and weapons is sometimes a fine one...
[12:31] <KenMan> so where do we go on this conn management issue ?
[12:32] <toad_> but what I do know: if we provide the ability to censor freenet, then we will be compelled to block all illegal content
[12:32] * TLF (~francisco@164.Red-81-40-113.pooles.rima-tde.net) has joined #freenet
[12:32] <toad_> and probably a lot of content that is borderline, but that nobody has the money to fight it out
[12:33] <KenMan> since i always talk about rates, we want to rate-limit the number of new connections. But upon what do we base this rate ?
[12:34] <KenMan> If all nodes received the same rate of samples, a simple answer would be forthcoming. But we don't like that model.
[12:34] <KenMan> Plus, we have mRI on top of less than perfectly balanced routing.
[12:35] <toad_> brb
[12:35] <plixed> maybe there should be connection fluctiation and the "better" connections (i.e. expierenced) are be a bit more sticky than new ones?
[12:36] <KenMan> ie. even without mRI, we still would not have perfectly distributed routing
[12:36] <KenMan> what is wrong with using raw success rates per node ? we have some newbie threshold, beyond which we use success as our stickiness factor
[12:37] <KenMan> this node resolved 13 of 300 requests, that one did 11 of 4200 requests. Dump the loser.
[12:37] <plixed> sounds resonable to me, but i don't know much about freenet internals ;)
[12:40] <KenMan> one problem is that in real life, success is so low... making it harder to meaningfully compare peers. Perhaps our newbie threshold could be "X requests or Y minutes, whichever comes first"
[12:43] <KenMan> having a large RT means lower average query rate per route. Smaller RT would mean we could evaluate peers more quickly, thus we could have a higher wallclock rate of new conns.
[12:43] <KenMan> this assumes we have a relatively constant rate of incoming queries, and that mRI doesn't mess it ALL up
[12:44] <KenMan> a larger RT counteracts/lessens the negative effects of mRI
[13:05] * jay (jcl@ool-18bf6dac.dyn.optonline.net) Quit ("Client exiting")
[13:06] * d-ArkAngel (~Rob@cpc2-midd2-5-0-cust16.midd.cable.ntl.com) has joined #freenet
[13:22] <toad_> KenMan: because raw success rate doesn't take into account number of successes, e.g. MRI
[13:22] <toad_> so you go to something like MRI*typicalEstimate
[13:22] <toad_> which is what we have now
[13:23] <toad_> but LRU is probably optimal
[13:23] <toad_> but all this is irrelevant; what we want is not which conn to replace, but WHEN can we replace ANY conn
[13:23] <toad_> s
[13:26] <toad_> KenMan: here?
[13:28] <sanity> toad: are user-sent messages still going to the announce list?
[13:29] <toad_> sanity: no
[13:30] <sanity> what about those emails from terryg and marc tallec?
[13:30] <sanity> ie. how did marc tallec get terryg's email if not through the list?
[13:31] * spaetz_ (~spaetz@80-218-147-20.dclient.hispeed.ch) Quit (Read error: 242 (No route to host))
[13:31] <toad_> from who?
[13:31] <toad_> i had a thread in my inbox
[13:31] <toad_> but that wasn't on the list afaics
[13:32] <sanity> ok
[13:33] <toad_> are they in the archives?
[13:33] <toad_> sanity: what can we do about the problems with dropping nodes and newbieness?
[13:34] <toad_> sanity: the idea I proposed on devl, and was simulating, won't work in practice
[13:34] <toad_> the idea being don't drop any nodes until you have good data on all nodes
[13:34] <sanity> toad: yes, they *are* in the archives, including one that was just sent a few minutes ago
[13:34] <toad_> the problem is, if you impose a cutoff of N hits on each node, and one of your nodes has a 5,000,000,000 ms MRI...
[13:34] <toad_> sanity: wtf?
[13:34] <sanity> we need to stop these ASAP or we will quickly start to lose subscribers to announce
[13:34] * toad_ checks...
[13:35] <toad_> why did they end up in inbox then?
[13:35] <sanity> http://dodo.freenetproject.org/pipermail/announce/2004-November/date.html
[13:35] <toad_> oh, the header changed... hmmm
[13:35] <sanity> because you were cc'd
[13:35] * cbreak (~cbreak@dhcp-32-094.via-eth.ch) has joined #freenet
[13:35] <toad_> hmmm
[13:38] * toad_ enables emergency moderation...
[13:39] <toad_> sanity: where should the reply-to address be for announce? support or chat?
[13:39] <sanity> me or you?
[13:39] <toad_> huh?
[13:39] <sanity> reply to sender?
[13:40] <toad_> perhaps
[13:40] <sanity> doesn't it let you make reply-to point to the sender of the email?
[13:40] <toad_> i don['t get it
[13:40] <toad_> default_member_moderation = 1
[13:41] <toad_> so why are people's mails getting through?
[13:41] <toad_> member_moderation_action = 0, meaning hold
[13:43] <toad_> and yes, we are losing people from announce
[13:43] <toad_> anyway, what now?
[13:43] <KenMan> what's the trouble with announce ? people's addresses are being given away ? or what ? too many unauthorized posts ?
[13:43] <toad_> any unauthorized posts is a problem
[13:43] <KenMan> why are they leaving ?
[13:43] <toad_> it's supposed to be low traffic
[13:44] <toad_> that means absolutely no posts except when we make them
[13:44] <toad_> and hand-approve them, since worms spoof our addresses :(
[13:44] <toad_> sanity: what now? i think it's fixed...
[13:45] <toad_> sanity: if that is not sufficient, what is? if that is sufficient, could you please provide an opinion on the when-should-we-drop-nodes issue?
[13:45] <sanity> perhaps send an email apologising for the spam and saying that there shouldn't be any more - and that if people wish to have discussions to use one of the other lists?
[13:45] <toad_> sanity: would that help?
[13:45] <sanity> perha
[13:46] <sanity> ps
[13:46] <toad_> or would that just be even more reason to unsubscribe?
[13:46] * toad_ suspects the latter..
[13:46] <KenMan> no, it is meant as a 1 way comm channel. State as much in your post.
[13:48] <sanity> toad: well, we could ditch all that complexity and think about it from scratch (re: dropping nodes)
[13:49] <toad_> sanity: I'm writing a mail, I should include a brief summary of what we've been doing lately
[13:49] <toad_> ideas?
[13:49] <sanity> ok
[13:49] <toad_> or do you want to send it? or do you want me to send you a draft?
[13:49] <sanity> i will read and respond to your mail
[13:50] <toad_> sanity: to announce
[13:50] <sanity> oh, no, just send that
[13:50] <toad_> sanity: better we send one definitive mail to announce
[13:50] <sanity> keep it brief
[13:50] <sanity> ok, well if you want me to review it before you send it - forward it to me
[13:50] <KenMan> then we can discuss peer-management ...
[13:52] <sanity> ok, so lets go back to basics - what is wrong with simple LRU for peer management?
[13:54] * Redb3ard (~oylerj@c-24-125-12-101.va.client2.attbi.com) has joined #freenet
[13:54] <sanity> i mean - did the whole newbie thing, for all its added complexity, actually solve any problems?
[13:55] <Redb3ard> yes
[13:55] <Redb3ard> it prevents newbies, which is the biggest problem freenet now faces
[13:55] * toad_ back
[13:55] <toad_> sanity: okay
[13:55] <toad_> I have NO problem with LRU
[13:55] <Redb3ard> the internet was cool before all these new people joined, what, in 93 was it?
[13:55] <sanity> Redb3ard: huh?
[13:55] <Redb3ard> freenet needs less new people
[13:55] <sanity> Redb3ard: i agree
[13:56] <toad_> the question is, do we want to allow absolutely every incoming connection request? and do we want to open a new connection EVERY time a request completes successfully and gives us a new reference?
[13:56] <Redb3ard> there has to be time for them to become old-timers before more new ones can be absorbed
[13:56] <toad_> LRU for deciding which connection to drop is absolutely correct
[13:56] <sanity> it doesn't have to do it with 100% probability
[13:56] <toad_> sure, but it has to do it sometimes and not other times
[13:56] <toad_> how do we determine the probability?
[13:57] <sanity> well, it should be 1.0 until the RT is full
[13:57] <sanity> then it depends on what rate of turnover we want
[13:57] <toad_> ordinary probabilistic referencing is unsuitable, for example, because if we get enough connections it'll still be flooded
[13:57] <toad_> absolutely, we should accept connections until the RT is full
[13:57] <sanity> we could decide, for example, that one new connection every 5 minutes is appropriate, and tailor the probability accordingly
[13:58] <toad_> and on what do we base THAT alchemy?
[13:58] <Redb3ard> do you have any historical data?
[13:58] <sanity> well, its a tradeoff
[13:58] <toad_> which inevitably varies from node to node just on bandwidth alone
[13:58] <Redb3ard> and what does it suggest will happen?
[13:58] <cbreak> maybe increase accept probability with time, and set it to zero when accepted a newbie.
[13:58] <toad_> but on a zillion other things too
[13:58] <toad_> how do we calibrate the "5 minutes" ?
[13:58] <sanity> too much turnover and we must deal with the overhead of having to learn all about a new node, too little and the network stagnates and we don't get the correct connection distribution
[13:59] <toad_> yes, and doing a binary search to find the right number on the real network is simply not practical
[13:59] <toad_> we need a theoretical basis for it
[14:00] <sanity> well, how many requests does it take for NGR to learn anything useful about a node?
[14:00] <toad_> one option would be to say "we accept a new connection when we have reasonable info on half the RT"
[14:00] <toad_> and then drop the strict LRU node (not the LRU inexperienced node)
[14:01] <toad_> sanity: that is the subject of some uncertainty
[14:01] <toad_> 20 seemed reasonable once upon a time
[14:01] <toad_> on the simulated network, 200 seems better
[14:01] <toad_> in terms of getting better success rates than 100 or 500
[14:02] <sanity> well, lets start by using the simulator to track the accuracy of an estimator over time as it learns
[14:02] <sanity> we need to get a feel for how accurate estimators are, and how long it takes them to become accurate
[14:03] <toad_> that helps us to work out N
[14:03] <toad_> that does not help us to work out what to do with N
[14:03] <sanity> its a start
[14:04] <toad_> no, it's useless, because the architecture is still fundamentally broken
[14:04] <toad_> that means any success numbers we get are crap
[14:04] <sanity> what is broken about it?
[14:04] <toad_> and probably just show the biases of the broken architecture
[14:04] <toad_> well, which architecture are you referring to? the current one is that we only drop experienced nodes; the result of this is that we end up dropping the best node
[14:05] <toad_> if we go for pure LRU, then that is also a big problem, because even with estimator passing, learning will be slow; and there are security issues with it
[14:05] <sanity> if it is broken, then it is only through this kind of investigation that we stand a chance of finding out why, yet you are saying that we shouldn't try to investigate because the architecture is broken. that is circular
[14:05] <cbreak> a node should accept connections if he needs them, if he can connect to them without drop an other, or if he can gain something by dropping an other and connect.
[14:05] <toad_> i.e. we want to verify the passed estimators
[14:05] <toad_> NO
[14:05] <toad_> I'm saying we should investigate
[14:06] <toad_> but I'm saying we should investigate more than the number of hits it takes for an estimator to learn, because that is just a parameter
[14:06] <sanity> LOOK - the viability of NGR requires that the estimators are capable of learning - it is therefore *essential* that we discover 1) whether estimators learn and 2) how quickly they learn
[14:06] <toad_> cbreak: and how exactly does he determine that?
[14:06] <toad_> sanity: we KNOW that estimators DO learn
[14:06] <sanity> toad: ok, but how quickly, and how accurately?
[14:06] <cbreak> well... :) I don't know. The first two are fairly easy, but the last one is tricky.
[14:07] <sanity> toad: the first step is to answer those two questions - then we can proceed
[14:07] <toad_> sanity: that is NOT the only issue here
[14:07] <toad_> yes, we need to find that out
[14:07] <toad_> but that is a parameter
[14:07] <toad_> which we plug into whatever architecture we use
[14:07] <toad_> we haven't got an architecture
[14:07] <sanity> toad: did i say it was? it is not the only issue, but we must deal with one thing at a time
[14:08] <toad_> which means that the simulations are going to be hopelessly distorted, and anything we find won't be applicable
[14:08] <sanity> what does "we haven't got an architecture" mean?
[14:08] <sanity> what "architecture"?
[14:08] <toad_> it means we haven't got an architecture
[14:08] <toad_> an algorithm for deciding when to accept a connection, or open a connection to a new reference
[14:08] <toad_> we agree on "open a connection if the RT isn't full", that much is established
[14:08] <toad_> but what of the rest?
[14:09] <toad_> we even agree that LRU is nice
[14:10] <toad_> I can show trivially that what is implemented in the simulations and to a large degree on the real network sucks
[14:10] <toad_> so we need something else
[14:10] <cbreak> so you would have much less problems if announcements would work better?
[14:10] <toad_> one option is "allow a connection if and only if either the RT is not full, or we have N hits on every node", where N is a parameter we can determine empirically/by simulation
[14:10] <toad_> however, that will not work in the real world
[14:11] <toad_> because we have nodes that we can't route to because they, intentionally or accidentally, have very high MRIs
[14:13] * Hory (~Miranda@82.76.81.56) Quit (Read error: 232 (Connection reset by peer))
[14:15] <toad_> cbreak: announcements working better is a good thing, but it's not the only problem
[14:18] * toad_ leans towards "allow a connection if either the RT is not full, or we have N hits on X% of all nodes in the RT"
[14:18] <toad_> we'd end up dropping one of the bottom half, which is less experienced..
[14:18] * robilad (~topic@mpiat2320.ag2.mpi-sb.mpg.de) Quit ("ChatZilla 0.9.52B [Mozilla rv:1.6/1]")
[14:18] <toad_> with the result that it might never get experienced
[14:18] <toad_> but it does solve some problems
[14:21] <cbreak> is there a possibility to drop the node which itself has lots of connections?
[14:21] <toad_> eh?
[14:21] <cbreak> drop a well connected node since he would not suffer
[14:22] <toad_> http://amphibian.dyndns.org/simulations2-epass-0.025-400x50-supersmooth.png
[14:24] <toad_> http://amphibian.dyndns.org/simulations2-epass-0.025-400x50.png
[14:24] <toad_> curious
[14:24] <toad_> i expected the new code to get rid of the jerkiness...
[14:24] <toad_> there does seem to be an improvement anyhow..
[14:24] * guido^pe (~unknown@dsl-213-023-246-188.arcor-ip.net) has joined #freenet
[14:29] * toad_ fixes announcements, if this works it should go right back up...
[14:36] <toad_> hmm, fosdem 2005 will be 26-27 Feb 2005...
[14:37] <jepel_tailweaver> hello all
[14:38] <toad_> hi
[14:38] <toad_> sanity: for the time being, i'm going to simulate the proposed drop-only-if-all-nodes-have-N-hits
[14:39] <toad_> sanity: so far, that looks positive
[14:39] <toad_> jepel_tailweaver: hi
[14:43] <KenMan> toad_: sounds like a good approach for sim. We can revisit this later when attempting to apply it to real life.
[14:43] * TLF (~francisco@164.Red-81-40-113.pooles.rima-tde.net) Quit ("Adi?s a todos y a todas, y que les vaya bien. || http://www.vitaliano.esp.cc")
[14:44] <KenMan> meantime, pay attention to the load distribution for peers when a new one is finally let in.
[14:44] <toad_> Success ratio on last 10K reqs: 0.9997938144329896 (2 0s, 9698 1s, 9700 total)
[14:44] <toad_> Success ratio on last 10K reqs: 0.9991878172588833 (16 0s, 19684 1s, 19700 total)
[14:44] <toad_> KenMan: yeah, i suspect we can do something close enough IRL
[14:44] <KenMan> heh, twiddling the knobs, eh ?
[14:44] * d-ArkAngel (~Rob@cpc2-midd2-5-0-cust16.midd.cable.ntl.com) Quit (Read error: 232 (Connection reset by peer))
[14:44] <toad_> this is with 400x50, with the new code, with announcements apparently working
[14:45] <toad_> the first two cycles
[14:45] <KenMan> much lower conn turnover gives you higher success so far ? or what ?
[14:45] <toad_> and it's pretty impressive...
[14:45] <KenMan> first two cycles, jeesh! come back in a week's time and share !!
[14:45] <KenMan> ;)
[14:46] <KenMan> i'm running mine out to 200M unique CHK requests, and I'm not sure that is sufficient :p
[14:47] <toad_> Success ratio on last 10K reqs: 0.99365 (127 0s, 19873 1s, 20000 total)
[14:47] <toad_> third cycle
[14:47] <toad_> so far, 0 connections added...
[14:48] <toad_> this is with we only accept/make connections when each node in the RT has at least 80 hits
[14:49] <KenMan> and your HTL ??
[14:49] <KenMan> you may be waiting a while...
[14:50] <toad_> well, I compare it to previous simulations
[14:50] <toad_> it's dropping a lot less slowly with the new code
[14:50] <toad_> probably because it hasn't added any new connections...
[14:50] <KenMan> i found that with the "half RT must be experienced" , the permitted rate caused it to stay at "half are experienced"...
[14:51] <toad_> KenMan: that's a useful if perhaps predictable result
[14:51] <toad_> that is with estimator passing, of course? and pcaching?
[14:51] <toad_> well preffing, technically...
[14:51] <toad_> preffing is enabled by default, so unless you've changed it...
[14:52] <KenMan> trouble is, you are fighting the distribution curve of hits per node, in order to get the smallest one up to 80...
[14:52] <KenMan> checking on status of estimator passing - i think i do that.
[14:53] <KenMan> no, i'm not doing it so far.
[14:53] <toad_> okay well that partly explains your poor performance
[14:53] <toad_> estimator passing is a vast improvement
[14:54] <KenMan> i'll get there, i'm working up to it...
[14:54] <toad_> you do have fastestimators enabled, right? that speeds up simulations by ~ a factor of 10...
[14:54] <KenMan> yes, that's why i no feel the big need to replace CHK with unsigned int :)
[14:55] <KenMan> what htl are you using ? 400x50 seems pretty well 'connected'
[14:55] <toad_> 22...
[14:55] <KenMan> damn, boy !!
[14:56] <toad_> I wonder if it might actually work better with adding nodes...
[14:56] <toad_> last time I tried adding nodes it was a disaster, but that was before estimator passing
[14:56] <KenMan> dynamic set of nodes you mean ?
[14:56] <toad_> yeah, I mean growing from 50 to 400 over a period, with lots of requests in the process
[14:57] <toad_> okay, this is what i've got:
[14:57] <KenMan> try more nodes, fewer peers, lower htl... 400x50 is pretty busy, and with 22 HTL your key redundancy is going to mess with you
[14:57] <toad_> for one of the nice graphs of 400x50, which eventually goes up to around 96%
[14:57] <toad_> KenMan: 20% success is not realistic
[14:57] <toad_> nobody will use the network if the psuccess is only 20%
[14:57] <toad_> anyway
[14:58] <toad_> one of the good graphs:
[14:58] <KenMan> i am looking for network effects at all levels of success
[14:58] <toad_> Success ratio on last 10K reqs: 0.9752577319587629 (240 0s, 9460 1s, 9700 total)
[14:58] <toad_> Success ratio on last 10K reqs: 0.9681725888324874 (627 0s, 19073 1s, 19700 tota
[14:58] <toad_> l)
[14:58] <toad_> Success ratio on last 10K reqs: 0.93 (1400 0s, 18600 1s, 20000 total)
[14:58] <toad_> Success ratio on last 10K reqs: 0.87745 (2451 0s, 17549 1s, 20000 total)
[14:58] <toad_> Success ratio on last 10K reqs: 0.8514 (2972 0s, 17028 1s, 20000 total)
[14:58] <toad_> the new graph:
[14:58] <toad_> Success ratio on last 10K reqs: 0.9997938144329896 (2 0s, 9698 1s, 9700 total)
[14:58] <toad_> Success ratio on last 10K reqs: 0.9991878172588833 (16 0s, 19684 1s, 19700 total)
[14:58] <toad_> Success ratio on last 10K reqs: 0.99365 (127 0s, 19873 1s, 20000 total)
[14:58] <toad_> Success ratio on last 10K reqs: 0.9835 (330 0s, 19670 1s, 20000 total)
[14:58] <toad_> Success ratio on last 10K reqs: 0.9718 (564 0s, 19436 1s, 20000 total)
[14:58] <toad_> Success ratio on last 10K reqs: 0.96515 (697 0s, 19303 1s, 20000 total)
[14:58] <toad_> it's falling a LOT slower..
[14:58] <toad_> still no conns added whatsoever
[14:59] <KenMan> PS if you could achieve 20% success you would get 10x more users than you have today ;)
[14:59] <toad_> so that's promising
[14:59] <toad_> but need to get back to it when it's got better
[14:59] <toad_> KenMan: :|
[14:59] <KenMan> start counting keys...
[14:59] <toad_> KenMan: if we only get 20% _in the simulation_, then in real life we'll only get 5%
[14:59] <KenMan> yeah yeah, i know what you say ...
[15:00] <toad_> and what do you mean by key redundancy effects?
[15:00] <toad_> also i should find a remote PC to run this sim on or I won't get any work done...
[15:00] <KenMan> if the key you search for lives on 87 of the 400 nodes, you will find it. If it lives on only 3 nodes, the odds are much different.
[15:01] <KenMan> regardless of all the other factors...
[15:01] <toad_> KenMan: but there is ongoing data churn
[15:01] <toad_> i think that offsets that
[15:01] <toad_> well i know that offsets that, or the psuccess would be much higher :)
[15:02] <KenMan> the redundancy of each key (which we select uniformly at random) fits a normal curve anyway.
[15:03] <KenMan> with HTL=22, you should have a very high average level of redundancy. What is your DS size ?
[15:05] <KenMan> 400x50x22x(??)
[15:05] * sanity (~ian@81-178-109-149.dsl.pipex.com) Quit (Read error: 104 (Connection reset by peer))
[15:07] <toad_> KenMan: not necessarily, it depends on data churn
[15:07] <toad_> and like I said, 20% success is worthless
[15:07] <KenMan> suppose you have 100,000 DS slots, but your data-churning network can only support success for 721 uniquq CHKs. Would that bother you ?
[15:08] <toad_> I have a fixed ratio of slots to data requested; there is constant churn in the data requested set as more data is inserted
[15:08] <toad_> the fixed ratio is 2.5%
[15:08] <toad_> so with 100,000 slots, we have 100,000/40 = 2,500
[15:09] <toad_> okay, that's not great
[15:09] <toad_> but i'm not going to even consider simulations at the sort of success rates you deal with
[15:09] <KenMan> this i know :) I am just trying to "learn things"
[15:10] <toad_> :)
[15:10] * sanity (~ian@81-178-109-149.dsl.pipex.com) has joined #freenet
[15:10] <toad_> wb sanity
[15:10] <KenMan> your math (like mine) is flawed. 2500 is probably much higher than the actual number of unique keys in your network at any given moment...
[15:10] <toad_> of course you're not really here
[15:11] <toad_> yes, indeed
[15:11] <toad_> but we only request the last 2,500 inserted files
[15:11] <KenMan> he already went over his weekly quota of IRC typing earlier 8)
[15:11] <toad_> which is admittedly a gross approximation to what actually happens
[15:11] <toad_> LOL
[15:12] <KenMan> I think your approximation is okay in some circumstances.
[15:12] <KenMan> But I've found that HTL most directly influences key redundancy
[15:13] <KenMan> when you get a hit, it replicates that key an average of HTL/2 times. Depending on DS size, that can have a major effect.
[15:14] <KenMan> Tell me your net DS size...
[15:15] <KenMan> and your average successful HTL... that will allow us to (bad math) compute how fast you overwrite the global DS
[15:15] <toad_> aha
[15:15] <toad_> i'm still doing tons of logging
[15:15] * flatface (~flatface@69.158.10.104) Quit ("My other computer is a 4000 node beowulf cluster")
[15:15] <toad_> i think the ds size is 100 but it could be 1000
[15:15] <KenMan> nothing new there - you like to test those MTBF figures for your HD seeks
[15:16] <KenMan> okay, 400nodesx100DS, 40,000 . Each hit (in 25K cycle) affects 11 DSs. 250,000 slots being updated per cycle... that's a lot.
[15:17] <KenMan> but wait. You keep 2500 unique keys available for searching, and manage 90%+ ? Then you must have 2500+ keys available.
[15:17] <KenMan> some of those slots are written multiple times, but you basically are flushing the entire storage within a single cycle
[15:20] <toad_> well, logging costs CPU. a lot of CPU
[15:20] <KenMan> what are your cycles again ? 400 inserts per 25000 requests ?
[15:20] <toad_> not sure
[15:21] <KenMan> I'm doing 100ins, 10Kreqs , but I separated them. I must interleave them again...
[15:21] <toad_> 400 inserts, 10,000 requests, i think
[15:21] <KenMan> I don't think it matters much , but i must prove it to myself
[15:21] <toad_> okay, i've taken the logging out...
[15:21] <KenMan> woo hoo
[15:22] <toad_> and my first cycle took 34 seconds and got 100% success
[15:22] <toad_> hmm, that's impossible...
[15:23] <toad_> Success ratio on last 10K reqs: 1.0 (0 0s, 9700 1s, 9700 total)
[15:23] <toad_> Success ratio on last 10K reqs: 0.9993401015228427 (13 0s, 19687 1s, 19700 total)
[15:23] <toad_> well i suppose maybe it's possible...
[15:23] <KenMan> that's kickin' butt. Once you translate it to real life, we got it made ;)
[15:23] <toad_> should i check it out in more detail?
[15:23] <toad_> heh
[15:23] <toad_> it'll decline now
[15:23] <toad_> quite quickly
[15:23] <toad_> Success ratio on last 10K reqs: 0.99295 (141 0s, 19859 1s, 20000 total)
[15:23] <KenMan> naw, just feel good it's a nice big number
[15:24] <KenMan> i don't see how it's possible for the first cycle, but ...
[15:25] <KenMan> 1.0 is impossible unless HTL = netsize
[15:26] <toad_> no, it's possible
[15:26] <KenMan> bull output
[15:26] <toad_> this is with announcements, remember?
[15:26] <KenMan> ahh, not there yet :)
[15:26] <toad_> and it's at the very beginning, we haven't filled all stores yet
[15:26] <toad_> Success ratio on last 10K reqs: 1.0 (0 0s, 9700 1s, 9700 total)
[15:26] <toad_> Success ratio on last 10K reqs: 0.9993401015228427 (13 0s, 19687 1s, 19700 total)
[15:26] <toad_> Success ratio on last 10K reqs: 0.99295 (141 0s, 19859 1s, 20000 total)
[15:26] <toad_> Success ratio on last 10K reqs: 0.98305 (339 0s, 19661 1s, 20000 total)
[15:26] <toad_> Success ratio on last 10K reqs: 0.9736 (528 0s, 19472 1s, 20000 total)
[15:28] <toad_> Success ratio on last 10K reqs: 0.9663 (674 0s, 19326 1s, 20000 total)
[15:29] <toad_> 0 conns added...
[15:29] <KenMan> no surprise there...
[15:29] <toad_> Success ratio on last 10K reqs: 0.96405 (719 0s, 19281 1s, 20000 total)
[15:29] <KenMan> this is the first time you've run with 'RT must be fully experienced' ??
[15:30] <toad_> I doubt that it's settling this early...
[15:30] <toad_> KenMan: RT must be fully experienced I tried before, got some data; this is that plus a bugfix (announcements weren't working)
[15:30] <toad_> http://amphibian.dyndns.org/simulations2-epass-0.025-400x50-supersmooth.png
[15:30] <toad_> the bottom two are buggy
[15:30] <toad_> the brown one is buggy + RT-must-be-fully-experienced
[15:30] <toad_> the top two are the ones without the new coee
[15:30] <toad_> code
[15:31] <toad_> Success ratio on last 10K reqs: 0.96275 (745 0s, 19255 1s, 20000 total)
[15:31] <toad_> it can't possibly be settling... it's an illusion... it'll fall all the way down, then come back up...
[15:32] <cbreak> hmm... looks different than the others... :)
[15:33] <toad_> Success ratio on last 10K reqs: 0.95775 (845 0s, 19155 1s, 20000 total)
[15:33] <toad_> yeah, it was an illusion...
[15:35] <toad_> Success ratio on last 10K reqs: 0.9499 (1002 0s, 18998 1s, 20000 total)
[15:36] <KenMan> yeah, i accept your math for computing the size of the keyset on which to search. It is a good approximation.
[15:36] <toad_> 80*50*400 = 4000*400 = 1,600,000... *20 (newbieprobe) = 32,000,000...
[15:36] <toad_> so will it take 32M requests before it accepts any connections? :)
[15:36] <toad_> Success ratio on last 10K reqs: 0.9475 (1050 0s, 18950 1s, 20000 total)
[15:37] <KenMan> you should start getting new connections around 7M...
[15:37] <toad_> why?
[15:37] <KenMan> because they don't spill out in a perfectly balanced manner
[15:38] <toad_> is 80 too high, or is the simulator too slow?
[15:38] <KenMan> some node should get more load. but then again, his worst peer must reach 80 ...
[15:38] <KenMan> sim too slow, you've already told us this :)
[15:38] <toad_> Success ratio on last 10K reqs: 0.94845 (1031 0s, 18969 1s, 20000 total)
[15:38] <toad_> well, obviously it'd be better if the sim was faster, but there are limitations IRL on how fast we can learn too
[15:39] <toad_> hmmm
[15:39] <toad_> the psuccess actually ROSE
[15:39] <toad_> it INCREASED
[15:39] <KenMan> heh, i certainly have my limitations ;)
[15:39] <toad_> that is peculiar...
[15:39] <toad_> I mean how fast the network can learn...
[15:40] <toad_> well if it's going to settle at around 95% on a 400x50 node network with random connections, that's nice
[15:40] <toad_> we'll see
[15:40] <toad_> random FIXED connections
[15:40] <toad_> Success ratio on last 10K reqs: 0.9447 (1106 0s, 18894 1s, 20000 total)
[15:40] <toad_> it fell again :|
[15:40] <KenMan> oh, yeah. That's what I meant - my hardware is slow. I was most definitely not admitting to any deficiencies in my mental processes! Never.
[15:41] <toad_> LOL
[15:41] * nextgens (~toto@d80-170-24-39.cust.tele2.fr) has joined #freenet
[15:41] <toad_> well
[15:41] <KenMan> my bet is on 7M...
[15:41] <toad_> lets be a little more realistic and project it to the real network...
[15:42] <toad_> if we implement this, and assume no MRI issues (pretty major assumption!), or we implement something functionally similar which is immune to them...
[15:42] <toad_> Success ratio on last 10K reqs: 0.93805 (1239 0s, 18761 1s, 20000 total)
[15:42] <nextgens> hi
[15:42] <toad_> then we get for a node to accept conns, it must send out at least RTnodes (100) * perNodeNewbieCount (80)...
[15:43] <toad_> in practice, this is going to be closer to 100*80*20
[15:43] <toad_> although it won't be quite that much
[15:43] <toad_> so it's between 8000 requests and 160,000 requests
[15:43] <toad_> that's a hell of a lot of requests
[15:43] <KenMan> won't some nodes get a lot more than others (even ignoring mRI) ?
[15:43] <KenMan> indeed, that's a lot
[15:44] <toad_> now if we assume each request is 32K (perhaps we can cut it down a lot further, but 32K is definitely achievable)...
[15:44] <KenMan> rather, for a given node's RT, won't some PEERS get many more requests than other PEERS ?
[15:44] <toad_> 8000*32K = 256000K = 256MB
[15:44] <toad_> 160000*32K = 256M*20=5,120MB
[15:45] <toad_> so that is in fact just about feasible
[15:45] <toad_> still, the point at which it starts dropping nodes is not the end point, it's near the beginning...
[15:45] <toad_> KenMan: yes, the good peers get more requests...
[15:45] <toad_> he who has will be given more etc
[15:45] <toad_> Success ratio on last 10K reqs: 0.93335 (1333 0s, 18667 1s, 20000 total)
[15:46] <toad_> Success ratio on last 10K reqs: 0.9282 (1436 0s, 18564 1s, 20000 total)
[15:46] <toad_> how much data do we have irl?
[15:46] <KenMan> too much. Definitely too much.
[15:46] <toad_> 10K*3600s = 36000K = 36MB/hr
[15:46] <KenMan> I can't seem to manage all of my data.
[15:46] <toad_> ah
[15:46] <toad_> even if it's 20K, that's only 72MB/hr
[15:47] <toad_> okay, so this definitely won't work
[15:47] <toad_> even if we go for half the RT has 80 hits
[15:47] <KenMan> starting point (where conns begin) will be between 7M and 8M ...
[15:47] <KenMan> I bet you one chicken.
[15:48] <toad_> if we cut the block size to 4K (hypothetically), then 256MB becomes 64MB, and 5120MB becomes ~ 1280MB...
[15:48] <toad_> Success ratio on last 10K reqs: 0.9282 (1436 0s, 18564 1s, 20000 total)
[15:48] <toad_> Success ratio on last 10K reqs: 0.92465 (1507 0s, 18493 1s, 20000 total)
[15:48] <toad_> hmm, ignore the first one
[15:48] <toad_> so that's between an hour and 20 hours, roughly
[15:48] <toad_> hmmm
[15:49] <toad_> if we say half the RT, then halve that; if we cut the threshold to 20 hits, divide by another 4; you get, 8MB and 160MB...
[15:50] <toad_> THAT might be feasible
[15:50] <toad_> maybe
[15:50] <toad_> depending on a string of maybes
[15:50] <KenMan> i'm running 400x50x22x100 out of curiosity (no announce, no est passing) and see 80% success, around 1900 unique keys in 40,000 global DS
[15:50] <toad_> sure but it falls, right?
[15:50] <toad_> it falls a long way?
[15:50] <toad_> Success ratio on last 10K reqs: 0.9243 (1514 0s, 18486 1s, 20000 total)
[15:51] <toad_> KenMan: also you are using fetchfrac=0.025?
[15:51] <KenMan> if i let it run for a long time, probably... but it has 'settled' around 80% already (started 98%).
[15:51] <toad_> not the default, which is based on HTL and sucks?
[15:51] <KenMan> no, i insert 100 for every 10000 requests
[15:51] <toad_> well sucks is wrong, it's wierd
[15:51] <toad_> sure, but how big is the fetch set?
[15:51] <KenMan> complete
[15:52] <toad_> huh?
[15:52] <KenMan> i track every key in the net, plus counts of each key
[15:52] <toad_> ah
[15:52] * toad_ does not regard that as realistic :)
[15:52] <toad_> Success ratio on last 10K reqs: 0.92565 (1487 0s, 18513 1s, 20000 total)
[15:52] <KenMan> it costs a little more :(
[15:52] <toad_> e.g. if a lot of the keys are lost on insert, that won't show up in your sim psuccess numbers
[15:52] <toad_> Success ratio on last 10K reqs: 0.92235 (1553 0s, 18447 1s, 20000 total)
[15:53] <KenMan> no, when i perform a search, i know that key exists within the net
[15:53] <KenMan> for every single search. Is that what you mean ?
[15:53] <toad_> KenMan: I mean that it's not simulating push/pull tests; you can get 100% success and it's only because all but 1 key has fallen out
[15:54] <mmllaabb> does it work inkaffe?
[15:54] <KenMan> Oh, i get what you mean - you are measuring something different.
[15:54] <toad_> okay, the results so far suggest that at least on the present network, it will take roughly forever for the node to learn enough to drop nodes
[15:54] <toad_> even if we cut the threshold
[15:54] <toad_> mmllaabb: which?
[15:54] <toad_> mmllaabb: the simulation does, but Freenet itself does not
[15:54] <KenMan> i got one chicken riding on 7-8M request
[15:54] <mmllaabb> do you spite me
[15:54] <toad_> at present
[15:55] <mmllaabb> toad_, 1.5 years ago kaffe worked
[15:55] <mmllaabb> what happened
[15:55] <cbreak> spite?
[15:55] <toad_> mmllaabb: we implemented non-blocking I/O
[15:55] <KenMan> (szechuan chicken if you win)
[15:55] <toad_> the free JVMs have support for this, but it's very buggy
[15:55] <toad_> nobody has gotten around to fixing it yet
[15:55] <mmllaabb> could you fix it?
[15:55] <toad_> the only way to fix it is for me and dalibor to do it collaboratively
[15:55] <mmllaabb> being the excellent coder you are
[15:55] <toad_> sure, donate $500 and we'll do it
[15:56] <mmllaabb> no
[15:56] <toad_> otherwise it's not a priority
[15:56] <mmllaabb> so this is about money now
[15:56] <toad_> as I said, it's not a priority
[15:56] <toad_> Success ratio on last 10K reqs: 0.9173 (1654 0s, 18346 1s, 20000 total)
[15:56] <toad_> Success ratio on last 10K reqs: 0.92025 (1595 0s, 18405 1s, 20000 total)
[15:56] <mmllaabb> is freenet still uber slow?
[15:57] <mmllaabb> (it was fast for about a month last year)
[15:57] <toad_> moderately
[15:57] <cbreak> hmm... I got an unexpected exception...
[15:57] <cbreak> ---Unexpected Exception------------------
[15:57] <toad_> try it out, it's free ;)
[15:57] <cbreak> java.lang.IllegalArgumentException: null rp for freenet.client.inFlightRequestTrackingAutoRequester@1f271b:(not requesting)()
[15:57] <mmllaabb> has anything improved since last year
[15:57] <toad_> i think so
[15:57] <cbreak> mmllaabb: It certainly has.
[15:57] <mmllaabb> toad_, sorry sun and ibms VMs dont work on my machine
[15:57] <mmllaabb> they just fail
[15:57] <mmllaabb> but kaffe works
[15:57] <mmllaabb> its a hardware prob
[15:57] <mmllaabb> a strange one
[15:57] <toad_> mmllaabb: peculiar..
[15:57] <toad_> OS?
[15:58] <toad_> CPU?
[15:58] <mmllaabb> debian stable
[15:58] <mmllaabb> dual AMD-XPs
[15:58] <mmllaabb> 2Ghz
[15:58] <mmllaabb> on a tiger board
[15:58] <toad_> modded up to MPs, I presume?
[15:58] <toad_> I mean, they're cache coherent?
[15:58] <toad_> Success ratio on last 10K reqs: 0.91775 (1645 0s, 18355 1s, 20000 total)
[15:59] <mmllaabb> O
[15:59] <mmllaabb> yea they are MPs
[15:59] <mmllaabb> but XP and MPs are basically the same
[15:59] <toad_> hmmm
[15:59] <toad_> unfortunate
[15:59] * pittaman (~mistery_b@d51A410CA.kabel.telenet.be) has joined #freenet
[15:59] <mmllaabb> anyway only kaffe dosn't die on that box
[15:59] <toad_> what happens when you try to run the sun jre?
[15:59] <toad_> or blackdown?
[16:00] <mmllaabb> and my 2 250gb drives want to be filled
[16:00] <toad_> Success ratio on last 10K reqs: 0.9101 (1798 0s, 18202 1s, 20000 total)
[16:00] <mmllaabb> it errors stangely
[16:00] <mmllaabb> i had one of the devs come on the box
[16:00] <toad_> well it's one more argument for getting it working on kaffe
[16:00] <toad_> there are plenty
[16:00] <mmllaabb> and he couldnt figure it out
[16:00] <toad_> but there's a lot else that needs to be done too
[16:00] <mmllaabb> is 256 bit encryption in?
[16:00] <toad_> it segfaults or something?
[16:00] <mmllaabb> no
[16:01] <toad_> no, we still use 128 for symmetric IIRC
[16:01] <mmllaabb> ah
[16:01] <toad_> that hasn't been cracked but is looking a little shaky
[16:01] <mmllaabb> yea
[16:01] <toad_> AES and Twofish
[16:01] <mmllaabb> i use AES 256 inside DES3 for my ssh sessions
[16:01] <mmllaabb> tunnel inside of a tunnel
[16:02] <mmllaabb> got an automated tool for it if you want
[16:02] <mmllaabb> https://caethaver2.ath.cx/cat2/programs.html#SNFT
[16:03] <mmllaabb> toad_, are their any sym cyphers that do 512 yet?
[16:03] <toad_> i don't know that superencrypting is always better
[16:03] <mmllaabb> toad_, 3des and AES don't overlap
[16:03] <mmllaabb> i read the papers on the pros and cons of layering
[16:03] <cbreak> 3des is weak.
[16:03] <toad_> mmllaabb: it doesn't matter that much; 256 bits will not be brute forced. the problem is not brute force, it's algorithmic attacks.
[16:04] <mmllaabb> cbreak, as weak as any 128 cypher.. + some keyspace attacks I believe
[16:04] <mmllaabb> AES also has a keyspace attack :(
[16:04] <toad_> Success ratio on last 10K reqs: 0.909 (1820 0s, 18180 1s, 20000 total)
[16:04] <toad_> Success ratio on last 10K reqs: 0.91015 (1797 0s, 18203 1s, 20000 total)
[16:05] <cbreak> I think 3DES can only use 112 bits of a key due to some design flaws.
[16:05] <toad_> 256 bits will not be brute forced without quantum computers anyway
[16:05] <mmllaabb> but ssh dosn't give massive amounts of choice
[16:05] <mmllaabb> toad_, sym cyphers wont be brute forced by any computers i believe
[16:05] <mmllaabb> quantum would only do pub key killing
[16:06] <toad_> mmllaabb: why not?
[16:06] <mmllaabb> i don't know, just what i heard
[16:06] <KenMan> yeah, my quantum computer does both. Of course, it hasn't arrived yet, but my pre-order is placed.
[16:06] <toad_> pub key breaking is the more high profile app
[16:06] <toad_> Success ratio on last 10K reqs: 0.9085 (1830 0s, 18170 1s, 20000 total)
[16:07] <mmllaabb> i read many times that symmetrical cyphers wouldnt be broken tho
[16:07] <KenMan> still no conns ? how many you up to ? about 2.5M reqs ?
[16:07] <toad_> mmllaabb: from where?
[16:07] <toad_> mmllaabb: we don't really know what quantum computers can do yet
[16:07] <mmllaabb> toad_, any place they had a forum discussion on QCs
[16:07] <mmllaabb> but yea that's true
[16:08] <mmllaabb> there will probably be a way
[16:08] <toad_> yeah, and this was "I'm working on quantum computer algorithms and I'm a professional cryptographer, and I tell you quantum computers can't break symmetric crypto"... or was it "oh yeah, nobody ever said anything about breaking symmetric"...
[16:08] <toad_> ;)
[16:09] <KenMan> I'm gonna be real pissed if toad's conns start around 6.8M unique requests, and I have to buy him a szechuan chicken !!
[16:09] <toad_> Success ratio on last 10K reqs: 0.90785 (1843 0s, 18157 1s, 20000 total)
[16:09] <mmllaabb> it was the first one kinds
[16:09] <mmllaabb> like
[16:09] <toad_> and yes, still no connections
[16:09] <mmllaabb> "im studdying this and it dosn't look like..."
[16:09] <toad_> Success ratio on last 10K reqs: 0.9106 (1788 0s, 18212 1s, 20000 total)
[16:09] <toad_> maybe it'll stabilize at 91%?
[16:09] <toad_> brb
[16:10] <mmllaabb> btw
[16:10] <mmllaabb> defeats encryption:
[16:10] <mmllaabb> https://caethaver2.ath.cx/cat2/programs.html#MRAMSPY
[16:10] <mmllaabb> :P
[16:10] <mmllaabb> assuming the app ever decrpypts it
[16:14] <toad_> KenMan: the problem is the factor of 20
[16:14] <toad_> KenMan: if the distribution wasn't so distorted as I'm assuming, we wouldn't need the factor of 20, and it'd happen closer to #nodes threshold to #nodes threshold * 20
[16:15] <KenMan> yes, i understand
[16:15] <toad_> good
[16:15] <mmllaabb> ok so whats on freenet today
[16:15] <KenMan> I still bet on 7-8M
[16:15] <mmllaabb> remeber when i used to run that news site
[16:15] <toad_> still have 0 added conns...
[16:15] <mmllaabb> Binary42dot
[16:15] <KenMan> where are you at ? number of requests ?
[16:15] <mmllaabb> with it's working forums and all
[16:15] <toad_> Total inserts: 12800
[16:15] <toad_> Total requests: 319700
[16:16] <toad_> hmm, that's pretty low...
[16:16] <KenMan> yeah, in the meantime, I'm going to get my own chicken for dinner !
[16:16] <toad_> :)
[16:16] <toad_> well i'm going to have dinner
[16:16] <KenMan> bbl
[16:16] <toad_> bbl
[16:16] <toad_> Success ratio on last 10K reqs: 0.9187 (1626 0s, 18374 1s, 20000 total)
[16:16] <toad_> Success ratio on last 10K reqs: 0.91805 (1639 0s, 18361 1s, 20000 total)
[16:16] <toad_> Success ratio on last 10K reqs: 0.91485 (1703 0s, 18297 1s, 20000 total)
[16:17] <mmllaabb> toad_, whats the popular freesites now?
[16:17] <toad_> mmllaabb: depends
[16:17] <toad_> the ones on the gateway obviously are quite popular :)
[16:17] <toad_> bbl
[16:19] <mmllaabb> any other popular ones?
[16:19] <cbreak> the most popular use for freenet is frost.
[16:22] <mmllaabb> o
[16:22] <mmllaabb> btw
[16:22] <mmllaabb> where can i get specs on AMD-XP/MP machine code
[16:22] <mmllaabb> i want to write a couple of ditties in it
[16:23] <cbreak> AMD maybe?
[16:23] <mmllaabb> i looked
[16:24] <cbreak> asuming you still asked the question you didn't find anything :)
[16:25] <toad_> it's very very similar to intel machine code...
[16:28] <toad_> Connections added: 2, tried 12432, force tried 0, passed estimators: 2
[16:28] <toad_> yay
[16:28] <toad_> Total requests: 379700
[16:28] <toad_> ;)
[16:28] <mmllaabb> found something
[16:28] <mmllaabb> but i just want a look up table
[16:29] <mmllaabb> not a final theroum
[16:29] <mmllaabb> toad_, know anywhere i can get a lookup table
[16:29] <toad_> first connection added around 349,700 requests
[16:29] <toad_> mmllaabb: why do you want machine code? not assembler?
[16:29] <mmllaabb> just going to do simple things so i guess an intel machine code table would be fine
[16:29] <toad_> well, 3dnow is amd-specific
[16:29] <mmllaabb> toad_, want to try my had at some low level stuff
[16:29] <toad_> but i'm sure you can find the specs for that on AMD
[16:29] <mmllaabb> not doing 3d things
[16:30] <toad_> sure, but even so..
[16:30] * hobx_ (~chatzilla@ankh.math.chalmers.se) Quit (Remote closed the connection)
[16:30] <toad_> if you want to get THAT low level, write an assembler
[16:30] <mmllaabb> i found something but it dosnt have a look up table it seems
[16:30] <toad_> don't code directly in machine code!
[16:30] <mmllaabb> no i want lower
[16:30] <toad_> why?
[16:30] <mmllaabb> just to see if it works
[16:30] <mmllaabb> we were doing some on a VM
[16:30] <mmllaabb> it was easy
[16:30] <toad_> between 339,700 and 349,700 requests
[16:30] <toad_> well, VM p-code is SERIOUSLY RISC
[16:31] <toad_> whereas x86 is seriously CISC
[16:31] <toad_> KenMan: you'd owe me a chicken, if it wasn't for the fact that I never agreed to the bet ;)
[16:31] <mmllaabb> toad_, yea :) I want to try tho :P
[16:32] <toad_> in any case, it's getting a few connections here and there, and it's rising..
[16:32] <mmllaabb> just load and do a couple math ops
[16:32] <mmllaabb> and say i did it
[16:32] <cbreak> look at the source of gcc.
[16:32] <mmllaabb> but i cant find usefull lookup tables :(
[16:32] <cbreak> it should contain some kind of mapping.
[16:32] <toad_> cbreak: even gcc goes to assembler
[16:32] <toad_> get the source to gas or nasm
[16:32] <mmllaabb> does anyone know where a lookup table
[16:32] <cbreak> yes, but then it creates byte code :)
[16:33] <toad_> cbreak: huh?
[16:33] <cbreak> well, maybe not the app gcc, but the whole packet gcc :)
[16:33] * yonkel-cluster (~yonkeltro@pcp04665066pcs.wilog501.pa.comcast.net) has joined #freenet
[16:33] <toad_> hi yonkel-cluster
[16:33] <toad_> (I wonder why he's called that..)
[16:33] <toad_> maybe he's another AI supercluster, like people used to say I was...
[16:34] * greycat (~wooledg@192.35.79.70) Quit ("This time the bullet cold rocked ya / A yellow ribbon instead of a swastika")
[16:34] <mmllaabb> toad_, 386 code will work on amd right?
[16:35] <toad_> yup
[16:35] <toad_> even 8086 code in the right mode on the right OS
[16:36] <yonkel-cluster> toad: my normal nick is yonkeltron
[16:36] <yonkel-cluster> but i am speaking from my cluster
[16:36] <mmllaabb> toad_ do you know of a 386 lookup table?
[16:36] <toad_> not offhand, but the internet's a big place
[16:36] <toad_> yonkel-cluster: SSI or Beowulf?
[16:37] <mmllaabb> toad_, im seaching and cannot find
[16:37] <yonkel-cluster> toad_: gonna be openmosix as soon as i manage to recompile my kernels
[16:37] * yonkel-cluster goes off to fix his wlan conn
[16:37] <toad_> yonkel-cluster: only mosix? why not SSI...?
[16:39] <toad_> of course this is completely off topic :) bt
[16:39] * toad_ -bt
[16:42] <toad_> http://amphibian.dyndns.org/simulations2-epass-0.025-400x50.png
[16:42] <toad_> http://amphibian.dyndns.org/simulations2-epass-0.025-400x50-smooth.png
[16:42] <toad_> http://amphibian.dyndns.org/simulations2-epass-0.025-400x50-supersmooth.png
[16:42] <toad_> bbl
[17:17] * cbreak (~cbreak@dhcp-32-094.via-eth.ch) Quit ("leaving")
[17:22] * FallingBuzzard (~srademach@207.152.112.129) has left #freenet
[17:33] <KenMan> aahg - i owe toad one chicken !
[17:34] * Zorix (Brandon@fl-198-70-222-58.dyn.sprint-hsd.net) Quit (Read error: 238 (Connection timed out))
[17:39] * Usurp (~Usurp@trudaine-8-82-230-34-86.fbx.proxad.net) has joined #freenet
[17:41] * pittaman (~mistery_b@d51A410CA.kabel.telenet.be) Quit (Client Quit)
[17:43] <toad_> hi ppl
[17:48] * nextgens (~toto@d80-170-24-39.cust.tele2.fr) Quit ("bye")
[17:51] * toad__ (toad@82-32-18-233.cable.ubr03.azte.blueyonder.co.uk) has joined #freenet
[17:51] <toad__> grrr
[17:51] <toad__> other irc client crashed
[17:52] * yonkel-cluster (~yonkeltro@pcp04665066pcs.wilog501.pa.comcast.net) Quit (Client Quit)
[17:54] <toad__> hmmm
[17:54] <toad__> it's not going as well as i would like...
[17:54] <toad__> either there's some problem with the code still, or the change is a problem, or it just needs more time...
[17:59] <toad__> http://amphibian.dyndns.org/simulations2-epass-0.025-400x50-new.png
[17:59] <toad__> hrrrrm
[18:02] * plixed_ (~plixed@p5082343D.dip.t-dialin.net) has joined #freenet
[18:03] <KenMan> it may need a LOT of time, if you are dependent on those changing connections, no ??
[18:03] <toad__> on both systems, the native code fails to load..
[18:03] <KenMan> that sucks!
[18:04] <toad__> but it works fine on my node
[18:04] <toad__> hmmm
[18:04] <toad__> but not on my sim
[18:04] <toad__> and it's running significantly slower on the a64 3000+ than it does on my xp 2800+
[18:04] <KenMan> your bottom axis is total unique requests ? I thought you were sampling on a 25K interval, but there are more points than that.
[18:04] <toad__> which is strange
[18:05] <toad__> and which is irritating because i want to run things other than simulations on this
[18:05] <KenMan> it looks like you have data for every 5K events
[18:05] <toad__> hmmm
[18:05] <toad__> perhaps the scale is wrong?
[18:06] <toad__> yup, the scale is definitely wrong
[18:06] <KenMan> or even every 2500 requests
[18:06] <toad__> yes, indeed, it's plotting it as every 2500 requests
[18:08] <toad__> when it's actually every 10,000 requests
[18:08] <toad__> so multiply by a factor of 4
[18:08] <KenMan> this is almost a meaningless observation, but 400x50x22x100 yields an average redundancy of 20 copies per live key
[18:09] <KenMan> you are running at 10K now ? I thought it was 25K ?
[18:09] <toad__> that depends on how many keys you pump into it
[18:09] <KenMan> 1:100 inserts to retrieves, right !
[18:09] <toad__> 10,000 requests, 400 inserts, per cycle
[18:10] <KenMan> what about fetch-frac - is that related ?
[18:10] * KenMan is just double checking your numbers...
[18:10] <toad__> http://amphibian.dyndns.org/simulations2-epass-0.025-400x50-new.png
[18:10] <toad__> is that rising?
[18:10] <toad__> KenMan: fetchfrac is related, yes, of course
[18:11] <toad__> we only fetch the last fetchfrac*slots inserts
[18:11] <KenMan> it is too soon to say whether it is rising.
[18:11] <KenMan> the full effect of connection shifting is still quite a ways off
[18:12] <toad__> yeah, only 800K requests...
[18:12] * yonkel-cluster (~yonkeltro@pcp04665066pcs.wilog501.pa.comcast.net) has joined #freenet
[18:12] <toad__> connection shifting?
[18:12] <toad__> what do you mean by connection shifting?
[18:12] <KenMan> changing, 'if you like'
[18:12] <toad__> yeah
[18:12] <toad__> the concerning thing is that I got better results for the old code with these config options
[18:13] <toad__> there may still be more bugs, or it may be caused by the uber-low connection churn
[18:13] <KenMan> the primary difference is no new conns until all are 80 exp
[18:13] <toad__> that's not the only difference
[18:13] <toad__> there is major new code and there could easily be more bugs
[18:13] <KenMan> your charts used to make a quick dip and then rise, correct ?
[18:14] <KenMan> but you generally ran your sims for a much longer time as well, right ?
[18:14] <toad__> sorta
[18:14] <KenMan> in any case, the dip was quick, no wiggling about at the bottom point of the curve, before
[18:15] <mmllaabb> do daily sites work good
[18:15] <toad__> in context: http://amphibian.dyndns.org/simulations2-epass-0.025-400x50.png
[18:16] <toad__> mmllaabb: possibly
[18:16] <toad__> there are several daily sites that work reasonably
[18:16] <toad__> the ones on the Web Interface by default, for example
[18:16] <toad__> KenMan: I'm also concerned that it has not gotten rid of the wiggle
[18:16] <KenMan> i can't even make out the yellow line in there
[18:17] <toad__> KenMan: I expected these changes to get rid of the wiggle
[18:17] <toad__> KenMan: you're looking for dark blue
[18:17] <toad__> yellow is a distraction
[18:17] <toad__> really dark blue
[18:17] <KenMan> yeah, i thought blue was your current run
[18:17] <toad__> there's 3 blues on that graph...
[18:17] <toad__> xcodelocal...
[18:17] <KenMan> oh, i see dark blue now
[18:17] <toad__> xnewcodelocal in fact
[18:18] <toad__> well, it seems to be wiggling a lot...
[18:18] <toad__> far too much
[18:18] <KenMan> when did that start ?
[18:19] <toad__> after it stopped falling
[18:19] * plixed (~plixed@pD95512DA.dip.t-dialin.net) Quit (Read error: 110 (Connection timed out))
[18:19] <KenMan> no, i mean what sim change caused the wiggling to appear ?
[18:19] <toad__> they always wiggled
[18:19] <toad__> at least ever since estimator passing
[18:19] <KenMan> you've had it about a week, i thought
[18:19] <KenMan> i get it without estimator passing
[18:19] <toad__> I thought it was a product of the newbie stuff
[18:19] <toad__> but apparently not
[18:20] <KenMan> i think it is normal, the wiggle stays within a reasonable overall range to indicate the average value
[18:20] <toad__> the numbers seem to suggest that it'll just wiggle forever and never make any progress
[18:20] <toad__> quite possibly with ever increasing wiggles
[18:20] <toad__> and if it doesn't do that, it'll fall...
[18:21] <KenMan> it's +/- 2% here , with my <20% runs. I've run some of them way far out, and they don't really widen the wiggle any
[18:21] <toad__> so the reason it hasn't reached the levels of the previous simulations is simply that it adds less connections per cycle than they do?
[18:22] <KenMan> fwiw, 2 pictures with wiggling : http://mywebpages.comcast.net/jkcorson/FreeNet/1000x20x8x50.png and x100.png
[18:22] <KenMan> they are both run out to 200M requests
[18:23] * bluephile (bluephile@69-160-215-63.clvdoh.adelphia.net) has joined #freenet
[18:23] * toad__ checks an old one...
[18:23] <toad__> on average we were adding around 250 connections per cycle
[18:23] <KenMan> i think , if changing conns has an effect, you will need to run much further now to observe that effect. Possibly. 250 is an awful lot of changes.
[18:24] <KenMan> How many total connections do you think you had, when you saw 250/10Kreqs
[18:24] <toad__> 400*50 = 20,000
[18:24] <KenMan> (over 2)
[18:24] <toad__> yeah
[18:25] <KenMan> 2% changing per cycle, huh ? Maybe not so bad. I am working with around 20/10Kreqs
[18:26] <toad__> 51 cycles on the A64 3000+
[18:26] <toad__> 94 cycles on the XP 2800+
[18:26] <mmllaabb> omg
[18:26] <toad__> why?
[18:26] <mmllaabb> 20k
[18:26] <toad__> 20k what?
[18:26] <mmllaabb> connnections
[18:26] <mmllaabb> thats alot
[18:26] <toad__> it's a simulation
[18:26] <toad__> they're not real connections
[18:26] <mmllaabb> oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
[18:26] <KenMan> yeah, mmllaabb, you need to set your node to at least 20K connections to get anywhere ;)
[18:26] <mmllaabb> KenMan, thats how i remeber it ;)
[18:27] <toad__> also the periods it spends under 0.91 seem to be getting longer..
[18:27] <toad__> maybe it's still buggy... the announcements bug was a major problem though
[18:28] <toad__> I suppose I ought to find somewhere to run the same code minus the connection churn changes... compare that to the old results...
[18:29] <KenMan> http://mywebpages.comcast.net/jkcorson/FreeNet/400x50x22x100.png
[18:30] <KenMan> no announcements, no estpassing, approx 60 new cons/10K reqs
[18:30] <toad__> Not Found
[18:30] <toad__> The requested object does not exist on this server. The link you followed is either outdated, inaccurate, or the server has been instructed not to let you have it.
[18:30] <toad__> besides, that's different code
[18:31] <toad__> right?
[18:31] <KenMan> sorry, image there now
[18:31] <KenMan> well, it is a comparison, that is all.
[18:32] <KenMan> I can say with high confidence that it is different code. Not necessarily better or worse, just *different*
[18:32] * Usurp (~Usurp@trudaine-8-82-230-34-86.fbx.proxad.net) Quit (Remote closed the connection)
[18:32] <KenMan> I had to work it all around to get familiar with what it is doing :)
[18:33] * NeUtr0N (~wightghos@danyzone.net) has joined #freenet
[18:34] <toad__> KenMan: I have plenty of graphs with different code
[18:34] <KenMan> heh, me too !
[18:34] <toad__> the problem is I'm not sure whether this particular code works :<
[18:34] <toad__> and I know that it DID have some major bugs
[18:34] <toad__> and there probably are more
[18:36] <KenMan> I am content (almost intimate) with what my base is doing, so far. I believe I understand everything it is doing. That's why it took me this long just to try dynamic conns
[18:37] <KenMan> besides, you are looking for representative of a perfect 'real world' net. I just want to turn knobs and see how they affect things. It is all relative for me...
[18:37] * guido^pe (~unknown@dsl-213-023-246-188.arcor-ip.net) Quit (Remote closed the connection)
[18:38] <toad__> no, i want to see whether the code is completely fubarred by my recent refactoring, and thence whether my recent changes' simulations are valid or not
[18:38] * guido^pe (~unknown@dsl-213-023-246-188.arcor-ip.net) has joined #freenet
[18:38] <KenMan> Oh, well then you can compare it to me, sort of. I took your code about 4 weeks back, i think. before serialization anyway.
[18:38] <toad__> no, I have that too
[18:39] <toad__> I have graphs from the old code
[18:41] * toad__ runs a comparison graph on scabserver...
[18:41] <KenMan> I deleted my old graphs, after the last bug I found on the 30th.
[18:41] <toad__> uh oh..
[18:41] <toad__> it starts _VERY_ high
[18:45] <KenMan> well, just out of curiosity, what is your average successful HTL at 400x50x22x100 ??
[18:45] <KenMan> if anything, it should be lower than my '7' which is equivalent to your '8'
[18:46] <KenMan> (if the key gets found on the very first node, i count that as zero hops)
[18:46] <toad__> 6.4, so far
[18:46] <toad__> that's with only a million or so reqs though
[18:47] <KenMan> do you ever see the one-time dip like in my 200M graphs ? i've never seen that before.
[18:48] <KenMan> I attribute your shorter HTL's to your "add-ons" like announcements and est passing.
[18:48] <toad__> I can however tell you that lower HTL even slightly and you get significantly lower success
[18:49] <KenMan> hmmm, yes, i remember you trying to determine htl for netsize to reach a certain success %
[18:49] <toad__> and it was really stupid of me to run it on my xp 2800+ because while it's running i won't be able to get any non-simulation-related work done thanks to the stupid 2.6 scheduler
[18:49] <KenMan> reprioritizing don't help any ?
[18:49] <toad__> but that's only a problem because it runs at half the speed on the A64 3000+ as on the XP 2800+
[18:49] <toad__> which is REALLY surreal
[18:49] <toad__> KenMan: no
[18:50] <toad__> not enough
[18:50] <KenMan> Qwll, I'm working with a single 256M chip nowadays, thanks to these damned sims... ;)
[18:50] <KenMan> the heat killed my second module.
[18:50] <toad__> your computer is broken
[18:50] <toad__> don't blame the simulations
[18:50] <KenMan> it is fixed now... but only 256M :(
[18:51] <toad__> okay, i should go to bed
[18:51] <KenMan> seriously, it was going to happen whenever something bumped up the heat, by coincidence it was a sim. G night
[18:51] <toad__> will be back tomorrow around 14.00, maybe earlier, probably
[18:51] <hobx> whats wrong?
[18:51] <hobx> toad to bed at midnight?
[18:52] <KenMan> he needs to let the sims crank. "sims" not "Sims"
[18:52] <toad__> no, i have to get up for something or other
[18:52] <toad__> distinct from work
[18:52] <KenMan> A guy only says that when his gf has arranged something stupid for him. Otherwise you would remember what it was.
[18:52] <toad__> heh
[18:53] <toad__> that's faith
[18:53] <KenMan> Sometimes, I guess I remember "person on sunday" but not what I am doing for that person...
[18:53] <hobx> I think Toad has community service.
[18:53] * NeUtr0N (~wightghos@danyzone.net) Quit (Remote closed the connection)
[18:53] <toad__> the concept that i.. nevermind
[18:53] <KenMan> good night, drop some results for us here in the channel...
[18:53] <toad__> hobx: so what did I get caught for?
[18:54] * guido^pe (~unknown@dsl-213-023-246-188.arcor-ip.net) Quit (Remote closed the connection)
[18:54] <hobx> toad_: Last time you tried to get a gf!
[18:54] <KenMan> you are doing your little bro's community service ?
[18:54] <toad__> lol
[18:54] <hobx> cs is teaching gender relations to highschoolers
[18:54] <toad__> KenMan: i don't think they'd let me do that..
[18:54] <KenMan> but he certainly would, no ??
[18:55] <toad__> err would what?
[18:55] <KenMan> yeah, i'll never forget this one blond chick in one of my cs classes. She sat next to me, and rubbed one of her nipples for at least 5 minutes straight. It was extremely distracting, and entertaining too...
[18:56] * Zorix (Brandon@fl-198-70-222-58.dyn.sprint-hsd.net) has joined #freenet
[18:56] <hobx> that is the other cs, isn't it?
[18:56] <KenMan> She wasn't doing it for my benefit, of that I am sure. But there was no question what she was doing...
[18:56] <toad__> community service..
[18:56] <KenMan> oh, yeah. Nevermind.
[18:56] <hobx> she could have a fungul infection...
[18:56] <hobx> (/me is good at spoiling sexy stuff...)
[18:56] <KenMan> there were a lot of explanations, but all i knew was that I couldn't pay attention to anything else.
[18:57] <KenMan> There was nothing sexy about it, except that there was NO QUESTION what she was doing physically. Why, I'll never know. But it really took my attention away from the class.
[18:57] * Usurp (~Usurp@trudaine-8-82-230-34-86.fbx.proxad.net) has joined #freenet
[18:58] <hobx> I'm teaching a group with only men this semester.
[18:58] <toad__> heh
[18:58] <hobx> Stupid electrical engineers. I need to get back to chemistry.
[18:58] <toad__> they're onto you...
[18:58] <KenMan> oh god. Well, at least you can't see below their desks from the front of the room, right ?
[18:58] <hobx> ?
[18:58] <KenMan> i am kidding with you.
[18:58] <toad__> they've been hearing complaints...
[18:59] <hobx> KenMan: Well, it went way over my head and landed somewhere far away.
[18:59] <toad__> one too many threatened lawsuits..
[18:59] <hobx> perhaps
[18:59] <hobx> Except I don't see how it is sexism to offer women "oppertunities" I don't offer the men!
[19:00] <KenMan> that's what causes the threatened lawsuits 8)
[19:00] <toad__> okay, i'm outta here. please don't get incarcerated in your vain attempts to obtain a girlfriend, it's good to have you here occasionally
[19:00] <KenMan> toad and hobx on typing terms this week, must make a note of it.
[19:00] <hobx> well, incracerated at least one'll make a "friend"
[19:01] * patryk (~patryk@c-67-172-54-80.client.comcast.net) has joined #freenet
[19:01] <toad__> hobx: so go rob a bank
[19:01] <toad__> bbl zzz
[19:04] <patryk> Other than making sure my port is open on my firewall and the IP is correct, how can I diagnose problems? [Getting "Freenet could not find the data"]
[19:04] <mmllaabb> hobx, f u and ure bankz
[19:04] <KenMan> patryk: you should keep an eye on how many connections you have
[19:05] <KenMan> Getting "Freenet could not find the data" is fairly normal
[19:05] <patryk> KenMan: Ok -- when I netstat -tn | grep portNumber, I see 10 connections to my port
[19:05] <patryk> KenMan: Is that informative of something?
[19:06] <KenMan> cool, that shows how many incoming connections you have. There should be more on other ports for the outgoing ones...
[19:06] <mmllaabb> KenMan, what is your fav freesite ?
[19:06] <KenMan> i guess it would be CofE , although I haven't run a node for a while since I got a hold of toad's sim tool
[19:07] <KenMan> i think at least half of the content is only available through frost...
[19:07] <patryk> KenMan: Hmm, should there be outgoing ones to the *same* IPs as the incoming ones?
[19:07] <KenMan> generally , no.
[19:08] * lolo-laptop (~lostlogic@68.251.84.226) Quit ("Client exiting")
[19:08] <patryk> KenMan: Ok =) Just making sure. So, typically, after requesting a page, one gets "cannot find" and just lets it keep retrying?
[19:08] <KenMan> yeah. It happens sometimes.
[19:08] <patryk> Very good. Will keep at it
[19:31] <patryk> If I have downloaded 5099 today, is it worth it for me to download a new seednodes.ref? (Or just go with the bundled one?)
[19:34] * jepel_tailweaver (~jepeltw@CPE0004e28cd3c1-CM014340105960.cpe.net.cable.rogers.com) Quit (Remote closed the connection)
[19:43] * lostlogic (~lostlogic@node-4024215a.mdw.onnet.us.uu.net) has joined #freenet
[20:19] <KenMan> patryk: wait a day, see how many connections you get, and then decide whether to fetch a new seednodes file
[20:24] <KenMan> given how many peers you had earlier, you should be fine as is
[21:15] <patryk> KenMan: thanks for reply
[22:31] <Ash-Fox> http://kano.net/javabench/
[23:32] * Zorix (Brandon@fl-198-70-222-58.dyn.sprint-hsd.net) Quit (Read error: 110 (Connection timed out))
[23:37] * patryk (~patryk@c-67-172-54-80.client.comcast.net) Quit ("using sirc version 2.211+ssfe")
These logs were automatically created by Jay Oliveri with his gimp hapi on irc.freenode.net.
