Timestamps are in GMT/BST.
[0:00] <hirvox> nevermind
[0:07] * Zorix (Brandon@fl-64-45-226-120.dyn.sprint-hsd.net) Quit (Read error: 110 (Connection timed out))
[0:09] * goatee (~goatee@ip216-239-87-183.vif.net) has joined #freenet
[0:24] <KenMan> ahh, back to the relatively calm world of "stable FreeNet"
[0:25] <KenMan> now, is 5091 comparable to 5084 ? I think it pushes more queries, and finds more data, leaving the overall success rate about the same.
[0:26] <KenMan> it certainly manages to push (pull?) about 25% more total successes by volume through my node, but the overall query level fluctuates a bit more, and drifts higher, than 5084 ever did.
[0:27] <KenMan> i think overall it is comparable to 5084 - not a major gain, but also (importantly) not a major loss.
[0:27] <KenMan> can anyone argue to the contrary ? anecdotal evidence or personal perceived experience is welcome.
[0:34] * mikeDOTd_ (~mikedotd@plutonium.nexthop.net) has joined #freenet
[0:34] * mikeDOTd (~mikedotd@plutonium.nexthop.net) Quit (Read error: 54 (Connection reset by peer))
[0:34] * Hamled|Erp| (~hamled@pool-68-162-123-225.phil.east.verizon.net) has joined #freenet
[0:41] * Naked (naked@naked.iki.fi) has joined #freenet
[0:41] * Naked is now known as Hadaka
[0:52] * Hamled|Erp (~hamled@pool-68-162-123-225.phil.east.verizon.net) Quit (Read error: 110 (Connection timed out))
[1:01] * hirvox (~hirvox@cs181027153.pp.htv.fi) Quit (Read error: 54 (Connection reset by peer))
[1:03] * Zorix (Brandon@fl-64-45-226-120.dyn.sprint-hsd.net) has joined #freenet
[2:04] * Hamled|Erp (~hamled@pool-68-163-53-37.phil.east.verizon.net) has joined #freenet
[2:20] * Hamled|Erp| (~hamled@pool-68-162-123-225.phil.east.verizon.net) Quit (Read error: 110 (Connection timed out))
[2:23] * Zorix (Brandon@fl-64-45-226-120.dyn.sprint-hsd.net) Quit (Read error: 110 (Connection timed out))
[2:37] * galt is now known as galt-away
[2:50] * nassmoo (~chatzilla@217.167.208.229) has joined #freenet
[2:58] <mazzanet> toad_
[2:58] <mazzanet> KenMan
[2:58] <KenMan> mazzanet
[2:58] <KenMan> ozzy
[2:58] <mazzanet> we need to fix up the seednodes
[2:59] <KenMan> what's wrong wid them ?
[2:59] <mazzanet> seednodes.ref 10-Aug-2004 19:47 25.9M
[2:59] <mazzanet> 26mb seednodes isn't healthy
[2:59] <mazzanet> and breaks a lot of things
[2:59] <KenMan> yeah, well, no comment on that. Toad has written some code that can process unlimited size seednodes.ref files
[2:59] <KenMan> try out 5091
[3:00] <mazzanet> but think of the poor dialup users trying to update their seednodes
[3:00] <mazzanet> !
[3:00] <KenMan> can we really have 'too many seednodes' ?? Maybe it will encourage them to get a better connection, or stop hurting Freenet then.
[3:00] <mazzanet> lol
[3:01] <KenMan> i really can't imagine a dialup user helping the network in any way. I can imagine them hurting things. But, we should try to be inclusive,
[3:01] <KenMan> and I agree that the 25M file doesn't actually contain 25M worth of data
[3:01] * hirvox (~hirvox@cs181027153.pp.htv.fi) has joined #freenet
[3:02] <mazzanet> even on dsl, downloading 25M files is still a pain
[3:02] <KenMan> anyway, that is just par for the course with this 'research project' . Someday in the future, reducing the size of a seed node may become more important...
[3:02] <mazzanet> let alone trying to edit them in vi
[3:02] <KenMan> so trim them with the 'head' command. And feel sorry for those windows users.
[3:02] <mazzanet> heh
[3:03] <mazzanet> pity i just editted the windows installer to make it grab new seednodes each update
[3:03] <mazzanet> :P
[3:03] * Zorix (Brandon@fl-64-45-226-120.dyn.sprint-hsd.net) has joined #freenet
[3:04] <KenMan> aw, fignuts. I don't know, take it up with toad, he's head code-master, not me !!
[3:04] <KenMan> i think he was considering binary seeds which would reduce the size considerably...
[3:05] <KenMan> prod him on that, he's likely to go for it, if he is willing to spend the time there.
[3:05] <mazzanet> but then they wouldn't be really editable
[3:05] <KenMan> that's for sure. Something worth considering. What do you think about it all ? I suppose the script(s) that collect them could set a max size, no ?
[3:06] <KenMan> and make an effort to rotate different nodes in and out every 2 hours, or however often it runs...
[3:06] <mazzanet> feck
[3:06] <mazzanet> h*
[3:06] <KenMan> that might be the 'right' place to fix the whole issue. I don't know.
[3:07] <mazzanet> the current seednodes has 62 nodes running 5084
[3:07] <mazzanet> 56 running 5088, 54 running 5089
[3:07] <mazzanet> and 483 running 5090
[3:07] <mazzanet> and 2 running 5076
[3:07] <mazzanet> heh
[3:07] <KenMan> blecch. That happens, i guess. Well, it is nice to see so many that are up to date.
[3:08] <mazzanet> i get the feeling the ones running 5076 could be removed
[3:08] <mazzanet> as could be the 5084 ones..
[3:08] <KenMan> your first step is to figure out who maintains the harvester scripts, and see what they think.
[3:09] <KenMan> it's probably toad, although sanity might know something about it. Or someone else I can't even think of...
[3:29] * mazzanet (~mazzanet@mazzanet.registered) Quit (Read error: 54 (Connection reset by peer))
[3:29] * mazzanet (~mazzanet@mazzanet.registered) has joined #freenet
[3:47] * totatite (~bla@195.25.40.253) has joined #freenet
[3:47] <totatite> hi what are the new stuffs in 5091 compared to the 5090 ?
[3:48] * lostlogic (~lostlogic@node-402418b2.mdw.onnet.us.uu.net) has joined #freenet
[3:48] * lostlogic (~lostlogic@node-402418b2.mdw.onnet.us.uu.net) Quit (Read error: 104 (Connection reset by peer))
[3:54] * TheSeeker (~yeahright@user-24-214-134-203.knology.net) has joined #Freenet
[3:55] <TheSeeker> I haven't isntalled Freenet yet on my new laptop... but I installed XP SP2, and heard it has a 10 socket per port limitation... anyone have any evidence of this, or workarounds?
[3:56] <totatite> yes it is microsoft who made this limit
[3:56] <totatite> to limit the p2p software
[3:56] <totatite> cf www.ratiatum.com
[3:56] <totatite> where you have some stuff in register database
[3:56] <totatite> to update to unlimit this limit
[4:09] <TheSeeker> zzzz
[4:09] * TheSeeker (~yeahright@user-24-214-134-203.knology.net) Quit ()
[4:18] * moskau23 (~Miranda@dsl-213-023-250-039.arcor-ip.net) Quit (Read error: 110 (Connection timed out))
[4:44] <Iakin3> [10:15] <totatite> cf www.ratiatum.com
[4:44] <Iakin3> [10:15] <totatite> where you have some stuff in register database
[4:44] <Iakin3> Hmmmm... french
[4:44] <Iakin3> panique de p2p maybe?
[4:45] <Iakin3> http://www.ratiatum.com/p2p.php?article=1646
[4:46] <totatite> yes exactly
[4:46] <Iakin3> totatite: Care to send an email about it ti devl and support?
[4:47] <Iakin3> I am totally unskilled in french.. I cannot properly express what that article really says ;)
[4:47] * leex-sleeping is now known as leex-work
[4:47] <leex-work> Zzzz
[4:49] * moskau23 (~Miranda@dsl-213-023-250-039.arcor-ip.net) has joined #freenet
[4:51] * ninja_ (~im10ninja@3ffe:bc0:8000:0:0:0:0:3fab) Quit (Remote closed the connection)
[4:52] * ninja_ (~im10ninja@3ffe:bc0:8000:0:0:0:0:3fab) has joined #freenet
[5:10] * Orasis (~orasis@c-66-41-30-194.mn.client2.attbi.com) Quit (Remote closed the connection)
[5:18] * perexxx (~perexxx@a80-186-1-10.elisa-laajakaista.fi) has joined #freenet
[5:19] * perexxx (~perexxx@a80-186-1-10.elisa-laajakaista.fi) Quit (Client Quit)
[5:20] <leex-work> lots dodaing here
[5:47] * sanity (~ian@81-178-89-164.dsl.pipex.com) has left #freenet
[5:51] * hobx__ (~chatzilla@ankh.math.chalmers.se) has joined #freenet
[5:58] * Hory (~Miranda@81.196.25.110) has joined #FreeNet
[6:03] * tyesinclair (~tyesincla@170.252.64.1) has joined #freenet
[6:05] * nassmoo (~chatzilla@217.167.208.229) Quit ("ChatZilla 0.9.61 [Mozilla rv:1.7.1/20040707]")
[6:05] * tyesinclair (~tyesincla@170.252.64.1) has left #freenet
[6:18] * hobx__ (~chatzilla@ankh.math.chalmers.se) Quit (Remote closed the connection)
[6:21] * guido^pe (~unknown@dsl-082-082-158-185.arcor-ip.net) has joined #freenet
[6:27] <iip_i2p> <oOo> Use the englih article at http://forum.edonkey.com/viewtopic.php?t=68189 then ;)
[6:30] * spaetz_ (~spaetz@217-162-197-136.dclient.hispeed.ch) has joined #freenet
[6:35] <Iakin3> toad_: Would it be possible to remove the MuxTrailerReadManager.consumed(TrailerChunkMessage) method? I dont see the value of it really..
[6:35] * spaetz (~spaetz@217-162-197-136.dclient.hispeed.ch) Quit (Read error: 110 (Connection timed out))
[6:57] * ninja_ (~im10ninja@3ffe:bc0:8000:0:0:0:0:3fab) Quit (Remote closed the connection)
[6:58] * ninja_ (~im10ninja@3ffe:bc0:8000:0:0:0:0:3fab) has joined #freenet
[7:16] * sanity (~ian@81-178-89-164.dsl.pipex.com) has joined #freenet
[7:16] * ChanServ sets mode +o sanity
[7:35] * Zorix (Brandon@fl-64-45-226-120.dyn.sprint-hsd.net) Quit (Read error: 110 (Connection timed out))
[7:36] * totatite (~bla@195.25.40.253) Quit (Remote closed the connection)
[7:47] * nassmoo (~chatzilla@217.167.208.229) has joined #freenet
[8:06] * hobx_ (~chatzilla@ankh.math.chalmers.se) Quit (Remote closed the connection)
[8:06] * hobx_ (~chatzilla@ankh.math.chalmers.se) has joined #freenet
[8:06] <hobx_> flooop
[8:17] <toad_> hobx_: ian's response on tech:
[8:17] <toad_> Actually I have discussed this with Dingledine, just last week in-fact.
[8:17] <toad_> Perhaps it was just because he was drunk, but the general consensus
[8:17] <toad_> seemed to be that social engineering attacks where the weakest link in
[8:18] <toad_> the chain, implying that there is little point in inconveniencing users
[8:18] <toad_> by strengthening other links in the chain.
[8:18] <toad_> >anyone can shed a light on this? Was he a devl, why does he make those
[8:18] <toad_> >claims if he is...or is he just a troll?
[8:18] <toad_> The two are not mutually exclusive :-)
[8:19] <hobx_> Sure, one can argue that one can social engineer anyways.
[8:20] <toad_> :)
[8:20] <hobx_> But you know I'm right when I say it is probably possible to track readers of sites, and possibly even publishers.
[8:20] <toad_> I admit that at present there are many things that can be improved on freenet w.r.t. anonymity
[8:21] <toad_> however I'm not convinced that it's that bad, with most threat models
[8:22] <toad_> readers of sites, and splitfiles - maybe, if you're lucky
[8:22] <toad_> if you happen to be the first node in the chain, and you see an abnormally large number of reqs from a single splitfile
[8:23] <toad_> at max htl (which btw it stays at for the first 2 hops on average)
[8:23] <hobx_> Anyways, I covered this with Newsbyte here yesterday.
[8:23] <hobx_> Before we got into space travel
[8:23] <toad_> my log doesn't go back that far :)
[8:24] <toad_> hobx_: so how do you track down requestors?
[8:24] <toad_> I can see how you might get lucky
[8:24] * Iakin3 (~nb@212.105.104.162) Quit (Read error: 110 (Connection timed out))
[8:25] <hobx_> I think just fishing is probably enough. Run several nodes and try to effect your specialisation for better effect.
[8:25] <toad_> try to make them all specialize on TargetKey?
[8:25] <hobx_> Look for queries for the key with high HTL (I never bought the whole "possible deniability" thing. It is easy to make the risk of false positive low enough to satisfy even a fair court (and those are not typically the problem))
[8:26] <toad_> well, making the risk of false positives low enough means correlating multiple reqs from a splitfile, right?
[8:26] <hobx_> toad_: I don't know how easy that is, either before or under NG routing. I wanted to believe it was difficult, but I don't any more.
[8:27] <hobx_> that is one method.
[8:27] <toad_> hobx_: you don't believe anything any more. i'm interested in arguments not feelings.
[8:27] <toad_> hobx_: gimme another
[8:27] <hobx_> Just queries two or three of HTL 25 (or whatever you start at) should be enough.
[8:27] * toad_ wonders if pcaching increases our vulnerability...
[8:28] <hobx_> toad_: I don't need to give evidence for a negative.
[8:28] <toad_> hobx_: the maximum is 20, however a) we perturb it slightly at the start, and b) if the htl is the max there's a 50
[8:28] <toad_> % chance of not decrementing it
[8:28] <hobx_> It is up to you to prove that Freenet works.
[8:28] <toad_> so on average we have 2 extra hops
[8:29] * ninja_ (~im10ninja@3ffe:bc0:8000:0:0:0:0:3fab) Quit (Connection timed out)
[8:29] <hobx_> toad_: Actually that adds one more.
[8:29] <toad_> hobx_: do you think that pcaching without a client cache increases our vulnerability?
[8:29] <toad_> hobx_: okay, one more hop
[8:29] <toad_> either way it's an improvement
[8:30] <hobx_> But I think in practice it matters very little. if you can do it once, you can do it twice, and three times, and by repetition eliminate the risk of false positive.
[8:30] <toad_> hobx_: by repetition?
[8:30] <hobx_> toad: Don't know for sure and won't share my feelings :-)
[8:31] <toad_> what do you mean? repeated requests from the same node?
[8:31] <hobx_> toad_: FIshed once, can fish again.
[8:31] <toad_> repeated requests from the same node to different fishing nets on the same specialization?
[8:32] <toad_> is that the idea?
[8:32] <hobx_> Somebody publishes a site every day for a year. By fishing the whole time I find that 20 times during the year I believe that it was Alice, but there is a 30% risk of false positive.
[8:32] <hobx_> But (1-.30)^20 == negligable
[8:33] <toad_> okay, so you adjust your spec so you are The Node for the key it happens to land on, each day
[8:33] <toad_> or you run a LOT of nodes
[8:34] <toad_> either way, you try to catch the insert at low htl
[8:34] <hobx_> I would say there are two ways around this: 1) Add onion routing steps before routing. 2) Get rid of HTL and have a probability that the request terminates at every step instead (say 5%). This would mean that some things would end too early, which ties in with my "don't try to hard" ideal for how things should be.
[8:35] <toad_> i can tell you that that probably won't work on the present network because routing is so poor that you'll rarely see it at min htl :)
[8:35] <hobx_> toad: You mean high HTL, but yeah.
[8:35] <toad_> max htl yeah
[8:35] <toad_> onion routing steps before routing are a good idea
[8:36] <hobx_> Yes. TOR only uses three steps, so it doesn't have to be that bad.
[8:36] <toad_> probabilistic termination is an interesting one, it allows requests to run indefinitely...
[8:36] <toad_> there will be some practical difficulties e.g. where do you get the onion intermediaries from
[8:36] <toad_> but they will be solved
[8:36] <hobx_> And the fact that freenet is full of other traffic helps against traffic analysis (note that this needs work too - we pad data to 2^n but not messages (including the storable fields, which I fear gives away data (especially if you plant data with stuff in the storable fields to give it unique size))
[8:37] <toad_> hobx_: we pad messages too now
[8:37] <toad_> fixed that a couple weeks ago
[8:37] <toad_> packets are now multiples of 160 bytes
[8:37] <hobx_> good, that is something.
[8:38] <toad_> we were vulnerable re using really small chain IDs
[8:38] <toad_> also we were really vulnerable to detection of the MRI, because we sent it as a float in text form on every message!
[8:38] <toad_> and a common value was 600,000
[8:38] <hobx_> Unless it got lost some time the code on one end was already there IIRC. I added code to swallow nulls between messages when we considered using block cipher modes.
[8:38] <toad_> precisely 600,000
[8:39] <toad_> hobx_: yeah, i did it differently after muxing
[8:39] <toad_> but there's always been the possibility and intention
[8:39] <toad_> I just send a void low level message, just swallow it... 4 bytes header (type, length), the rest junk
[8:39] <hobx_> Well, padding the messages was good. That always nagged me.
[8:39] * toad_ did some traffic analysis with an ethernet tapping tool and got really scared :)
[8:40] <toad_> traffic analysis is also too easy because of variable file sizes
[8:40] <toad_> we should use fixed file sizes
[8:40] <toad_> we will, there are tons of advantages
[8:40] <toad_> soon, but not immediately
[8:41] <toad_> it'll need a content reset, so we need to get the users in a slightly better mood first, but not put it off until we have a lot more of them
[8:41] <hobx_> I came up with an idea yesterday for a simple "semi-structured" model for DHT type overheads.
[8:41] <hobx_> s/overheads/overlays/
[8:41] <toad_> hobx_: hmm?
[8:41] <toad_> brb
[8:42] <hobx_> I can calculate a nice logarithmic concentration bound on the number of steps, and it is a little different than earlier approaches.
[8:43] <toad_> hobx_: hmm?
[8:43] <hobx_> hmm, hmm?
[8:44] <toad_> oh, re padding... there's one message that takes two chunks, and StoreData is _enormous_, it takes 34kB, because we include the estimators
[8:44] <toad_> we're going to do binary presentation though
[8:44] <hobx_> I think one could probably identify storedata by timing anyways
[8:44] <toad_> hobx_: perhaps so
[8:44] <toad_> hobx_: you were going to tell me about what you came up with yesterday?
[8:45] <toad_> hobx_: then you were going to convince me to get rid of HTL?
[8:45] <hobx_> Hey, maybe my pluggable layers will come to use... Though I suppose you people already ran all over that.
[8:45] <toad_> pluggable layers?
[8:46] <hobx_> toad_: Yes, the way it was written binary presentation could have been implemented with a single class, and the old presentation could have been kept around but given lower priority.
[8:46] <toad_> ah
[8:46] <toad_> yes
[8:46] <toad_> we do plan to use that
[8:46] <toad_> thelema put some of the compression code in FieldSet, which imho is a little blech
[8:47] <toad_> but yes we have an FNCRawMessage and so on
[8:47] <toad_> we also supported mux and non-mux simultaneously for a while
[8:47] <toad_> via different Protocol's
[8:47] <hobx_> I don't remember how it worked. Not as good as it should have IIRC because they wouldn't let me define datatypes.
[8:48] <toad_> huh?
[8:48] <hobx_> Like I said, I don't remember. But there was a lot of debate about those things.
[8:48] <toad_> hobx_: okay, if we abolish the HTL: a) What does it do to inserts? b) It will make some requests really short. We simply retry. b) It will make some requests really long. This is a PITA.
[8:49] <toad_> really short inserts would be a problem
[8:49] <hobx_> the second b) is not a problem
[8:49] <toad_> b) is not a problem, agreed
[8:49] <hobx_> exponential walls and all that.
[8:49] <toad_> what's an exponential wall?
[8:50] <toad_> uhh
[8:50] <toad_> the first b) is not a problem
[8:50] <toad_> why is the second b) (supposed to be c) ) not a problem?
[8:51] <hobx_> Because an accordingly small number of messages will live very long.
[8:51] <hobx_> Only the average matters for the traffic level on the network.
[8:51] <toad_> yes but we need to deal with them for timeouts etc
[8:52] <hobx_> Yeah, throwing out timeouts would be part of it.
[8:52] <toad_> well, NGRouting is another issue - it sees that some requests take crazy times, this is unwelcome noise
[8:52] <toad_> but it would average out
[8:53] <hobx_> You can still calculate timeouts, I can do the math to get a 95% confidence interval on the length of time a not-lost query takes.
[8:53] * cehteh (foobar@cehteh.homeunix.org) has joined #freenet
[8:53] <toad_> right, we just do the timeout at 2 std deviations
[8:53] * cehteh (foobar@cehteh.homeunix.org) has left #freenet
[8:54] <hobx_> But you couldn't really have timeouts at every node, since by it's nature the expected survival time of the message at node n would be the same as that at node n+1
[8:54] <toad_> of course that means an RNF, and we're trying to prevent such things :|
[8:54] <hobx_> So the first node would time out first instead of the last.
[8:55] <toad_> well yeah but we have to stop it at some point if only because it uses up memory
[8:55] * greycat (~wooledg@192.35.79.70) has joined #freenet
[8:55] <toad_> also the client might like a response at some point so it can retry
[8:55] <hobx_> stop what?
[8:55] <toad_> the request
[8:56] <hobx_> But my opinion for a long time has been that you should move all restarts to the client. (This would make your life easier)
[8:56] <toad_> hobx_: perhaps so
[8:56] <toad_> I suppose we should RNF when we get such a timeout
[8:56] <hobx_> Part of the "Don't try so hard!" initiative.
[8:56] <toad_> I don't like RNFs, but sometimes we have to.. and it should be rare
[8:57] <toad_> hobx_: well, imho if we just abolish retries on RNF now, it would reduce traffic, but it would also mean most queries RNFed
[8:57] <toad_> hobx_: I'd consider it after we've implemented queueing though
[8:57] <toad_> we did a little (pathetic) simulation
[8:57] <toad_> we assumed we had two routes
[8:57] <toad_> 1 could accept 1 request every 10 seconds
[8:57] <toad_> the other could accept 1 every 30
[8:58] <toad_> the second was preferred
[8:58] <toad_> and we had 8 requests per minute incoming at random times
[8:58] * Zorix (Brandon@fl-64-45-226-120.dyn.sprint-hsd.net) has joined #freenet
[8:58] <toad_> the sim said on average we will get 25% to 35% instant RNFs (due to neither node being reachable)
[8:58] <toad_> we are therefore working on queueing
[8:59] * toad_ thinks queueing is the other half of rate limiting
[8:59] <toad_> without which it sucks, but not quite as badly as not having rate limiting :)
[8:59] <toad_> after queueing, I'll consider not retrying on RNFs
[8:59] <hobx_> so what is queuing?
[9:00] <toad_> hobx_: okay, first, rate limiting: a node tells us how often we can send it requests. this is the Minimum Request Interval. this is our primary means of controlling load. we generally accept 90%+ because of it.
[9:01] <toad_> the result of this is that at any one time a lot of nodes are "backed off" i.e. we've sent a request to them and we can't send another for some period of time
[9:01] <toad_> but when we have a request to route we have to send it immediately, without delay - so we can only choose between nodes that aren't backed off
[9:01] <toad_> often there aren't any
[9:02] <toad_> the average backoff is 90% on KenMan's node
[9:02] * sanity (~ian@81-178-89-164.dsl.pipex.com) Quit (Read error: 104 (Connection reset by peer))
[9:02] <toad_> we think we can reduce this by queueing
[9:02] <toad_> and we KNOW we can reduce the number of queries that instantly RNF by queueing
[9:02] * sanity (~ian@81-178-89-164.dsl.pipex.com) has joined #freenet
[9:02] * ChanServ sets mode +o sanity
[9:02] <toad_> queueing: keep a request for a while, say 10 seconds, so we can try to send it to a better node, or to send it to a node at all
[9:02] <hobx_> Because rate limiting does not fix the problem as much as handle it more gracefully.
[9:03] <toad_> hobx_: make any sense?
[9:03] <toad_> hobx_: well, we've tried all manner of approaches
[9:03] <hobx_> I mean, reject means that the prefered path is missed, and so does never routing to it because it backed off.
[9:03] <toad_> I don't think that QueryRejected's and probabilities of QRs are any better
[9:04] <hobx_> I had some hope that perhaps if nodes didn't spend so much time going "not now I'm busy" they might get more work done.
[9:04] <hobx_> But I think we can safely say then that this is not the case.
[9:04] <toad_> hobx_: quick QR Reason=No route found usually means we don't have any non-backed off routes to send it to
[9:04] <toad_> no, I think QR is pretty nasty
[9:04] <toad_> even QR reason=node overloaded
[9:04] <toad_> rate limiting is quite nice
[9:05] <toad_> the fundamental problem is one of load versus routing, how to make them combine instead of fighting
[9:05] <toad_> and we've struggled with that ever since freenet started
[9:05] <hobx_> But the root problem is: we can't route where we want because those nodes are too busy.
[9:05] <toad_> indeed
[9:05] <toad_> part of the reason is that we retry on RNFs
[9:05] <toad_> it's self-generating
[9:06] <toad_> we RNF so we retry so we RNF...
[9:06] <hobx_> I believe the reason for this problem is clear as day and shows up perfectly in both my and Theo's simulation results.
[9:06] <toad_> what is it?
[9:06] <hobx_> The mesh created by the freenet algorithm is power law.
[9:07] <toad_> of course
[9:07] <hobx_> Meaning that most (used) edges go to a small number of nodes.
[9:07] <toad_> it's not going to work otherwise
[9:07] <toad_> oh you mean...
[9:07] <toad_> ah
[9:07] <toad_> I thought you meant one node's links are power law
[9:08] <hobx_> no.
[9:08] <toad_> this is the part that was carefully left out of the papers so as to get hacking quickly?
[9:08] <hobx_> Go back and look at Theo's sims. 90% of the nodes did no work.
[9:08] <hobx_> toad_: I thought load balancing would fix it.
[9:08] <toad_> hobx_: it might yet
[9:08] <toad_> it's just hard
[9:09] <hobx_> yes, but you want to why I became so dissillusioned?
[9:09] <toad_> why?
[9:10] * lolo-laptop (~lostlogic@68.251.84.226) has joined #freenet
[9:10] <hobx_> Random walks with caching in graphs with this degree distribution display the exact log scabability of step length we were seeing.
[9:11] <hobx_> When I went back, I realized that all the simulation results could have been due to this only and that I had no reason to believe Freenet beats completely random routing.
[9:11] <toad_> I thought you said it wasn't logarithmic, it was polynomial?
[9:11] <toad_> O(n^0.28)?
[9:11] <hobx_> They are hard to differentiate with limited data.
[9:12] <toad_> so you accept it could be log?
[9:12] <toad_> in which case, why does it work BETTER with LESS caching, according to those very same simulations?
[9:12] <hobx_> But either way (with random walks on power law graphs it depends on the degree distribution - at 1/k^2 it is log, below that sublinear)
[9:12] <toad_> that's the flaw... random routing ought to work better with MORE caching...
[9:12] <hobx_> toad_: It worked better with less caching because of the LRU queues of course.
[9:13] <toad_> what's k?
[9:13] <toad_> hobx_: huh?
[9:13] <hobx_> Number of neighbors.
[9:13] <hobx_> So the probability would be M to have one M/4 to have two M/9 to have three, etc.
[9:14] <hobx_> For some constant M
[9:14] <toad_> if you have N nodes, and you want to find a document, and it's only on 1 of them, then you probably won't find it with random routing
[9:14] <hobx_> (M is simply a normalizer since probability distributions have to sum to 1)
[9:15] <hobx_> toad_: No, but if the degree distribution is sufficiently skewed, the probability that two paths (insert and request) will intersect is good.
[9:15] <toad_> less caching => random routing is less likely to find the data, right?
[9:15] <toad_> hobx_: hmm, because you cache near the core nodes?
[9:15] <toad_> yeah... hmmm
[9:15] <hobx_> not if more caching => data gets lost
[9:15] <toad_> but it'll get evicted from them fast
[9:16] <toad_> doesn't more data get lost if we have less caching?
[9:16] <hobx_> I mean, I'm not saying that Freenet is random walking.
[9:16] * Iakin4 (~nb@212.105.104.162) has joined #freenet
[9:16] <hobx_> Maybe it almost is IRL because of overload, but the sims didn't.
[9:16] <toad_> if it's only cached on one core node, then it's only findable as long as it's on that node
[9:16] <toad_> which isn't very long
[9:16] <hobx_> The nodes definitely specialized.
[9:17] <toad_> hobx_: okay, so it wasn't a random walk, it was just indistinguishable from one?
[9:17] <hobx_> I'm just saying, that when you take the load distribution into account, the results weren't meaningful.
[9:17] * sanity (~ian@81-178-89-164.dsl.pipex.com) Quit (Read error: 54 (Connection reset by peer))
[9:17] <hobx_> toad_: It just didn't perform noticibly better.
[9:17] <toad_> well, I've been of the opinion that the simulation results leave a lot to be desired for some time
[9:17] * sanity (~ian@81-178-89-164.dsl.pipex.com) has joined #freenet
[9:17] * ChanServ sets mode +o sanity
[9:17] <toad_> but what do you expect with 1 node announcement, 1 insert, for every 5 requests? that simply isn't realistic
[9:18] <toad_> it's not realistic in PROPORTION
[9:18] <hobx_> My mistake was I never implemented load balancing in the simulator.
[9:18] <hobx_> Using whatever method possible, force the load distribution to be flat.
[9:18] <hobx_> Then see if still routes.
[9:18] <toad_> queueing will have an element of sending the best request to the best node, even if you have to wait a little, I'm hoping this will help
[9:19] <toad_> hobx_: hmmm
[9:19] <Iakin> [14:58] <toad_> it'll need a content reset, so we need to get the users in a slightly better mood first, but not put it off until we have a lot more of them
[9:20] <Iakin> Why not implement it for the insertion part of the code first, then ~6 months later remove the variable-size support from the request code
[9:20] <Iakin> ?
[9:20] <toad_> hobx_: so you propose that I resurrect Serapis, run it, see what happens, adapt it to take into account rate limiting or something, and see what happens?
[9:20] <toad_> Iakin: because removing the support would in itself require a reset of routing
[9:20] <hobx_> nah
[9:21] <hobx_> Serapis should stay dead
[9:21] <toad_> no?
[9:21] <hobx_> I started a rewrite
[9:21] <hobx_> It was too low level.
[9:21] <toad_> in what sense?
[9:22] <toad_> did you ever try running serapis with 1 node announce : 5 inserts : 25 requests?
[9:22] <hobx_> It attempted to simulate stuff like network traffic and interlocked queries and stuff.
[9:22] <toad_> or some other reasonably realistic traffic level?
[9:22] <hobx_> I think we have more requests per node than that.
[9:22] <hobx_> But not the thousands you have in RL.
[9:23] <toad_> i think the simulations used something silly
[9:23] <toad_> 1:1:1 or 1:1:5
[9:23] <toad_> according to the papers...
[9:24] <hobx_> The best simulations were probably the ones I did for presentation at O'Reilly in Washington
[9:24] <toad_> hobx_: what you are saying is that we don't need to simulate individual messages, right?
[9:24] <hobx_> That data is all lost though.
[9:24] <toad_> hobx_: why?!
[9:24] <hobx_> toad_: Theo was right with the high level sim. Simulate one queury at a time, run the whole query and then the next.
[9:25] <hobx_> No passing around message objects.
[9:25] <toad_> ahhh
[9:25] <hobx_> why?! what?
[9:25] <toad_> so how do you estimate bandwidth load in that case?
[9:26] <toad_> hobx_: why was the data lost?
[9:26] <hobx_> Use quotas, and deduct when it passes messages.
[9:26] <toad_> or data
[9:27] <hobx_> When I came back from the conference real life invaded and I never wrote the paper I intended.
[9:27] <toad_> heh
[9:27] <hobx_> Since then, well, bitrot you know.
[9:27] <toad_> IT'S ALL YOUR FAULT ! :)
[9:28] <toad_> hobx_: if freenet routing can't work, how come the postal experiment worked? that's the same principle... admittedly it had zero load...
[9:29] * toad_ sees freenet more as a superposition of lots of power law networks than as one power law network...
[9:29] <hobx_> You mean small world thing?
[9:30] <toad_> the "get this package to Mr XYZ, in state X, address X, only forwarding it via your friends"
[9:30] <toad_> thing
[9:30] <hobx_> I read an interesting paper about that just yesterday.
[9:30] <toad_> was he arguing it was random routing on a power law network?
[9:30] <toad_> or that it was a fluke?
[9:31] <hobx_> No
[9:31] <toad_> nah, it wasn't a fluke, it was repeatable
[9:31] <hobx_> but it isn't the same problem as freenet
[9:31] <hobx_> The people knew the package was going to new york
[9:31] <toad_> well, it's routing to a location, not a key which is essentially location independant...
[9:32] <toad_> but that IS similar to "route this to nodes good near the key"
[9:32] <hobx_> There is a paper from 2000 that I read yesterday
[9:32] <hobx_> He showed that if you consider a rectangular grid
[9:33] <hobx_> and you want to route from one point to another, then you can get there in log n steps if you have the right friend distribution
[9:33] <toad_> eh?
[9:33] <toad_> if you want to get from one end of the grid to the other in steps of unit length it will take O(n)
[9:33] <hobx_> If your friends are uniformly distributed on the grid (equal probability that they are anywhere) then you try to get the letter somewhere, it bounces back and forth and it takes many steps to find the way.
[9:34] <toad_> ohhh
[9:34] <toad_> I see
[9:34] <toad_> your friends are not your neighbours
[9:34] <toad_> ok
[9:34] <hobx_> If your friends are all nearby on the grid (for instance the probability that you have a friend goes down exponentially with distance) then the letter doesn't get anywhere, and it takes many steps
[9:35] <hobx_> The paper showed that if you have the right distribution - which he shows is that the probability of being friends with somebody is proporsional to 1/k^2 of the distance - then you can send the package in log n steps by simply sending it to the friend who is nearest.
[9:35] <toad_> okay. analogy with freenet: I expect that we will have a number of nodes close to our specialization, a number of nodes scattered over the keyspace (with some transition maybe), and a number of newbie nodes we're not sure about. this should satisfy the power law criteria - a few long haul jumps, a lot of close links - right?
[9:35] <hobx_> It is by Kleinberg, here: http://www.cs.cornell.edu/home/kleinber/swn.pdf
[9:36] <hobx_> I was very surprised that I had never heard of it before.
[9:37] <toad_> at least we would if we ever got to route anything close to our first choice :)
[9:37] <toad_> we may be able to ensure backoff of 50% or less with queueing...
[9:37] <toad_> hobx_: what is wrong with my analogy?
[9:38] <hobx_> The problem is that it is handwaving.
[9:38] <toad_> it is?
[9:38] <hobx_> We have to evidence that the graph freenet produces really meets the criteria.
[9:39] <toad_> well, if we get request specialization, then it should
[9:39] <hobx_> "It should have some short and some long links" isn't really sufficient. And remember that freenet uses numerical distance not grid distance, so that changes things as well.
[9:39] <hobx_> toad_: No, you have specialization and simply end up wrong.
[9:39] <toad_> hobx_: huh?
[9:39] <toad_> a) we don't generally have specialization
[9:39] <toad_> b) what do you mean by "simply end up wrong" ?
[9:40] <hobx_> And note that the Kleinberg model has a predermined "landscape" (a grid). Freenet doesn't - I have always worried that freenet develops competing clusters for the same keyspace.
[9:40] <hobx_> I meant "You can have specialization and still end up wrong"
[9:41] <toad_> it may come down to the connection ordering strategy...
[9:43] <toad_> well IMHO one of the fundamental problems from day 1 has always been load versus routing, and I think the logical next step is to implement queueing, so that is precisely what I am going to do
[9:45] <toad_> I have to work on the assumption that it CAN work
[9:46] <toad_> bbiab
[9:47] <hobx_> Yeah, but perhaps you see my point when I say that the priority ought to be showing that it CAN work in an ideal environment, instead of working in a less than ideal one where that becomes an unfalsifiable assumption.
[9:48] <toad_> if, having implemented queueing, fixed size files, and local traffic management, it still doesn't work...
[9:48] <toad_> we'll probably build a testnet :)
[9:48] <toad_> hobx_: well, I can see that, however I'm not an academic
[9:48] <hobx_> When I say that, you always respond that an ideal environment can't prove that it will work IRL. But it can prove that it won't...
[9:48] <toad_> it's not my job to prove that it can work
[9:49] <toad_> and I'm not sure the simulations can prove that it can't work either
[9:49] <hobx_> Well, I'm going to write a new simulator to run this idea I had yesterday.
[9:49] <toad_> what idea?
[9:49] <hobx_> With some calculations and sim results there is a enough for a paper there. Nothing seminal, but still.
[9:50] * orasis (~orasis@c-66-41-30-194.mn.client2.attbi.com) has joined #freenet
[9:50] <toad_> well okay, what's the scenario here? I prove that it probably can't work by simulation, Ian blindly continues in faith, I go fork and build something else, and try to get the money to follow me?
[9:51] <hobx_> toad_: Basically relaxing the type of routing they use in DHT systems and walking more or less randomly most of the time. It can still get logarithmic times, and I can even concentration bound it (meaning giving an exponentially small probability that it takes much more time than expected).
[9:51] <toad_> freenet is sexy precisely because you only need local knowledge...
[9:51] <hobx_> Well, if I write a sim for this, I'll make it modular.
[9:51] <hobx_> So that freenet type logic can be plugged in.
[9:51] <toad_> classical DHTs need more global knowledge than freenet
[9:52] <toad_> e.g. I2P
[9:52] <toad_> there are many situations where global knowledge is a big problem
[9:52] <hobx_> Chord, Kamdelia, Tapestry, Skipnets etc work without global knowledge.
[9:52] <toad_> they do?
[9:53] <hobx_> But I find them two rigid - in an environment where nodes come and go they have to work to hard to maintain integrity.
[9:54] <vsalento> I'm thinking about a simple application that uses freenet, but need some help with it
[9:54] <vsalento> 1) if I have file data read into a String and wan't to find out it's CHK-value if it is inserted into Freenet how can I do it?
[9:54] <hobx_> Though Kademlia is pretty good at it.
[9:54] <vsalento> 2) also what class should I use to insert it / request it? (FCP?)
[9:54] * tyesinclair (~tyesincla@170.252.64.1) has joined #freenet
[9:54] <toad_> vsalento: AutoRequester probably
[9:55] <toad_> and why do you need 1)?
[9:55] <hobx_> I think, after some autotranslate, that he wants to precalculate the CHK value.
[9:55] <toad_> hobx_: well, you go write your simulator
[9:55] <hobx_> There is an FNP command for it, isn't there?
[9:55] <toad_> you'll need more than you've given me if you want to convince me to work on simulations
[9:55] <hobx_> Can't. I have to work on my thesis subject.
[9:56] <hobx_> And it bores me so I don't work.
[9:56] <toad_> then convince me!
[9:56] <hobx_> I didn't tell you to go away...
[9:56] <toad_> what is to be gained by proving either a) freenet might just work on a blue moon? or b) freenet probably won't work?
[9:57] <vsalento> well I might be able to live without it but if it can be calculated before insert that would help
[9:57] <toad_> vsalento: it can...
[9:57] <toad_> check the API on AutoRequestor
[9:57] <toad_> i think it's there somewhere
[9:57] <toad_> you have to be careful to have the same metadata though
[9:58] <vsalento> ok, thanks
[9:58] <toad_> half the problem is that we can't really simulate NGR... but I do like NGR...
[9:58] <toad_> well maybe we can actually
[9:58] * Iakin4 (~nb@212.105.104.162) Quit (Read error: 110 (Connection timed out))
[9:59] <toad_> we can make time depend on hop distance or something
[9:59] <toad_> classic routing with load balancing would probably be a good place to start
[10:00] <hobx_> Did I spell that wrong?
[10:00] <hobx_> AutoRequestor?
[10:00] <toad_> hobx_: huh?
[10:00] <hobx_> hmm.
[10:00] <toad_> IMHO load balancing via QueryRejected and contact probabilities is crazy
[10:00] <hobx_> 291000 gogole hits for "requester" and 284000 for "requester"
[10:01] <hobx_> is that a US/UK thing?
[10:01] <toad_> if we can simulate load balancing via rate limiting, perhaps with queueing, then that would be more sensible...
[10:01] <toad_> hobx_: don't know
[10:01] <hobx_> I meant 281000 for requestor
[10:01] <toad_> just a common spelling mistake
[10:01] <toad_> my dicts show up 1 hit for requester
[10:01] <toad_> and none for requestor
[10:02] <hobx_> same with dictionary.com
[10:02] <toad_> hobx_: what is the key difference between a simulation and a testnet?
[10:03] <hobx_> well, for one thing one is a hell of lot easier to write.
[10:03] <toad_> perhaps
[10:04] <toad_> an actual deployed network has actual users
[10:04] <hobx_> And easier to deploy since all you need to do is recompile and rerun
[10:04] <toad_> and is therefore profitable
[10:04] <toad_> a simulation is useful to the devs but not directly to the users
[10:04] <hobx_> And the simulator lives in a less complicated environment where it is easier to identify the cause of problems.
[10:05] <toad_> but the simplifications made to make the sim work mean that fixes aren't necessarily portable back to the real net
[10:08] <toad_> i dunno
[10:08] <toad_> what i'm trying to say is: what are we going to get out of it?
[10:10] <hobx_> We are going to find out how to balance load and still get good results from routing.
[10:10] <hobx_> Which may or may not be possible. If it isn't, I would think everyone would rather know now.
[10:10] <toad_> only if the sim is so detailed as to be impossibly slow
[10:11] <toad_> would they? I'm not sure about that
[10:11] <toad_> e.g. Ian won't give up no matter what
[10:11] <hobx_> well, let Ian be Ian.
[10:12] <hobx_> I'm going to go eat something. BBL.
[10:12] <toad_> what happens if I discover that we can't? Well, the first step is to try different options to see if we can...
[10:12] <toad_> but if we really can't, I'm in the shit.. and I'm not the only one...
[10:13] <toad_> I suppose I could fork and make something with kademlia :|
[10:14] <toad_> i dunno, I suppose I have a responsibility not to let that be a major consideration :)
[10:15] <toad_> how can we factor out queueing, and still have rate limiting?
[10:16] <toad_> if we simulate rate limiting, we need to deal with the backed off nodes situation
[10:16] <toad_> writing a good simulator would probably take a month, and probably be against direct orders from ian
[10:16] <toad_> i need one hell of a good reason to do it
[10:21] <hobx_> Not doing politics.
[10:21] <iip_i2p> <Newsbyte> I am!
[10:21] <hobx_> Just saying what I think. I have no expectation that you will listen to me.
[10:22] <joltrushsoon> toad: what's this backed off nodes situation?
[10:23] <hobx_> I mean, as long as you are happy doing this, people are happy to donate your salary, and users like Newsbyte are happy with how well it works, everything is good.
[10:24] * guido^pe (~unknown@dsl-082-082-158-185.arcor-ip.net) Quit (Read error: 110 (Connection timed out))
[10:24] * hobx_ is now known as e^ip
[10:24] * guido^pe (~unknown@dsl-213-023-232-216.arcor-ip.net) has joined #freenet
[10:24] * e^ip is now known as e^ip-1
[10:25] <e^ip-1> can one have an = in a nick?
[10:25] * lolo-laptop is now known as lost
[10:25] <lost> no
[10:25] * lost is now known as lolo-laptop
[10:26] * e^ip-1 is now known as e^ip
[10:26] <e^ip> bla
[10:26] <e^ip> no + either
[10:26] * e^ip is now known as hobx
[10:26] * hobx is now known as hobx_
[10:26] <hobx_> Funny. Where is home me?
[10:27] * thelema|away is now known as thelema
[10:28] <hobx_> Rick Fox, Vanessa Williams divorcing
[10:33] <thelema> <toad_> thelema put some of the compression code in FieldSet, which imho is a little blech
[10:33] <thelema> hey! It's the most efficient thing I can think of, short of redesigning the protocol.
[10:43] * robilad (~topic@mpiat2313.ag2.mpi-sb.mpg.de) Quit (Remote closed the connection)
[10:44] * robilad (~topic@mpiat2313.ag2.mpi-sb.mpg.de) has joined #freenet
[10:50] <thelema> is anyone interested in testing FloatBucketStore on the real network?
[10:51] <toad_> thelema: well it's blech because it means ReadInputStream has to have support for reading either binary or UTF
[10:51] <toad_> that's the problem
[10:51] <toad_> it's ugly ugly ugly
[10:51] <toad_> and it means we can't carve up seednodes files without parsing them, if they are binary (which they probably should be!)
[10:52] <toad_> thelema: in unrelated news, hobx_ is trying to convince me to write a simulator that takes load into account
[10:52] <toad_> and me and KenMan have decided to implement queueing, and more or less specced it out
[10:52] <toad_> i'm going to post a summary to devl later
[10:53] <toad_> (as a fait accompli; I'm doing it, no matter what people think ;) )
[10:53] <toad_> I think we will ultimately have to get into test networks and even simulations
[10:54] <toad_> i just think that now is not yet the appropriate time
[10:54] <toad_> but there will be one...
[10:56] * mazzanet (~mazzanet@mazzanet.registered) Quit (Read error: 60 (Operation timed out))
[10:56] <thelema> interesting...
[10:57] <nassmoo> QUIT
[10:57] <thelema> Why did you decide to queue?
[10:57] * nassmoo (~chatzilla@217.167.208.229) has left #freenet
[10:57] <toad_> thelema: KenMan convinced me that we need to drastically reduce the number of RNFs
[10:57] <thelema> I agree. Will queueing do that?
[10:58] <toad_> a simulation showed we'll get a hell of a lot of RNFs unless we queue, if our incoming query rate is approximately equal to our allowed outgoing query rate from MRIs
[10:58] <toad_> and I'm not willing to make RNFs fatal
[10:58] <toad_> not yet
[10:58] * mazzanet (~mazzanet@mazzanet.registered) has joined #freenet
[10:58] <toad_> err our allowed outgoing query rate from MRIs
[10:59] <thelema> RNFs shouldn't be fatal, but they are a sign of connectivity problems (counting having all your nodes backed off as lack of connections)
[10:59] <toad_> granted that's a bit pessimistic, but we want to improve the current backoff situation - KenMan's node routinely gets 90%+ nodes backed off
[10:59] <toad_> right
[11:00] <toad_> but just by queueing, IMHO we can reduce RNFs significantly... and RNFs cause more RNFs
[11:00] <thelema> that's because nodes are too busy. The greatest limiting factor for nodes is bandwidth.
[11:00] <toad_> and improve routing at the same time
[11:00] <thelema> and the thing that that limits the most is simultaneous xfers
[11:00] <thelema> (well, it doesn't really limit it, but it makes things suck when there
[11:00] <toad_> well, i'm not ready to go for chording yet thanks muchly :)
[11:00] <thelema> 's too many)
[11:01] <thelema> I'm just laying out a perspective.
[11:01] <thelema> requests that DNF (should or will soon) take next to no resources
[11:02] <toad_> two or three other steps we identified - fixed size files, limiting/queueing/preventing RNFs of queries at the client end, and perhaps making RNFs fatal - AFTER we have queueing
[11:02] <thelema> so it's not just accepting queries that makes us overloaded, it's accepting queries that succeed, and having to pipe the data.
[11:02] <toad_> sure but i still see no good reason why freenet shouldn't be able to work without chording
[11:03] <thelema> Honestly I'm not sure I like fixed file sizes. It opens up too much correlation in datastore contents.
[11:03] <toad_> correlation?
[11:03] <toad_> what do you mean?
[11:03] <thelema> yeah, if every key is in several pieces, the presence of those pieces in one's store is a pretty sure sign that you requested that key.
[11:03] * |ux (~|ux@82-37-17-24.cable.ubr03.wolv.blueyonder.co.uk) has joined #freenet
[11:03] <toad_> is it? pcaching :)
[11:04] <toad_> if your store is full there's a good chance it won't be cached there
[11:04] <|ux> where to get 5091 from?
[11:04] <|ux> freenetproject.org?
[11:04] <toad_> and there's at least half a dozen great reasons for it - not least traffic analysis
[11:04] <toad_> |ux: yes
[11:04] <|ux> or u got a newer version toad?
[11:04] <thelema> pcaching will just make re-requesting suck. Imagine having to fetch a control document for every image on a page.
[11:04] <toad_> thelema: yes, I know
[11:05] <toad_> the answer to that is a client-cache
[11:05] <toad_> and request coalescing
[11:05] <thelema> client cache that's wiped on shutdown? hmm, I guess that'd work.
[11:05] <toad_> yup
[11:05] <toad_> use some of the temp space in the store
[11:06] <toad_> use random filenames and random crypto keys that are only kept in RAM
[11:06] <|ux> toad: just freenet.jar?
[11:06] <toad_> limit it to 10% of the store or so
[11:06] <toad_> maybe even 5%
[11:06] <toad_> |ux: yes
[11:06] <|ux> toad: t/y
[11:06] <thelema> how soon do you want the binary xport?
[11:06] <thelema> s/xport/presentation/
[11:06] <toad_> of course you can't completely prevent correlation attaclks without premix routing
[11:07] <toad_> thelema: it'd be nice to have it reasonably soon
[11:07] <toad_> it would make it a lot easier to sell fixed keysizes :)
[11:07] <toad_> but it'd be a nice gain in itself
[11:07] <thelema> There's no way to do fixed key sizes without compressing StoreData
[11:07] <thelema> btw, how much of storeData is the bigints?
[11:07] <thelema> (or is a lot of it the RAs?)
[11:08] <toad_> thelema: if we want to make seednodes binary, we'll need a method on FieldSet to skip a FieldSet without parsing it, and return it in a byte[]
[11:08] <toad_> if you make one of those that'd be a bonus
[11:08] <toad_> thelema: I don't know
[11:08] <toad_> quite a lot
[11:08] <toad_> especially after compression
[11:09] <toad_> sometime it may be worth trying 32 or 64 bits fixed point instead of the full 160 bit accuracy
[11:09] <toad_> maybe as a config option
[11:09] <toad_> i don't think floats are the best way to do it
[11:09] <thelema> I already coded FloatBucketStore (using doubles), if you want to give that a go.
[11:09] * toad_ suggests you build something as a config option
[11:09] <toad_> well, i'd prefer fixed point
[11:10] <toad_> and right now routing is pretty messed up anyway
[11:10] <toad_> because of rate limiting, because of huge files, probably because of routing bugs...
[11:10] <thelema> It basically is. very rarely will Edge keys be small enough for the extra precision of small doubles to matter.
[11:11] <toad_> well i'd still prefer fixed point :)
[11:11] <toad_> anything else you need to talk to me about?
[11:11] <thelema> fair enough. I'll finish making FloatBucketStore network compatible and make it a config option.
[11:11] <thelema> I guess that's all.
[11:11] <toad_> okay
[11:11] <toad_> bbiab
[11:12] * toad_ restarts X
[11:12] * toad_ (toad@82-32-16-91.cable.ubr03.azte.blueyonder.co.uk) Quit (Remote closed the connection)
[11:15] * toad_ (toad@82-32-16-91.cable.ubr03.azte.blueyonder.co.uk) has joined #freenet
[11:15] <thelema> does anyone here know if Double.doubleValue will correctly parse a BigInteger.toString?
[11:16] <toad_> thelema: on most JVMs, that's how BigInteger.doubleValue() works
[11:16] <toad_> thelema: it sucks, so we use a native alternative
[11:16] <toad_> also on a lot of 1.4.2's it either NPEs or NFEs
[11:16] * toad_ recommends you use BigInteger.doubleValue()
[11:18] <thelema> oh yeah. thanks.
[11:18] <toad_> (actually NativeBigInteger.doubleValue()) :)
[11:18] <thelema> I realized that it's a BigInt.toString(16) anyway, so it won't be in the correct form to just be parsed as a double)
[11:19] <thelema> I'm not doing conversion from one to the other, I'm being given a string that used to be parsed as a bigint, and I'm parsing it as a double.
[11:19] <toad_> hmm, why? compressed fieldsets?
[11:20] <toad_> oh, serializing from an Estimator fieldset
[11:20] <toad_> check
[11:20] <toad_> bbiab
[11:23] <thelema> grr, converting double -> bigDecimal -> BigInteger seems to be pretty rough. is there an easier way..
[11:26] * Hory (~Miranda@81.196.25.110) Quit ("Miranda IM! Smaller, Faster, Easier. http://miranda-im.org")
[11:34] <Iakin> 17:45:42 Timed out waiting for chunk in freenet.TrailerReaderInputStream@3115f9: tr =freenet.MuxTrailerReader@99c1d2: ID=56405, curPos=16405, 0 chunks pending, 119 chunks recieved, wantChunk=true, ph=freenet.PeerHandler@f7d5a6 (DSA(e539 2eb4 c930 7e27 e717 6199 36d8 b7c6 deec 3ac3),tcp/192.33.96.94:20131, sessions=1, presentations=3, ID=DSA(e539 2eb4 c930 7e27 e717 6199 36d8 b7c6 deec 3ac3), version=Fred,0.6,1.51,60190): outbound attempts=1:0/1, buffer
[11:34] <Iakin> 17:45:42 Connection died with only 1033495/16405 bytes read, suck=freenet.support.io.VerifyingInputStream@83ec5b, datalength=1049900, bytesRead=16405
[11:35] <Iakin> You recognize that former erro message from your log?
[11:35] <Iakin> I added the second log message..
[11:36] <Iakin> Seems like _all_ of those timeouts happens with 16405 bytes read for some reason
[11:37] <thelema> 1033495/16405 bytes read <- reading too much?
[11:38] <Iakin> No.. I put the int's the wrong way round
[11:38] * moskau23 (~Miranda@dsl-213-023-250-039.arcor-ip.net) Quit (Read error: 113 (No route to host))
[11:38] <thelema> or is that backwards, and it's only one 16K chunk (plus headers) that's being sent on a 1MB skey...
[11:38] <Iakin> Check the latter part of the message
[11:39] <thelema> VIS is blocking on its verification?
[11:40] <Iakin> wc -l 16405 freenet.log
[11:40] <Iakin> 177
[11:40] <Iakin> for 30 minutes of uptime
[11:40] <Iakin> thelma: Well.. or some chunks doesn't arrive
[11:40] <thelema> or aq few bytes why of verification are being sent, and something's trying to wait for verified data to come out before letting anymore data through?
[11:40] <hobx_> I wrote a response to anonymity thing: http://www.math.chalmers.se/~ossa/temp/fn_response.txt
[11:41] <thelema> s/why/shy/
[11:41] <hobx_> Somebody can forward it to tech if they feel like helping.
[11:41] <thelema> or there's a problem sending where it may take minutes (>5) to build the VIS?
[11:41] <thelema> (although that's done on the client/inserter side...
[11:43] * jay (jcl@ool-18bf6dac.dyn.optonline.net) has joined #freenet
[11:43] <Iakin> thelema: That verifyinginputStream wraps a TrailerReaderInputStream
[11:43] <Iakin> Which recieves trailer-mux-chunks from the TRailerReaderManager
[11:44] <Iakin> And returns from its 'read' method when it recieves another chunk
[11:44] <thelema> bah, too many layers...
[11:44] <Iakin> And, for some reason it seems like chunks get lost..
[11:45] <thelema> some wierd inter-layer buffering issue?
[11:46] <Iakin> I have committed the logging changes
[11:46] <Iakin> thelema: PRobably..
[11:47] <Iakin> But.. one that seems to result in _many_ of those 'Times out waiting for chunk ' messages..
[11:47] <Iakin> Well.. I gtg, bbl
[11:47] * mazzanet (~mazzanet@mazzanet.registered) Quit (Read error: 60 (Operation timed out))
[11:47] * mazzanet (~mazzanet@mazzanet.registered) has joined #freenet
[11:47] <Iakin> please try running with those logging enhancements and see if your result is similar to mine
[11:47] <Iakin> bbl
[11:50] <toad_> Iakin: eeeek
[11:55] <thelema> toad_: it's been suspected for a long time (at least by me) that all the xfer failures we have were not caused just by nodes going down.
[11:56] * |ux (~|ux@82-37-17-24.cable.ubr03.wolv.blueyonder.co.uk) Quit ("Leaving")
[11:57] * mazzanet (~mazzanet@mazzanet.registered) Quit (Read error: 60 (Operation timed out))
[11:57] * mazzanet (~mazzanet@mazzanet.registered) has joined #freenet
[11:58] * ninja_ (~im10ninja@3ffe:bc0:8000:0:0:0:0:3fab) has joined #freenet
[11:59] <thelema> biab
[12:06] * spaetz_ (~spaetz@217-162-197-136.dclient.hispeed.ch) Quit (Remote closed the connection)
[12:07] * TLF (francisco@104.Red-81-40-113.pooles.rima-tde.net) has joined #freenet
[12:17] * ninja_ (~im10ninja@3ffe:bc0:8000:0:0:0:0:3fab) Quit (Read error: 60 (Operation timed out))
[12:26] * joltrushsoon (josh@mir.stevecole.org) has left #freenet
[12:39] * TLF (francisco@104.Red-81-40-113.pooles.rima-tde.net) Quit ("http://www.vitaliano.esp.cc || Adi?s a todos y a todas y que les vaya bien.")
[12:42] <thelema> back
[12:42] <thelema> toad_: ant ideas on iakin's errors?
[12:45] * jabawok_h2 (~rob@203-59-105-146.dyn.iinet.net.au) Quit (Read error: 110 (Connection timed out))
[12:52] <thelema> why didn't anyone tell me SBKE was still in its reverted state?
[12:55] <thelema> now I can
[12:55] <thelema> 't just replace BucketStore with FloatBucketStore... :(
[13:11] * salahx (salahx@sc1-24.217.174.147.charter-stl.com) has joined #freenet
[13:13] * hobx_ (~chatzilla@ankh.math.chalmers.se) Quit (Remote closed the connection)
[13:24] * Gunnar (~gunnar@34.80-203-209.nextgentel.com) Quit (Read error: 110 (Connection timed out))
[13:37] <jokern> 19:53:54 Not our NoStoreData: freenet.node.states.request.NoStoreData@11024d5@9cd20d03b2c9b157,true, expected
[13:37] <salahx> soemnoen else say that too
[13:37] <salahx> err saw that too
[13:41] * TheSeeker (~yeahright@user-24-214-134-203.knology.net) has joined #Freenet
[13:42] <TheSeeker> ok, I found a tcpip.sys patcher for SP2 to allow more than 10 connections on a port... Freenet is probably the most intensive applicatino I'll be running, so should I just set my max to what I have in Freenet's config for MacConnections ?
[13:42] * TLF (francisco@225.Red-81-40-113.pooles.rima-tde.net) has joined #freenet
[13:48] <salahx> I Wasn't aware the was a 10-connection limit per port in the first place ?
[13:50] <TheSeeker> yeah, new security measuer to help keep viruses and worms from spreading very fast... after 10 half-open connections the rest are queued and opened at some slow rate
[13:52] <salahx> oh great
[13:54] <salahx> that sounds more liek a misfeature to me...
[13:54] <thelema> Honestly, if freenet is running into that, freenet is designed poorly.
[13:54] <thelema> 10 simultaneous connectionopenings is plenty.
[13:54] <TheSeeker> Quote:
[13:54] <TheSeeker> - A TCP connection is etablished by a three way handshake.
[13:54] <TheSeeker> Until the handshake is completed the connection stays 'halfopen'. when the other side doesn't answer it will be closed after a time out. The change in SP2 is the number of allowed 'halfopen' connections. The limit is 10. When the limit is reached the computer have to wait until a 'halfopen' connection times out before he can try to open a new TCP connection. This might interfere with P2P programs that may open a lot of connection that
[13:55] <salahx> well the problem being is that there are misconfigured nodes out there, with firewall that drop everything
[13:55] <thelema> It'll only be a problem if 1) connection openings take a while or 2) if we're trying to connect to a lot of nodes that aren't there.
[13:56] <thelema> we should solve taht problem by having those know that they're not globally reachable, and not give out contact info. sorta like lowID on edonkey.
[13:56] <TheSeeker> 2 is possible for new nodes isn't it?
[13:56] <thelema> TheSeeker: yes it is. and if we have a short enough timeout, it won't be atht big of a deal.
[13:57] <TheSeeker> well, I'm restarting to apply the hack. I guess other P2P software is more likely to pose a problem since I get large download lists with hosts that frequently aren't there anymore ...
[13:58] <TheSeeker> this can generate a lot more than 10 halfopen conenctions in a short amount of time, and explains why I've lost the ability to connect on occasion.
[13:58] * TheSeeker (~yeahright@user-24-214-134-203.knology.net) Quit (Read error: 54 (Connection reset by peer))
[14:04] * KenMan is surprised MS didn't just set a limit of 10 connections on a port... that would be more their style
[14:05] <KenMan> and can't virus writers compensate by using more than one port ? it is primarily intended to reduce outgoing scans, right ?
[14:06] * tyesinclair (~tyesincla@170.252.64.1) Quit (Read error: 110 (Connection timed out))
[14:06] <KenMan> but, oh yeah. This information wasn't supposed to see daylight for a year or two. Talk about security :o
[14:06] <salahx> 10 connection IS the maximum nubmer of conenction Microsoft allows to XP Home/Pro for fior the purposed of SMB, etc...
[14:12] * TLF (francisco@225.Red-81-40-113.pooles.rima-tde.net) Quit ("http://www.vitaliano.esp.cc || Adi?s a todos y a todas y que les vaya bien.")
[14:12] <KenMan> could this be related to what Iakin is reporting ? look at 14:00 - http://mywebpages.comcast.net/jkcorson/FreeNet/5090bw.png
[14:13] * TLF (francisco@25.Red-81-40-113.pooles.rima-tde.net) has joined #freenet
[14:13] <KenMan> both in and out xfers suddenly drop from ~55 to ~25 active in the blink of a minute
[14:16] <thelema> KenMan: hmm.. that could be.,
[14:16] <thelema> anything interesting happen in that minute?
[14:17] <KenMan> no, just that funny drop. I was in bed sleeping at the time. Those rate limiting spikes are curious also.
[14:18] * Gunnar (~gunnar@34.80-203-209.nextgentel.com) has joined #freenet
[14:18] <KenMan> my system is MUCH better behaved since I stopped using linux ACPI. NTP actually works, without any resets (big jumps in clock time that destroy Fred)
[14:19] <salahx> 2.4 or 2.6 ?
[14:19] <KenMan> 2.6.7 is working wonderfully now. so it wasn't one of those NTP hiccups to blame (i checked the ntp logs/stats)
[14:19] * TLF (francisco@25.Red-81-40-113.pooles.rima-tde.net) Quit ("http://www.vitaliano.esp.cc || Adi?s a todos y a todas y que les vaya bien.")
[14:20] <greycat> what's ACPI?
[14:20] <KenMan> maybe i'll try using voluntary preemption again - it was a big suspect in my mind while I was bug-hunting
[14:20] <KenMan> Advanced Control of Programmable Interrupts ( or something equally stupid )
[14:21] <KenMan> somehow it gets lumped into power management, but its purpose has to do with rerouting IRQs
[14:21] <greycat> oh, the other APM.
[14:21] <KenMan> yeah, sort of. Except it covers a lot.
[14:21] <salahx> ACPI = Advanced Configuration & Power Interface
[14:21] <salahx>
[14:21] <KenMan> thank you
[14:21] <salahx> You;re thing APIC, which has to do with interrupt routing
[14:22] <KenMan> no, I turned that back on. The "Configuration" part of ACPI is very similar
[14:22] <salahx> ACPI suppsoedely works a lot better in 2.6
[14:23] <KenMan> heh, not in my book. I think it means I have to reach over and hit the power button with my hand now. But I never turn off anyway, so...
[14:24] <KenMan> maybe I left APIC off, i can't remember. But both of them are out of my hair now, and my life is much better for it.
[14:25] <KenMan> my problem of having an unstable system was isolated to use of ACPI. One man's report, take it at face value.
[14:25] <salahx> you';ren to the first
[14:25] <KenMan> huh ? no, i usually raise myself to the third power, not the first.
[14:26] * KenMan wonders if that makes him a cubist
[14:27] <KenMan> oh, did i just decode your secret message properly ?? : "You're not the first"
[14:27] <salahx> right
[14:27] <KenMan> It sure would make our lives easier if you would take the time to spell things right !!! but no pressure :)
[14:27] * Hory (~Miranda@81.196.25.110) has joined #FreeNet
[14:28] <KenMan> i saw some reports of certain brands of laptops causing troubles with ACPI, and others that worked wonderfully...
[14:30] <KenMan> the tradeoff - i can no longer make the machine turn itself off (no loss there), I can't read the CPU temperature (i usually don't want to anyway!)
[14:31] <salahx> I thoguh getting cpu temp was part of lm_sensors
[14:31] <KenMan> it is, which is part of ACPI in 2.6
[14:31] * thelema wants to implement a 23-byute modular type
[14:31] <thelema> *byte
[14:32] <iip_i2p> <west234> why i most of the time have:Attempts were made to contact 0 nodes.
[14:32] <thelema> so that we can stop using bigints for key manipulation
[14:32] <thelema> west234: get new seeds from http://www.freenetproject.org/snapshots
[14:33] <salahx> Unstable doesn't work anymore; none of the nodes in unstable.ref respond
[14:33] <thelema> salahx: really?
[14:33] <salahx> yep
[14:33] <iip_i2p> <west234> why i most of the time have:Attempts were made to contact 0 nodes,0 were totally unreachable,0 restarted,0 cleanly rejected,14 backed off
[14:34] <iip_i2p> <west234> it is new node
[14:34] <KenMan> well, everybody knows that stable is where it's at , nowadays
[14:34] <KenMan> ;)
[14:34] <salahx> Well I'll just stick with 5091 a bit longer
[14:35] <iip_i2p> <west234> i am transparent mode
[14:35] <thelema> west: the problem is that your node is only connected to 14 other nodes.
[14:35] <KenMan> say, what kind of numbers do you see for searchFailedCount/hr salahx ? for average, and for max ...
[14:35] <salahx> lemem check
[14:35] <thelema> west: with only 14 neighbors, you can't make requests often
[14:36] <thelema> west: you can either leave your node running and let it find more nodes by itself
[14:36] <thelema> west: or you can get some more seeds from someone.
[14:36] <salahx> well I jsut reseeded with 5091 so I donl;t hav any good stats currently
[14:37] <iip_i2p> <west234> ok i will wait
[14:37] <KenMan> salahx: cool, it is hard to interpret sFC anyway, with RNFs wrapped around them...
[14:42] <thelema> 62 bits per long * 3 longs = 186 bits. 8 bits per byte * 23 bytes = 184 bits. Just enough space to put a Key in a long[3]
[14:48] * JustMe_ (JustMe_@cs6669187-74.houston.rr.com) Quit ()
[14:54] * Kyon (~Kyon@pcd396122.netvigator.com) Quit (Read error: 104 (Connection reset by peer))
[14:56] * salahx (salahx@sc1-24.217.174.147.charter-stl.com) Quit ()
[14:56] * Kyon (~Kyon@pcd396122.netvigator.com) has joined #freenet
[15:04] <KenMan> argh... I have a file, but I don't know what it is (it is a business card). On unix, 'file' reports it as "Microsoft Office Document"
[15:04] <greycat> what did you want to do with it (today)?
[15:04] <KenMan> so I try to load it in word 97, but no good. Now I wonder if it is a .doc file for a newer version of word, or if it is from some other MS publishing application
[15:05] <KenMan> it has encapsulated postscript inside it, and that's about all I know.
[15:06] <KenMan> I'm going to proceed on the assumption that it is for a newer version of Word, and see if the copier store can print it.
[15:06] <KenMan> The file was originally created with their equipment 2 years ago, but of course they change their computers, software, and printers every 4 months...
[15:06] <hirvox> AFAIK MS also has a desktop publishing program, but I can't rember it's name, Postscript sure would lean towards DTP..
[15:07] <KenMan> yeah, that's the odd part. I can't remember the name either :( not Visio. Something similar to PageMaker, to steal market share.
[15:07] <KenMan> PageBreaker ? PageCreator ? PageWanker ? PageShaker ? PageTaker ? i dunno
[15:08] <hirvox> Microsoft Publisher (google to the rescue)
[15:08] <KenMan> thank you !
[15:09] <KenMan> now how would I find out the expected file extension ? I have a strange layout on the floppy, an .eps file , a resource.wrk subdirectory , ... hmm. If they actually still had MS Publisher their, it should read it correctly.
[15:10] * KenMan rings the local Kinko's copier store
[15:13] <hirvox> bbl, sleep
[15:13] * hobx (~hobx@h196n1fls21o1077.bredband.comhem.se) has joined #freenet
[15:13] * ChanServ sets mode +o hobx
[15:15] <jokern> Still getting these errors:
[15:15] <jokern> 21:28:42 Type wrong on reporting: transfer rate should be time java.lang.Exception: debug
[15:21] <jokern> Build 5090 , Uptime 2 days 19 hours 33 minutes - Total amount of data transmitted/received 5,745 MiB/6,967 MiB
[15:22] <jokern> So something must be working ...
[15:23] <thelema> what's the rest of that error?
[15:24] <jokern> Just a sec.
[15:26] <jokern> Aug 11, 2004 6:54:53 AM (freenet.node.rt.OptimizingSlidingBucketsKeyspaceEstimator, YThread-1059, ERROR): Type wrong on reporting: transfer rate should be time
[15:26] <jokern> java.lang.Exception: debug
[15:26] <jokern> at freenet.node.rt.SlidingBucketsKeyspaceEstimator.checkType(SlidingBucketsKeyspaceEstimator.java:408)
[15:26] <jokern>
[15:27] <jokern> at freenet.node.rt.SlidingBucketsKeyspaceEstimator.reportTransferRate(SlidingBucketsKeyspaceEstimator.java:419)
[15:27] <jokern> at freenet.node.rt.NGRouting.transferFailed(NGRouting.java:274)
[15:27] <jokern> at freenet.node.states.request.TransferReply.reportTransferFailed(TransferReply.java:270)
[15:27] <jokern> at freenet.node.states.request.TransferReply.receivedMessage(TransferReply.java:228)
[15:27] <jokern> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
[15:27] <jokern> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
[15:27] <jokern> at java.lang.reflect.Method.invoke(Unknown Source)
[15:27] <jokern> at freenet.node.State.received(State.java:143)
[15:27] <jokern> at freenet.node.StateChain.received(StateChain.java:177)
[15:27] <jokern> at freenet.node.StateChain.received(StateChain.java:61)
[15:28] <jokern> at freenet.node.StateChainManagingMessageHandler$ChainContainer.run(StateChainManagingMessageHandler.java:332)
[15:28] <jokern> at freenet.node.StateChainManagingMessageHandler$ChainContainer.received(StateChainManagingMessageHandler.java:285)
[15:28] <jokern> at freenet.node.StateChainManagingMessageHandler$ChainContainer.access$100(StateChainManagingMessageHandler.java:204)
[15:28] <jokern> at freenet.node.StateChainManagingMessageHandler.handle(StateChainManagingMessageHandler.java:96)
[15:28] <jokern> at freenet.Ticker$Event.run(Ticker.java:323)
[15:28] <jokern> at freenet.thread.YThreadFactory$YThread.run(YThreadFactory.java:285)
[15:28] <jokern> Thats all I got
[15:33] <thelema> what build?
[15:33] <jokern> 5090
[15:34] <thelema> :( I have some rewrites I was doing on a friend's laptop to fix this
[15:34] <thelema> but those rewrites are across the state right now, and couldn't be checked in because they're unfinished.
[15:35] <jokern> Get the car and be back in 2 hours :)
[15:37] <thelema> It's not a two hour state.
[15:38] * salahx (salahx@sc1-24.217.174.147.charter-stl.com) has joined #freenet
[15:41] <jokern> Just kidding .
[15:46] * eMBee (~mbaehr@sirius.iaeste.tuwien.ac.at) Quit (Read error: 110 (Connection timed out))
[15:46] * eMBee (~mbaehr@sirius.iaeste.tuwien.ac.at) has joined #freenet
[16:00] * studdlybiggi (~noneya@12-215-171-38.client.mchsi.com) has joined #freenet
[16:00] <studdlybiggi> so whats better to run, stable or unstable?
[16:02] <salahx> stable
[16:05] <studdlybiggi> did they ever fix the 26meg seednode problem on stable?
[16:06] <studdlybiggi> what do you have to do to run both stable and unstable on the same machine?
[16:07] <salahx> nothing special
[16:07] <salahx> you'll jsut have 2 Freenet instalttions
[16:07] <salahx> You'll probably have ot modify the startup script slightly though
[16:07] <salahx> Just make sure you have a LOT of memory, you'll need it!
[16:08] <salahx> right now though, usntable is broken
[16:09] <salahx> I assume you are running Linux?
[16:10] <studdlybiggi> I have xp pro and I have 1 gig of ram
[16:10] <salahx> ok well 1 gig or ram is plenty
[16:10] <salahx> You'll have to create your own startup file for the other node though
[16:10] <studdlybiggi> dont I need to change the ports too?
[16:11] <salahx> yes
[16:11] <studdlybiggi> does it matter what I change it to?
[16:11] <salahx> you'll have 2 differnt Freenet installtions in 2 differnet place
[16:11] <salahx> no
[16:11] <studdlybiggi> ok
[16:11] <studdlybiggi> the damn stable seednode is huge still I see
[16:11] <salahx> yeagh
[16:12] <salahx> Borrow mine if you need to: http://home.earthlink.net/~salahx/freenet/unstable.ref.gz
[16:12] <studdlybiggi> ok thanks
[16:12] <salahx> despotei the filename the refs are for stable
[16:12] <studdlybiggi> thats an unstable one, you have a stable one?
[16:13] <salahx> it for stable, despite the name
[16:13] <studdlybiggi> ok
[16:13] <salahx> normally I run an unstable node but I switched to stable for the "netowrk reset"
[16:13] <studdlybiggi> you dont run them both at once?
[16:14] <salahx> no
[16:15] <salahx> Stable is for users; unstable is for developrs, testers and the insane :)
[16:17] <salahx> (Most useres of unstable fit in all the 3 catagoies at the same time too :) )
[16:17] <studdlybiggi> hmm
[16:25] <studdlybiggi> what do I change storeSize to in order to allocate 1 gig?
[16:26] <thelema> 1G
[16:26] <salahx> 1G
[16:27] <studdlybiggi> it will recognize that? it dont have to be in byteS?
[16:27] <salahx> It'll recognize it
[16:27] <salahx> I wonder if the config file accept Ki, Mi, Gi and the like though ?
[16:27] <studdlybiggi> ok thanks, should I change anything else?
[16:27] <salahx> listenPort
[16:28] <studdlybiggi> I did that, and mainport too
[16:29] <salahx> don;t change mainPort
[16:29] <salahx> well acutally if ou're runnign 2 nodes you'll ahve to
[16:29] <salahx> and fcpPort too
[16:30] <studdlybiggi> ok
[16:30] <salahx> and hte location of the datatore, logfile, etc...
[16:33] <studdlybiggi> cannot seem to get them both started, I have stable running, but when I try to launch unstable it just brings me to fred on stable
[16:34] <salahx> you have to point to the unstabble node mainPort
[16:34] <studdlybiggi> I know but it connection refused cause its not running
[16:34] <salahx> i.e. http://nodeip:mainport (something other than 8888)
[16:35] <salahx> it might take a few seconds to start
[16:35] <studdlybiggi> 'stable is on 8889
[16:35] <studdlybiggi> the other is 8888
[16:37] <salahx> well it shoud lbe owrking by now
[16:37] <salahx> if not, check the log file
[16:38] <thelema> hmm, how to do the average (midpoint) of modular values...
[16:38] <thelema> adding the two and dividing by 2 doesn't work because of wraparound
[16:39] <thelema> and dividing each by two and then adding also doesn't work.
[16:39] <thelema> There's two midpoints; how do I do this so it's ordered, so that I get the one in the correct 'between'?
[16:40] <thelema> is the only way to take the difference, halve it, and add it to the left?
[16:43] <studdlybiggi> well, if I start unstable I can view it on 8888. But in order to run stable I have to exit unstable then start stable. Cannot seem to get them BOTH to run at once...
[16:44] <studdlybiggi> I changed all the ports like we discussed above and I checked for comment lines
[16:44] <salahx> did you point them to differnt confilg files ?
[16:45] <studdlybiggi> I have them in 2 different directories
[16:45] <salahx> but by defaiyl they'll try to read frmo the same config file
[16:45] <studdlybiggi> ok
[16:45] <salahx> you have to add -p confgilename to chosoe an alternate
[16:47] <studdlybiggi> no they cannot be using the same config file because when I start unstable it is on 8889 like its config file says, but the other runs in 8888 so they cannot be looking at the same file
[16:47] <studdlybiggi> they both run ok, just not at the same time
[16:48] <salahx> odd
[16:49] * thelema is now known as thelema|away
[16:59] * greycat (~wooledg@192.35.79.70) Quit ("Lick Bush in '04")
[16:59] * goatee (~goatee@ip216-239-87-183.vif.net) Quit (Remote closed the connection)
[17:01] * goatee (~goatee@ip216-239-87-183.vif.net) has joined #freenet
[17:02] * Hory (~Miranda@81.196.25.110) Quit (Read error: 54 (Connection reset by peer))
[17:05] * studdlybiggi (~noneya@12-215-171-38.client.mchsi.com) Quit ()
[17:21] * jay (jcl@ool-18bf6dac.dyn.optonline.net) Quit (".")
[17:22] <KenMan> you know, if you count a retry as a request (and why wouldn't you?) then we really don't even have 1% success :p
[17:26] * leex-work is now known as leex-R
[17:26] * MrNeo (~MrNeo@68.117.127.231) has joined #freenet
[17:27] <leex-R> lo
[17:27] <MrNeo> hello
[17:28] <leex-R> he some to talk
[17:28] <leex-R> heh
[17:28] * moskau23 (~Miranda@dsl-082-082-237-021.arcor-ip.net) has joined #freenet
[17:30] <MrNeo> I've got a question .. whenever I try to bring up a freenet page, I get a couldn't retrieve key error. Im a bit of a newb so, does this just mean that its looking for it and it could take a few minutes?
[17:31] <salahx> yes
[17:31] <MrNeo> ok :)
[17:31] <salahx> with Freenet you need patience. LOTS of it.
[17:31] <salahx> I recommend watching paint dry of at least an hour a day for the next month to nget used to it :)
[17:32] <MrNeo> haha
[17:32] * studdlybiggi (~noneya@12-215-171-38.client.mchsi.com) has joined #Freenet
[17:33] <studdlybiggi> can I get a seednode ref please
[17:33] <salahx> for stable or unstable ?
[17:33] <studdlybiggi> stable
[17:33] <salahx> sure!
[17:33] <salahx> http://home.earthlink.net/~salahx/freenet/unstable.ref.gz
[17:34] <salahx> Desptie the name, it IS for stable
[17:34] <leex-R> what about useing the one on the web site
[17:34] <studdlybiggi> thanks, you gave it to me earlier but accidently deleted it
[17:34] <salahx> its 25+M and it tends to blow up nodes due to its size :)
[17:34] <studdlybiggi> its 27 megs and causes memory errors
[17:34] <leex-R> heh big
[17:35] <leex-R> unstable is 236k not much of an unstable network
[17:37] <salahx> well usntable seems broken right now
[17:37] <leex-R> i got 3 nodes running
[17:37] <leex-R> none seem to be working
[17:38] <leex-R> RNF
[17:43] <leex-R> Fred,0.6,1.51,60179 1
[17:43] <leex-R> Fred,0.6,1.52,60187 6
[17:43] <leex-R> Fred,0.6,1.52,60188 1
[17:44] <leex-R> all my old nodes before had 52 on them
[17:44] <leex-R> 52
[17:44] <leex-R> even
[17:44] <leex-R> shit kaybord
[17:45] <leex-R> ms natrue keybord
[17:45] <studdlybiggi>
[17:45] <studdlybiggi> Couldn't connect to the network. Are you sure you have configured Freenet correctly? Also make sure that you are connected to the internet.
[17:46] <leex-R> ?
[17:46] <studdlybiggi> since I got a icon for "FIND" I assume that I am connected eh?
[17:46] <leex-R> ?
[17:46] <studdlybiggi> ?
[17:46] <leex-R> find ?
[17:47] <studdlybiggi> yeah where dolphin is on unstable
[17:47] <leex-R> heh o yer
[17:47] * MrNeo (~MrNeo@68.117.127.231) Quit ("All your base are belong to us!")
[17:47] <leex-R> my main node has been on for 1 day and its still not working
[17:48] <studdlybiggi> You get used to the ms natural keyboard eventually, I hated it at first now I cannot imagine not having it
[17:48] <studdlybiggi> are you stable or unstable
[17:48] <leex-R> not at my house
[17:48] <leex-R> unstable
[17:50] <leex-R> ?
[17:52] <studdlybiggi> I have 2 nodes on two different comps, the other comp seems to be able to surf a little, but this one is the shit
[17:52] <salahx> Unstable is broken. None of the seednopdes work
[17:52] <studdlybiggi> both nodes are stable
[17:52] <salahx> oh
[17:53] <studdlybiggi> when I export refs on the good one, it does not output a file... strange
[17:55] <salahx> hmmm
[18:00] <leex-R> so its not working then
[18:01] <leex-R> the test networked worked
[18:07] <Iakin> Heh.. it really _is_ so that the mixtrailerstreams dies after 16405 bytes read
[18:08] <Iakin> I have _many_ examples of that in my log now
[18:08] <Iakin> s/mix/mux
[18:08] <Iakin> I assume that not all transfers dies though...
[18:09] <leex-R> stuff i got is whtsit stoed in my store
[18:17] <leex-R> toad_ must be working hard been qite tonight
[18:27] <toad_> hobx: here?
[18:27] * salahx (salahx@sc1-24.217.174.147.charter-stl.com) Quit ()
[18:27] <toad_> <thelema> toad_: it's been suspected for a long time (at least by me) that all the xfer failures we have were not caused just by nodes going down. - well, with typical transfer times of 800 seconds or so, that's not completely implausible
[18:28] <toad_> <thelema> toad_: ant ideas on iakin's errors? - don't know...
[18:28] <leex-R> lo
[18:28] <toad_> it's one of these bastard things that can probably only be sorted out by seeing both sides of the transaction at heavy logging
[18:29] <toad_> <thelema> Honestly, if freenet is running into that, freenet is designed poorly.
[18:29] <toad_> <thelema> 10 simultaneous connectionopenings is plenty.
[18:29] <toad_> uhh, is it 10 simultaneous OPENS
[18:29] <toad_> or is it 10 simultaneous open conns?
[18:29] <toad_> <thelema> It'll only be a problem if 1) connection openings take a while or 2) if we're trying to connect to a lot of nodes that aren't there.
[18:29] <toad_> sadly they do
[18:29] <toad_> so we try to open 50 at once
[18:30] <Iakin> toad: opens they say
[18:31] * guido^pe (~unknown@dsl-213-023-232-216.arcor-ip.net) Quit (Remote closed the connection)
[18:33] <Iakin> cat freenet.log>grep 16405>wc -l
[18:33] <Iakin> 6033
[18:33] <Iakin> For 5 hrs of node uptime
[18:34] <Iakin> Meaning.., slightly above 6k broken trailertransfers
[18:34] <Iakin> thats 1k per hour... just by _my_ node
[18:35] <Iakin> And every single one after ~16k bytes
[18:36] <toad_> <thelema> so that we can stop using bigints for key manipulation - uhh, no, it won't be faster. at least you'll still need to use BigInts for modPow and divide/multiply.
[18:36] <Iakin> sorry, I take that back.. it would seem to be 2.2k, not 6k
[18:36] <toad_> <salahx> Unstable doesn't work anymore; none of the nodes in unstable.ref respond
[18:36] <toad_> <thelema> salahx: really?
[18:36] <toad_> <salahx> yep
[18:36] <Iakin> (had some other 16405's too)
[18:36] <toad_> what??
[18:38] <toad_> <thelema> 62 bits per long 3 longs = 186 bits. 8 bits per byte 23 bytes = 184 bits. Just enough space to put a Key in a long[3] - why 62 bits per long? and what can you possibly gain from NOT using BigInteger anyway? you will need to do nasty stuff like multiplies and divides
[18:38] <toad_> and modPow
[18:38] <toad_> and the way you clip it to 62 bits just to avoid sign issues and so you can carry is blechable
[18:39] <toad_> disgusting
[18:39] <leex-R> i cant get unstable to work now
[18:39] <Iakin> ok, correction.. not _all_ times out after 16405 bytes, some after other amounts of bytes too..
[18:39] <Iakin> but _most_ seems to time out after 16405 bytes for some reason
[18:39] <toad_> Iakin: okay, so it gets a corrupt byte somewhere in the 1st segment, and only realizes when the segs completed, right?
[18:40] <Iakin> toad: Absolutely no idea
[18:40] <Iakin> I haven't been into the muxing code
[18:40] <toad_> <jokern> Still getting these errors:
[18:40] <toad_> <jokern> 21:28:42 Type wrong on reporting: transfer rate should be time java.lang.Exception: debug
[18:40] <toad_> I FIXED IT !!! AAAARGH!!!!
[18:41] <Iakin> But.. when you say 'segment'.. do you mean like Rolling hash segments or something?
[18:41] <toad_> jokern: it's possible that bug is fixed in 5091
[18:42] <Iakin> toad: Thing it.. it doesn't realize at _all_.. It just continues waiting for the next chunk.. until it times out
[18:42] <toad_> <salahx> Stable is for users; unstable is for developrs, testers and the insane :)
[18:42] <toad_> <salahx> (Most useres of unstable fit in all the 3 catagoies at the same time too :) )
[18:42] * lolo-laptop (~lostlogic@68.251.84.226) Quit ("Client exiting")
[18:42] <toad_> hehehehehe
[18:43] <toad_> <salahx> I wonder if the config file accept Ki, Mi, Gi and the like though ? - no, K = Ki = 1024, k = 1000
[18:44] * Redb3ard (~oylerj@c-24-125-12-101.va.client2.attbi.com) Quit (Read error: 113 (No route to host))
[18:44] <Iakin> toad, one thing worth noticing.. why is that corruption only occurring in the _first_ segment?
[18:44] <leex-R> :)
[18:44] <Iakin> I havent seen any large bytesRead values yet.. they are all less than 17k
[18:44] <toad_> <thelema> hmm, how to do the average (midpoint) of modular values...
[18:44] <toad_> <thelema> adding the two and dividing by 2 doesn't work because of wraparound
[18:44] <toad_> <thelema> and dividing each by two and then adding also doesn't work.
[18:44] <toad_> <thelema> There's two midpoints; how do I do this so it's ordered, so that I get the one in the correct 'between'?
[18:44] <toad_> <thelema> is the only way to take the difference, halve it, and add it to the left?
[18:44] <toad_> it's HARD
[18:45] <toad_> read the code to do it in SlidingBucketsKeyspaceEstimator
[18:48] <toad_> <Iakin> toad: Thing it.. it doesn't realize at _all_.. It just continues waiting for the next chunk.. until it times out - hmmm
[18:49] <toad_> so the first segment was VALID and it STILL timed out?
[18:49] <toad_> Iakin: you need to find this happening on the SEND side
[18:49] <toad_> otherwise you'll never get anywhere
[18:50] <toad_> now what's this about unstable being busted?
[18:50] <toad_> there are only 8 nodes in it...
[18:51] <toad_> and all but 1 are on 1.52 i.e. the testnet
[18:51] <Iakin> toad: Cant it be so that some code on the receiving side looses a chunk or something?
[18:51] <toad_> shit, this is a matter of there being no seednodes sources again isn't it? :<
[18:51] * toad_ dutifully starts his unstable node...
[18:52] <toad_> really, if people aren't going to run seednodes sources... then the project ought to pay me a good deal more for hardware
[18:52] <toad_> Iakin: can you prove that?
[18:52] <toad_> search for the chunk ID in the log file...
[18:53] <Iakin> Hmm...
[18:53] <Iakin> trailer id you mean?
[18:53] <toad_> Iakin: how do I find one of these error messages?
[18:53] <toad_> Iakin: yes
[18:59] * Iakin4 (~nb@212.105.104.162) has joined #freenet
[18:59] <Iakin4> curPos=16405, 0 chunks pending, 119 chunks recieved
[18:59] <Iakin4> Ehhh.. how small chunks do we use?
[19:00] <Iakin4> Generally it seems to be the 120:th or 121:th chunk that is missing
[19:00] <toad_> Iakin4: pretty small now
[19:01] <Iakin4> (but not always.. I see chunk #6 missing on a couple of occations too
[19:01] <Iakin4> )
[19:01] <toad_> in the name of defeating traffic analysius
[19:01] <toad_> just under the padding chunk size...
[19:01] <toad_> so that there's ambiguity as to whether it's a message packet or a chunk
[19:01] <Iakin> toad: search for 'eating'
[19:01] <Iakin> ..in the log
[19:01] <Iakin> (with lastest unstable code)
[19:02] <Iakin> or well, at max 5 hrs old code at least
[19:02] * toad_ attempts to resuscitate the unstable network...
[19:02] <leex-R> heh
[19:03] <Iakin> toad: There is no mention of the failing trailers id in the log.. other than in the actual failure message
[19:04] <toad_> yay, i have 2 connections...
[19:04] <toad_> Iakin: the actual log message includes the failing trailer ID or at least the toString() on the TRIS or the MTR?
[19:05] <toad_> that should be enough to find it...
[19:05] <Iakin> Yes, it does..
[19:05] <toad_> hmmm
[19:05] * toad_ needs some code changes to do this...
[19:06] <Iakin> but that was the only mention of it..
[19:06] <toad_> especially: try to reconnect to nodes with invalid versions if either a) we have not contacted them since startup or b) we have not contacted them for aaaaages
[19:06] <toad_> Iakin: not enough logging them
[19:06] <toad_> then
[19:07] <toad_> logLevelDetail=freenet.MuxTrailerRead:debug,freenet.TrailerRead:debug
[19:07] <leex-R> Connections open (Inbound/Outbound/Limit) 56 (41/15/200)
[19:07] <leex-R> Transfers active (Transmitting/Receiving) 31 (14/17)
[19:07] <leex-R> Data waiting to be transferred 123 Bytes
[19:07] <leex-R> Total amount of data transferred 1,608 MiB
[19:07] <leex-R> after running unstable 90 for 1 day
[19:09] <Iakin4> ok, I am off to bed..
[19:09] * sanity (~ian@81-178-89-164.dsl.pipex.com) Quit (Read error: 54 (Connection reset by peer))
[19:10] * sanity (~ian@81-178-89-164.dsl.pipex.com) has joined #freenet
[19:10] * ChanServ sets mode +o sanity
[19:10] <toad_> hi sanity :)
[19:10] <Iakin4> will run my node with more verbose logging for the night
[19:10] <Iakin4> cu tomorrow
[19:10] <toad_> Iakin4: show me the error
[19:10] <toad_> before you go
[19:10] <toad_> so i can find it here?
[19:10] <Iakin4> ok, w8
[19:11] <Iakin4> This is the combo you should look for:
[19:11] * Iakin4 (~nb@212.105.104.162) Quit (Excess Flood)
[19:12] <toad_> Unhandled exception java.lang.ArrayIndexOutOfBoundsException: -21474836
[19:12] <toad_> 48 in job freenet.interfaces.LocalNIOInterface$ConnectionShell@1d89cb4
[19:12] <toad_> java.lang.ArrayIndexOutOfBoundsException: -2147483648
[19:12] <toad_> at java.text.SimpleDateFormat.subFormat(SimpleDateFormat.java:934)
[19:12] <toad_> at java.text.SimpleDateFormat.format(SimpleDateFormat.java:806)
[19:12] <toad_> at java.text.SimpleDateFormat.format(SimpleDateFormat.java:778)
[19:12] <toad_> at java.text.DateFormat.format(DateFormat.java:314)
[19:12] <toad_> at freenet.support.servlet.http.HttpServletResponseImpl.addDateHeader(Ht
[19:12] <toad_> tpServletResponseImpl.java:153)
[19:12] <toad_> at freenet.client.http.ImageServlet.returnImage(ImageServlet.java:60)
[19:12] * Iakin5 (~nb@212.105.104.162) has joined #freenet
[19:12] <toad_> at freenet.client.http.ImageServlet.doGet(ImageServlet.java:32)
[19:12] <Iakin5> grrr
[19:12] <toad_> at javax.servlet.http.HttpServlet.service(Unknown Source)
[19:13] <toad_> at javax.servlet.http.HttpServlet.service(Unknown Source)
[19:13] <toad_> at freenet.interfaces.servlet.ServletContainer.handle(ServletContainer.j
[19:13] <toad_> ava:82)
[19:13] <toad_> at freenet.interfaces.LocalNIOInterface$ConnectionShell.run(LocalNIOInte
[19:13] <Iakin5> Aug 11, 2004 8:08:26 PM (freenet.TrailerReaderInputStream, YThread-51, NORMAL): Timed out waiting for chunk in freenet.TrailerReaderInputStream@10743a7: tr =freenet.MuxTrailerReader@1eaaa4b: ID=22827, curPos=16405, 0 chunks pending, 119 chunks recieved, wantChunk=true, ph=freenet.PeerHandler@bda96b (DSA(f8b1 8308 d0d9 3b38 4dd9 fba2 8109 738b 4dc1 7ece),tcp/binasys.ath.cx:34209, sessions=1, presentations=3, ID=DSA(f8b1 8308 d0d9 3b38 4dd9 fba2 8109 738b 4dc1
[19:13] <Iakin5> at freenet.TrailerReaderInputStream.read(TrailerReaderInputStream.java:85)
[19:13] <Iakin5> at freenet.support.io.VerifyingInputStream.read(VerifyingInputStream.java:86)
[19:13] <toad_> rface.java:268)
[19:13] <toad_> at freenet.thread.YThreadFactory$YThread.run(YThreadFactory.java:285)
[19:13] <Iakin5> :)
[19:13] <toad_> argh, i hate library bugs
[19:13] <toad_> if indeed it is one, which it looks like...
[19:13] <leex-R> food!
[19:13] <toad_> Iakin5: IndexOutOfBoundsException in DateFormat.format()
[19:13] <toad_> very strange
[19:13] <Iakin5> And Aug 11, 2004 8:08:26 PM (freenet.node.states.data.EatData, YThread-51, NORMAL): Connection died while eating data, suck=freenet.support.io.VerifyingInputStream@15244c2, datalength=1049900, bytesRead=16405, in=freenet.TrailerReaderInputStream@10743a7: tr =freenet.MuxTrailerReader@1eaaa4b: ID=22827, curPos=16405, 0 chunks pending, 119 chunks recieved, wantChunk=true, ph=freenet.PeerHandler@bda96b (DSA(f8b1 8308 d0d9 3b38 4dd9 fba2 8109 738b 4dc1 7ece),tcp/b
[19:13] <toad_> anyway seeya
[19:13] <leex-R> hay i am at 0.1FPS sec here
[19:14] <Iakin5> There you have those two relevant log messages
[19:14] <toad_> okay
[19:14] <toad_> will find
[19:14] <toad_> later
[19:14] <Iakin5> They always come in that pair
[19:14] <Iakin5> cu
[19:14] <leex-R> cya
[19:15] <toad_> dateFormat.format(new Date(date))
[19:15] <toad_> how can that POSSIBLY generate an out of bounds error?
[19:17] <toad_> if that happens in the logger, we're buggered
[19:17] <toad_> so why does it only happen there?
[19:18] <leex-R> dono
[19:19] <leex-R> what does it do
[19:32] <toad_> hmmm
[19:33] <toad_> partly the unstable messup was caused by my unstable node somehow being reconfigged not to accept conns from dodo on HTTP...
[19:33] <toad_> i think i lost the config file at some point...
[19:34] <leex-R> so toad_ broke unstable ?
[19:34] * Gunnar (~gunnar@34.80-203-209.nextgentel.com) Quit (Remote closed the connection)
[19:34] <toad_> leex-R: well partly
[19:34] <leex-R> heh
[19:34] <toad_> however the seednodes should not rely on my node alone
[19:34] <leex-R> how can one node do that
[19:34] <toad_> exactly
[19:34] <leex-R> thats very bad
[19:34] * lostlogic (~lostlogic@node-402418b2.mdw.onnet.us.uu.net) has joined #freenet
[19:35] <toad_> leex-R: it's because all the seednodes sources went to stable
[19:36] <toad_> hi lostlogic !!
[19:37] <leex-R> i need to turn back on join and part
[19:37] <lostlogic> hi
[19:37] <lostlogic> what's up?
[19:38] * greycat (rfc1413@wooledge.org) has joined #freenet
[19:39] <toad_> unstable seednodes got messed up
[19:39] <toad_> because of a bunch of factors
[19:39] <leex-R> that why it was so small
[19:39] <toad_> mostly because the seednodes sources all went to stable
[19:39] <lostlogic> ewps
[19:39] <toad_> and I had also turned my unstable node off
[19:39] <toad_> Timed out waiting for chunk in freenet.TrailerReaderInputStream: tr =freen
[19:39] <toad_> et.MuxTrailerReader@18b753c: ID=24768, curPos=485070
[19:40] <toad_> okay, so they are NOT all at the beginning
[19:41] <leex-R> i gess you fix soon
[19:41] <leex-R> it soon
[19:41] * studdlybiggi (~noneya@12-215-171-38.client.mchsi.com) Quit (Read error: 60 (Operation timed out))
[19:41] <leex-R> any way i am off to home agane
[19:41] <toad_> yes, working on it
[19:42] <toad_> waiting for chunk in freenet.TrailerReaderInputStream: tr =freen
[19:42] <toad_> et.MuxTrailerReader@7c802f: ID=7228, curPos=126960
[19:42] <leex-R> back in antoher 18hrs or so
[19:42] <leex-R> i gess thats an error then
[19:42] <toad_> the point is these are late timeouts
[19:43] <toad_> whereas Iakin was getting all early timeouts
[19:43] <lostlogic> toad_: so I was trying to keep my node up for a long time to see if it would crash even if I"m not doing splitfiles, but then my DSL went down for 14 hours
[19:43] <toad_> also i'm not getting many of them
[19:43] <leex-R> 12-Aug-2004 00:55:12 (freenet.node.states.announcing.Announcing, YThread-3, NORMAL): Found 0 announcement targets for this node.
[19:43] <toad_> makes me think maybe it's due to some recent code changes...
[19:44] <leex-R> toad_ you got an new unstableref file
[19:44] <toad_> leex-R: hmm?
[19:44] <leex-R> as my second node will not find any
[19:44] <toad_> yes, i will have soon
[19:45] <toad_> otherwise i can dcc you one now
[19:45] <leex-R> my fist one has 100 (i think) with 50 open connects
[19:45] <leex-R> 1 sec
[19:45] <leex-R> ready
[19:46] <toad_> leex-R: your what?
[19:46] * remote-lex (~le@spc1-warr1-3-0-cust198.bagu.broadband.ntl.com) has joined #freenet
[19:46] <toad_> you have 50 open conns on unstable?
[19:46] <toad_> if so, you have more than i do !
[19:46] <leex-R> yep
[19:46] <toad_> i have 16
[19:46] <leex-R> but its mixed vers
[19:47] <leex-R> dam my server BW is been battered
[19:47] <leex-R> VNC is super slow
[19:47] <remote-lex> :)
[19:48] <leex-R> node 1
[19:48] <leex-R> Connections open (Inbound/Outbound/Limit) 55 (40/15/200)
[19:48] <leex-R> Transfers active (Transmitting/Receiving) 30 (14/16)
[19:48] <leex-R> Data waiting to be transferred 12 Bytes
[19:48] <leex-R> Total amount of data transferred 1,661 MiB
[19:48] <remote-lex> node 2
[19:48] <remote-lex> ok not past 0
[19:48] <remote-lex> connectons
[19:48] <remote-lex> paste
[19:48] <remote-lex> how i gve you mine
[19:49] <remote-lex> from node 1
[19:51] <leex-R> ..
[19:51] <leex-R> # nodes: 110
[19:51] <leex-R> Fred,0.5,STABLE-1.50,50901
[19:51] <leex-R> Fred,0.6,1.51,60158 1
[19:51] <leex-R> Fred,0.6,1.51,60161 3
[19:51] <leex-R> Fred,0.6,1.51,60164 1
[19:51] <leex-R> Fred,0.6,1.51,60167 2
[19:51] <leex-R> Fred,0.6,1.51,60168 5
[19:51] <leex-R> Fred,0.6,1.51,60174 3
[19:51] <leex-R> Fred,0.6,1.51,60177 1
[19:51] <leex-R> Fred,0.6,1.51,60178 2
[19:51] <leex-R> Fred,0.6,1.51,60179 1
[19:51] <leex-R> Fred,0.6,1.51,60180 4
[19:51] <leex-R> Fred,0.6,1.51,60182 2
[19:51] <leex-R> Fred,0.6,1.51,60184 10
[19:51] <leex-R> Fred,0.6,1.51,60185 45
[19:51] <leex-R> Fred,0.6,1.51,60190 26
[19:51] <leex-R> Fred,0.6,1.51,60192 1
[19:51] <leex-R> Fred,0.6,1.51,60193 1
[19:51] <leex-R> Fred,0.6,1.52,60187 1
[19:52] <leex-R> don't think peeps are updateing
[19:52] <leex-R> but where are 91 92 an ver 52,60287 comming from
[19:52] <leex-R> (93)
[19:53] <leex-R> as well
[19:53] <toad_> leex-R: hmm?
[19:53] <toad_> it's updated
[19:54] * toad_ changes topic to 'Upgrade to 5091 (5089: mandatory reset, 5090: RNF bug fixed!) | Unstable: Upgrade to 60193 | Experimental unstable test fork terminated | Logs: http://newton.matcmp.ncc.edu/~lockej/freenet/chanlog/ | #freenet-politics and #freenet-chat are available for offtopic discussions'
[19:54] <leex-R> yes hmm
[19:54] <lostlogic> toad_: is latest unstable CVS incompatible with the stable network?
[19:55] <toad_> lostlogic: yes
[19:56] <lostlogic> tragic
[19:56] <lostlogic> I have to choose again...
[19:56] <remote-lex> updite time
[19:56] <leex-R> hmm
[19:57] <leex-R> server not updated
[19:57] <remote-lex> wana update before i sleep on keybord
[19:59] * salahx (salahx@sc1-24.217.174.147.charter-stl.com) has joined #freenet
[20:01] <remote-lex> lo salahx
[20:01] <salahx> Hi remote-lex
[20:02] <leex-R> lo
[20:02] <leex-R> thowing stones here
[20:04] <remote-lex> yorn
[20:04] <remote-lex> i will update tomoro
[20:04] <remote-lex> need sleep
[20:04] * remote-lex slaps leex-R around a bit with a large trout
[20:05] * leex-R throws a keyboard at remote-lex
[20:05] <leex-R> ops i forgot this pc has no keybord
[20:05] <toad_> eek
[20:05] <leex-R> or monitor or mouse
[20:06] <remote-lex> who says windows can be shit some times
[20:06] * remote-lex that be me more then likey
[20:06] <toad_> remote-lex: that's not controversial. the question is is it so ALL the time?
[20:07] <remote-lex> no
[20:07] <leex-R> (OS) Windows 2000 Enterprise Server, Service Pack 4 (5.0 - 2195), (installed for) 77w 3d 17h 40m, (uptime) 4w 4d 2h 37m, (record) 10w 1d 42m 38s (set on 06.02.2004 23:34 using Win2K)
[20:07] <remote-lex> mite try and bracky my record
[20:08] <remote-lex> (can i blame this keybord agane)
[20:08] <salahx> ok, there, NOW I have connection on unstable :)
[20:09] <remote-lex> salahx think you found me
[20:09] <remote-lex> my open connection just whent up by one
[20:09] <salahx> probably
[20:10] <remote-lex> grr
[20:10] <remote-lex> say agane
[20:11] <remote-lex> with mirc open
[20:11] <remote-lex> cya tomoro
[20:11] * remote-lex (~le@spc1-warr1-3-0-cust198.bagu.broadband.ntl.com) Quit ()
[20:11] <leex-R> new ver tomoro
[20:11] * leex-R is now known as leex-Zzzzz
[20:12] * leex-Zzzzz no trying to kill me server :))
[20:12] <leex-Zzzzz> nn
[20:51] <toad_> WTF? 404 on http://freenetproject.org/seednodes/unstable.ref
[20:52] <toad_> bbl zzz
[20:59] <salahx> nto good
[20:59] <salahx> well I got 'em before that so they cna borrow mine in the meantime :)
[21:00] <salahx> http://home.earthlink.net/~salahx/freenet/unstable.ref.gz - if anyone needs unstable seednodes
[21:04] <salahx> Connections open (Inbound/Outbound/Limit) 27 (22/5/200)
[21:04] <salahx> Transfers active (Transmitting/Receiving) 175 (106/69)
[21:04] <salahx> Whoa
[21:07] <salahx> and my I'm not runnign anything!
[21:09] * Zorix (Brandon@fl-64-45-226-120.dyn.sprint-hsd.net) Quit (Read error: 110 (Connection timed out))
[21:16] * Gunnar (~gunnar@34.80-203-209.nextgentel.com) has joined #freenet
[21:57] * lostlogic (~lostlogic@node-402418b2.mdw.onnet.us.uu.net) Quit ("Going to the moon")
[22:01] * greycat (rfc1413@wooledge.org) Quit ("Lick Bush in '04")
[22:13] * Zorix (Brandon@fl-64-45-226-120.dyn.sprint-hsd.net) has joined #freenet
[22:27] * jay (jcl@ool-18bf6dac.dyn.optonline.net) has joined #freenet
[22:46] * thelema|away is now known as thelema
[22:47] <thelema> salahx: 175 transfers over 27 nodes? I wonder if they're all hogs or if it's just one.
[22:47] <KenMan> impressive, in any case... :(
[22:51] <salahx> dunno
[22:51] <thelema> actually, # transfers per node isn't really that important... it's # requests...
[22:51] <salahx> Connections open (Inbound/Outbound/Limit) 35 (27/8/200)
[22:51] <salahx> Transfers active (Transmitting/Receiving) 152 (121/31)
[22:51] <salahx> Data waiting to be transferred 454 KiB
[22:51] <salahx> Total amount of data transferred 300 MiB
[22:51] <thelema> # requests can lead to a lot of xfers, but so can a smaller # of good requests.
[22:51] <thelema> as opposed to lots of probes.
[22:52] <thelema> 300MB in 152 xfers?
[22:52] <thelema> something's wrong.
[22:52] <salahx> yep
[22:52] <salahx> and my node is udle
[22:52] <thelema> oh wait, what's wrong is the number.
[22:52] <thelema> 454K in 152 xfers. meaning the average xfer doesn't have much data to send.
[22:53] <thelema> so why aren't the transfers finishing? maybe the 23 K of StoreData?
[22:53] <salahx> so about 4k per tranfer then ?
[22:54] <salahx> if everyone is running Frostm tha'ts reasonabl
[22:55] <salahx> unfeoruntly my log is full ogf timed out adn killed requests, not surprsiingly
[22:55] <salahx> plus the usual "No point 0" and "Minimum request inveral violated!" messages
[22:57] <KenMan> thelema: isn't Data waiting to be xferred relative to the transport layer, or something like that ? Like buffers allocated, and blocked on 'outgoingBWlimit' setting ??
[22:58] <salahx> Transfers active (Transmitting/Receiving) 145 (120/25)
[22:58] <salahx> Data waiting to be transferred 7,012 Bytes
[22:58] <KenMan> and not the amount of data remaining unsent, added over for all active transfers ?
[22:58] <KenMan> yeah, that is a normal / typical report I'm used to looking at without wondering what it means... !
[23:00] <thelema> salahx: 3K
[23:01] <thelema> KenMan: I don't think so because it includes (iirc) data yet to be recieved
[23:01] <salahx> well 4k IS about 3k (I chose 4k becasue all CHK sizes are powerse of 2)
[23:02] <thelema> yeah, but iirc, keys are all >=8K
[23:02] <KenMan> I have some that are 1205 bytes in the store, fwiw
[23:03] <KenMan> valid range is 2^10 - 2^20 ...
[23:03] <thelema> really? hmm... I may have to revise that
[23:03] <thelema> (in my memory, not in fred)
[23:03] <KenMan> there was some quirkiness about it though. I never saw 1K keys being requested, only 2K and up.
[23:04] * Zorix (Brandon@fl-64-45-226-120.dyn.sprint-hsd.net) Quit (Read error: 110 (Connection timed out))
[23:04] <KenMan> but 1K is a valid size ?! that 21st byte can go from 10 to 63, i think. But in reality it is bound at 20.
[23:04] <salahx> yeah, its capped at 20
[23:04] <salahx> there are no 1k keys in my datastore
[23:05] <salahx> maybe there's an off-by-1 error somewhere
[23:05] <salahx> Which may explain why short Frost message seem to disappear in the ether...
[23:05] * KenMan hates trying to learn how things are 'supposed to work' by reading fred's source code
[23:06] <KenMan> it's never any more accurate than the 3 conflicting responses you can find here in this channel :p
[23:06] <thelema> feh, of course it is; as long as you read it right.
[23:06] <KenMan> well, source code IS the ultimate reference, but sometimes it is not the intended logic :(
[23:07] * KenMan pulls out his rose colored shades :)
[23:07] <thelema> it's a bit harder than reading the channel.
[23:07] <salahx> well a flwochat woudl be handy
[23:08] <thelema> That's not a too bad idea for most of the subsystems at least.
[23:08] <KenMan> it is probably no coincidence that no high level design ever existed...
[23:08] <thelema> Hobx and Scipient just weren't very good at writing documentation. And they were best at working on the fly, and not from a plan.
[23:08] <salahx> side effect of OOP maybe
[23:09] <KenMan> typical engineering, for sure ;)
[23:10] <KenMan> my last job assumed that all engineers were perfectly interchangable and replaceable, thus they demanded we spend 4 times more on design than implementation. And 8 times sitting in meetings , watching non-technical people assert their superiority...
[23:11] <KenMan> design and documentation. The typical "but if you got hit by a bus, how would we make the software run?" without much concern for the thing that slowed the bus
[23:12] * jay is fetching "Outfoxed"
[23:12] <thelema> salahx: part of it was they were having way too much fun making freenet as java as possible (using reflection and nesting classes many many deep)
[23:13] <salahx> i dunno what reflection is
[23:14] <KenMan> it allows you to write more complicated software, so that an object can ask another object "what kind of object are you ? what are you doing here ? why am I responsible for you ? is there someone else I can pass you off on ? "
[23:14] <KenMan> things like that
[23:14] <salahx> like RTTI ?
[23:15] <jay> i recall using reflection to call dynamically constructed method names
[23:15] <KenMan> at least, yeah, i am pretty sure it is like RTTI
[23:15] <salahx> THat doesn't sounds liek a very good idea though, woudln't leak too much information abotu the class heiarchy
[23:16] <jay> salahx: oop is more about minimizing the ripple effect, not hiding code
[23:17] <thelema> jay: if only the early developers knew about that. stupid interfaces for everything.
[23:17] <jay> eww
[23:17] <jay> yeah the code is pretty ugle in places
[23:17] <KenMan> interfaces are wonderful when used in certain contexts. And not so wonderful when over-applied.
[23:17] <jay> ugly
[23:18] <jay> i browse it occasionally looking for hints on FCP related stuff
[23:18] <KenMan> interfaces are a software designer's wet dream, at least, for those designers who don't actually have to write any of the implementations...
[23:18] <thelema> jay: re your suggestion on metadataredirect
[23:18] <jay> KenMan: heh
[23:19] <KenMan> you can earn super-big bucks, and leave a trail of pain and misery in your wake. Who wouldn't want a job like that ??
[23:19] <thelema> I think it's a bad idea. I can't quite put my finger on problems with it. Maybe it has to do with me thinking that standardizing key sizes is a bad idea, but...
[23:19] <KenMan> right now, 1M keys are the most commonly transferred size
[23:20] <KenMan> followed by 4K keys
[23:20] <KenMan> all the rest combined make up the other 5%
[23:20] <jay> KenMan: quite a spread
[23:20] <KenMan> and in the past month or two, 1M keys reached about 2:1 versus 4K keys
[23:21] <KenMan> which means, if you chop 1M keys into 32 little 32KB keys, well, then you have 32 times as many keys that need to be pushed around
[23:21] <jay> thelema: at this point (in fcpput) it's an option; by default it stores the metadata directly in the proper key
[23:21] <KenMan> just to accomplish what you already can do today. Is routing really that good ??
[23:21] <jay> KenMan: yeah im not sure
[23:22] <salahx> on one hand, that means more overhead. OTOH, lots of tranfers get killed and we have to start over...
[23:22] <jay> i don't fully understand routing
[23:22] <jay> my CS theory is pretty bad
[23:22] <KenMan> don't worry jay , apparently none of us do just yet :(
[23:22] <jay> eheh
[23:22] <jay> im not sure what effect having more files or smaller size will have
[23:23] <KenMan> all i'm saying, is that if you have to have 32 times as many successes , just to obtain the same data, in a sea of 32 times the requests... well, i don't know what that means.
[23:23] <jay> my typing is bad..
[23:23] <KenMan> but it makes me nervous as hell
[23:23] <jay> more files of smaller size.. not sure
[23:24] <KenMan> the most likely benefit is that all transfers will exhibit more consistent behavior - more similar transfer times, and , uh, in NGR where it attempts to normalize 20 different key sizes in some weird algorithms... well...
[23:24] <jay> how do other comparable systems store data?
[23:24] <KenMan> comparable ?
[23:24] <jay> other p2p systems.. as far as transferring data
[23:24] <jay> fixed vs. variable sizes
[23:25] <jay> i understand toad's argument for it
[23:25] <salahx> I think the only other netowrk that does this kind of thing is eDonkey
[23:25] <KenMan> some really really respectable and important paper written by some dude with a degree said that 128KB was the exact perfect answer for all P2P systems, regardless of the topology. Aren't YOU impressed ?
[23:26] <jay> KenMan: anything is possible as far as im concerned
[23:26] <jay> if someone proved that i would easily accept it
[23:26] <salahx> Trailers only get 5 minutes to finsh before they are killed
[23:26] <salahx> even wirth no other traffic, it woulk at least 1 minute to get a 1MB key
[23:27] <jay> salahx: any specifics?
[23:27] <jay> salahx: err u talking about edonkey?
[23:27] <KenMan> trailers only get 5 minutes to DO NOTHING. They can eek out a basic living by skulking along at 2bytes/sec for the next decade.
[23:28] <salahx> oh, my bad then
[23:28] <KenMan> at least, that's how i understand it - 5 minutes idle time is the chopping point
[23:28] <KenMan> i don't even begin to pretend to know this stuff. Just as soon as I think I have some basic concept down, I come to find I am critically wrong in my perception.
[23:28] <salahx> now I'm not sure
[23:29] <KenMan> i get used to it
[23:29] <jay> i never am
[23:29] <KenMan> wait, actually, that's my only reason for being here. i LIKE to pretend that I understand the whole deal :o
[23:29] <jay> i figure one day something will click and ill learn it
[23:30] <salahx> the logs are jstu FULL of trailer timeouts
[23:30] <jay> im focused on other mundane things for the moment including fcplib
[23:30] <salahx> you have any idea how the FEC stuff works ?
[23:30] <jay> salahx: yeah.. on 5090?
[23:30] <salahx> the algoritm in general
[23:30] <jay> FEC-FCP yes
[23:30] <jay> oh well a little
[23:31] <jay> the actual algorithm i've wanted to implement to do native encoding/decoding i haven't gotten around to understanding
[23:31] <salahx> i understand a bit
[23:31] <jay> at least so far to integrate it with fcplib internally
[23:32] <jay> if i knew how to encode a file into x data blocks and y check blocks using the code from Onion it would work
[23:32] <salahx> I certinaly understand why its so slow though - all the plynomials and matrix inversions....
[23:33] <jay> now with the native integers it's real quick.. but i still want fcplib to do it natively
[23:33] <jay> instead of sending the file over the wire
[23:33] <jay> just to FEC
[23:35] <jay> salahx: so i divide up the file into X-byte blocks and do all kinds of matrix operations ?
[23:35] <salahx> Soemthing like that
[23:35] <salahx> Essenially, the data is geernated into a row matrix of n blocks eack q bytes each
[23:35] <salahx> Exactly what "q" shoudl be, I dunno
[23:36] <jay> does it need to load an entire segment at a time?
[23:36] <jay> sounds like it would
[23:37] <jay> then we'd have to change the algorithm perhaps to handle more blocks per segment
[23:37] <jay> or a non-segmented scheme
[23:37] <jay> dunno about that
[23:37] <salahx> teh segments exist because it really slow
[23:37] <salahx> the forula can onyl handle so much, beyond that; its too slow to bother
[23:38] <salahx> there's a 8-buit version and 126-bit version
[23:38] <salahx> What the 8 and 16 refer to, I don;t know
[23:38] * thelema returns and notices conversation on FCP
[23:38] * thelema is the FCP guru
[23:38] <jay> FEC really
[23:38] <salahx> the 8-bit version is al ot faster because we can "pre-cook" the tables, so instead of have to do the math we can jsut look it up on table
[23:38] <thelema> on yeah, and the FEC guru
[23:39] <jay> thelema: does the current algorithm need to load an entire segment to encode in the worse case?
[23:39] <thelema> the number of bits is the size of the finite field the math is done in.
[23:39] <thelema> jay: in best case it does. There's no way to have any check block be able to recover any data block without it.
[23:40] <jay> right now that means a splitfile can eat 128M of ram while being encoded / decoded
[23:41] <jay> well memory to be precise
[23:41] <thelema> yup.
[23:42] <jay> so if a fixed-block size is implemented, our current FEC algo will be obsolete?
[23:42] <thelema> one nice side-effect of the xor-style FEC that is specced in the metadata standard is that it doesn't require such overheads; decoding can be done in little memory.
[23:42] <toad_> <KenMan> just to accomplish what you already can do today. Is routing really that good ?? - no, but 32x more keys means routing is 32x BETTER, not worse! anyway I'm not here.
[23:43] <thelema> yes, if we go fixed size, I'll have to implement a good FEC in fred.
[23:43] <jay> thelema: any idea what type of algo may be used?
[23:43] <thelema> yes. I have implementations in ocaml. How much detail do you want?
[23:43] <jay> toad_: not hi
[23:44] <jay> thelema: general knowledge first
[23:44] <jay> i don't know how the current one works precisely even
[23:44] <salahx> i have a vague idea how ti works
[23:44] <jay> thelema: anything online i would read?
[23:45] <thelema> jay: basic idea is to generate check blocks such that each check block is the bitwise xor of some set of data blocks
[23:45] <thelema> to decoe, get all but one of those data blocks together with the check block, XOR what you have, and out pops what you don't.
[23:45] <thelema> *decode
[23:46] <thelema> jay: most of the literature is a bit obtuse. Although the "rateless codes" papers aren'
[23:46] <thelema> t that bad.
[23:46] <salahx> I beleive this is the formula used: http://search.cpan.org/src/MLEHMANN/Algorithm-FEC-0.5/README.fec
[23:46] <jay> thelema: i tried lookin for info but didn't know what specifically i was even interested in
[23:47] <jay> i think the 'g' key on this keyboard is a bit broke
[23:48] <jay> ah got some hits
[23:49] <jay> Rateless codes are like FEC, but they are infinite and can guarantee the file is downloaded successfully. For more info look at the relevant discussion on freenet-devl.
[23:49] <jay> heh from a frost page
[23:49] <thelema> salahx: that's correct; ]
[23:49] <jay> the current algo yes?
[23:49] <thelema> jayL: yes, the current implementation uses vandermonde codes.
[23:50] <jay> thelema: i actually brought that webpage (printout) to a math professor at work (school) and asked him a bunch of questions
[23:50] <salahx> we take the data, create it into x blocks of quest size; do something (matrix multplication), with a Vandermonde martix based on the original number of block and how many check block
[23:50] <jay> salahx: that README i mean
[23:51] <salahx> so if we 10 data blocks, adn wants 50% reducuancy, we create a 15x10 matrix
[23:51] <thelema> salahx: the thing with that file is that it doesn't really explain how they work; that description is the description of *all* linear FEC.
[23:51] <jay> do rateless code algo's differentiate between data and check blocks; everything is spread over the segment?
[23:51] <salahx> ohhh
[23:51] * Ash-Fox (Hal-9000@aao173.neoplus.adsl.tpnet.pl) has joined #FreeNET
[23:52] <jay> i mean can any set X of Y blocks be used to reconstruct the whole?
[23:52] <salahx> Yes
[23:52] <thelema> jay: no. rateless codes are local; only certain data blocks can be reconstructed by any given check block.
[23:52] <salahx> so if we genentate 15 block of the original 10, and get an 10, we can reconsutct the whole file
[23:53] <salahx> i duno abotu ratless codecs
[23:53] <salahx> that
[23:53] <salahx> tha'ts how the current formual works though
[23:53] <thelema> vandermonde codes aren't local, so they require the whole segment to encode a single check block, but can use any X of Y
[23:54] <thelema> rateless codes come really close to X of Y when the blocks are randomly given, and I believe in freenet, since we get to choose what blocks are requested, we can get an average case of X of Y (even though theoretically that's improbable).
[23:55] <thelema> s/theoretically/in the standard models of FEC communication/
[23:57] <jay> ill do some reading
[23:58] <toad_> thelema: they have to be chosen randomly for propagation. oh and if you could think about adapting FCP to be a) multiplexed with a nice simple protocol so we have a single FCP conn per client and we can rate limit them as single entities, and b) have explicit support for queueing and checking status of live requests, then that'd be something useful as a basis of future discussions on devl re FCP and rate limiting. Oh and I'm STILL
[23:58] <toad_> just don't like going to bed, and I get reading stuff... bad mistake). bye.
[23:59] <thelema> toad_: they can be random but still directed when it matters
These logs were automatically created by Jay Oliveri with his gimp hapi on irc.freenode.net.
