2016-07-30T00:18:24Z rme: pierpa: Have I ever had the pleasure of meeting you in real life at a Lisp conference or something? 2016-07-30T00:45:48Z ekinmur joined #ccl 2016-07-30T00:45:54Z ekinmur quit (Client Quit) 2016-07-30T00:52:29Z ekinmur joined #ccl 2016-07-30T01:01:02Z ekinmur quit (Quit: My Mac has gone to sleep. ZZZzzz…) 2016-07-30T01:06:32Z DougNYC quit 2016-07-30T02:02:50Z pierpa: hi rme, no, we never met 2016-07-30T02:04:50Z pierpa: (my name is Pierpaolo Bernardi, btw) 2016-07-30T02:27:54Z rme: pierpa: OK, thank you. I've been able to go to several European lisp meetings, and since the Lisp world is not all that large (at least in my experience), I just wondered. Maybe one day. 2016-07-30T02:29:31Z pierpa: the only lisp conference I attended was Euro Lisp 2009 in Milan. 2016-07-30T02:31:38Z rme: I wasn't at that one. 2016-07-30T02:32:03Z pierpa: yeah, I would have remembered you :) 2016-07-30T02:33:01Z rme: I suppose Marco Antoniotti must have helped organize that. 2016-07-30T02:33:06Z pierpa: yes, him 2016-07-30T02:33:35Z pierpa: it was at his home univ 2016-07-30T03:09:17Z pierpa quit (Ping timeout: 258 seconds) 2016-07-30T08:57:01Z edgar-rft quit (Quit: edgar-rft) 2016-07-30T09:30:57Z rme quit (Quit: rme) 2016-07-30T09:30:57Z rme quit (Quit: rme) 2016-07-30T10:11:04Z DavidGu joined #ccl 2016-07-30T10:45:43Z DavidGu quit (Read error: Connection reset by peer) 2016-07-30T10:45:51Z DavidGu joined #ccl 2016-07-30T11:37:33Z DavidGu1 joined #ccl 2016-07-30T11:37:33Z DavidGu quit (Read error: Connection reset by peer) 2016-07-30T11:37:34Z DavidGu1 is now known as DavidGu 2016-07-30T11:48:27Z DavidGu1 joined #ccl 2016-07-30T11:48:28Z DavidGu quit (Read error: Connection reset by peer) 2016-07-30T11:48:34Z DavidGu1 is now known as DavidGu 2016-07-30T12:16:35Z DavidGu quit (Read error: Connection reset by peer) 2016-07-30T12:17:40Z rme joined #ccl 2016-07-30T12:18:56Z DavidGu joined #ccl 2016-07-30T12:23:51Z DavidGu quit (Ping timeout: 264 seconds) 2016-07-30T12:25:45Z DavidGu joined #ccl 2016-07-30T12:29:52Z DavidGu quit (Ping timeout: 252 seconds) 2016-07-30T12:37:53Z DavidGu joined #ccl 2016-07-30T13:59:43Z DavidGu quit (Quit: DavidGu) 2016-07-30T14:08:17Z hhdave joined #ccl 2016-07-30T15:03:07Z rme quit (Quit: rme) 2016-07-30T15:09:07Z ekinmur joined #ccl 2016-07-30T15:34:13Z rme joined #ccl 2016-07-30T16:34:17Z ekinmur quit (Quit: My Mac has gone to sleep. ZZZzzz…) 2016-07-30T16:41:11Z edgar-rft joined #ccl 2016-07-30T17:14:05Z rme quit (Quit: rme) 2016-07-30T17:14:05Z rme quit (Quit: rme) 2016-07-30T17:55:14Z pierpa joined #ccl 2016-07-30T18:40:15Z rumbler31 joined #ccl 2016-07-30T20:06:03Z rumbler31: I can see how (get-internal-run-time can be used to profile my code in regards to execution speed, but what can I do to understand more of why my code may be taking up so much time idoing GC 2016-07-30T20:06:07Z rumbler31: doing* 2016-07-30T20:06:50Z pierpa: there's no simple recipe, afaik 2016-07-30T20:27:23Z rumbler31: i'm on really slow internet, if this paste would ever come up I could provide some context 2016-07-30T20:34:29Z rumbler31: http://paste.lisp.org/display/321725#1 2016-07-30T20:35:07Z rumbler31: this is what i'm doing. similar (but worse) than pcl's chapter on binary files and id3 tag parsing. I take a description of a class and produce this 2016-07-30T20:36:32Z rumbler31: with code i'm working with, I have two such classes defined, and methods to dispatch to them when my code detects the right field, otherwise a simple array is passed along. I make a list of all of these, either objects of arrays, and my code spends up to 80% of its time in GC 2016-07-30T20:36:53Z pierpa: hmmm 2016-07-30T20:37:00Z rumbler31: the two classes i've made have 50+ fields, the paste is a demo of all of the field types I support 2016-07-30T20:37:14Z pierpa: k 2016-07-30T20:37:45Z pierpa: I can't spot any OBVIOUS inefficiency... 2016-07-30T20:38:25Z rumbler31: i'm about to write code that just reads 1k chunks of a file and makes arrays and passes them up, to see what the theoretical "fastest" case 2016-07-30T20:41:14Z pierpa: you could try declaring the fixnum variables as such 2016-07-30T20:42:24Z pierpa: could change (expt 2 x) to (ash 1 x) (but this won't change allocation) 2016-07-30T20:45:20Z pierpa: I don't think that declaring NEWBYTE dynamic-extent has any effect. Declaring it fixnum instead may have a (small) effect. 2016-07-30T20:46:57Z pierpa: same for the second dynamic-extent declaration 2016-07-30T21:23:41Z ekinmur joined #ccl 2016-07-30T21:56:33Z |3b|: yeah, would expect types to help more than dynamic-extent for small numeric variables 2016-07-30T21:59:39Z |3b| would try removing byte-size as an argument and using 8 directly, and use logbitp instead of (> newbyte ...) 2016-07-30T22:00:25Z |3b|: (if using 8 directly helps and you actually use other sizes, either have separate functions or a special case for 8 and any other common sizes) 2016-07-30T22:02:04Z |3b|: rumbler31: actually your use of dynamic-extent is actively wrong :p you declare OUTPUT dynamic-extent then return it from the dynamic extent of the corresponding scope 2016-07-30T22:02:37Z rumbler31: i was thinking about making read-int-from-stream as a macro that returns a loop that incf's directly into the slot instead 2016-07-30T22:02:45Z |3b|: and FORMAT is probably a very inefficient way to do whatever you are doing there 2016-07-30T22:03:53Z |3b|: ah, i guess might not be too horrible, but probably faster to just store characters directly into the string yourself 2016-07-30T22:04:42Z rumbler31: i just prophylactically allowed for specifying the byte size. and I wrote that string code a while ago, I should take another look at that 2016-07-30T22:05:30Z rumbler31: oh, yea I guess I don't fully understand the dynamic-extent business 2016-07-30T22:05:55Z |3b|: yeah, it isn't unreasonable, there are lots of formats with values in strange sizes, but i wouldn't be surprised if the compiler could generate better code for known sizes 2016-07-30T22:06:18Z |3b| would expect it to in SBCL for example, not as sure about CCL 2016-07-30T22:07:20Z |3b|: limiting NEWBYTE to some fixed size <= word size would probably help also if possible (again possibly handling larger sizes separately if needed) 2016-07-30T22:08:27Z |3b|: possibly inline READ-INTEGER-FROM-STREAM as well, particularly if word-sized values are common (since implementations usually have to store word-sized values in memory to pass them into or out of functions, even if they could be in a register within a function) 2016-07-30T22:09:10Z |3b|: and similarly it might be able to optimize for specific values of the size and signed arguments at usage sites 2016-07-30T22:15:58Z rumbler31: i remember trying to do this once, and sbcl and ccl differed in what they expected in terms of type declarations etc 2016-07-30T22:16:35Z rumbler31: like i guess what you're saying is that if I were to declare things like, newbyte and byte-size were always fixnums, or even '(unsigned-byte 8) 2016-07-30T22:16:55Z rumbler31: er *byte-count 2016-07-30T22:17:17Z pierpa: yes 2016-07-30T22:18:19Z |3b|: newbyte will usually need to be larger than 8 bits 2016-07-30T22:18:45Z pierpa: I think that that dynamic-extent declarations are noops. You could try comparing the disassemblies with and without the declaration 2016-07-30T22:19:00Z |3b|: (if it isn't you might want to add a special case for 1-byte values) 2016-07-30T22:22:03Z |3b|: it would be hard to stack-allocate an unknown-sized/variable-sized variable like an arbitrary numeric value (not that it even knows that in advance in that code) 2016-07-30T22:22:29Z |3b|: and a fixed size integer <= word sized would probably be on stack or in a register anyway 2016-07-30T22:22:56Z |3b|: the array could be stack allocated, but then you have a problem when it returns a value on the stack after unwinding past it 2016-07-30T22:23:33Z |3b|: (a particular compiler might be smart enough to make a copy or refuse to stack allocate it in that case, but no guarantees of that) 2016-07-30T22:27:29Z rumbler31: oh right newbyte's final bit length is unknown... 2016-07-30T22:28:04Z rumbler31: i'm almost certain I'm spending the majority of my time in this function 2016-07-30T22:28:14Z |3b|: yeah, sounds plausible 2016-07-30T22:28:32Z rumbler31: for 1 byte values I use read-byte directly 2016-07-30T22:30:21Z rumbler31: well then I think I don't know what to do 2016-07-30T22:30:22Z |3b|: that top part is generated by a macro or something? 2016-07-30T22:33:30Z |3b|: top part returns a value declared dynamic-extent also, which is pretty much the definition of "not dynamic-extent" :p 2016-07-30T22:33:48Z rumbler31: yea 2016-07-30T22:34:58Z rumbler31: I guess I could make explicit functions for each byte size 2016-07-30T22:35:03Z |3b|: if you are generating the code that calls READ-INTEGER-FROM-STREAM, it would probably be reasonable to generate specialized code for at least the common cases 2016-07-30T22:35:33Z rumbler31: yea, I am. I could do that 2016-07-30T22:37:15Z |3b|: (maybe keep the old function around for unusual cases, and add specific ones as you find formats that need them and affect performance) 2016-07-30T22:37:57Z rumbler31: I wish I knew enough to know why this code isn't holding up. I didn't know this in advance and wrote it 2016-07-30T22:40:48Z rumbler3_ joined #ccl 2016-07-30T22:40:57Z rumbler3_: so in the cases where I make new read-u2 -u4 etc, does it make sense to declare the temporary value as '(unsigned-byte 16) 32) and so forth? 2016-07-30T22:43:45Z rumbler31 quit (Ping timeout: 244 seconds) 2016-07-30T22:44:07Z ekinmur quit (Quit: My Mac has gone to sleep. ZZZzzz…) 2016-07-30T22:49:32Z |3b|: yeah, that should help 2016-07-30T22:51:48Z |3b|: and is part of the reason for suggesting separate functions 2016-07-30T22:57:34Z rumbler3_: whats the other part 2016-07-30T22:57:43Z |3b|: fixing the constants 2016-07-30T22:58:07Z |3b|: and possibly knowing it can return the value without allocating storage for it 2016-07-30T22:58:35Z |3b|: where "fixing the constants" might also include manually unrolling the loop 2016-07-30T22:59:02Z rumbler3_: (declare (type '(unsigned-byte 16) u2)) :unknown-type, declaration ignored. So I guess this isn't an n-type thing, only '(unsigned-byte 8) is defined 2016-07-30T22:59:05Z |3b|: (or better, letting a macro do so for you, if you generate the specific read-foo functions) 2016-07-30T22:59:17Z |3b|: don't quote it there 2016-07-30T22:59:30Z rumbler3_: oh 2016-07-30T22:59:32Z |3b|: (declare (type (unsigned-byte 16) u2)) 2016-07-30T22:59:48Z |3b|: you only need to quote types when they will be evaluated, like when passing to a function 2016-07-30T23:00:06Z |3b|: sorry, missed that in the earlier question 2016-07-30T23:00:41Z |3b|: and you can use any N there, though some N are more useful than others :) 2016-07-30T23:02:26Z rumbler3_: ugh that doesn't seem to be doing it, while it certainly looks useful, I must be missing something else. 2016-07-30T23:02:57Z |3b|: what does the current code look like? 2016-07-30T23:04:09Z rumbler3_: one moment 2016-07-30T23:04:11Z |3b|: might also try inlining the read-foo functions, before the function definition put (declaim (inline read-foo)) 2016-07-30T23:04:52Z |3b|: (just make sure to recompile everything that uses it when you change an inlined function) 2016-07-30T23:05:08Z rumbler3_: my emacs is hung.... i think ccl also expects something else in addition to the type declaration 2016-07-30T23:09:38Z rumbler3_: http://paste.lisp.org/display/321725#2 2016-07-30T23:10:15Z rumbler3_: basically this, where container-header-stream-reader is the result of my macro expansion, and isn't a large class, only 5 fields, 3 of them doubles 2016-07-30T23:10:55Z rumbler3_: this right now I explicitly want to use the generic method, that reads in the payload as an array 2016-07-30T23:11:19Z rumbler3_: and even doing just this has us spending beaucoup cpu time in gc 2016-07-30T23:20:38Z |3b|: does READ-SEQUENCE signal END-OF-FILE? 2016-07-30T23:22:16Z |3b|: looks like it should just return 0 and leave the sequence unmodified if it hits EOF 2016-07-30T23:22:59Z |3b|: how many times does it go through that loop, and what is the average size of the container-payload array 2016-07-30T23:27:36Z rumbler3_: i'm dumb. I used the handler-case in higher level code and forgot. read-sequence returns the number of bytes read, I need to check this return value to see if we are prematurely at eof 2016-07-30T23:30:21Z rumbler3_: one file i'm reading now has 989700 container/payload pairs 2016-07-30T23:32:22Z rumbler3_: 759.02075 avg length sequences 2016-07-30T23:36:50Z |3b|: and you are sure you are spending lots of time in GC still? 2016-07-30T23:36:59Z rumbler3_: i've annotated with what the stream-reader function would look lie 2016-07-30T23:37:39Z hhdave quit (Quit: hhdave) 2016-07-30T23:37:57Z rumbler3_: annotated with the time from the run of reading the file in question 2016-07-30T23:39:13Z |3b|: "would look like" isn't as useful as "does look like" :p 2016-07-30T23:39:35Z |3b|: for example doubles might have different effects on GC time than 16 bit ints 2016-07-30T23:40:05Z |3b|: (since a 16 bit int is a fixnum so doesn't need extra storage, while a double probably does need separate storage) 2016-07-30T23:40:44Z |3b|: doesn't have to be the actual code if it is proprietary or whatever, but something more representative of the actual types being stored in the class 2016-07-30T23:41:50Z rumbler3_: right sorry, I consider the exact binary spec to be sensitive at this time, 2016-07-30T23:43:04Z rumbler3_: well without giving much else away, the demo2 represents full code coverage of the cases i'm testing 2016-07-30T23:43:07Z |3b|: what is the output of ROOM after whatever that TIME output was from? (in a freshly started lisp if convenient) 2016-07-30T23:43:27Z rumbler3_: one second (few seconds ;-) 2016-07-30T23:43:57Z |3b|: is that actually reading a single u8 and storing it as a double? 2016-07-30T23:44:08Z |3b|: ah, 8-byte not 8-bit 2016-07-30T23:44:49Z |3b|: what is the definition of read-u8? 2016-07-30T23:45:17Z rumbler3_: i'll include that in the next paste 2016-07-30T23:46:21Z |3b|: so you have 989700 elements, each is 3 doubles, length (i assume 32bit unsigned int?), what type is the other value? 2016-07-30T23:48:33Z rumbler3_: i have that many paris of container object/payload lists, object is a few slots consisting of some doubles some int32s some strings, payload is a simple octet array 2016-07-30T23:48:37Z rumbler3_: and the other value? 2016-07-30T23:50:34Z rumbler3_: annotated, 2016-07-30T23:50:47Z rumbler3_: and dit looks like something has changed because it got faster.... 2016-07-30T23:50:55Z |3b|: you said 5 values, 3 of which are doubles. i assumed that referred to the container-header, and payload-length was one of the non-double values 2016-07-30T23:51:23Z rumbler3_: maybe I spend too much time messing around in my image and things get gunked up for reasons i don't understand 2016-07-30T23:52:09Z |3b|: yeah, might have had some old data laying around that was hard to GC or something, or old code that didn't get recompiled when it should 2016-07-30T23:52:22Z |3b|: try (room t) and see if it prints more info 2016-07-30T23:52:25Z rumbler3_: container-header itself has a handful of slots, some doubles some strings, yes a payload length of uint32 2016-07-30T23:53:21Z |3b|: and is that read-u8 declaimed inline? 2016-07-30T23:53:54Z |3b|: though i guess you pass it to another function immediately anyway, so might not help 2016-07-30T23:57:05Z rumbler3_: no it isn't 2016-07-30T23:57:49Z rumbler3_: i'm trying to make small changes at a time to see what they effect 2016-07-30T23:58:46Z rumbler3_: while everything is faster, of the group of files i'm reading in, most are over 60% gc time 2016-07-30T23:59:26Z rumbler3_: i'm appending the lists made by each file read together, and doing (time on each file read, 2016-07-30T23:59:41Z |3b| would be curious how much effect it has on time if you replace the double fields with 2 uint32 fields