have the feeling of being put under magnifying glass or microscope sometimes at work and elsewhere (some recent joking about this on the physics stackexchange hbar chat room among long oldtimers). in worst case scenario as mentioned last month it feels like being micromanaged, bullied, stalked, or hunted. maybe this is my coming-home-to-roost karma for having a cyber alter ego—decades old now. lets face it no matter how human corporations pretend to be, theyre fundamentally soulless at heart.
theres some greatness that psychology is starting to understand the negative effects of the corporate world. could really relate to this latest headline/ study, Greedy bosses are bad for business, study finds. but could it really be true? bet theres some other study that says that soulless bosses can help drive up the bottom line. but ah, also trying not to be egocentric and put it all in perspective, this is a very old complaint in historical terms, ie roughly as old as capitalism, and intensified with the industrial revolution and so-called late stage capitalism (luv that phrase! what does/ can it mean?! reminds me of the term postmodernism…). overall what one might call 1st world problems… time to take a vac…
this code has optimization and analysis sections. the optimization is to push down mxl, mxr the max 0/1 lengths on both left/ right of the glide. then it looks at the binary structure of the largest trajectories. it takes a same # of iterates starting from left and right sides of the glides, concatenates them in binary, and then analyzes the 0/1 runs. in the 1st graph its shown that the histograms (4) for left/ right 0/1 run lengths are nearly identical as found awhile back with another generation scheme, (cant recall exactly, maybe the long-examined ‘w’ widthdiff). in the 2nd graph there seems to be some slight differentiation, but this seems a rough 1st cut on finding it. strangely there seems to be a difference between odd and even lengths seen in the apparent alternating/ thrashing pattern in the graph. the 2nd graph is 0 run length histogram difference and the 3rd is 1 lengths.
overall it needs some further investigation/ polishing but seems to be real. as was working on this it occurs to me immediately that both on the left and right side, there is not a “control” for varying trajectories starting at the intermediate positions and was wondering if thats causing the results to be more uniform/ undifferentiable. from prior experiments its known that many of the intermediate trajectories or “subtrajectories” (actually subglides) starting from the intermediate points tend to have a much different aspect of terminating quickly eg on the left side. another aspect that occurs to me is that the (“quick-and-dirty”) concatenation idea might bias the measurements slightly versus the alternative of concatenating all the separate 0/1 iterate string splits because the start and end of the binary iterate are always 1s and join to make a larger combined 1-run than either separately.
(8/10) coded up some multiple experiments in a dash and they all came up with null results. am thinking it is at least worthwhile to refactor + consolidate them all because the analysis is nontrivial and the analysis subroutine grouping is coherent/ convenient so to speak (they “go together,” ie like tools in a toolbox). the idea is to look at ½-density iterate evolution for large 1000 bit width iterates. the idea with the large size is that breaking down into 0/1 runs gives more samples for histograms. there is a significant spread in the postdetermined iterates after 1000 iterations wrt eg bit size and other statistics such as glide peak index and glide length etc. but none of this seems to be quantifiable/ predictable in terms of the known binary features of the original seeds.
this is all again a very strong demonstration of the now-familiar curse of undifferentiability. many of the prior features/ analysis was applied and it all comes up empty. it is reminiscent of how in cryptography, arbitrary signals can be encoded in what appears, under statistical examination, to be uniform random noise. this seems somewhat in contrast to similar exercises for Terras-generated distributions studied at length on 2/2019. in particular the very notable
construct9c experiment was able to find a very subtle but identifiable/ differentiable bias in the 0/1 runs histograms. it appears to me from latest analysis this doesnt occur in the random ½-density iterates. and it would be very worthwhile to try to pin this down a little further. it seems to show that even though Terras-generated iterates seem to run through the ½-density region theyre somehow different.
pondering these results am thinking more about the
vary1 experiment from 12/2018. that showed some identifiable trend for ½-density seeds in density/ entropy distances. am thinking now that very few experiments have shown anything similar and want to re-survey prior analysis to find any, but think they are very scattered. this seems to be some key to understanding differentiability in ½-density seeds.
(8/12) ok here it is after substantial refactoring effort. this has some nice streamlining of a lot of scattered code. now have that feeling of something like cleaning up a room and being a little tired/ sweaty afterwards… some entropy of the universe has been converted to order thru human effort/ intervention. one nice feature is how it converts all the analysis code to output separate self-contained graphs on the filestream ie a more reusable pattern (and there is a lot of call for that given all the scattered/ specialized/ nonuniform related graph code). it also includes the
"histcurve" algorithm associated with
construct9c code to look at/ normalize histogram trends. theres also an idea about/ code for combining the 0/1 runs into pairs for similar analysis as was seen in an isolated older experiment. but, am omitting the graphs for now to save time because they seem to be mostly null results (but not to be taken as an indication that theyre not worth looking at, ie/ aka “exercise for the reader”).
some caveat applies to the known overplotting bias for some of the diagrams ie sometimes patterns seem to appear that dont exist. overall anyway these are all powerful statistical analysis routines that have all revealed significant structure in different contexts but, despite substantial variation in trajectory/ glide statistics incl initial postdetermined iterate bit width (as seen in the earlier experiment, 1st graph output), here they all come up empty. but from long experience, theres no shame in null results esp with this problem. done well, keeps the knife shiny/ sharpened.
vary1 code just mentioned. a basic property long noticed is a gradual uptrend in density and entropy variance at the end of trajectories. this can be seen in many experiments, eg
review124 from more recently at 10/2018 sticks in my mind but was probably 1st noticed years ago almost immediately starting with density measurements. on closer examination it looks like
vary1 code is just reflecting this in a slightly different way. apparently all trajectories tend to “end” in this similar way. it looks like nearly all the differentiation found in that experiment can be attributed to this “end differentiation.” but what is needed is some differentiable function for the beginning of the trajectory, not the end. there is some hint of this in the
vary1 diagrams esp the end. it looks like its gonna take some more careful analysis to isolate something here. is any of this attributable to initial-trajectory instead of end-trajectory? ie exploitable…
(8/15) 💡 ❗ had some idea to look at this density/ entropy variance “wedge” in closer detail, dont recall doing so beforehand, although there were some similar experiments. this generates 100 ½ density seeds and sorts by trajectory lengths, color coding by that. then it overplots density and entropy trends in 1st 2 graphs, aligned by start of sequence. in 2nd two graphs the sequences are aligned by ends instead. the 3rd and 4th pair of graphs have a new idea. its interesting to look at density by seed width. and then one can also look at the 1st value by width or the last value by width using a hashmap for replacement. surprisingly in the 3rd pair for density, the 2 graphs are different! in the left leading top edge, from downsloping in 1st case to flatter in 2nd case. in the 4th pair by entropy, there is not much difference. this is suggesting there is some trend to density as it “enters and exits” by bit width of iterates! ie some new differentiable feature, in brief “entry has higher variance than exit”, and needs more investigation (it would definitely help to plot the diference in a single graph). this reminds me of stock market candlestick graphs and maybe will try to build something like that in this case. its also a surprise that the density/entropy-by-width graphs are generally differently looking/ shaped than the density/entropy-by-sequence. needs more investigation. ❓
⭐ ⭐ ⭐
💡 😎 ❗ 😮 this is a very straightfwd idea carried out and seems to have deep implications. this generates the 100 ½ density distribution, sorts by trajectory length ‘c’ red, and then looks for trends in density effecting the overall outcome, graphed right side scale. the results were striking so quickly hooked up 100 sample averages with the nice/ very handy
avgs routine to take out the noise. ‘da’ is average density in green and roughly tracks glide length, although glide length is a sigmoidal trend and ‘da’ is concave downward. an earlier experiment and variants about ½ yr ago 2/2016 looked at density parity and this is now shown to be pivotal. density parity ‘dp’ blue closely tracks the total glide length ‘c’.
and heres a new extraordinary finding! the collatz operation was seen as a sampling operation from a random bit string, and this reveals it in a new way. a “power sequence” is constructed n·3x where ‘n’ is an initial iterate and ‘x’ is the sequence length (position). this sequence has a density (for each iterate) and a density parity (count or average) associated with it. the average density ‘d3a’ is flat, but the density parity ‘d3p’ closely tracks the collatz sequence density parity. remarkable! something very deep is revealed here. there must be big implications to all this, have to assimilate it and build on it. at least, it does seem to confirm that density spread at initial part of trajectory can discriminate trajectory lengths, for the drain “region” (or integer set). ❓
❗ 💡 immediate idea! these experiments seem to be showing that the drain set has a range of different nearly-linear slopes based on the initial entry iterate so to speak (probably around the 1st postdetermined iterate). this seems utterly basic, is hinted at in probably many prior diagrams, but has been missed until now. holy cow! 😮 ⭐
(8/27) 💡 its a shiny new toy to play with, and thinking it over with initial optimism, maybe something much deeper, aka scratching the surface/ tip of the iceberg. there was some old analysis that did look at the msbs of the starting iterate multiplied by 3^n and this new “3^n density parity pattern” certainly ties in with it/ is reminiscent. there are different directions to go but my 1st idea/ immediate question was wondering if this “3^n density parity pattern” holds outside the drain ie predetermined region. then was thinking about some “simple glide pattern” to try out as a test. the old heavy workhorse ‘w’ glide generation strategy came to mind, basically as an easy “steep” climb generator. have referenced it quite a bit over the last year in analysis ideas but looking back, dont think have run it in a long time!
in retrospect had some aversion to this scheme given that it seemed to be somewhat “unrepresentative” in some ways but also it turned out to be a disappointment wrt generalization ideas aka “fallen out of favor” esp wrt lsb 1-runs/ triangles (as found last year, even key finding, long climbs/ glides are possible without them). however, another way of looking at it, now more clear in retrospect but not mentioned previously, is that they (ie ‘w’ type glides) seems to be similar to the “high-density climbs” for Terras-generated glides where “high density” refers to the corresponding selected parity sequence of the glide, and maybe (ie it seems plausible) the 1-runs/ triangles are characteristic of those. looking back over old posts and searching, the last time the ‘w’ type glide was actually used in experiments was 9/2018
bitwise5 (+ bitwise7, bitwise8) almost an exact year ago!
another old earlier idea touched on/ invoked by these latest findings: finding a “nearby approximation” of the collatz function with presumably more analytical/ accessible/ exploitable properties and working with it instead. for the idea to work some kind of difference bound/ limit on the two functions has to be constructed. it seems an especially intriguing/ plausible idea is if the difference bound turns out to be nearly randomly distributed. this is very similar to the Perron linear approximation stability ideas found 10/2017 and pursued at many months length with the matrix differential equation framework etc.
(8/28) heres the code. its ‘w’
widthdiff implemented in the new bitwise format. technically ‘w’ format optimizes
widthdiff while minimizing density distance here expressed as 2-multivariate optimization. this is not exactly the same as earlier implementations that did a list sort on 2-element lists of
widthdiff/ density distance, and it would be subtle to try to statistically isolate the difference, it has to do with in particular when there are identical
widthdiffs but different density distances, but anyway its probably a close approximation.
there was both signal and null results. the null results were oriented around the idea of building random walks with the same density parity as that of the 3^n sequence and collatz sequence, with +1/-1 increments based on density parity, or also 3n+1 or div2 ops. unexpectedly these walks did not seem to match the glide much. but another key metric turns out to be bang on the money. one can look at the correlation in density parity between the collatz sequence and the 3^n sequence. maybe not surprisingly, it turned out to be critical to match/ line up the 3^n count to the actual 3x multiplications in the sequence otherwise the signal disappears. but the end result is rather striking.
green line ‘x’ is the sequence/ glide found after 5k iterations. red line ‘ij’ is the matching rate between the sequence and the 3^n sequence, right side scale. remarkably there seem to be 2 distinct slopes, 1st a high-correlation range for the climb region and the 2nd for the descent region, maybe closer to random, but maybe still above random. note the correlation function is very noisy in the beginning due to small sample size, this graph was hand-selected, among others that spike downward at the start but still seem to exhibit the overall trend. there seems to be a concept of “decay” in the match rate as the signal gets increasingly mixed/ scrambled/ disordered over time so to speak (again clearly an “entropy-increasing” related concept/ phenomenon/ manifestation).
also note technically the ‘w’ scheme needs restart logic (or multi-generation + discarding smallest glides) to find the longer glides because of intermittent paint-into-corner on short-to-nonexistent glides, here instead the code is just manually rerun. another consideration is that this code doesnt necessarily use the same optimal iterate found from the optimization for the analysis due to different ordering, the multivariate optimation is by
widthdiff ‘w’ and density distance ‘da’ and the final analysis finds the iterate by optimal
widthdiff ‘w’ only. also on further analysis/ examples am wondering if the idea of a 2-region slope change is misleading/ wishful thinking/ selection bias and its actually a more continuous trend from steep to gradual, because saw the same trend for a short
widthdiff/ glide case.
(8/29) this seems to lend more plausibility to the continuous decay trend idea. an obvious idea is to do a running average on the matching sequence. this does a 20 point average, ‘ma’ magenta right side scale. its noisy and generally decreasing but on multiple runs doesnt noticeably seem to have regions lining up with the climb. blue line ‘i’ is just cumulative match count. note the optimization is shifted to ‘cm’ glide max index and (minimizing) ‘da’ density distance. the code is modified to keep the same optimal iterate from generation to analysis by saving in a separate file. there was some trickiness wrt formatting/ escape characters in getting the multiline title to display correctly in gnuplot. have another idea along this theme that would use Terras generated glides over climb-density-varying range as used in a pivotal earlier/ recent experiment.
❓ but now there seems to be some mystery. the collatz curve density parity is roughly correlated with the 3^n sequence density parity (at the initial elevated, nonrandom rate of the decaying matching curve eg ‘ij’ red in the last 2 graphs), but random walks based on the 3^n sequence density parity didnt seem to track the collatz curve very closely. either some kind of noise is obscuring a correlation, or maybe there is some kind of anti correlation going on. because in the climb region, one must have higher parity density, ie roughly speaking more up steps than down steps (adjusted for a different # of up steps matching down steps, ie a derivable ratio found/ outlined in earlier experiments). now wondering, are the up steps in the climb region more associated with the mismatching density parities? these questions should be relatively easy to answer/ resolve with statistical averaging/ high-sample-counts attacks. the general outline/ trends seems to point to some kind of specific dynamic to be isolated/ uncovered.
(8/30) 💡 ❗ this is a very quick realization of the Terras generation idea variant. this looks at 3^n match ratio versus different Terras-generated density varying glides and finds a very strong signal. the short glides (cooler) have (start with) low match ratio (ie anticorrelated) over the trajectory and long glides (hotter) have high match ratio, and all of them converge or “decay” to a random 50/50 match ratio over time. this is reminiscent of some earlier graphs based on completely different analysis. it was hard to find this effect in recent prior code to the point that was wondering if there was any signal. it might seem like this is showing the match rate is different in the predetermined range vs postdetermined range/ drain but actually all these iterates have the same 50-length predetermined range/ bit width.
construct20 from 2/2019 seems related, have to think how it fits together. it found correlation vs anticorrelation in the density parity statistics.
another observation is that while density parity was mostly considered starting only earlier this year, there are (several year) older ideas relating to density parity such as studying the density crossover dynamics.
there seems to be some “relatively natural” estimate/ function implied here/ based on the metric/ feature for determining whether the trajectory has remaining glide. what does it look like?
(8/31) 😳 looking again closer at
construct20, the code isnt checking whether the trajectories are (at least) as long as the extracted sections. maybe manually looked into this, believe it may not be an issue maybe with all trajectories longer than the extracted sections, even the shorter ones, but its something of an oversight to not address it specifically in the code/ writeup. an easy way of checking is to estimate the length of a drain versus the starting seed bit size, this slope is roughly constant, although havent really quantified it more exactly previously, something of an oversight.
also further thinking, it seems what is desired is a “drain distance estimator” ie at any point in the sequence, an estimate how far in the “future” sequence the drain appears. and then one would have to try to prove the drain distance estimator is correct within some (noise) bound and monotonically decreasing. there seem to be some kind of hints of such an estimator in these findings.
an idea occurred to me of studying “head fake” glides (so to speak). these would be glides artificially constructed to fall for a long ways and then climb above the starting point. they would presumably be the “hard” cases for a drain distance estimator. such glides would be simple to construct using the Terras approach, just have equal length sequences of ½ – d parity density and ½ + d parity density, ie causing nearly equivalent downslope followed by upslope, comprising the entire predetermined range. a straightfwd idea definitely worth looking into further. or something to delegate to my lurking audience, lol! 🙄