Category Archives: bigdata

big data summer 2017, revolution + firehose

hi all, have blogging up a storm for years now, coming up on the ½ decade anniversary. at times as far as topics its been like “kid in a candy store” for me. as the chinese say “may you live in interesting times”. there is so much going on these days, so many moving parts. we seem to be in the middle of major simultaneous revolutions in multiple fields, some of my favorites like CS+ physics. CS advancements in particular are rippling across many fields. CS is historically the field of infotech. but as noted in this blog, science is becoming increasingly about analysis of info/ data. a hybrid of CS+ statistical science is leading the way in analyzing Big Data. Big Data covers many areas and for years there was mostly a lot of focus on the hardware aspects. theres been huge innovation there and weve got awesome capabilities, systems, libraries for stuff like that, like nosql and graph databases. very impressive!

so now with those solid foundations in place, some of the applications/ advances/ breakthrus can commence. and we’re already seeing the “early” fruits all over the place. yet my feeling is that wrt the “long run” all of this is maybe only in 2nd gear.

have been tracking Big Data in my massive bookmark/ link piles. as you can imagine, thats a very difficult challenge, but am largely up to it with my hardcore cybersurfing and hacker tools eg bookmark export. and so a “huge” ~215 link list is just a drop in the bucket…!

made a conscious decision to hold off on writing up the links “for awhile” because AI has been taking center stage with deep learning and its largely merged with a lot of the cutting edge big data field. its getting increasingly hard to separate/ draw boundaries between them. but then, holy cow! THREE YEARS flew by in a flash. have been tracking a few big data related links in AI posts otherwise, but now my big data links runneth over and time to unleash/ release them. was somewhat waiting for a headline-grabbing event, but in a way they are happening constantly…

Continue reading


AI 2017 — poker milestone passed/ crumbled

hi all. AI technology is really exploding in the last few years. the last big post/ compilation on the subj here was ~½ year ago and the links piled up in a blur since then. the main trigger for this post: the game of poker now seems to have “folded” to computer supremacy. a new paper was published on Deepstack and its highly competitive play, and Libratus is $800K up in a recent match against top experts (top players). my understanding is that there is still some weakness in multiplayer games and that the new breakthru is for 1-1 games, human vs computer, but presumably that razor-thin human edge might also melt away quickly.[a]

poker was a very good game for humans wrt our inherent/ evolved psychology. we (top humans that is) seem to have an intuitive grasp of how to bet based on the strength of cards, including the use of bluffing. it took computers until the 21st century to master this stuff. but it looks like they just passed the threshhold again. in a small surprise, it wasnt done by Deepmind but which is behind many other near-monthly, even verging on weekly breakthroughs.[c]

maybe not by total coincidence, the winning Libratus algorithm involves training a neural network to accurately estimate the search tree, quite similar to the Deepmind Go strategy that made huge headlines just a year ago. the media hasnt picked up on the poker competition as much as it did with Go… is it because cautious/ publicity shy academics have less PR instinct than google? or less budget? but maybe that “relatively low profile” will change in the weeks/ months ahead. hopefully there will be a very high profile contest that again captures widespread public interest/ imagination.

it seems the top poker competitions are typically held in Las Vegas afaik… what would it take to get the computers in that? wouldnt be cool if say Vegas (or some other high profile gambling center) decided to publicize it to attract attn/ tourism? but would the computer algorithms be competitive in the top multiplayer games? there have been increasing/ huge audiences for poker over last few years, not sure what all the factors are in in this surge (internet gambling might play a role…)

its neat to see academia still at the top of competitive research in AI. but that seems to be thinning somewhat over last few years as the massive corporations Google[b], Microsoft, Apple,[g] Facebook, Intel [f] and misc other corps [e] are snapping up AI talent like its a feverish arms race, and to some degree it is. theres also very fast/ dynamic startup/ other merger activity going on, and new research laboratories being founded.[h]

Continue reading

alphaGo victorious 4-1 over Sedol in SKorea, NKorea crushes Warmbier 15-0

hi all the go match was very eventful and turned out to have massive media coverage worldwide, with huge interest from technology publications, and theres a sizeable AI/ ML/ singulatarian crowd on the internet that follows these types of developments quite avidly.

wish that google would transcribe the press conferences. there was a lot of interesting details but it takes a long time to watch them and the midstream translation interruption (korean to english, english to korean) slows down the presentation also.

saw some interesting press questions in the post-3rd match conference iirc that tie in with some angles explored on this blog.

Continue reading

battle of the brains midmatch pause: alphaGo 2-0 over Sedol

hassabis-sedol-schmidthi all. feeling blog-overwhelmed by recent Turing Machine-type events but just cannot resist blogging again at this historic moment (my blog frequency is up lately and some at expense of other activities). caught some of the 2nd match live late wed eve. do not know go heads from tails myself but found the commentary by Redmond quite engaging. it was interesting in postgame analysis that Sedol felt at no time was he winning, but Redmond saw the game as fairly even even far into the middle. but near the end something shifted, possibly a single move, and Redmond said that Black (alphaGo) had a major ~10stone advantage based on rough count.

korea_heraldthe long ~4hr match made me feel a bit sorry for the commentators attempting to say something meaningful the whole time sometimes when the moves were very slow. there was a ~30m delay in the midgame as sedol pondered a weird/ unusual move by alphago. at one point Redmond called alphago a “he” and they reacted briefly on that, redmond said it seemed natural to him. (at my job a guy also sometimes talks about computational processes in terms of “he”…)

this game is quite interesting in that, for apparently nearly even positions (which are possibly frequently the case in very advanced level games) one does not know clearly if one is winning or losing and single moves can significantly tip the balance. the single moves seem to be about unifying separate regions and strengthening major separate areas such that they reinforce each other. it seems to be about simultaneously playing out multiple strategies in separate regions and then tying them together in the end.

the game seems to me to have a strong fractal quality, apparently not noted by many. explaining exactly what this means is not quite possible at this moment in scientific history. fractals are very difficult to describe.

Continue reading

battle of the brains! real vs artificial! google AlphaGo vs top human Go at it next week

low-Go - 3next wednesday march 9th is the 1st match between an improved google Go and the top human player Lee Sedol. it will be livestreamed on youtube with commentary by several experts.[a1]

as has been noted in a lot of articles on the subject, the recent match between european champion Hui tested a human at ELO ranking 2750. Sedol is ranked at 2940. Hui is ranked 633rd in the world, Sedol ranked 5th. Sedol [and other cohort experts] is already on record as saying he is confident he will win at least this 1st match based on analysis of the prior Hui match and apparent mistakes in the computer gameplay. the prior Hui match would seem to definitively show the Go machine is already playing substantially better than ELO 2750.

but Sedol maybe seems not to have realized/ anticipated/ acknowledged that Google is already throwing significantly more hardware and software improvements (obtained via reinforcement learning) in the new match, therefore the new system will not be the same as the prior one; it will presumably be substantially improved and maybe even its style of play will diverge some.

Continue reading