Thread by @tamaramunzner

Great start to #ieeevis on Sun morning at #belivws: fiery keynote on open science from @Protohedgehog so good and dense that I just sat back to absorb inspiration for new and better ways to fight the good fight.

@Protohedgehog Good point from @amcrisan following up @Protohedgehog 'penguin metaphor' - not fair that junior penguins asked to test the waters for killer whales while senior tenured penguins wait comfortably on the shore
#ieeevis #belivws

@Protohedgehog @amcrisan Appreciated @Protohedgehog emphasis on all four pillars of OS: in addition to reproducibility, also access, serials, evaluation. Articulate calling out perpetuators of paywalls including commercial (El$evier) and academic (IEEE is major offender) #ieeevis #belivws

@Protohedgehog @amcrisan Liked the detailed and discussion of both threats to validity and possible solutions from @sharoz & @eagereyes, see paper at osf.io/f8qey/ #ieeevis #belivws

@Protohedgehog @amcrisan @sharoz @eagereyes #belivws panel is so great and galvanizing that I'm compelled to start typing. Huge props to @sharoz for making oavis.steveharoz.com in the first place and then writing up more as followup, in osf.io/8ag3w/. #ieeevis

Theme: *scrutiny* as underlying goal of replication, from @sharoz and @jsndyks and @dalbersszafir - then lots of extremely interesting commentary on generalizing these ideas to qualitative research from the latter two.

#ieeevis #belivws

@sharoz @jsndyks @dalbersszafir @ngehlenborg suggests we adopt idea of 'research parasite' award - reusing previous data rewarded and publicized. great, except i want to instead call it the 'research symbiote' approach, since it's good for both parties. framing matters!
#ieeevis #belivws

@sharoz @jsndyks @dalbersszafir @ngehlenborg @jsndykes calls for 'reviewing by methods but conference organization by topic'. Then can see triangulation towards truth in a given session. #ieeevis #belivws

@sharoz suggestions for carrots towards reproducibility, plus panel discussion of the nuances of what can and can't be posted #ieeevis #belivws

TIL: micro-phenomenology. Set of rigorous interview and analysis methods to make implicit aspects of experience *explicit*. #ieeevis #belivws paper from Nowak, Bartram, and Schiphorst

.@amcrisan makes point that we should not conflate qualitative/quantitative with subjective/objective #ieeevis #believws, much more in methods paper at amcrisan.github.io/assets/files/p…

@amcrisan Aha, thanks to @amcrisan for clean version of qual/quant subj/obj deconflation table #ieeevis #belivws

Nice #belivws talk by @LacePadilla arguing that conceptual replication would benefit from cognitive model to consider whether new experiment design captures same phenomenon as original. A Case for Cognitive Models in Visualization Research,
lacepadilla.com/Downloads/publ…
#ieeevis

@LacePadilla #vds keynote from Pat Hanrahan: Automating Analysis?

Starting in a few moments at #ieeevis

@LacePadilla pat: uses keynotes to talk about what's on his mind, and these days it's AI.
at stanford most popular course in university is AI/ML, with over 1K students
#ieeevis #vds

@LacePadilla natural question: can we automate analysis with AI? if we could, do we need this conference? an existential crisis he thinks about. doesn't believe that, but it's a commonly held belief.
#ieeevis #vds

vis people might say of course not. but it's the number one question he gets. either from customers investing in various products, or "analysts". here's his answers to that question
#ieeevis #vds

a little review of ai: many amazing advances and he's an enthusiast. watson jeopary win was stunning milestone. imagenet and cnns / deep learning have become very common.
#ieeevis #vds

huge corporate investments, toolkits like tensorflow and pytorch that provide off the shelf libraries. systematized for easy application of ai to their problems. cnns are quite general.
#ieeevis #vds

also now revolution in hardware for ML, w/ google tensor processing unit (tpu) & nvidia's Turing chip. unbelievable compute power.
#ieeevis #vds

large datasets, algorithmic breakthroughs, well engineered software systems that works at scale, and specialized hardware. it's impressive and it's changing the world.
#ieeevis #vds

but wait, it gets better. these systems were supervised, requiring training, but now there's unsupervised. even more amazing was alphago beating lee sedol.
#ieeevis #vds

read that paper - they train on games from human experts & then game play against itself. got 20% improvement from that. alphagozero *only* played against itself, no human-expert training data, in relatively modest amount of time they trained it to beat alphago.
#ieeevis #vds

siggraph papers that stood out: deepmimic, example guided deep reinforcement learning of physical based character skills, xue bin peng, pieter eabbeel, sergey levine, michiel van de panne (of @ubc_cs!) arxiv.org/abs/1804.02717
#ieeevis #vds

siggraph papers that stood out: deepmimic, example guided deep reinforcement learning of physical based character skills, xue bin peng, pieter eabbeel, sergey levine, michiel van de panne (of @ubccs!) arxiv.org/abs/1804.02717
#ieeevis #vds

take mocap data, have well defined goal, run RL (reinforcement learning) on it.
RL is more amazing since don't need training data up front, it's automatically generated.
#ieeevis #vds

natl acad sciences report: "the expectation is that machines and algorithms will continue to improve in terms of the scale and complexity of the tasks they can accomplish. so the natural flow is that more of the human role will be moved over time to machines." #ieeevis #vds

natl acad sciences report: "the expectation is that machines and algorithms will continue to improve in terms of the scale and xomplexity of the taks they can accomplish. so the natural flow is that more of the human role will be moved over time to machines." #ieeevis #vds

analytical thinking, his defn: "a structured approach to answering questions and making decisions based on facts and data".
pat loves analysis, prev keynote on 'systems of thought' also covered this.
#ieeevis #vds

analytical thinking seems to fit within AI paradigm: they're structured and algorithmic and clearly based on data. and they make decisions.
#ieeevis #vds

prediction is uncertain. ML is way to make predictions. and they don't tell you how good they are. clearly sometimes get good results. but sometimes don't.
#ieeevis #vds

(uncertainty visualization is fundamental to ML. ML is about stats/probabalistic model...)
#ieeevis #vds

examples of how hard it is to make predictions. fivethirtyeight.com has a prediction of US election results, forecast 5/6 chance that Dems take the House... but 2 years ago, saw similar diagram, didn't come out this way!
#ieeevis #vds

hurricane simulation as prediction is another example. hurricane sandy - huge amount of variation in predictions. most said it wouldn't even hit the us.
#ieeevis #vds

reverse of prediction: certainty visualization.
read collins and evans 2008 on hawk-eye. they know it's uncertain, marketing ploy to blame on computer so players don't argue with ump. sites.cardiff.ac.uk/harrycollins/w…
#ieeevis #vds

sources of uncertainty:
- poor predictive model
- no means to collect the data
- incomplete data
- data collection is biased
- data is not publicly available (privacy)
#ieeevis #vds

it's very hard to get the data you want to make the predictions. sometimes it's whatever you can get, not what you need to make prediction - inherently biased. whatever data people have is what they use. don't question whether it's right or appropriate.
#ieeevis #vds

note often predictions not very good. might be 60%. at some level impressive you do it at all...
#ieeevis #vds

second issue:
decisions affect people
- should we leave our home?
- is this person a threat?
- should i undergo chemotherapy?
decision theory: weight consequences by their likelihood
#ieeevis #vds

attempt to use watson for medicine, mp anderson cancer site had signed contract, found it was very difficult to transfer. you can read more about it.
#ieeevis #vds

important thing about predictions - should put in decision-theoretic context about consequences. recs of what book to buy is one thing. followed amazon rec for SF book, he bought it, but it was terrible. so he lost $4.99, it's only slightly annoying. #ieeevis #vds

quality of life is a much more important issue. self-driving cars killing people will have huge consequences. as consequences increase, systems have to get much much better.
#ieeevis #vds

responsible analysis
- misleading? deceptive?
- fair? biased?
- explainable? understandable? transparent?
- vetted?
- ethical?

"who" is accountable? responsible?

you need to be thinking about these things constantly if you're deploying ML for analysis.

#ieeevis #vds

good example: climate change. what's involved in understanding the underlying science. ipcc report released lately. serious effort to analyze what's going on.
#ieeevis #vds

they didn't just use data lying around on their computers, they did targeted data collection after figuring out what they know, what they don't know, and what they need to know. deploy submarines etc to get more data.
#ieeevis #vds

analysis is hard

CORRELATION IS NOT CAUSATION

example: (age of miss america correlates with murders by steam, hot vapours, and hot objects)
tylervigen.com/view_correlati…

#ieeevis #vds

pat *really* loves analysis. founded tableau because want people to do it better.

learned about it as a kid from reading sherlock holmes. detective searching for clues from data. with magnifying glass. trying to deduce through reason. didn't find data on desk.
#ieeevis #vds

note sherlock considered eccentric. analysis is not something that most people are good at without effort. takes training and skill to be a good analyst. test of analytical thinking, amazing how poorly most people do. analytical thinking is hard.
#ieeevis #vds

would be nice if didn't need analysts so much, since there aren't so many sherlock holmes out there. but instead of conveniently handing over to AI program, try to make people be better analytical thinkers. that should be goal with visualization
#ieeevis #vds

people, analysis, and ai. how do we do this? how to take these very automated techniques and help people? five things:
#ieeevis #vds

1. map information into person's mental model
2. provide context
3. recommendations
4. Meta-AI
5. pragmatics
#ieeevis #vds

1. map information into person's mental model

first question: how do you think about your problem. what's important to you?

read great paper by marti hearst. faced metadata for image search and browsing.
flamenco.berkeley.edu/papers/flamenc…

#ieeevis #vds

previously would cluster results and show you the categories. hearst pointed out doesn't work very well. should use facets: multidimensional hierarchies of common ontologies. everybody uses facets now. logical hierarchy is crucial because that's how people think
#ieeevis #vds

2. provide context
business intelligence grew up. people collecting data and store in warehouses, after transactions. this customer brought this product at this store, this time, at this price. somebody thought to augment transactions with semantic model : context
#ieeevis #vds

then point back into record about customers - name, gender, addres. that info added into transaction and you can do analysis.
(business objects was the company that did this.)
#ieeevis #vds

dimensional modelling: birth of BI. star schema. dimensions are how people think about it: customer, store, product, time, price, units. these dimensions could be used with multiple databases. once you understand your customers in one db, can share across others
#ieeevis #vds

created a whole industry (BI) by adding semantic context to data. people underappreciate how important that is.
#ieeevis #vds

to analyze data have to embed within global context. ai can help us build up these contexts. he gave a sigmod talk several years ago saying data integration is really AI.
#ieeevis #vds

data integration is AI
- ontology/types with methods and calculations
- corresponding attributes )columns)
- corresponding entities (rows)
- functional dependencies and 4th normal form
#ieeevis #vds

you have to figure out what the types are and what calculations you can do on the types. eg if you know it's currency, can calculate how to convert between different ones and use CPI correction.

data integration is huge.

#ieeevis #vds

3. recommendations
leverage recommendations. could go with it or not. lightweight and unobtrusive way to give information. movie or book rec, he does look but doesn't care if imperfect.
#ieeevis #vds

tableau was a giant recommendation system: users don't understand things about visualization, they'll recommend everything. automatic marks.
#ieeevis #vds

graphic design recs:
- mark & encoding
- choose visual mark based on type of fields on axes
- choose other default visual attributes based on properties of field
- where to add field to visualization
- encode w/ bertin-like rules
- recommending a new vis

#ieeevis #vds

more in InfoVis2007 paper, graphics.stanford.edu/projects/polar…

why are recs so great? people can accept or reject your defaults. always can override.
#ieeevis #vds

4. Meta-AI
jeopardy example: should i ring the buzzer and try to answer or not? is your recommendation any good?
is this a good idea? consider confidence before recommend!
#ieeevis #vds

if good to do this, do it
if bad to do this - don't do it
if could be good, could be bad... principle: don't do it.

example: geocoding. sometimes it's easy, sometimes it's hard.

#ieeevis #vds

if you do a bad thing, you confuse user or generate a lot of extra work for them, which offsets potential gains.

tufte quote - "above all, do no harm". humbling how easy it is to confuse user in some way.

#ieeevis #vds

tricky - how do you know if you're making good recs or not? do this triage, and err on the side of not making recs if unclear.
#ieeevis #vds

5. pragmatics:
- ambiguity
- intent
- discourse

great progress in AI lately on this front. build up model of user. only tell them stuff they don't already know. if can get intent right, can make better recs.
#ieeevis #vds

these five things he thinks are the most promising, we could consider. sort of involves visualization but mostly about meaning. have to make sure visualizations are meaningful

#ieeevis #vds

pat hanrahan #vds keynote summary:

applying AI to analysis
- interfaces based on meaning
- recommendations and defaults
- provide context
- aware of limitations
- pragmatics: intent and discourse

subtleties
- analysis is hard
- decisions affect people

#ieeevis

just because you have some predictive model doesn't mean you have good analysis. decisions you make that affect people, have to be responsible. duty of teachers: train students to understand subtleties of analysis; of analysts: that analysis is employed responsibly.
#ieeevis #vds

correll Q: sometimes correct response is healthy skepticism or doing nothing. how do you market that kind of skepticism? one tool does something totally wrong but looks cool, vs another tells you go to back and get more data.
#ieeevis #vds

pat A: do have some faith in users. they are trying to really solve the problem. he's seen many buzzword-complete systems that don't do anything, after deployment people do figure that out eventually.
#ieeevis #vds

Another Q, Pat A:

everything is biased, don't know ultimate truth. all people are biased. science is also biased: it's a creation of people.

must learn to deal with bias. make sure to include multiple sources of bias. must always be aware of bias.
#ieeevis #vds

(Whew. Netlag turned this livetweeting attempt into undead-zombie-lurching-tweeting, but finally got all of this out after regained connectivity!... End of Pat keynote section.)
#ieeevis #vds

A second fantastic keynote from Kirk Goldsberry.
Analytics Illustrated - How Visualization Is Changing Sports Forever
- who an i and why am i here?
- how visualization is changing pro sports?
- why did it take so long?
#ieeevis #vds

.@kirkgoldsberry: Geography professor working in the NBA. He's a social scientist and cartographer by training and identifies as a geographer, not a data scientist. Wow, an all-star geographic lineage: Waldo Tobler and Sara Fabrikant and Cynthia Brewer!
#ieeevis #vds

@kirkgoldsberry march 2012. at harvard as a visiting prof, teaching geography. sorta awkward: in 400 years harvard has eliminated exactly one academic department, and that was geography. other places have also eliminated. oof
#ieeevis #vds

@kirkgoldsberry they wanted him to teach geography and gis and all this stuff that seemed useful. not to difficult, he had a lot of spare time. started to work on basketball. culmination of nights-and-weekends side project was a paper on his passion, basketball. #ieeevis #vds

@kirkgoldsberry 'courtvision', at sloan sports analytics conf.
sloansportsconference.com/wp-content/upl…

remarkable that no visualization in sports analytics at that point, esp basketball.

#ieeevis #vds

@kirkgoldsberry 3 reasons why visualization hadn't been more common
- constraints of media. (print)
- analysts didn't know how to make visualizations.
- executives don't demand visualizations - not part of culture

all of those reasons are changing

#ieeevis #vds

@kirkgoldsberry official scorers started recording spatial reference for shot attempts in 2000. he did this in 2012. why so long in between?
#ieeevis #vds

challenge was political, and that's not unique to sports.

pic: "i've always believed that analytics was CRAP"

@kirkgoldsberry: "guess he's more of a qualitative social scientist :-)"

#ieeevis #vds

every year NBA shooters take 200K shots, across whole league. but who cares about a shitload of dots?
geographers have techniques to address! #ieeevis #vda

A Multiresolution Stochastic Process Model for Predicting Basketball Possession Outcomes. arxiv.org/abs/1408.0777

microeconomics of the NBA based on second by second decisions
#ieeevis #vda

implications of univ. geography cuts:
- early egocentric view: lack of opportunities for academic jobs for him personally
- as geography programs faded from american universities in the 20th c, american learners were decreasingly exposed to spatial concepts
#ieeevis #vda

impacts on visual data science:
- lack of collaborators on american campuses
- lack of institutional lineage
- lack of undergrad education in cartography
- lack of awareness in other parts of university
#ieeevis #vda

VDS is changing sports

Challenges more political than technological:
- building collaborations across domains
- fluencies in two languages
- science
- sports

Scientists really underestimate how hard it is to be a good communicator in a sports environment

#ieeevis #vda

[missing many great visuals @kirkgoldsberry showed, incl]
"graphic at the heart of what's happening to aesthetic of professional basketball in 2018. bimodal reality where shots by the rim and behind the line are good, and everything else is a foolish investment"
#ieeevis #vda

Monster thread continues with Tuesday morning #ieeevis main opening. Attendance figures: descriptive stats, regression line, and beating the prediction with record attendance of 1252!

#ieeevis governance updates: some upcoming turnover on both VEC and VGTC ExCom. On ExCom list the people rotating out are in red, and new members in blue.

Previous year lists up in more readable form at
vgtc.org/executive-comm…
ieeevis.org/year/2018/info…

VGTC Chair elections coming right up on Nov 1-26. See vgtc.org/election2018 for info on candidates Jim Ahrens and Doug Bowman. See also how to become eligible to vote, double check it worked, opt in to communication to get the ballot.

Please vote, #ieeevis governance matters!

Please also participate in the Wed 1-2:20 lunchtime #ieeevis restructuring discussion (Conv 1, Sec C).

Give us your feedback on recommendations & reports at ieeevis.org/governance/res…

And now my favorite - awards!
Sheelagh Carpendale wins the Career Award & Anders Ynnerman wins the Technical Achievement award.
#ieeevis

.@andersynnerman quips on what users of medical imaging might be saying. Two possibilities:

#ieeevis

#ieeevis VAST 10-year Test of Time Award to Gennady and Nathalie Adrienko for
Spatio-temporal Aggregation for Visual Analysis of Movements

#ieeevis InfoVis 20-year Test of Time Award to Ed Chi and John Riedl for
An Operator Interaction Framework For Visualization Systems

#ieeevis InfoVis 10-year Test of Time award to
Effectiveness of Animation in Trend Visualization from George Robertson, Roland Fernandez, Danyel Fisher, Bongshin Lee, and John Stasko.
cc.gatech.edu/~stasko/papers…

A long-time favorite of mine, I still assign it to students regularly!

#ieeevis SciVis 25-year Test of Time award to
Roger Crawfis & Nelson Max for
Texture Splats for 3D Scalar and Vector Field Visualization

web.cse.ohio-state.edu/~crawfis.3/Pub…

#ieeevis SciVis 15-year Test of Time award to Jens Krüger & Rüdiger Westermann for

Acceleration Techniques for GPU-based Volume Rendering

Nice acceptance talk contrasting change (technical details) and persistence (basic idea)

Paper page:
wwwcg.in.tum.de/research/resea…

An Operator Interaction Framework For Visualization Systems (paywall version) at

doi.org/10.1109/INFVIS…

#ieeevis InfoVis 20-year TOT award

Paper link for
Spatio-temporal Aggregation for Visual Analysis of Movements by
Gennady & Nathalia Adrienko is

geoanalytics.net/and/papers/vas…

#ieeevis VAST 10-year TOT award

#ieeevis VAST Best Paper 2018 award to

TPFlow: Progressive Partition and Multidimensional Pattern Extraction for Large-Scale Spatio- Temporal Data Analysis,

by Dongyu Liu, Panpan Xu, Liu Ren

Paper:
vis.cse.ust.hk/papers/dongyu/…

Video:
lliquid.github.io/homepage/files…

Really interesting data abstraction choice in TPFlow paper: model multidimensional spatiotemporal data as tensors, then decompose piecewise to slice & dice for pattern finding. #ieeevis

#ieeevis InfoVis 2018 Best Paper to Draco!

Code
uwdata.github.io/draco/

Paper
domoritz.de/papers/2018-Dr…

Blog
medium.com/@uwdata/64ce20…

Authors:
@domoritz, Chenglong Wang, Greg L. Nelson, Halden Lin, Adam M. Smith, Bill Howe, @jeffrey_heer

More great & open science from @uw_data

@domoritz @jeffrey_heer Draco is:

A formal model that represents visualizations as sets of logical facts and design guidelines as a collection of hard and soft constraints over these facts.

#ieeevis

@domoritz @jeffrey_heer Draco constraints are:
- attribute domain
- integrity
- preferences.

Built on top of the compassQL rec engine for voyager & voyager2

#ieeevis

@domoritz @jeffrey_heer Draco first author @domoritz is:

- a great speaker
- on the academic job market

#ieeevis

#ieeevis SciVis 2018 Best Paper

Deadeye: A Novel Preattentive Visualization Technique Based on Dichoptic Presentation.

by @andreykrekhov & Jens Krüger

Wow, spectacularly good talk slide design (& delivery) and impressively careful scientific work!

ieeexplore.ieee.org/document/84400…

@AndreyKrekhov After great #ieeevis talk, rolling my eyes at Q "is this a scivis paper?"

Good answers:
Andrey: "No idea where to submit to VIS for first time, saw previous SciVis session on stereo & perception"
Laidlaw: "The science was impressive and it's about visualization, so it's SciVis"

@AndreyKrekhov The entire exchange speaks volumes (pun intended) about why there's a need to seriously consider the current structure of #ieeevis and whether these historical walls of VAST, InfoVis, and SciVis still serve us. Come talk on Wednesday at the #ieeevis #restructure session!

And now for #ieeevis keynote "When VIS met AR"!

Dieter Schmalstieg opens with a When Harry Met Sally pic, noting a ...complicated... relationship.

Dieter has a longstanding interest in visualization but also admits to his dark side: augmented reality

I really appreciate this thoughtful discussion. It's a topic that I've personally been extremely skeptical about, but I feel my mind opening up a bit more with each successive slide. #ieeevis

Key point: referent = physical object/locale with meaning. AR requires spatial registration with meaningful referent.

#ieeevis

Referent could be:
- hidden inside physical object
- own body
- second person

Also - idea of X-ray vision giving people superpowers is probably the real reason he started doing this work in the first place :-)

#ieeevis