At the recent #neurips4creativity panel someone asked what areas of machine learning were recommend for newcomers, and I suggested exploring atypical algos such as image segmentation and how they might be repurposed.
let me explain further with using some of my students' work.👇
let me explain further with using some of my students' work.👇
Semantic Image Segmentation is arguably one of the more mundane success stories of modern computer vision. The task is to label each pixel with a label - @jeremyjordan provides a good overview if you are unfamiliar: jeremyjordan.me/semantic-segme…
This past year I introduced it to my @vicuniwgtn design 2nd year undergraduates as part of a digital photography assignment. It seemed completely natural to students that (of course!) images can be decomposed into not only Red, Green, and Blue but also imperfect semantic labels.
Students made lots of innovative projects based on their own photos. For example, Amy Wilson examined the negative impacts of designer dogs with a @p5xjs generative algorithm that juxtaposes a blurred unchosen dog against a diverse matrix of possibilities. bl.ocks.org/missfabulous/e…
My advanced class also looked at Image Segmentation including building a custom dataset and labelling data. This assignment ended up being a great way to sneak in heaps of general ML concepts like dataset cleaning, overfitting, out of domain test data, etc.
Students got this task right away and with fewer preconceptions did things I would not have imagined. For example Michael Kelly (@Normcore_Mikey) trained a model to recognize contextually relevant items from a cooking video based mainly on hand proximity. vimeo.com/301118446
And in a computational photography project Zala Habib (@zyjerah) built a labelled dataset specific to her own picture frame, and then engineered a pipeline around the segmentation model to automatically apply neural style transfer on areas of new photos. zyjerah.github.io/Frame-of-Fancy/
TL;DR:
1) Image Segmentation is quite approachable for students, is useful for teaching general ML concepts, and has lots of creative possibilities.
2) There are likely other overlooked CV algorithms that have many untapped creative uses waiting to be more fully explored.
1) Image Segmentation is quite approachable for students, is useful for teaching general ML concepts, and has lots of creative possibilities.
2) There are likely other overlooked CV algorithms that have many untapped creative uses waiting to be more fully explored.
