Thread: a day in the life of @BFI National Archive #digipres using #FOSS @FFmpeg @MediaArea_Net #RAWcooked @MatroskaOrg to 1) preserve and 2) (yay) save cash...
Yesterday was a normalish day, oceans of data swilling around the digipres network, some servers working, some busted, some scripts working, some silently failing, some loudly failing. It’s ok, we have smart and caring human stewards to take care of this.
Anyway, we use #RAWCooked to convert our #DPX to #FFV1 #Matroska, and we use @FFmpeg to do the same for videotape digitisation, V210 MOVs to FFV1_MKVs..
Yesterday we ingested 189 MKVs at 3.9 TB from DPX. That would have been 7.8 TB of DPX in TAR. And we ingested 110 MKV / 2.4 TB from videotape source. That would’ve been 4.8 TB of MOV. So in total 6.3 TB of Matroska ingested yesterday, would’ve been 12.6 TB... Here’s a dataviz...
So 6.3 TB data size reduction. But of course we store two instances, in two data tape libraries, so it’s really 12.6 TB storage cost saving. LTO6 yields 2.8 TB usable capacity, so in one day, using #FOSS, we saved 4 LTO data tapes. If we repeat each day, one month = 120 tapes. 😍
And this all runs through a pipeline connecting infrastructure components from @axiellalm @Imagen_io @spectralogic - with LTO DPX help from @OvationData
All with REST APIs to push and pull data and send jobs from #Python scripts all day #NoSystemWithoutAPIPlease
I got the LTO6 usable capacity wrong - it's 2.2, not 2.8... So in fact it's 5.7 LTO6 tapes saved in one day, or 171 in a month. 😍😎 #ArithmeticFail
