Eliminating superfluous photos in RenunionAppPhotos in Dropbox

[cponeill]

Is there anyway to determine which photos currently stored in ReunionAppPhotos in Dropbox are no longer used in your family file? I have a bazillion photos & pdfs stored in my ReunionAppPhotos folder that I am sure are no longer relevant to my family file. Is there any automated routine I can use to find those superfluous files?

[SGilbert]

IMHO, only you can determine whether a picture is relevant or not. Not even AI.

[cponeill]

Well, what I want to do is to move out of my multimedia folder - Reunion App Photos in my case - all the files (records) that are not being used as an active input to my FamilyFile. After 20 years (almost) of adding sources, photos, documents to my two family files, I’ve got a folder of stuff no longer necessary for my FamilyFiles.

[cponeill]

I should add that said folder also includes all the files, documents, photos, etc, that are relevant to (used by) my familyl files.

[cponeill]

So - what I’d like to do is to use some automated routine that will list all the files in the folder, and another list of all the files in use by my FamilyFile, and then move out of the first (all encompassing) folder (to another folder) all the files that are no longer used in my FamilyFile.

if I could program, I would write a simple routine to do that ((last time I wrote a computer program was 60 years ago using Fortran IV). But… I don’t have clue how to write programs today.

[Bob White]

I might have the answer for you...maybe. For many years, I have used an app named PrintWindows to produce file listings when needed. Just now experimented by dropping the Reunion Files folder onto its dialog box. It produced, within 2-3 minutes, a 165 page report listing everything. Attached screenshot is page 1. It is available on the Mac App Store for $20.

PrintWindows.png

[theKiwi]

You could try this...

Take the top most folder that contains the media items, and make a .zip copy of it.

Then delete the folder so that there are no image files on your computer.

Then go to Reunion ------> Settings ------> Media ------> Repair Media Links

That will produce a list of all the media items that Reunion is now not finding.

Save that list

Unzip the folder and then go through it deleting, or moving to another folder, everything EXCEPT what is in that list.

Roger

[A Ginn]

Another option - If you can’t find the program you want and have Excel, you are welcome to try the procedure I use involving:

1) Exporting a Reunion Media usage report as a TXT file.

2) Using Apple’s Terminal application to save a TXT file of the contents of the media folder and its subfolders.

3) Loading the two text files into two worksheets in an Excel workbook, then running three included macros. Two clean the lists and the third runs a comparison to identify the Finder files that are not used in Reunion. The macros take just over 1.5 minutes each to process over 6200 media records on my old MacBook Air.

The missing piece would be a script to automatically move the orphaned files to a different location. That would likely require an AppleScript, and my AppleScript days ended 25+ years ago.

If you have some experience with Excel and want to try this as far as it goes, send me an email at aginn@ne.rr.com. I will send the Excel Macro-Enabled Workbook and all instructions. It has been tested with media folders under the /Users/[username] tree and also with folders on external volumes.

[cponeill]

Bob - re your PrintWindows app capable of action - that is comparing one list to another and subtracting one from another - doesn't appear to be.

[cponeill]

Roger -

I may end up doing something like that - but I have thousands of media items - and it would take forever to do that manually. I can pretty much do that now by using Reunion's capability of producing lists - but again these are not operable. But thanks for the suggestion.

[cponeill]

and A Ginn - thanks. I'd like to have your input if I could. I'm not a big player in the Apple Terminal ballpark, so any suggestions there would be helpful. If there were someway to magically turn the Reunion produced lists into .csvs I might be able to work it that way.

[bmohr]

and A Ginn - thanks. I'd like to have your input if I could. I'm not a big player in the Apple Terminal ballpark, so any suggestions there would be helpful. If there were someway to magically turn the Reunion produced lists into .csvs I might be able to work it that way.

To get a list of all files in your ReunionAppPhotos folder, you can use the "find" command in the terminal. Open Terminal.app and type "cd" (without the quotes) and a space at the terminal prompt. Don't hit return yet! Drag your ReunionAppPhotos folder from the finder into the Terminal window. This will insert that folder's path into Terminal. Then hit return. This sets up the terminal so that it's "working directory" is your ReunionAppPhotos folder. Now type

find . -type f > files.txt

in terminal and hit return. This will create a plaintext file called "files.txt" in your ReunionAppPhotos folder that lists every file in that folder (and any subfolders).

On the Reunion side, I would suggest starting with a GEDCOM export rather than a Media Usage report. This may seem like a roundabout way to go, but both approaches will require editing to get a straightforward list of files and the GEDCOM file is in a format that I find much easier to massage with simple search/replace logic. It will take a few steps to get to the end result we want, but each step is easy.

First, export a GEDCOM of all people, being sure to enable "Include media links." Open the resulting file in a text editor such as BBEdit (the free/unlicensed version will work fine). I'm going to give instructions for BBEdit specifically, but these can be adapted to another text editor pretty easily. Note that you need a text editor not a word processor, that is, an app that edits plain text. First, you need to strip away everything in the GEDCOM that isn't a media link. In BBEdit, you would use the "Process lines containing" command in the Text menu, finding lines that do not contain "2 FILE" and "deleting matched lines."
processlines.png

Next, do a search-and-replace to get rid of that leading "2 FILE" on every line. In BBEdit, you would use "Find" in the Search menu and replace "2 FILE " (note that there's a space after that final E) with nothing: replace.png
Hit "Replace all" and you're almost done! Now you have a list of all the media Reunion knows about, but any file linked to more than one person (or source) will be duplicated, so let's delete those duplicates. BBEdit has you covered here, too. Just use "Process Duplicate Lines" (again in the Text menu). You want to "leave one" and "delete duplicate lines." While we're at it, we can also have BBEdit sort the remaining lines, which will be handy later:
duplicate.png
Bingo. Save the file (perhaps changing the name so it ends in ".txt" rather than ".ged") and proceed.

You now have a pair of text files, one listing every file in your ReunionAppPhotos folder and the other listing every file Reunion links to. This may be enough for your purposes, but to directly compare these files, you may need to do another search-and-replace to get the path names to match up. Reunion lists paths starting with your user directory (shown using a tilde), but the terminal's find command will show paths starting with your ReunionAppPhotos folder. I would inspect each file to figure out what you need to search-for and replace-to, but it will be something like searching for "~/Dropbox/folder/folder/ReunionAppPhotos" and replacing with ".".

[A Ginn]

The GEDCOM approach is very good. I have actually been using that longer than the usage report because you also have accesss to the description, comments, _CROP coordinates, etc. for archiving purposes. I don't link media to any of my sources, however, and it wasn't until I dusted off the old scripts and ran a test that I discovered I can't get the GEDCOM to report media linked to sources. I must have missed something in the setup.

Here are details of the process above.

Finder Data

Launch the Terminal application. Without quotes, type “find”, a space, then drag your multimedia folder to the Terminal window. This will load the folder's path followed by a space. Next type “-type f > myfilelist.txt” (or any other name to save the text file output). By dropping the folder on the Terminal window the path is added automatically. If you type it manually, know that spaces in the path must be preceded by backslashes. Also, if you type the path manually and use the tilde for the user’s folder the output still returns the full /Users/[username] prefix (see macro note below).

The text file will appear in your user folder. I run a macro in Excel to correct a couple of issues with this output:

1) Reunion reports the linked file names with the tilde character representing the home folder of the logged-in user, but the Terminal program reports the Finder’s file names with the full /Users/[username] prefix. This would prevent Excel from recognizing a match. The macro fixes this. (This is not a problem for files stored on an external volume – both programs begin the path with /Volume.)

2) The command I use for the Terminal program returns the folder names as well as the file names. The macro removes the folders.

Reunion Data

In Reunion use Reports - Media Usage - Show All Files. Ensure the settings include all people, all sources, and all media types. When the list appears, sort by Location then Option-Click the Share button and select Plain Text File as the Destination.

Note: If the Text File destination is selected instead then the result is an RTF with its formatting code, and in that case the file would have to be opened with TextEdit, Format - Make Plain Text chosen, and the file saved again.

I run a macro to extract just the file paths from the rest of the usage report formatting. It first looks at the start of the file names in the Finder text report above to determine whether we are interested in file names starting with ~/ or /V for extraction.

Comparing of the two lists

The last step is running a macro that takes each file in the Finder list and looks for it in the Reunion list. If it is not found, the macro flags it with the word MISSING. It could just as easily delete all the names that aren’t missing.

[theKiwi]

After the end of Brad's method so that you have two files listing the file names without the prefix from the GEDCOM file, copy one file, and paste it into the bottom of the other file.

Then in BBEdit use the Text menu ------> Process duplicate lines

Set it to "leaving one" and in the right side list check "unique lines to new document"

This should then give you a list of files not linked in Reunion.

Roger

[bmohr]

Excellent addition, Roger. For those comfortable using the command line, the "comm" command can be helpful, too.

The last step of the process, of course, is to actually delete the unwanted files. I'm personally very wary of doing such a task automatically with any software or command that isn't well tested. That being said, there are ways it can be done.