Your question really has nothing to do with CSV files.
Once you have the image name (String) taken from the CSV file, the next step is to find the right relative path to the image file, including the folder name, etc.
Then if you have that path, your question is how to load/display an image in JavaFX from a given file path.
This question might help, although there are different approaches depending on how you want to display the image in the UI.
https://lovelace.augustana.edu/q2a/index.php/7241/image-in-javafx