To clarify, when I click an icon in plank it opens a window of the respective app. If I click that icon again it gets minimized like it should. But when I do the same with Nautilus it opens up a new window with new session of the application. I don't want that to happen and I remember that it wasn't always like this.
When I open it through Gnome dash (on the right) it doesn't show this behavior. The problem is only in Plank.
Ubuntu Gnome 17.04
The icons in the Dash and in the Launcher uses different .desktop-files. The icon in the launcher uses org.gnome.Nautilus.desktop, the icon in the Dash uses nautilus.desktop. Starting Nautilus from the different location will add different icons (with the same look) to the plank-dock.
I added both icons to the dock to get some meaningful output and they have different behavior. The org.gnome.Nautilus.desktopicon will behave as you want, it will minimize all open Nautilus-windows and rise all off them clicking the icon again. The nautilus.desktop-icon behaves as you describe, it will always open a new Nautilus-window.
You will find the launcher-entries for the plank-dock in
~/.config/plank/dock1/launchers
Here some output from my configuration:
$ ls -l '/home/mook/.config/plank/dock1/launchers'
total 20
-rw-rw-r-- 1 mook mook 83 พ.ค. 15 19:46 firefox.dockitem
-rw-rw-r-- 1 mook mook 84 พ.ค. 15 20:12 nautilus.dockitem
-rw-rw-r-- 1 mook mook 91 พ.ค. 15 19:50 org.gnome.gedit.dockitem
-rw-rw-r-- 1 mook mook 94 พ.ค. 15 19:47 org.gnome.Nautilus.dockitem
-rw-rw-r-- 1 mook mook 94 พ.ค. 15 19:47 org.gnome.Terminal.dockitem</pre>
Content of org.gnome.Nautilus.dockitem:
[PlankDockItemPreferences] Launcher=file:///usr/share/applications/org.gnome.Nautilus.desktop
Content of nautilus.dockitem:
nautilus.dockitem
[PlankDockItemPreferences] Launcher=file:///usr/share/applications/nautilus.desktop
When you check the files in ~/.config/plank/dock1/launchers, I am sure you will find nautilus.dockitem as this is the one which behave as you described. To remove the icon from the dock, close all Nautilus-windows, right-click the icon in the plank-dock and un-check Keep in Dock. Then start Nautilus from the Launcher (you will probably have to add Nautilus to favorites before) and lock the new Nautilus-icon in the plank-dock with right-click and checking Keep in dock.
Confusing, isn't it?
A funny thing is that I can't find the files org.gnome.Nautilus.desktop and nautilus.desktop with Nautilus, but they are listed when I usels -a /usr/share/applications in the terminal...
Tested in Ubuntu Gnome 17.04
