configuration - How can I send a 404 when user tries to access a protected directory? -

I am running Apache 2.4, and there is a directory that can only be reached from a certain IP address via the following :

  & lt; Directory / path / to / private / diamond & gt; IP 11.22.33.44 & lt; / Directory & gt;   
 I think anyone trying to use that URL to get a 404 Not Found file instead of 403 Access Denied. Is there any way to do this through Apache?   
 
 
 I believe that you can do something like