The memory values inside the container are gathered from the cgroups accounting for the container. The memory values you see in free command are setted by LXCFS in your container (by overmounting /proc/meminfo with the container cgroups values) and that’s why you see it as is from inside your container. However, as @stgraber explained, the cache are global to the system, and in this case, you only see more or less what you container uses, but you can’t interact with it specifically just for your container.
Containers are not clearly isolated from a memory point of view compared to virtual machines, your containers shares the global memory pool of your host, they just have a limited view of their available resources thanks to LXCFS.