A process is consuming 200MB, Is there any way to restrict its usage only to 100MB without using cgroup. (Remaining 100MB it can use from swap space)
You may be able to do this with
ulimit -m. That restricts the maximum resident set. However, the manual entry states that a lot of systems don't honor (i.e. implement) this limit. Note that the limits set this way only apply to new processes started by the current shell. (The ulimit command is a shell built-in command.)
prlimit command allows you to alter the resource limits for an existing process.