My program is using huge pages. For doing, it open files as follows:
oflags = O_RDWR | O_CREAT | O_TRUNC;
fd = open(filename, oflag, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
cat /proc/meminfo | grep Huge
AnonHugePages: 741376 kB
Hugepagesize: 2048 kB
oflags = O_RDWR | O_TMPFILE;
fd = open(pathname, oflag, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
open failed for /dev/hugepages: Operation not supported
Looks like O_TMPFILE is not implemented yet for hugetlbfs; indeed, this option requires support of the underlying file-system:
O_TMPFILE requires support by the underlying filesystem; only a subset of Linux filesystems provide that support. In the initial implementation, support was provided in the ex2, ext3, ext4, UDF, Minix, and shmem filesystems. XFS support was added in Linux 3.15.
This is confirmed by looking at the kernel source code where there's no inode_ops->tmpfile() implementation in hugetlbfs.
I believe that the right answer here is to work on this implementation...
I noticed your comment about the unlink() option, however, maybe the following approach is not that risky:
If your program gets killed in the middle, worst case is to leave an empty file.