Linux Kernel Programming–Memory Allocation / Kmalloc VS Vmalloc

Memory allocation in Linux kernel is different from the user space counterpart. The following facts are noteworthy,

• Kernel memory is not pageable.
• Kernel memory allocation mistakes can cause system oops (system crash) easily.
• Kernel memory has limited hard stack size limit.

There’re two ways to allocate memory space for a kernel process, statically from the stack or dynamically from the heap.
Static Memory Allocation
The static memory allocation is normally used you know how much memory space you’ll need. For example,

#define BUF_LEN    2048

char buf[BUF_LEN];

However, the kernel stack size is fixed and limited (the limit is architecture dependent, but normally it’s only tens of kilobytes). Therefore people seldom request big chunk of memory in the stack. The better way is to allocate the memory dynamically from heap.
Dynamic Memory Allocation
There’re two functions available to allocate memory from heap in Linux kernel process,
1. vmalloc
The vmalloc function is defined in /lib/modules/$(uname -r)/build/include/linux/vmalloc.h as below,

void *vmalloc(unsigned long size);

It’s Linux kernel’s version of malloc() function, which is used in user space. Like malloc, the function allocates virtually contiguous memory that may or may not physically contiguous.
To free the memory space allocated by vmalloc, one simply call vfree(), which is defined as,

void vfree(const void *addr);

2. kmalloc
The most commonly used memory allocation function in kernel is kmalloc, which is defined in /lib/modules/$(uname -r)/build/include/linux/slab.h as below,

void * kmalloc(size_t size, int flags);

kmalloc allocates a region of physically contiguous (also virtually contiguous) memory and return the pointer to the allocated memory. It returns NULL when the operation fails.
The behavior of kmalloc is dependent on the second parameter flags. Here only the two most popular flags are introduced,
GFP_KERNEL: this flag indicates a normal kernel memory allocation. The kernel might block the requesting code, free up enough memory and then continue the allocation. It cannot be used in places where sleep is not allowed.
GFP_ATOMIC: this flag indicates the kmalloc function is atomic operation. Atomic operation means the function is performed entirely or not performed at all. It cannot block at the middle of execution. As the kernel cannot jump out of the allocation and free up memory to satisfy the function request, this function has a higher chance of failure with this flag passed in.
There’re several other flags, for example, GFP_DMA for allocation of memory space capable of undergoing Direct Memory Access. For more information, one can refer the links provided in the references section.
To free up the memory allocated by kmalloc, one can use kfree defined as below,

void kfree(const void *objp);

3. vmalloc VS kmalloc
vmalloc allocates virtually contiguous memory space (not necessarily physically contiguous), while kmalloc allocates physically contiguous memory (also virtually contiguous). Most of the memory allocations in Linux kernel are done using kmalloc, due to the following reasons:

• On many architectures, hardware devices don’t understand virtual address. Therefore, their device drivers can only allocate memory using kmalloc.
• kmalloc has better performance in most cases because physically contiguous memory region is more efficient than virtually contiguous memory. The reason behind this is not covered here, interested readers can search for Linux memory management articles.

But when a large chunk of memory is needed, vmalloc is used often as it doesn’t require physically contiguous memory and the kernel can satisfy the request with much less effort than using kmalloc.


malloc allocates physically contiguous memory, memory which
pages are laid consecutively in physical RAM. vmalloc allocates
memory which is contiguous in kernel virtual memory space (that means
pages allocated that way are not contiguous in RAM, but the kernel
sees them as one block).
kmalloc is the preffered way, as long as you don't need very big
areas. The trouble is, if you want to do DMA from/to some hardware
device, you'll need to use kmalloc, and you'll probably need bigger
chunk. The solution is to allocate memory as soon as possible, before
memory gets fragmented.



What are the advantages of having a contiguous block of memory? Specifically, why would I need to have a contiguous physical block of memory in a system call? Is there any reason I couldn't just use vmalloc?

You only need to worry about using physically contiguous memory if the buffer will be accessed by a DMA device on a physically addressed bus (like PCI). The trouble is that many system calls have no way to know whether their buffer will eventually be passed to a DMA device: once you pass the buffer to another kernel subsystem, you really cannot know where it is going to go. Even if the kernel does not use the buffer for DMA today, a future development might do so.
vmalloc is often slower than kmalloc, because it may have to remap the buffer space into a virtually contiguous range. kmalloc never remaps, though if not called with GFP_ATOMIC kmalloc can block.
kmalloc is limited in the size of buffer it can provide: 128 KBytes^*). If you need a really big buffer, you have to use vmalloc or some other mechanism like reserving high memory at boot.

This was true of earlier kernels. On recent kernels (I tested this on 2.6.33.2), max size of a single kmalloc is up to 4 MB!