Important Uboot Commands - Loady usage in Booting from RAM

The exact set of commands depends on the U-Boot configuration, however these are the important list of commands for reference. This also would be quite helpful for debugging Bootloader(Uboot) related issues :-

This post doesn't describe all the command in detail from usage wise however there has been two important example covered in the post.

• help and help command
• boot, runs the default boot command, stored in bootcmd
• bootm / bootz  starts a kernel image loaded at the given address in RAM
• ext2load, loads a file from an ext2 filesystem to RAM And also ext2ls to list files, ext2info for information
• fatload, loads a file from a FAT filesystem to RAM and also fatls and fatinfo
• tftp, loads a file from the network to RAM
• ping, to test the network
• loadb, loads, loady, load a file from the serial line to RAM
• usb, to initialize and control the USB subsystem, mainly used for USB storage devices such as USB keys
• mmc or mmc rescan , to initialize and control the MMC subsystem, used for SD and microSD cards
• nand, to erase, read and write contents to NAND flash
• erase, protect, cp, to erase, modify protection and write to NOR flash
• md, displays memory contents. Can be useful to check the contents loaded in memory, or to look at hardware registers.
• mm, modifies memory contents. Can be useful to modify directly hardware registers, for testing purposes.

Commands to manipulate environment variables:

• printenv  [ Shows the value of a variable ]
• setenv [ Changes the value of a variable, only in RAM ]
• editenv [ Edits the value of a variable, only in RAM ]
• saveenv  [ Saves the current state of the environment to flash ]

List of Important Environment variables :-

• bootcmd, contains the command that U-Boot will automatically execute at boot time after a configurable delay(bootdelay), if the process is not interrupted.
• bootargs, contains the arguments passed to the Linux kernel. 
• serverip, the IP address of the server that U-Boot will contact for network related commands.
• ipaddr, the IP address that U-Boot will use. 
• netmask, the network mask to contact the server.
• ethaddr, the MAC address, can only be set once. 
• autostart, if yes, U-Boot starts automatically an image that has been loaded into memory.
• filesize, the size of the latest copy to memory (from tftp, fat load, nand read...)

Example of Uboot commands to boot a Linux/Arm system from MMC (assuming that bootargs are not part of kernel configuration) :-

/* This commands is to set the bootargs */

setenv bootargs console=ttymxc1,115200 root=/dev/mmcblk0p7 ro rootwait rootfstype=ext4 earlyprintk no_console_suspend=1 consoleblank=0 init=/sbin/init log_buf_len=1M cma=128M mem=256M galcore.contiguousSize=67108864 lpj=3948544

Note an important configuration in bootargs is the "size of RAM"

The usual commands to boot into normal kernel from a MMC device :-

/* Select the MMC device */

mmc dev 2  

/* Read the MMC and keep the Linux kernel uImage(Assuming that DTB is already part of uImage) in 0x10500000 address in RAM */

mmc read 0x10500000 800 4000  

Note that 800 is the block number, 4000 number of blocks of MMC

/* The command to set the PC to 0x10500000 address and run */

bootm 0x10500000

Off Topic :-  the uImage for UBOOT is created in the two following steps :-

1. Copying the DTB to zImage(The Kernel Image) - This is optional as the uImage may or may not have the DTB.

2. Use the mkimage utility to create the uImage.

What If the DTB is not part of uImage?

Let have a look at these steps below which loads dtb into a particular address in the RAM and then boots the system,the DTB and uImage is loaded from USB to RAM. Below are the command to be run from Uboot prompt.

fatload usb 0 0x1070000 am335x-boneblack.dtb

fatload usb 0 0x1090000 uImage

bootm 0x1070000 - 0x1090000

How to boot Linux kernel image without flashing into EMMC or NAND and directly writing into RAM from the UBOOT prompt :-

1. loady 0x10500000 921600

2. Switch the baud rate for teraterm or console which your are using to 921600

3. Send kernel image over ymodem  : Go to File -> Transfer -> YMODEM -> Send -> browse and select the kernel image which you want to boot.

The pic below for reference.

4. Switch back to baud 115200 after the transfer is complete and press ESC to comeback to Uboot prompt.

5. bootm 0x10500000  // To Boot the Kernel (Assuming that DTB is already part of uImage)

Note that All UBOOT commands depends upon the overall porting and configuration of UBOOT which is beyond the scope of this blog post.

Linux Kernel Debug Configs

some important debug options of Linux kernel :-


CONFIG_DEBUG_SPINLOCK : With this option enabled, the kernel catches operations on uninitialized spinlocks and various other errors (such as unlocking a lock twice).

SPINLOCK_TIMEOUT and SPINLOCK_TIMEOUT_TIME: The spinlocks will timeout after X number of seconds.This is useful for catching deadlocks, and make sure locks are not held too long.

CONFIG_DEBUG_SPINLOCK_SLEEP : This option enables a check for attempts to sleep while holding a spinlock. In fact, it complains if you call a function that could potentially sleep, even if the call in question would not sleep.

CONFIG_DEBUG_SLAB: This crucial option turns on several types of checks in the kernel memory allocation functions; with these checks enabled, it is possible to detect a number of memory overrun and missing initialization errors

CONFIG_INPUT_EVBUG : This option (under "Device drivers/Input device support") turns on verbose logging of input events. If you are working on a driver for an input device, this option may be helpful. Be aware of the security implications of this option, however: it logs everything you type, including your passwords.

Linux Kernel Debugging using Ftrace - Basic Post

Ftrace is a tracing utility built directly into the Linux kernel. Many distributions already have various configurations of Ftrace enabled in their most recent releases. 

Note that this post doesn't cover all tracers in details.

Important Kernel Configs for Tracing and debugging :-

CONFIG_FUNCTION_TRACER
CONFIG_FUNCTION_DURATION_TRACER
CONFIG_FUNCTION_GRAPH_TRACER
CONFIG_STACK_TRACER
CONFIG_DYNAMIC_FTRACE

To Find what tracers are available :-
If the DEBUGFS is not mounted 
mount debugfs -t debugfs /debug

Go to the tracer directory as below 
[~]# cd /sys/kernel/debug/tracing 
[tracing]#   
[tracing]# cat available_tracers 
function_graph function sched_switch nop

Note that the Kernel has huge list of tracers, covering all that is not be possible for just a single blog post, however below are few links which describes all of them in detail.

https://www.kernel.org/doc/Documentation/trace/ftrace.txt

http://lwn.net/Articles/365835/

https://lwn.net/Articles/366796/

I will surely come back with a blog post on debugging critical moments of Linux kernel using FTRACE. I am working toward that right now.

BTW, There is another important tip as below :-

Use Ftrace's  trace_printk() instead of printks :-
printk() is the king of all debuggers, but it has a problem. If you are debugging a high volume area such as the timer interrupt, the scheduler, or the network, printk() can lead to bogging down the system or can even create a live lock. It is also quite common to see a bug "disappear" when adding a few printk()s. This is due to the sheer overhead that printk() introduces.

Ftrace introduces a new form of printk() called trace_printk(). It can be used just like printk(), and can also be used in any context (interrupt code, NMI code, and scheduler code). What is nice about trace_printk() is that it does not output to the console. Instead it writes to the Ftrace ring buffer and can be read via the trace file.

Writing into the ring buffer with trace_printk() only takes around a tenth of a microsecond or so. But using printk(), especially when writing to the serial console, may take several milliseconds per write. The performance advantage of trace_printk() lets you record the most sensitive areas of the kernel with very little impact.

For example you can add something like this to the kernel or module:

    trace_printk("read foo %d out of bar %p\n", bar->foo, bar);


tracing_on() and tracing_off() for Kernel Debugging :-

The driver gets stuck in a sleep state and never wakes up. You will not be able to disable the tracer from user space when a kernel event occurs is difficult and which can in a trace buffer overflow and loss of the relevant information before the user can stop the trace.

There are two functions that work well inside the kernel: tracing_on() and tracing_off(). 
These two act just like echoing "1" or "0" respectively into the tracing_on file. 

If there is some condition that can be checked for inside the kernel, then the tracer may be stopped by adding something like the following:

                               if (test_for_error())
                                           tracing_off();

This gives flexibility to turn on and off tracing in the particular driver, where you can enable tracing on some error and use trace_printk() till you disable it later.

After examining the trace, or saving it off in another file with:
cat trace > ~/trace.sav

How does Linux kernel handles shared IRQs?

Though this topic is discussed in many places and also covered in  Linux Device Drivers, 3rd edition, by Corbet et al. However I would still like to compile the topic in my own way.

The reason is that there are quite confusing things about this topic.

Basics :-

Each IRQ can be viewed via /proc/interrupts so the registered handlers come from the drivers that have invoked request_irq passing in the form:

irqreturn_t (*handler)(int, void *)

If the handler should not handle the particular interrupt it must return the kernel macro IRQ_NONE. Each Interrupt handler can get access to dev_id. Also usually there will be bit set in hardware for interrupts and also one may need to acknowledge the interrupt by clearning a bit in a hardware's memory mapped register. 

The Real Case :-

Notice that there will be multiple drivers registering  a single handler or multiple handlers, that means you have a single interrupt number or interrupt line and you have mapped multiple device drivers to listen to. Note that each device entry in the device tree will have a interrupt number which usually the drivers read while registering the interrupt. The case is like you have multiple device nodes in device tree with the same irq number representing a single interrupt line.

Key Points :-

1. All Interrupt handlers will be serviced.

2.Each driver will be aware of its own interrupts via memory mapped registers hence can easily check whether its his device is the cause of the interrupt or not. Each device virtual memory space is different so there is proper memory restriction already in place.

3.Each dev_id for the driver is unique however this may not be useful to check or differentiate shared interrupt handlers. Hence it wouldn't be correct to solve this problem from dev_id point of view as each irq handler gets the same value whatever by its own drivers.

Hence there is only one way to differentiate is a step(2) as mentioned in the article.

NOTE : Its a usual practice to trigger bottom-halves or any other logic in the IRQ handler only after checking the IRQ status from a memory mapped register. Hence the problem is default solved by a good programmer.

An Example :

Magnetic Card Driver's Interrupt handler:-

  static irqreturn_t timag_irq(int irq, void *dev)
{
            struct timag *mag_dev = dev;
            bool  over_or_under_flow = false;
            unsigned int status, irqclr = 0;

             status = timag_readl(mag_dev, REG_IRQSTATUS);

             if ((status & IRQENB_FIFO_IRQ) )  {
                timag_enable(mag_dev, true);

Linux Security Topic : Key Management/Service

The below text explains the basics of key service and relevant terms. It discusses in length the role of kernel in key service management. The Key service is an kernel's abstraction of creation of cryptographic keys, the actual interface for crypto hardware drivers  to the service is not covered here.

The Master References :- https://www.kernel.org/doc/Documentation/security/


===============================================================
HOW TO ENABLE KEY MANAGEMENT/SERVICE IN THE KERNEL/USERSPACE
===============================================================

The key service can be configured on by enabling: "Security options"/"Enable access key retention support" (CONFIG_KEYS) in the arch/../../defconfig or dynamacally using make menuconfig.

This service allows cryptographic keys, authentication tokens, cross-domain user mappings, and similar to be cached in the kernel for the use of filesystems and other kernel services

In the USERSPACE, the keyutils - In-kernel key management utilities is needed to support it. Refer :- http://man7.org/linux/man-pages/man7/keyutils.7.html for more details.


============
KEY BASICS
============

KEYS represent units of cryptographic data, authenticationtokens, keyrings, etc.. These are represented in the kernel by struct key.

Each key has a number of attributes:

A serial number :- Each key is issued a serial number of type key_serial_t that is unique for
the lifetime of that key. All serial numbers are positive non-zero 32-bit integers.

A type :- Each key is of a defined "type". Types must be registered inside the kernel by a kernel service (such as a filesystem) before keys of that type can be added or used. Userspace programs cannot define new types directly. Key types are represented in the kernel by struct key_type. This defines a number of operations that can be performed on a key of that type. Note that if a type be removed from the system, all the keys of that type will be invalidated.

A description (for matching a key in a search) : Each key has a description. This should be a printable string. The key type provides an operation to perform a match between the description on a key and a criterion string.

Access control information : Each key has an owner user ID, a group ID and a permissions mask. These are used to control what a process may do to a key from userspace, and whether a kernel service will  be able to find the key. 

An expiry time : Each key can be set to expire at a specific time by the key type's instantiation function. Keys can also be immortal.

A payload : This is a quantity of data that represent the actual "key".  In the case of a keyring, this is a list of keys to which the keyring links; In the case of a user-defined key, it's an arbitrary blob of data.


State :

 Each key can be in one of a number of basic states:

(a) Un-instantiated. The key exists, but does not have any data attached. Keys being requested from userspace will be in this state.

  (b) Instantiated. This is the normal state. The key is fully formed, and  has data attached.

  (c) Negative. This is a relatively short-lived state. The key acts as a
note saying that a previous call out to userspace failed, and acts as
a throttle on key lookups. A negative key can be updated to a normal
state.

  (d) Expired. Keys can have lifetimes set. If their lifetime is exceeded,
they traverse to this state. An expired key can be updated back to a
normal state.

  (e) Revoked. A key is put in this state by userspace action. It can't be
 found or operated upon (apart from by unlinking it).

  (f) Dead. The key's type was unregistered, and so the key is now useless.

 
Important Points : 

(a) A payload is not mandatory; and the payload can, in fact, just be a value stored in the struct key itself.

(b) When a key is instantiated, the key type's instantiation function is called with a blob of data, and that then creates the key's payload in  the kernel.

(c) when userspace wants to read back the contents of the key, if  permitted, another key type operation will be called to convert the key's attached payload back into a blob of data.

(d) Userspace programs can use a key's serial numbers as a way to gain access to it, subject to permission checking.

(e) Userspace programs cannot define new types directly. Key types are represented 
in the kernel by struct key_type.For further information on key_type refer :- https://www.kernel.org/doc/Documentation/security/keys.txt

(f) Kernel maintains all datastructures for key managemnent. Userspace can make system calls to use/update keys. We will discuss this in detail in the later part of the document.


=========
KEYRINGS
=========

These are a special type of key that can hold links to other keys. Processes each have three standard keyring subscriptions that a kernel service can search for relevant keys.


==================================================
USERSPACE SYSTEM CALL INTERFACE WITH THE KERNEL
==================================================

Userspace can manipulate keys directly through three new syscalls: add_key, request_key and keyctl. It is important to note here that mostly the key created and added by the kernel and user space just sees it and authenticates it.

For e.g. Trusted and Encrypted Keys are two new key types added to the existing kernel key ring service.  Both of these new types are variable length symmetric keys, and in both cases all keys are created in the kernel, and user space sees, stores, and loads only encrypted blobs.

1.Create a new key and add it to the nominated keyring:

key_serial_t add_key(const char *type, const char *desc,
    const void *payload, size_t plen,
    key_serial_t keyring);

2. When referring to a key directly, userspace programs should use the key's serial number to directly get the key. Search the process's keyrings for a key, potentially calling out to userspace to create it. It is important to note that this API is similar to the other API provided to kernel services(e.g Filesystem) to search or request a key.

key_serial_t request_key(const char *type, const char *description,
const char *callout_info,
key_serial_t dest_keyring); 

For more description of this API and its relavent usage related to processes , 
refer :-https://www.kernel.org/doc/Documentation/security/keys-request-key.txt


3. Read the payload data from a key - one of the important usage of keyctl.This function attempts to read the payload data  from the specified key into the buffer. The process must have read permission on the key to succeed.  

long keyctl(KEYCTL_READ, key_serial_t keyring, char *buffer, size_t buflen);

For all other types of keyctl,  please refer :- http://man7.org/linux/man-pages/man7/keyutils.7.html

The application of keyctl is brought out when a key is created for Filesystem and the filesystem is then mounted using the key.

Refer :- https://www.kernel.org/doc/Documentation/security/keys-ecryptfs.txt


============================================
USAGE OF KEYCTL for Filesystem key authentication
============================================

important commands for keyctl :-

keyctl add encrypted name "new ecryptfs key-type:master-key-name keylen" ring
keyctl add encrypted name "load hex_blob" ring
keyctl update keyid "update key-type:master-key-name"

Example of encrypted key usage with the eCryptfs filesystem:

Create an encrypted key "1000100010001000" of length 64 bytes with format
'ecryptfs' and save it using a previously loaded user key "test":

    $ keyctl add encrypted 1000100010001000 "new ecryptfs user:test 64" @u
    19184530

    $ keyctl print 19184530
    ecryptfs user:test 64 490045d4bfe48c99f0d465fbbbb79e7500da954178e2de0697
    dd85091f5450a0511219e9f7cd70dcd498038181466f78ac8d4c19504fcc72402bfc41c2
    f253a41b7507ccaa4b2b03fff19a69d1cc0b16e71746473f023a95488b6edfd86f7fdd40
    9d292e4bacded1258880122dd553a661

    $ keyctl pipe 19184530 > ecryptfs.blob

Mount an eCryptfs filesystem using the created encrypted key "1000100010001000"
into the '/secret' directory:

    $ mount -i -t ecryptfs -oecryptfs_sig=1000100010001000,\
      ecryptfs_cipher=aes,ecryptfs_key_bytes=32 /secret /secret


=====================================
KERNEL SERVICES FOR KEY MANAGEMENT
=====================================

The kernel services for key management can be broken down into two areas: keys and key types.

Firstly, the kernel service registers its type, then it searches for a key of that type. It should retain the key as long as it has need of it, and then it should release it. 

E.g. For a filesystem or device file, a search would probably be performed during the open call, and the key released upon close. 

To access the key manager, the following header must be included:
#include

Some of the important kernel functions are described below, for complete list you can refer : - https://www.kernel.org/doc/Documentation/security/keys.txt

1. A keyring can be created by:

struct key *keyring_alloc(const char *description, uid_t uid, gid_t gid,
const struct cred *cred,
key_perm_t perm,
unsigned long flags,
struct key *dest);

2. A kernel service may want to define its own key type. For instance, an AFS filesystem might want to define a key type. To do this, it author fills in  a key_type struct and registers it with the system.

(a)To register a key type, the following function should be called:

int register_key_type(struct key_type *type);

(b) To unregister a key type, call:

void unregister_key_type(struct key_type *type);


3. To get the Key, following are the functions to be called by Kernel service.

struct key *__key_get(struct key *key);
struct key *key_get(struct key *key);

4. A key's serial number can be obtained by calling:

key_serial_t key_serial(struct key *key);

5. To search for a key, call:

struct key *request_key(const struct key_type *type,
const char *description,
const char *callout_info);

References :- https://www.kernel.org/doc/Documentation/security/

All about FIQs(Fast Interrupt reQuest)

A traditional question asked in Interviews :- 

Explain FIQ and its usage 

I would like to cover this whole topic here. Though all information related to this topic is available on web however its highly scattered. I have accumulated all that here.

Fast Interrupt reQuest  is a higher priority interrupt. This means that it will always have precedence over regular interrupts and  regular interrupts won’t mask or interrupt an FIQ, while an FIQ will mask or interrupt any IRQ.  

Many systems do not use FIQs at all, or use it in a way analogous to the non-maskable (NMI) interrupt found on other processors (although FIQ is software maskable on most ARM processors).

Important  Difference between FIQ and IRQs :-

First, FIQs are executed in a dedicated execution mode, and this FIQ mode has 7 dedicated registers, from r8 to r14. This allows to have persistent values between each FIQ handler code, and avoids the overhead of pushing and popping in the handler. The second thing to know is that, unlike the regular IRQ handlers, the FIQ handler has to be written using ARM assembly, mostly because the C compiler won’t produce any code that can use only these r8 to r14 registers.

free-electrons demostrated the difference of efficiency by generating a 50kHz square waveform by bitbanging the GPIOs over IRQ and FIQ.

FIQs are mostly not used by Linux kernel, however there are exceptions. Please read on ...

Few Real applications of FIQ 

FIQ can perhaps best be used to eliminate the need for a DMA unit in certain low cost systems. In FIQ mode , the register used by ARM is a different set hence which can be used for a FIFO interrupt of data transfer just like the DMA. This is only possible because ARM provides extra number of registers for FIQ mode.

FIQ is used for Secure Worlds in ARM TrustZone implementations, to distinguish interrupts from "secure" interrupt sources. The precise determination of what might be a secure interrupt source and how that should be handled differently from a normal interrupt depends on the implementations.

Even ARM supports this;  From ARM infocenter :-

"The model recommended by ARM is the use of IRQ as a Normal world interrupt source, and FIQ as the Secure world source. IRQ is the most common interrupt source in use in most operating environments, so the use of FIQ as the secure interrupt should mean the fewest modifications to existing software. If the processor is running the correct virtual core when an interrupt occurs there is no switch to the monitor and the interrupt is handled locally in the current world. If the core is in the other world when an interrupt occurs the hardware traps to the monitor, the monitor software causes a context switch and jumps to the restored world, at which point the interrupt is taken"

FIQ for spinlock debugging and Hardlock up scenarios :- 

In simple words, if the a ARM system faces deadlock due to spinlocks, the spin lock debug will only show stack trace of the current CPU that stuck due to the lock. however at the same time , if we can get the trace of all other CPUs, it makes easy for developers to debug issues in multicore systems. 

To address this problem the FIQ was used for  arch_trigger_all_cpu_backtrace() 

All cpu backtrace is called by the spinlock debugging code when it thinks the system might have locked up. It works by sending IPIs (inter-processor interrupts - refer callback  send_IPI_all in Linux kernel code) that raise FIQ on the CPUs  and, because it uses FIQ, these processors respond and issue a stack trace even if they are locked up and have interrupts masked. Note that when spinlock locks out, the local interrupts gets disabled so  the same applies as most embedded system would like to go for a recovery even though the kernel locks out. The FIQs can also used as a recovery mechanism for many such cases in order to get more debug information which watchdog may not provide.

Note that the IPI gets only applied to multi-core or smp systems, Please check the patch implemented for avoiding sending IPI for UP(uniprocessor) systems.

FIQs are also used by Hardware performance monitoring (Please click on the link to read more on the topics) 

References :-

http://stackoverflow.com/questions/973933/what-is-the-difference-between-fiq-and-irq-interrupt-system

http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.prd29-genc-009492c/CACCDCDH.html

http://www.linaro.org/blog/core-dump/debugging-arm-kernels-using-nmifiq/

Testing Videos on Suspend/Resume cycle in Android

This is a suspend/resume cycle test on Android. A script which will play the video then go to suspend and  resume will be happened after 60 seconds followed by holding wake_lock.This is not platform independent as it is tightly coupled with the system and the way you have written your PM drivers.

rand_test_loop(){
am start -a android.intent.action.VIEW -d 
sleep 30
  echo "OFF Mode count $off_count ************!!!!!!!!!!!!!!!"

     while [ 1 ]

     do

  input keyevent 82  // unlock home screen

  input keyevent 3 // home screen

"file:///mnt/sdcard/AV_1080p.mp4"-t "video/*"//video is for 40 sec

  echo "calling suspend"

  echo 60 >/d/pm_debug/wakeup_timer_seconds

  #echo mem > /sys/power/state

echo test>/sys/power/wake_unlock

echo "call power button"

input keyevent 26 //power button

sleep 7

echo "test" >/sys/power/wake_lock

input keyevent 82

input keyevent 3

echo "home came***"

  off_count=`cat /d/pm_debug/count | grep DEVICE-OFF`

     done

}

rand_test_loop