Merge branch 'generic-dispatch' of git://brick.kernel.dk/data/git/linux-2.6-block

4. The I/O scheduler

I/O schedulers are now per queue. They should be runtime switchable and modular

but aren't yet. Jens has most bits to do this, but the sysfs implementation is

missing.

I/O scheduler, a.k.a. elevator, is implemented in two layers.  Generic dispatch

queue and specific I/O schedulers.  Unless stated otherwise, elevator is used

to refer to both parts and I/O scheduler to specific I/O schedulers.

Block layer implements generic dispatch queue in ll_rw_blk.c and elevator.c.

The generic dispatch queue is responsible for properly ordering barrier

requests, requeueing, handling non-fs requests and all other subtleties.

Specific I/O schedulers are responsible for ordering normal filesystem

requests.  They can also choose to delay certain requests to improve

throughput or whatever purpose.  As the plural form indicates, there are

multiple I/O schedulers.  They can be built as modules but at least one should

be built inside the kernel.  Each queue can choose different one and can also

change to another one dynamically.

A block layer call to the i/o scheduler follows the convention elv_xxx(). This

calls elevator_xxx_fn in the elevator switch (drivers/block/elevator.c). Oh,

The functions an elevator may implement are: (* are mandatory)

elevator_merge_fn		called to query requests for merge with a bio

elevator_merge_req_fn		" " "  with another request

elevator_merge_req_fn		called when two requests get merged. the one

				which gets merged into the other one will be

				never seen by I/O scheduler again. IOW, after

				being merged, the request is gone.

elevator_merged_fn		called when a request in the scheduler has been

				involved in a merge. It is used in the deadline

				scheduler for example, to reposition the request

				if its sorting order has changed.

*elevator_next_req_fn		returns the next scheduled request, or NULL

				if there are none (or none are ready).

elevator_dispatch_fn		fills the dispatch queue with ready requests.

				I/O schedulers are free to postpone requests by

				not filling the dispatch queue unless @force

				is non-zero.  Once dispatched, I/O schedulers

				are not allowed to manipulate the requests -

				they belong to generic dispatch queue.

*elevator_add_req_fn		called to add a new request into the scheduler

elevator_add_req_fn		called to add a new request into the scheduler

elevator_queue_empty_fn		returns true if the merge queue is empty.

				Drivers shouldn't use this, but rather check

				if elv_next_request is NULL (without losing the

				request if one exists!)

elevator_remove_req_fn		This is called when a driver claims ownership of

				the target request - it now belongs to the

				driver. It must not be modified or merged.

				Drivers must not lose the request! A subsequent

				call of elevator_next_req_fn must  return the

				_next_ request.

elevator_requeue_req_fn		called to add a request to the scheduler. This

				is used when the request has alrnadebeen

				returned by elv_next_request, but hasn't

				completed. If this is not implemented then

				elevator_add_req_fn is called instead.

elevator_former_req_fn

elevator_latter_req_fn		These return the request before or after the

				one specified in disk sort order. Used by the

				block layer to find merge possibilities.

elevator_completed_req_fn	called when a request is completed. This might

				come about due to being merged with another or

				when the device completes the request.

elevator_completed_req_fn	called when a request is completed.

elevator_may_queue_fn		returns true if the scheduler wants to allow the

				current context to queue a new request even if

elevator_set_req_fn

elevator_put_req_fn		Must be used to allocate and free any elevator

				specific storate for a request.

				specific storage for a request.

elevator_activate_req_fn	Called when device driver first sees a request.

				I/O schedulers can use this callback to

				determine when actual execution of a request

				starts.

elevator_deactivate_req_fn	Called when device driver decides to delay

				a request by requeueing it.

elevator_init_fn

elevator_exit_fn		Allocate and free any elevator specific storage

				for a queue.

4.2 I/O scheduler implementation

4.2 Request flows seen by I/O schedulers

All requests seens by I/O schedulers strictly follow one of the following three

flows.

 set_req_fn ->

 i.   add_req_fn -> (merged_fn ->)* -> dispatch_fn -> activate_req_fn ->

      (deactivate_req_fn -> activate_req_fn ->)* -> completed_req_fn

 ii.  add_req_fn -> (merged_fn ->)* -> merge_req_fn

 iii. [none]

 -> put_req_fn

4.3 I/O scheduler implementation

The generic i/o scheduler algorithm attempts to sort/merge/batch requests for

optimal disk scan and request servicing performance (based on generic

principles and device capabilities), optimized for:

This arrangement is not a generic block layer characteristic however, so

elevators may implement queues as they please.

ii. Last merge hint

The last merge hint is part of the generic queue layer. I/O schedulers must do

some management on it. For the most part, the most important thing is to make

sure q->last_merge is cleared (set to NULL) when the request on it is no longer

a candidate for merging (for example if it has been sent to the driver).

The last merge performed is cached as a hint for the subsequent request. If

sequential data is being submitted, the hint is used to perform merges without

any scanning. This is not sufficient when there are multiple processes doing

I/O though, so a "merge hash" is used by some schedulers.

iii. Merge hash

ii. Merge hash

AS and deadline use a hash table indexed by the last sector of a request. This

enables merging code to quickly look up "back merge" candidates, even when

multiple I/O streams are being performed at once on one disk.

are far less common than "back merges" due to the nature of most I/O patterns.

Front merges are handled by the binary trees in AS and deadline schedulers.

iv. Handling barrier cases

A request with flags REQ_HARDBARRIER or REQ_SOFTBARRIER must not be ordered

around. That is, they must be processed after all older requests, and before

any newer ones. This includes merges!

In AS and deadline schedulers, barriers have the effect of flushing the reorder

queue. The performance cost of this will vary from nothing to a lot depending

on i/o patterns and device characteristics. Obviously they won't improve

performance, so their use should be kept to a minimum.

v. Handling insertion position directives

A request may be inserted with a position directive. The directives are one of

ELEVATOR_INSERT_BACK, ELEVATOR_INSERT_FRONT, ELEVATOR_INSERT_SORT.

ELEVATOR_INSERT_SORT is a general directive for non-barrier requests.

ELEVATOR_INSERT_BACK is used to insert a barrier to the back of the queue.

ELEVATOR_INSERT_FRONT is used to insert a barrier to the front of the queue, and

overrides the ordering requested by any previous barriers. In practice this is

harmless and required, because it is used for SCSI requeueing. This does not

require flushing the reorder queue, so does not impose a performance penalty.

vi. Plugging the queue to batch requests in anticipation of opportunities for

iii. Plugging the queue to batch requests in anticipation of opportunities for

     merge/sort optimizations

This is just the same as in 2.4 so far, though per-device unplugging

  blk_kick_queue() to unplug a specific queue (right away ?)

  or optionally, all queues, is in the plan.

4.3 I/O contexts

4.4 I/O contexts

I/O contexts provide a dynamically allocated per process data area. They may

be used in I/O schedulers, and in the block layer (could be used for IO statis,

priorities for example). See *io_context in drivers/block/ll_rw_blk.c, and

	struct as_rq *next_arq[2];	/* next in sort order */

	sector_t last_sector[2];	/* last REQ_SYNC & REQ_ASYNC sectors */

	struct list_head *dispatch;	/* driver dispatch queue */

	struct list_head *hash;		/* request hash */

	unsigned long exit_prob;	/* probability a task will exit while

	return ioc;

}

static void as_put_io_context(struct as_rq *arq)

{

	struct as_io_context *aic;

	if (unlikely(!arq->io_context))

		return;

	aic = arq->io_context->aic;

	if (arq->is_sync == REQ_SYNC && aic) {

		spin_lock(&aic->lock);

		set_bit(AS_TASK_IORUNNING, &aic->state);

		aic->last_end_request = jiffies;

		spin_unlock(&aic->lock);

	}

	put_io_context(arq->io_context);

}

/*

 * the back merge hash support functions

 */

		__as_del_arq_hash(arq);

}

static void as_remove_merge_hints(request_queue_t *q, struct as_rq *arq)

{

	as_del_arq_hash(arq);

	if (q->last_merge == arq->request)

		q->last_merge = NULL;

}

static void as_add_arq_hash(struct as_data *ad, struct as_rq *arq)

{

	struct request *rq = arq->request;

		BUG_ON(!arq->on_hash);

		if (!rq_mergeable(__rq)) {

			as_remove_merge_hints(ad->q, arq);

			as_del_arq_hash(arq);

			continue;

		}

	WARN_ON(!list_empty(&rq->queuelist));

	if (arq->state == AS_RQ_PRESCHED) {

		WARN_ON(arq->io_context);

		goto out;

	}

	if (arq->state == AS_RQ_MERGED)

		goto out_ioc;

	if (arq->state != AS_RQ_REMOVED) {

		printk("arq->state %d\n", arq->state);

		WARN_ON(1);

		goto out;

	}

	if (!blk_fs_request(rq))

		goto out;

	if (ad->changed_batch && ad->nr_dispatched == 1) {

		kblockd_schedule_work(&ad->antic_work);

		ad->changed_batch = 0;

		}

	}

out_ioc:

	if (!arq->io_context)

		goto out;

	if (arq->is_sync == REQ_SYNC) {

		struct as_io_context *aic = arq->io_context->aic;

		if (aic) {

			spin_lock(&aic->lock);

			set_bit(AS_TASK_IORUNNING, &aic->state);

			aic->last_end_request = jiffies;

			spin_unlock(&aic->lock);

		}

	}

	put_io_context(arq->io_context);

	as_put_io_context(arq);

out:

	arq->state = AS_RQ_POSTSCHED;

}

		ad->next_arq[data_dir] = as_find_next_arq(ad, arq);

	list_del_init(&arq->fifo);

	as_remove_merge_hints(q, arq);

	as_del_arq_hash(arq);

	as_del_arq_rb(ad, arq);

}

/*

 * as_remove_dispatched_request is called to remove a request which has gone

 * to the dispatch list.

 */

static void as_remove_dispatched_request(request_queue_t *q, struct request *rq)

{

	struct as_rq *arq = RQ_DATA(rq);

	struct as_io_context *aic;

	if (!arq) {

		WARN_ON(1);

		return;

	}

	WARN_ON(arq->state != AS_RQ_DISPATCHED);

	WARN_ON(ON_RB(&arq->rb_node));

	if (arq->io_context && arq->io_context->aic) {

		aic = arq->io_context->aic;

		if (aic) {

			WARN_ON(!atomic_read(&aic->nr_dispatched));

			atomic_dec(&aic->nr_dispatched);

		}

	}

}

/*

 * as_remove_request is called when a driver has finished with a request.

 * This should be only called for dispatched requests, but for some reason

 * a POWER4 box running hwscan it does not.

 */

static void as_remove_request(request_queue_t *q, struct request *rq)

{

	struct as_rq *arq = RQ_DATA(rq);

	if (unlikely(arq->state == AS_RQ_NEW))

		goto out;

	if (ON_RB(&arq->rb_node)) {

		if (arq->state != AS_RQ_QUEUED) {

			printk("arq->state %d\n", arq->state);

			WARN_ON(1);

			goto out;

		}

		/*

		 * We'll lose the aliased request(s) here. I don't think this

		 * will ever happen, but if it does, hopefully someone will

		 * report it.

		 */

		WARN_ON(!list_empty(&rq->queuelist));

		as_remove_queued_request(q, rq);

	} else {

		if (arq->state != AS_RQ_DISPATCHED) {

			printk("arq->state %d\n", arq->state);

			WARN_ON(1);

			goto out;

		}

		as_remove_dispatched_request(q, rq);

	}

out:

	arq->state = AS_RQ_REMOVED;

}

/*

 * as_fifo_expired returns 0 if there are no expired reads on the fifo,

 * 1 otherwise.  It is ratelimited so that we only perform the check once per

static void as_move_to_dispatch(struct as_data *ad, struct as_rq *arq)

{

	struct request *rq = arq->request;

	struct list_head *insert;

	const int data_dir = arq->is_sync;

	BUG_ON(!ON_RB(&arq->rb_node));

	/*

	 * take it off the sort and fifo list, add to dispatch queue

	 */

	insert = ad->dispatch->prev;

	while (!list_empty(&rq->queuelist)) {

		struct request *__rq = list_entry_rq(rq->queuelist.next);

		struct as_rq *__arq = RQ_DATA(__rq);

		list_move_tail(&__rq->queuelist, ad->dispatch);

		list_del(&__rq->queuelist);

		elv_dispatch_add_tail(ad->q, __rq);

		if (__arq->io_context && __arq->io_context->aic)

			atomic_inc(&__arq->io_context->aic->nr_dispatched);

	as_remove_queued_request(ad->q, rq);

	WARN_ON(arq->state != AS_RQ_QUEUED);

	list_add(&rq->queuelist, insert);

	elv_dispatch_sort(ad->q, rq);

	arq->state = AS_RQ_DISPATCHED;

	if (arq->io_context && arq->io_context->aic)

		atomic_inc(&arq->io_context->aic->nr_dispatched);

 * read/write expire, batch expire, etc, and moves it to the dispatch

 * queue. Returns 1 if a request was found, 0 otherwise.

 */

static int as_dispatch_request(struct as_data *ad)

static int as_dispatch_request(request_queue_t *q, int force)

{

	struct as_data *ad = q->elevator->elevator_data;

	struct as_rq *arq;

	const int reads = !list_empty(&ad->fifo_list[REQ_SYNC]);

	const int writes = !list_empty(&ad->fifo_list[REQ_ASYNC]);

	if (unlikely(force)) {

		/*

		 * Forced dispatch, accounting is useless.  Reset

		 * accounting states and dump fifo_lists.  Note that

		 * batch_data_dir is reset to REQ_SYNC to avoid

		 * screwing write batch accounting as write batch

		 * accounting occurs on W->R transition.

		 */

		int dispatched = 0;

		ad->batch_data_dir = REQ_SYNC;

		ad->changed_batch = 0;

		ad->new_batch = 0;

		while (ad->next_arq[REQ_SYNC]) {

			as_move_to_dispatch(ad, ad->next_arq[REQ_SYNC]);

			dispatched++;

		}

		ad->last_check_fifo[REQ_SYNC] = jiffies;

		while (ad->next_arq[REQ_ASYNC]) {

			as_move_to_dispatch(ad, ad->next_arq[REQ_ASYNC]);

			dispatched++;

		}

		ad->last_check_fifo[REQ_ASYNC] = jiffies;

		return dispatched;

	}

	/* Signal that the write batch was uncontended, so we can't time it */

	if (ad->batch_data_dir == REQ_ASYNC && !reads) {

		if (ad->current_write_count == 0 || !writes)

	return 1;

}

static struct request *as_next_request(request_queue_t *q)

{

	struct as_data *ad = q->elevator->elevator_data;

	struct request *rq = NULL;

	/*

	 * if there are still requests on the dispatch queue, grab the first

	 */

	if (!list_empty(ad->dispatch) || as_dispatch_request(ad))

		rq = list_entry_rq(ad->dispatch->next);

	return rq;

}

/*

 * Add arq to a list behind alias

 */

	/*

	 * Don't want to have to handle merges.

	 */

	as_remove_merge_hints(ad->q, arq);

	as_del_arq_hash(arq);

}

/*

 * add arq to rbtree and fifo

 */

static void as_add_request(struct as_data *ad, struct as_rq *arq)

static void as_add_request(request_queue_t *q, struct request *rq)

{

	struct as_data *ad = q->elevator->elevator_data;

	struct as_rq *arq = RQ_DATA(rq);

	struct as_rq *alias;

	int data_dir;

	if (arq->state != AS_RQ_PRESCHED) {

		printk("arq->state: %d\n", arq->state);

		WARN_ON(1);

	}

	arq->state = AS_RQ_NEW;

	if (rq_data_dir(arq->request) == READ

			|| current->flags&PF_SYNCWRITE)

		arq->is_sync = 1;

		arq->expires = jiffies + ad->fifo_expire[data_dir];

		list_add_tail(&arq->fifo, &ad->fifo_list[data_dir]);

		if (rq_mergeable(arq->request)) {

		if (rq_mergeable(arq->request))

			as_add_arq_hash(ad, arq);

			if (!ad->q->last_merge)

				ad->q->last_merge = arq->request;

		}

		as_update_arq(ad, arq); /* keep state machine up to date */

	} else {

	arq->state = AS_RQ_QUEUED;

}

static void as_deactivate_request(request_queue_t *q, struct request *rq)

static void as_activate_request(request_queue_t *q, struct request *rq)

{

	struct as_data *ad = q->elevator->elevator_data;

	struct as_rq *arq = RQ_DATA(rq);

	if (arq) {

		if (arq->state == AS_RQ_REMOVED) {

			arq->state = AS_RQ_DISPATCHED;

	WARN_ON(arq->state != AS_RQ_DISPATCHED);

	arq->state = AS_RQ_REMOVED;

	if (arq->io_context && arq->io_context->aic)

				atomic_inc(&arq->io_context->aic->nr_dispatched);

		}

	} else

		WARN_ON(blk_fs_request(rq)

			&& (!(rq->flags & (REQ_HARDBARRIER|REQ_SOFTBARRIER))) );

	/* Stop anticipating - let this request get through */

	as_antic_stop(ad);

}

/*

 * requeue the request. The request has not been completed, nor is it a

 * new request, so don't touch accounting.

 */

static void as_requeue_request(request_queue_t *q, struct request *rq)

{

	as_deactivate_request(q, rq);

	list_add(&rq->queuelist, &q->queue_head);

		atomic_dec(&arq->io_context->aic->nr_dispatched);

}

/*

 * Account a request that is inserted directly onto the dispatch queue.

 * arq->io_context->aic->nr_dispatched should not need to be incremented

 * because only new requests should come through here: requeues go through

 * our explicit requeue handler.

 */

static void as_account_queued_request(struct as_data *ad, struct request *rq)

{

	if (blk_fs_request(rq)) {

		struct as_rq *arq = RQ_DATA(rq);

		arq->state = AS_RQ_DISPATCHED;

		ad->nr_dispatched++;

	}

}

static void

as_insert_request(request_queue_t *q, struct request *rq, int where)

static void as_deactivate_request(request_queue_t *q, struct request *rq)

{

	struct as_data *ad = q->elevator->elevator_data;

	struct as_rq *arq = RQ_DATA(rq);

	if (arq) {

		if (arq->state != AS_RQ_PRESCHED) {

			printk("arq->state: %d\n", arq->state);

			WARN_ON(1);

		}

		arq->state = AS_RQ_NEW;

	}

	/* barriers must flush the reorder queue */

	if (unlikely(rq->flags & (REQ_SOFTBARRIER | REQ_HARDBARRIER)

			&& where == ELEVATOR_INSERT_SORT)) {

		WARN_ON(1);

		where = ELEVATOR_INSERT_BACK;

	}

	switch (where) {

		case ELEVATOR_INSERT_BACK:

			while (ad->next_arq[REQ_SYNC])

				as_move_to_dispatch(ad, ad->next_arq[REQ_SYNC]);

			while (ad->next_arq[REQ_ASYNC])

				as_move_to_dispatch(ad, ad->next_arq[REQ_ASYNC]);

			list_add_tail(&rq->queuelist, ad->dispatch);

			as_account_queued_request(ad, rq);

			as_antic_stop(ad);

			break;

		case ELEVATOR_INSERT_FRONT:

			list_add(&rq->queuelist, ad->dispatch);

			as_account_queued_request(ad, rq);

			as_antic_stop(ad);

			break;

		case ELEVATOR_INSERT_SORT:

			BUG_ON(!blk_fs_request(rq));

			as_add_request(ad, arq);

			break;

		default:

			BUG();

			return;

	}

	WARN_ON(arq->state != AS_RQ_REMOVED);

	arq->state = AS_RQ_DISPATCHED;

	if (arq->io_context && arq->io_context->aic)

		atomic_inc(&arq->io_context->aic->nr_dispatched);

}

/*

{

	struct as_data *ad = q->elevator->elevator_data;

	if (!list_empty(&ad->fifo_list[REQ_ASYNC])

		|| !list_empty(&ad->fifo_list[REQ_SYNC])

		|| !list_empty(ad->dispatch))

			return 0;

	return 1;

	return list_empty(&ad->fifo_list[REQ_ASYNC])

		&& list_empty(&ad->fifo_list[REQ_SYNC]);

}

static struct request *

	struct request *__rq;

	int ret;

	/*

	 * try last_merge to avoid going to hash

	 */

	ret = elv_try_last_merge(q, bio);

	if (ret != ELEVATOR_NO_MERGE) {

		__rq = q->last_merge;

		goto out_insert;

	}

	/*

	 * see if the merge hash can satisfy a back merge

	 */

	return ELEVATOR_NO_MERGE;

out:

	if (rq_mergeable(__rq))

		q->last_merge = __rq;

out_insert:

	if (ret) {

		if (rq_mergeable(__rq))

			as_hot_arq_hash(ad, RQ_DATA(__rq));

		 * behind the disk head. We currently don't bother adjusting.

		 */

	}

	if (arq->on_hash)

		q->last_merge = req;

}

static void

	 * kill knowledge of next, this one is a goner

	 */

	as_remove_queued_request(q, next);

	as_put_io_context(anext);

	anext->state = AS_RQ_MERGED;

}

	unsigned long flags;

	spin_lock_irqsave(q->queue_lock, flags);

	if (as_next_request(q))

	if (!as_queue_empty(q))

		q->request_fn(q);

	spin_unlock_irqrestore(q->queue_lock, flags);

}

		return;

	}

	if (arq->state != AS_RQ_POSTSCHED && arq->state != AS_RQ_PRESCHED) {

	if (unlikely(arq->state != AS_RQ_POSTSCHED &&

		     arq->state != AS_RQ_PRESCHED &&

		     arq->state != AS_RQ_MERGED)) {

		printk("arq->state %d\n", arq->state);

		WARN_ON(1);

	}

	INIT_LIST_HEAD(&ad->fifo_list[REQ_ASYNC]);

	ad->sort_list[REQ_SYNC] = RB_ROOT;

	ad->sort_list[REQ_ASYNC] = RB_ROOT;

	ad->dispatch = &q->queue_head;

	ad->fifo_expire[REQ_SYNC] = default_read_expire;

	ad->fifo_expire[REQ_ASYNC] = default_write_expire;

	ad->antic_expire = default_antic_expire;

		.elevator_merge_fn = 		as_merge,

		.elevator_merged_fn =		as_merged_request,

		.elevator_merge_req_fn =	as_merged_requests,

		.elevator_next_req_fn =		as_next_request,

		.elevator_add_req_fn =		as_insert_request,

		.elevator_remove_req_fn =	as_remove_request,

		.elevator_requeue_req_fn = 	as_requeue_request,

		.elevator_dispatch_fn =		as_dispatch_request,

		.elevator_add_req_fn =		as_add_request,

		.elevator_activate_req_fn =	as_activate_request,

		.elevator_deactivate_req_fn = 	as_deactivate_request,

		.elevator_queue_empty_fn =	as_queue_empty,

		.elevator_completed_req_fn =	as_completed_request,