Merge tag 'for-6.5/dm-changes' of...

	Perform the replacement only if bio->bi_opf has all the

	selected flags set.

  random_read_corrupt <probability>

	During <down interval>, replace random byte in a read bio

	with a random value. probability is an integer between

	0 and 1000000000 meaning 0% to 100% probability of corruption.

  random_write_corrupt <probability>

	During <down interval>, replace random byte in a write bio

	with a random value. probability is an integer between

	0 and 1000000000 meaning 0% to 100% probability of corruption.

Examples:

Replaces the 32nd byte of READ bios with the value 1::

mode, the dm-integrity target can be used to detect silent data

corruption on the disk or in the I/O path.

There's an alternate mode of operation where dm-integrity uses bitmap

There's an alternate mode of operation where dm-integrity uses a bitmap

instead of a journal. If a bit in the bitmap is 1, the corresponding

region's data and integrity tags are not synchronized - if the machine

crashes, the unsynchronized regions will be recalculated. The bitmap mode

zeroes. If the superblock is neither valid nor zeroed, the dm-integrity

target can't be loaded.

Accesses to the on-disk metadata area containing checksums (aka tags) are

buffered using dm-bufio. When an access to any given metadata area

occurs, each unique metadata area gets its own buffer(s). The buffer size

is capped at the size of the metadata area, but may be smaller, thereby

requiring multiple buffers to represent the full metadata area. A smaller

buffer size will produce a smaller resulting read/write operation to the

metadata area for small reads/writes. The metadata is still read even in

a full write to the data covered by a single buffer.

To use the target for the first time:

1. overwrite the superblock with zeroes

	device. If the device is already formatted, the value from the

	superblock is used.

interleave_sectors:number

interleave_sectors:number (default 32768)

	The number of interleaved sectors. This values is rounded down to

	a power of two. If the device is already formatted, the value from

	the superblock is used.

	Don't interleave the data and metadata on the device. Use a

	separate device for metadata.

buffer_sectors:number

	The number of sectors in one buffer. The value is rounded down to

	a power of two.

	The tag area is accessed using buffers, the buffer size is

	configurable. The large buffer size means that the I/O size will

	be larger, but there could be less I/Os issued.

buffer_sectors:number (default 128)

	The number of sectors in one metadata buffer. The value is rounded

	down to a power of two.

journal_watermark:number

journal_watermark:number (default 50)

	The journal watermark in percents. When the size of the journal

	exceeds this watermark, the thread that flushes the journal will

	be started.

commit_time:number

commit_time:number (default 10000)

	Commit time in milliseconds. When this time passes, the journal is

	written. The journal is also written immediately if the FLUSH

	request is received.

	the journal. Thus, modified sector number would be detected at

	this stage.

block_size:number

block_size:number (default 512)

	The size of a data block in bytes. The larger the block size the

	less overhead there is for per-block integrity metadata.

	Supported values are 512, 1024, 2048 and 4096 bytes.  If not

	specified the default block size is 512 bytes.

	Supported values are 512, 1024, 2048 and 4096 bytes.

sectors_per_bit:number

	In the bitmap mode, this parameter specifies the number of

should not be changed when reloading the target because the layout of disk

data depend on them and the reloaded target would be non-functional.

For example, on a device using the default interleave_sectors of 32768, a

block_size of 512, and an internal_hash of crc32c with a tag size of 4

bytes, it will take 128 KiB of tags to track a full data area, requiring

256 sectors of metadata per data area. With the default buffer_sectors of

128, that means there will be 2 buffers per metadata area, or 2 buffers

per 16 MiB of data.

Status line:

    Each run contains:

	* tag area - it contains integrity tags. There is one tag for each

	  sector in the data area

	  sector in the data area. The size of this area is always 4KiB or

	  greater.

	* data area - it contains data sectors. The number of data sectors

	  in one run must be a power of two. log2 of this value is stored

	  in the superblock.

	*data_mode = DATA_MODE_VMALLOC;

	/*

	 * __vmalloc allocates the data pages and auxiliary structures with

	 * gfp_flags that were specified, but pagetables are always allocated

	 * with GFP_KERNEL, no matter what was specified as gfp_mask.

	 *

	 * Consequently, we must set per-process flag PF_MEMALLOC_NOIO so that

	 * all allocations done by this process (including pagetables) are done

	 * as if GFP_NOIO was specified.

	 */

	if (gfp_mask & __GFP_NORETRY) {

		unsigned int noio_flag = memalloc_noio_save();

		void *ptr = __vmalloc(c->block_size, gfp_mask);

		memalloc_noio_restore(noio_flag);

		return ptr;

	}

	return __vmalloc(c->block_size, gfp_mask);

}

}

EXPORT_SYMBOL_GPL(dm_bufio_client_destroy);

void dm_bufio_client_reset(struct dm_bufio_client *c)

{

	drop_buffers(c);

	flush_work(&c->shrink_work);

}

EXPORT_SYMBOL_GPL(dm_bufio_client_reset);

void dm_bufio_set_sector_offset(struct dm_bufio_client *c, sector_t start)

{

	c->start = start;

 */

enum {

	DM_IO_ACCOUNTED,

	DM_IO_WAS_SPLIT

	DM_IO_WAS_SPLIT,

	DM_IO_BLK_STAT

};

static inline bool dm_io_flagged(struct dm_io *io, unsigned int bit)

 * In order to not degrade performance with excessive locking, we try

 * non-blocking allocations without a mutex first but on failure we fallback

 * to blocking allocations with a mutex.

 *

 * In order to reduce allocation overhead, we try to allocate compound pages in

 * the first pass. If they are not available, we fall back to the mempool.

 */

static struct bio *crypt_alloc_buffer(struct dm_crypt_io *io, unsigned int size)

{

	struct bio *clone;

	unsigned int nr_iovecs = (size + PAGE_SIZE - 1) >> PAGE_SHIFT;

	gfp_t gfp_mask = GFP_NOWAIT | __GFP_HIGHMEM;

	unsigned int i, len, remaining_size;

	struct page *page;

	unsigned int remaining_size;

	unsigned int order = MAX_ORDER - 1;

retry:

	if (unlikely(gfp_mask & __GFP_DIRECT_RECLAIM))

	remaining_size = size;

	for (i = 0; i < nr_iovecs; i++) {

		page = mempool_alloc(&cc->page_pool, gfp_mask);

		if (!page) {

	while (remaining_size) {

		struct page *pages;

		unsigned size_to_add;

		unsigned remaining_order = __fls((remaining_size + PAGE_SIZE - 1) >> PAGE_SHIFT);

		order = min(order, remaining_order);

		while (order > 0) {

			pages = alloc_pages(gfp_mask

				| __GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN | __GFP_COMP,

				order);

			if (likely(pages != NULL))

				goto have_pages;

			order--;

		}

		pages = mempool_alloc(&cc->page_pool, gfp_mask);

		if (!pages) {

			crypt_free_buffer_pages(cc, clone);

			bio_put(clone);

			gfp_mask |= __GFP_DIRECT_RECLAIM;

			order = 0;

			goto retry;

		}

		len = (remaining_size > PAGE_SIZE) ? PAGE_SIZE : remaining_size;

		__bio_add_page(clone, page, len, 0);

		remaining_size -= len;

have_pages:

		size_to_add = min((unsigned)PAGE_SIZE << order, remaining_size);

		__bio_add_page(clone, pages, size_to_add, 0);

		remaining_size -= size_to_add;

	}

	/* Allocate space for integrity tags */

static void crypt_free_buffer_pages(struct crypt_config *cc, struct bio *clone)

{

	struct bio_vec *bv;

	struct bvec_iter_all iter_all;

	struct folio_iter fi;

	bio_for_each_segment_all(bv, clone, iter_all) {

		BUG_ON(!bv->bv_page);

		mempool_free(bv->bv_page, &cc->page_pool);

	if (clone->bi_vcnt > 0) { /* bio_for_each_folio_all crashes with an empty bio */

		bio_for_each_folio_all(fi, clone) {

			if (folio_test_large(fi.folio))

				folio_put(fi.folio);

			else

				mempool_free(&fi.folio->page, &cc->page_pool);

		}

	}

}

		ret = crypt_ctr_auth_cipher(cc, cipher_api);

		if (ret < 0) {

			ti->error = "Invalid AEAD cipher spec";

			return -ENOMEM;

			return ret;

		}

	}