* [PATCH v1 00/12] T10-DIF Initial support for iSER target
@ 2014-02-16 17:38 Sagi Grimberg
[not found] ` <1392572297-5333-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (8 more replies)
0 siblings, 9 replies; 17+ messages in thread
From: Sagi Grimberg @ 2014-02-16 17:38 UTC (permalink / raw
To: nab-IzHhD5pYlfBP7FQvKIMDCQ
Cc: target-devel-u79uwXL29TY76Z2rM5mHXA,
roland-DgEjT+Ai2ygdnm+yROfE0A, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
ogerlitz-VPRAkNaXOzVWk0Htik3J/w, oren-VPRAkNaXOzVWk0Htik3J/w
Hey Nic and folks,
This patchset introduces target side T10-PI offload support over
RDMA. Currently the implementation is for iSER transport but can
be easily extended to SRP (or FCoE in the future).
Should mention that this patchset depends on RDMA signature
verbs making it for inclusion which will hopefully happen in
near future.
This code was tested against iSER legacy initiator, and also
initiator that uses T10-PI offload as well. I'll clean up the
initiator code in the following days and submit that as well.
This code works under medium loads using backstores:
- FileIO with DIF emulation.
- RD with DIF emulation.
- iBLOCK (scsi_debug with DIF support).
Nic, Can you take this code to rdma_dif branch until RDMA
Signature verbs set will pass submission? (after review of
course...).
Chnages from v0:
- Rebased from for-next
- Target core:
- Don't minor fixes for check_prot - to support transports
that doesn't use submit_map_sgls.
- file format - use escape values.
- Removed redundant prot_handover.
- Added protection checks and operation set.
- iSER:
- Added preperation routines for mapping/unmapping buffers
to ease the amount of code in isert_reg_rdma.
- Fixed print of DIF error (sector instead of offset).
- Fix RDMA length for protection on wire domain.
- Refactored reg_sig_mr to use helper routines.
Sagi Grimberg (12):
Target/transport: Allocate protection sg if needed
Target/sbc: Set protection operation and relevant checks
Target/file: place escape values for protection information format
Target/sbc: don't return from sbc_check for non prot_sg
Target/core: Remove prot_handover use for now
Target/iscsi: Add T10-PI indication for iscsi_portal_group
Target/configfs: Expose iSCSI network portal group T10-PI support
Target/sbc: add debug print
IB/isert: Introduce isert_map/unmap_data_buf
IB/isert: Initialize T10-PI resources
IB/isert: Accept RDMA_WRITE completions
IB/isert: Support T10-PI protected transactions
drivers/infiniband/ulp/isert/ib_isert.c | 675 ++++++++++++++++++++------
drivers/infiniband/ulp/isert/ib_isert.h | 38 ++-
drivers/target/iscsi/iscsi_target_configfs.c | 6 +
drivers/target/iscsi/iscsi_target_core.h | 5 +-
drivers/target/iscsi/iscsi_target_tpg.c | 21 +
drivers/target/iscsi/iscsi_target_tpg.h | 1 +
drivers/target/target_core_file.c | 40 +--
drivers/target/target_core_sbc.c | 93 ++++-
drivers/target/target_core_transport.c | 33 ++-
include/target/target_core_base.h | 13 +-
10 files changed, 698 insertions(+), 227 deletions(-)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v1 01/12] Target/transport: Allocate protection sg if needed
[not found] ` <1392572297-5333-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2014-02-16 17:38 ` Sagi Grimberg
[not found] ` <1392572297-5333-2-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2014-02-16 17:38 ` [PATCH v1 02/12] Target/sbc: Set protection operation and relevant checks Sagi Grimberg
` (3 subsequent siblings)
4 siblings, 1 reply; 17+ messages in thread
From: Sagi Grimberg @ 2014-02-16 17:38 UTC (permalink / raw
To: nab-IzHhD5pYlfBP7FQvKIMDCQ
Cc: target-devel-u79uwXL29TY76Z2rM5mHXA,
roland-DgEjT+Ai2ygdnm+yROfE0A, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
ogerlitz-VPRAkNaXOzVWk0Htik3J/w, oren-VPRAkNaXOzVWk0Htik3J/w
In case protection information is involved, allocate
protection SG-list for transport.
Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/target/target_core_transport.c | 12 ++++++++++++
1 files changed, 12 insertions(+), 0 deletions(-)
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index 24b4f65..f029bb7 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -2036,6 +2036,10 @@ static inline void transport_free_pages(struct se_cmd *cmd)
transport_free_sgl(cmd->t_bidi_data_sg, cmd->t_bidi_data_nents);
cmd->t_bidi_data_sg = NULL;
cmd->t_bidi_data_nents = 0;
+
+ transport_free_sgl(cmd->t_prot_sg, cmd->t_prot_nents);
+ cmd->t_prot_sg = NULL;
+ cmd->t_prot_nents = 0;
}
/**
@@ -2199,6 +2203,14 @@ transport_generic_new_cmd(struct se_cmd *cmd)
return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
}
+ if (cmd->prot_type != TARGET_PROT_NORMAL) {
+ ret = target_alloc_sgl(&cmd->t_prot_sg,
+ &cmd->t_prot_nents,
+ cmd->prot_length, zero_flag);
+ if (ret < 0)
+ return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
+ }
+
ret = target_alloc_sgl(&cmd->t_data_sg, &cmd->t_data_nents,
cmd->data_length, zero_flag);
if (ret < 0)
--
1.7.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v1 02/12] Target/sbc: Set protection operation and relevant checks
[not found] ` <1392572297-5333-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2014-02-16 17:38 ` [PATCH v1 01/12] Target/transport: Allocate protection sg if needed Sagi Grimberg
@ 2014-02-16 17:38 ` Sagi Grimberg
2014-02-18 19:53 ` Nicholas A. Bellinger
2014-02-16 17:38 ` [PATCH v1 07/12] Target/configfs: Expose iSCSI network portal group T10-PI support Sagi Grimberg
` (2 subsequent siblings)
4 siblings, 1 reply; 17+ messages in thread
From: Sagi Grimberg @ 2014-02-16 17:38 UTC (permalink / raw
To: nab-IzHhD5pYlfBP7FQvKIMDCQ
Cc: target-devel-u79uwXL29TY76Z2rM5mHXA,
roland-DgEjT+Ai2ygdnm+yROfE0A, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
ogerlitz-VPRAkNaXOzVWk0Htik3J/w, oren-VPRAkNaXOzVWk0Htik3J/w
SBC-3 mandates the protection checks that must be
performed in the rdprotect/wrprotect field. Use them.
According to backstore device pi_attributes and
cdb rdprotect/wrprotect.
Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/target/target_core_sbc.c | 88 ++++++++++++++++++++++++++++++++++---
include/target/target_core_base.h | 7 +++
2 files changed, 88 insertions(+), 7 deletions(-)
diff --git a/drivers/target/target_core_sbc.c b/drivers/target/target_core_sbc.c
index a448944..dfeb1c2 100644
--- a/drivers/target/target_core_sbc.c
+++ b/drivers/target/target_core_sbc.c
@@ -240,6 +240,11 @@ static inline u32 transport_lba_32(unsigned char *cdb)
return (cdb[2] << 24) | (cdb[3] << 16) | (cdb[4] << 8) | cdb[5];
}
+static inline u8 rwprotect(unsigned char *cdb)
+{
+ return cdb[1] >> 5;
+}
+
static inline unsigned long long transport_lba_64(unsigned char *cdb)
{
unsigned int __v1, __v2;
@@ -569,30 +574,95 @@ sbc_compare_and_write(struct se_cmd *cmd)
return TCM_NO_SENSE;
}
+static int
+sbc_set_prot_op_checks(u8 protect, enum target_prot_type prot_type,
+ u8 op, struct se_cmd *cmd)
+{
+ switch (op) {
+ case READ_10:
+ case READ_12:
+ case READ_16:
+ cmd->prot_op = protect ? TARGET_PROT_DIN_PASS :
+ TARGET_PROT_DIN_STRIP;
+ switch (protect) {
+ case 0x0:
+ case 0x1:
+ case 0x5:
+ cmd->prot_checks = TARGET_DIF_CHECK_GUARD;
+ if (prot_type == TARGET_DIF_TYPE1_PROT)
+ cmd->prot_checks |= TARGET_DIF_CHECK_REFTAG;
+ break;
+ case 0x2:
+ if (prot_type == TARGET_DIF_TYPE1_PROT)
+ cmd->prot_checks = TARGET_DIF_CHECK_REFTAG;
+ break;
+ case 0x3:
+ cmd->prot_checks = 0;
+ break;
+ case 0x4:
+ cmd->prot_checks = TARGET_DIF_CHECK_GUARD;
+ break;
+ default:
+ pr_err("Unsupported protect field %d\n", protect);
+ return -EINVAL;
+ }
+ break;
+ case WRITE_10:
+ case WRITE_12:
+ case WRITE_16:
+ case WRITE_VERIFY:
+ cmd->prot_op = protect ? TARGET_PROT_DOUT_PASS :
+ TARGET_PROT_DOUT_INSERT;
+ switch (protect) {
+ case 0x0:
+ case 0x3:
+ cmd->prot_checks = 0;
+ case 0x1:
+ case 0x5:
+ cmd->prot_checks = TARGET_DIF_CHECK_GUARD;
+ if (prot_type == TARGET_DIF_TYPE1_PROT)
+ cmd->prot_checks |= TARGET_DIF_CHECK_REFTAG;
+ break;
+ case 0x2:
+ if (prot_type == TARGET_DIF_TYPE1_PROT)
+ cmd->prot_checks = TARGET_DIF_CHECK_REFTAG;
+ break;
+ case 0x4:
+ cmd->prot_checks = TARGET_DIF_CHECK_GUARD;
+ break;
+ default:
+ pr_err("Unsupported protect field %d\n", protect);
+ return -EINVAL;
+ }
+ break;
+ default:
+ pr_err("ERROR: bad opcode %d\n", op);
+ return TCM_INVALID_CDB_FIELD;
+ }
+
+ return 0;
+}
+
static bool
sbc_check_prot(struct se_device *dev, struct se_cmd *cmd, unsigned char *cdb,
u32 sectors)
{
+ u8 protect = rwprotect(cdb);
+
if (!cmd->t_prot_sg || !cmd->t_prot_nents)
return true;
switch (dev->dev_attrib.pi_prot_type) {
case TARGET_DIF_TYPE3_PROT:
- if (!(cdb[1] & 0xe0))
- return true;
-
cmd->reftag_seed = 0xffffffff;
break;
case TARGET_DIF_TYPE2_PROT:
- if (cdb[1] & 0xe0)
+ if (protect)
return false;
cmd->reftag_seed = cmd->t_task_lba;
break;
case TARGET_DIF_TYPE1_PROT:
- if (!(cdb[1] & 0xe0))
- return true;
-
cmd->reftag_seed = cmd->t_task_lba;
break;
case TARGET_DIF_TYPE0_PROT:
@@ -600,6 +670,10 @@ sbc_check_prot(struct se_device *dev, struct se_cmd *cmd, unsigned char *cdb,
return true;
}
+ if (sbc_set_prot_op_checks(protect, dev->dev_attrib.pi_prot_type,
+ cdb[0], cmd))
+ return false;
+
cmd->prot_type = dev->dev_attrib.pi_prot_type;
cmd->prot_length = dev->prot_length * sectors;
cmd->prot_handover = PROT_SEPERATED;
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 1772fad..5ae9249 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -463,6 +463,12 @@ enum target_prot_type {
TARGET_DIF_TYPE3_PROT,
};
+enum target_core_dif_check {
+ TARGET_DIF_CHECK_GUARD = 0x1 << 0,
+ TARGET_DIF_CHECK_APPTAG = 0x1 << 1,
+ TARGET_DIF_CHECK_REFTAG = 0x1 << 2,
+};
+
struct se_dif_v1_tuple {
__be16 guard_tag;
__be16 app_tag;
@@ -556,6 +562,7 @@ struct se_cmd {
/* DIF related members */
enum target_prot_op prot_op;
enum target_prot_type prot_type;
+ u8 prot_checks;
u32 prot_length;
u32 reftag_seed;
struct scatterlist *t_prot_sg;
--
1.7.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v1 03/12] Target/file: place escape values for protection information format
2014-02-16 17:38 [PATCH v1 00/12] T10-DIF Initial support for iSER target Sagi Grimberg
[not found] ` <1392572297-5333-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2014-02-16 17:38 ` Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 04/12] Target/sbc: don't return from sbc_check for non prot_sg Sagi Grimberg
` (6 subsequent siblings)
8 siblings, 0 replies; 17+ messages in thread
From: Sagi Grimberg @ 2014-02-16 17:38 UTC (permalink / raw
To: nab; +Cc: target-devel, roland, linux-rdma, ogerlitz, oren
No need to actually compute protection information when formatting
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
drivers/target/target_core_file.c | 40 +------------------------------------
1 files changed, 1 insertions(+), 39 deletions(-)
diff --git a/drivers/target/target_core_file.c b/drivers/target/target_core_file.c
index cf991a9..7d6cdda 100644
--- a/drivers/target/target_core_file.c
+++ b/drivers/target/target_core_file.c
@@ -854,25 +854,6 @@ static int fd_init_prot(struct se_device *dev)
return 0;
}
-static void fd_init_format_buf(struct se_device *dev, unsigned char *buf,
- u32 unit_size, u32 *ref_tag, u16 app_tag,
- bool inc_reftag)
-{
- unsigned char *p = buf;
- int i;
-
- for (i = 0; i < unit_size; i += dev->prot_length) {
- *((u16 *)&p[0]) = 0xffff;
- *((__be16 *)&p[2]) = cpu_to_be16(app_tag);
- *((__be32 *)&p[4]) = cpu_to_be32(*ref_tag);
-
- if (inc_reftag)
- (*ref_tag)++;
-
- p += dev->prot_length;
- }
-}
-
static int fd_format_prot(struct se_device *dev)
{
struct fd_dev *fd_dev = FD_DEV(dev);
@@ -880,10 +861,8 @@ static int fd_format_prot(struct se_device *dev)
sector_t prot_length, prot;
unsigned char *buf;
loff_t pos = 0;
- u32 ref_tag = 0;
int unit_size = FDBD_FORMAT_UNIT_SIZE * dev->dev_attrib.block_size;
int rc, ret = 0, size, len;
- bool inc_reftag = false;
if (!dev->dev_attrib.pi_prot_type) {
pr_err("Unable to format_prot while pi_prot_type == 0\n");
@@ -894,37 +873,20 @@ static int fd_format_prot(struct se_device *dev)
return -ENODEV;
}
- switch (dev->dev_attrib.pi_prot_type) {
- case TARGET_DIF_TYPE3_PROT:
- ref_tag = 0xffffffff;
- break;
- case TARGET_DIF_TYPE2_PROT:
- case TARGET_DIF_TYPE1_PROT:
- inc_reftag = true;
- break;
- default:
- break;
- }
-
buf = vzalloc(unit_size);
if (!buf) {
pr_err("Unable to allocate FILEIO prot buf\n");
return -ENOMEM;
}
-
prot_length = (dev->transport->get_blocks(dev) + 1) * dev->prot_length;
size = prot_length;
pr_debug("Using FILEIO prot_length: %llu\n",
(unsigned long long)prot_length);
+ memset(buf, 0xff, unit_size);
for (prot = 0; prot < prot_length; prot += unit_size) {
-
- fd_init_format_buf(dev, buf, unit_size, &ref_tag, 0xffff,
- inc_reftag);
-
len = min(unit_size, size);
-
rc = kernel_write(prot_fd, buf, len, pos);
if (rc != len) {
pr_err("vfs_write to prot file failed: %d\n", rc);
--
1.7.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v1 04/12] Target/sbc: don't return from sbc_check for non prot_sg
2014-02-16 17:38 [PATCH v1 00/12] T10-DIF Initial support for iSER target Sagi Grimberg
[not found] ` <1392572297-5333-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2014-02-16 17:38 ` [PATCH v1 03/12] Target/file: place escape values for protection information format Sagi Grimberg
@ 2014-02-16 17:38 ` Sagi Grimberg
2014-02-18 19:55 ` Nicholas A. Bellinger
2014-02-16 17:38 ` [PATCH v1 05/12] Target/core: Remove prot_handover use for now Sagi Grimberg
` (5 subsequent siblings)
8 siblings, 1 reply; 17+ messages in thread
From: Sagi Grimberg @ 2014-02-16 17:38 UTC (permalink / raw
To: nab; +Cc: target-devel, roland, linux-rdma, ogerlitz, oren
For transports which use generic new command these
buffers have yet to be allocated. Instead check afterwards
if command required prot buffers but none are provided.
Also this way, target may support protection information
against legacy initiators (writes are inserted and reads
are stripped).
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
drivers/target/target_core_sbc.c | 3 ---
drivers/target/target_core_transport.c | 21 +++++++++++++++++----
2 files changed, 17 insertions(+), 7 deletions(-)
diff --git a/drivers/target/target_core_sbc.c b/drivers/target/target_core_sbc.c
index dfeb1c2..3f12726 100644
--- a/drivers/target/target_core_sbc.c
+++ b/drivers/target/target_core_sbc.c
@@ -649,9 +649,6 @@ sbc_check_prot(struct se_device *dev, struct se_cmd *cmd, unsigned char *cdb,
{
u8 protect = rwprotect(cdb);
- if (!cmd->t_prot_sg || !cmd->t_prot_nents)
- return true;
-
switch (dev->dev_attrib.pi_prot_type) {
case TARGET_DIF_TYPE3_PROT:
cmd->reftag_seed = 0xffffffff;
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index f029bb7..3a5550a 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -1365,6 +1365,13 @@ int target_submit_cmd_map_sgls(struct se_cmd *se_cmd, struct se_session *se_sess
target_put_sess_cmd(se_sess, se_cmd);
return 0;
}
+
+ rc = target_setup_cmd_from_cdb(se_cmd, cdb);
+ if (rc != 0) {
+ transport_generic_request_failure(se_cmd, rc);
+ return 0;
+ }
+
/*
* Save pointers for SGLs containing protection information,
* if present.
@@ -1374,11 +1381,17 @@ int target_submit_cmd_map_sgls(struct se_cmd *se_cmd, struct se_session *se_sess
se_cmd->t_prot_nents = sgl_prot_count;
}
- rc = target_setup_cmd_from_cdb(se_cmd, cdb);
- if (rc != 0) {
- transport_generic_request_failure(se_cmd, rc);
- return 0;
+ /*
+ * Fail if protection operation requiers protection
+ * information buffers but None are provided!
+ */
+ if ((!se_cmd->t_prot_sg || !se_cmd->t_prot_nents) &&
+ (se_cmd->prot_op != TARGET_PROT_NORMAL)) {
+ pr_err("ERROR: protection information was requested but "
+ "protection buffers weren't provided.\n");
+ return -EINVAL;
}
+
/*
* When a non zero sgl_count has been passed perform SGL passthrough
* mapping for pre-allocated fabric memory instead of having target
--
1.7.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v1 05/12] Target/core: Remove prot_handover use for now
2014-02-16 17:38 [PATCH v1 00/12] T10-DIF Initial support for iSER target Sagi Grimberg
` (2 preceding siblings ...)
2014-02-16 17:38 ` [PATCH v1 04/12] Target/sbc: don't return from sbc_check for non prot_sg Sagi Grimberg
@ 2014-02-16 17:38 ` Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 06/12] Target/iscsi: Add T10-PI indication for iscsi_portal_group Sagi Grimberg
` (4 subsequent siblings)
8 siblings, 0 replies; 17+ messages in thread
From: Sagi Grimberg @ 2014-02-16 17:38 UTC (permalink / raw
To: nab; +Cc: target-devel, roland, linux-rdma, ogerlitz, oren
This is not going to be supported soon - so drop it.
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
drivers/target/target_core_sbc.c | 1 -
include/target/target_core_base.h | 6 ------
2 files changed, 0 insertions(+), 7 deletions(-)
diff --git a/drivers/target/target_core_sbc.c b/drivers/target/target_core_sbc.c
index 3f12726..3326acb 100644
--- a/drivers/target/target_core_sbc.c
+++ b/drivers/target/target_core_sbc.c
@@ -673,7 +673,6 @@ sbc_check_prot(struct se_device *dev, struct se_cmd *cmd, unsigned char *cdb,
cmd->prot_type = dev->dev_attrib.pi_prot_type;
cmd->prot_length = dev->prot_length * sectors;
- cmd->prot_handover = PROT_SEPERATED;
return true;
}
diff --git a/include/target/target_core_base.h b/include/target/target_core_base.h
index 5ae9249..344fd05 100644
--- a/include/target/target_core_base.h
+++ b/include/target/target_core_base.h
@@ -451,11 +451,6 @@ enum target_prot_op {
TARGET_PROT_DOUT_PASS,
};
-enum target_prot_ho {
- PROT_SEPERATED,
- PROT_INTERLEAVED,
-};
-
enum target_prot_type {
TARGET_DIF_TYPE0_PROT,
TARGET_DIF_TYPE1_PROT,
@@ -567,7 +562,6 @@ struct se_cmd {
u32 reftag_seed;
struct scatterlist *t_prot_sg;
unsigned int t_prot_nents;
- enum target_prot_ho prot_handover;
sense_reason_t pi_err;
sector_t bad_sector;
};
--
1.7.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v1 06/12] Target/iscsi: Add T10-PI indication for iscsi_portal_group
2014-02-16 17:38 [PATCH v1 00/12] T10-DIF Initial support for iSER target Sagi Grimberg
` (3 preceding siblings ...)
2014-02-16 17:38 ` [PATCH v1 05/12] Target/core: Remove prot_handover use for now Sagi Grimberg
@ 2014-02-16 17:38 ` Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 08/12] Target/sbc: add debug print Sagi Grimberg
` (3 subsequent siblings)
8 siblings, 0 replies; 17+ messages in thread
From: Sagi Grimberg @ 2014-02-16 17:38 UTC (permalink / raw
To: nab; +Cc: target-devel, roland, linux-rdma, ogerlitz, oren
In case an iscsi portal group will be defined as t10_pi enabled,
all connections on top of it will support protected transactions.
T10-PI support may require extra reource allocation and maintenance by
the transport layer, so we don't want to apply them on non-t10_pi network
portals. This is a hook for the iscsi target layer to signal the transport
at connection establishment that this connection will carry protected
transactions.
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
drivers/target/iscsi/iscsi_target_core.h | 5 ++++-
drivers/target/iscsi/iscsi_target_tpg.c | 2 ++
2 files changed, 6 insertions(+), 1 deletions(-)
diff --git a/drivers/target/iscsi/iscsi_target_core.h b/drivers/target/iscsi/iscsi_target_core.h
index 48f7b3b..886d74d 100644
--- a/drivers/target/iscsi/iscsi_target_core.h
+++ b/drivers/target/iscsi/iscsi_target_core.h
@@ -58,7 +58,8 @@
#define TA_DEMO_MODE_DISCOVERY 1
#define TA_DEFAULT_ERL 0
#define TA_CACHE_CORE_NPS 0
-
+/* T10 protection information disabled by default */
+#define TA_DEFAULT_T10_PI 0
#define ISCSI_IOV_DATA_BUFFER 5
@@ -765,6 +766,7 @@ struct iscsi_tpg_attrib {
u32 prod_mode_write_protect;
u32 demo_mode_discovery;
u32 default_erl;
+ u8 t10_pi;
struct iscsi_portal_group *tpg;
};
@@ -787,6 +789,7 @@ struct iscsi_np {
void *np_context;
struct iscsit_transport *np_transport;
struct list_head np_list;
+ struct iscsi_tpg_np *tpg_np;
} ____cacheline_aligned;
struct iscsi_tpg_np {
diff --git a/drivers/target/iscsi/iscsi_target_tpg.c b/drivers/target/iscsi/iscsi_target_tpg.c
index 3976183..80ae14c 100644
--- a/drivers/target/iscsi/iscsi_target_tpg.c
+++ b/drivers/target/iscsi/iscsi_target_tpg.c
@@ -225,6 +225,7 @@ static void iscsit_set_default_tpg_attribs(struct iscsi_portal_group *tpg)
a->prod_mode_write_protect = TA_PROD_MODE_WRITE_PROTECT;
a->demo_mode_discovery = TA_DEMO_MODE_DISCOVERY;
a->default_erl = TA_DEFAULT_ERL;
+ a->t10_pi = TA_DEFAULT_T10_PI;
}
int iscsit_tpg_add_portal_group(struct iscsi_tiqn *tiqn, struct iscsi_portal_group *tpg)
@@ -500,6 +501,7 @@ struct iscsi_tpg_np *iscsit_tpg_add_network_portal(
init_completion(&tpg_np->tpg_np_comp);
kref_init(&tpg_np->tpg_np_kref);
tpg_np->tpg_np = np;
+ np->tpg_np = tpg_np;
tpg_np->tpg = tpg;
spin_lock(&tpg->tpg_np_lock);
--
1.7.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v1 07/12] Target/configfs: Expose iSCSI network portal group T10-PI support
[not found] ` <1392572297-5333-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2014-02-16 17:38 ` [PATCH v1 01/12] Target/transport: Allocate protection sg if needed Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 02/12] Target/sbc: Set protection operation and relevant checks Sagi Grimberg
@ 2014-02-16 17:38 ` Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 09/12] IB/isert: Introduce isert_map/unmap_data_buf Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 11/12] IB/isert: Accept RDMA_WRITE completions Sagi Grimberg
4 siblings, 0 replies; 17+ messages in thread
From: Sagi Grimberg @ 2014-02-16 17:38 UTC (permalink / raw
To: nab-IzHhD5pYlfBP7FQvKIMDCQ
Cc: target-devel-u79uwXL29TY76Z2rM5mHXA,
roland-DgEjT+Ai2ygdnm+yROfE0A, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
ogerlitz-VPRAkNaXOzVWk0Htik3J/w, oren-VPRAkNaXOzVWk0Htik3J/w
User may enable T10-PI support per network portal group. any connection
established on top of it, will be required to serve protected transactions.
Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/target/iscsi/iscsi_target_configfs.c | 6 ++++++
drivers/target/iscsi/iscsi_target_tpg.c | 19 +++++++++++++++++++
drivers/target/iscsi/iscsi_target_tpg.h | 1 +
3 files changed, 26 insertions(+), 0 deletions(-)
diff --git a/drivers/target/iscsi/iscsi_target_configfs.c b/drivers/target/iscsi/iscsi_target_configfs.c
index 1c0088f..fe35dcb 100644
--- a/drivers/target/iscsi/iscsi_target_configfs.c
+++ b/drivers/target/iscsi/iscsi_target_configfs.c
@@ -1052,6 +1052,11 @@ TPG_ATTR(demo_mode_discovery, S_IRUGO | S_IWUSR);
*/
DEF_TPG_ATTRIB(default_erl);
TPG_ATTR(default_erl, S_IRUGO | S_IWUSR);
+/*
+ * Define iscsi_tpg_attrib_s_t10_pi
+ */
+DEF_TPG_ATTRIB(t10_pi);
+TPG_ATTR(t10_pi, S_IRUGO | S_IWUSR);
static struct configfs_attribute *lio_target_tpg_attrib_attrs[] = {
&iscsi_tpg_attrib_authentication.attr,
@@ -1064,6 +1069,7 @@ static struct configfs_attribute *lio_target_tpg_attrib_attrs[] = {
&iscsi_tpg_attrib_prod_mode_write_protect.attr,
&iscsi_tpg_attrib_demo_mode_discovery.attr,
&iscsi_tpg_attrib_default_erl.attr,
+ &iscsi_tpg_attrib_t10_pi.attr,
NULL,
};
diff --git a/drivers/target/iscsi/iscsi_target_tpg.c b/drivers/target/iscsi/iscsi_target_tpg.c
index 80ae14c..d95a5f2 100644
--- a/drivers/target/iscsi/iscsi_target_tpg.c
+++ b/drivers/target/iscsi/iscsi_target_tpg.c
@@ -860,3 +860,22 @@ int iscsit_ta_default_erl(
return 0;
}
+
+int iscsit_ta_t10_pi(
+ struct iscsi_portal_group *tpg,
+ u32 flag)
+{
+ struct iscsi_tpg_attrib *a = &tpg->tpg_attrib;
+
+ if ((flag != 0) && (flag != 1)) {
+ pr_err("Illegal value %d\n", flag);
+ return -EINVAL;
+ }
+
+ a->t10_pi = flag;
+ pr_debug("iSCSI_TPG[%hu] - T10 Protection information bit:"
+ " %s\n", tpg->tpgt, (a->t10_pi) ?
+ "ON" : "OFF");
+
+ return 0;
+}
diff --git a/drivers/target/iscsi/iscsi_target_tpg.h b/drivers/target/iscsi/iscsi_target_tpg.h
index 213c0fc..0a182f2 100644
--- a/drivers/target/iscsi/iscsi_target_tpg.h
+++ b/drivers/target/iscsi/iscsi_target_tpg.h
@@ -39,5 +39,6 @@ extern int iscsit_ta_demo_mode_write_protect(struct iscsi_portal_group *, u32);
extern int iscsit_ta_prod_mode_write_protect(struct iscsi_portal_group *, u32);
extern int iscsit_ta_demo_mode_discovery(struct iscsi_portal_group *, u32);
extern int iscsit_ta_default_erl(struct iscsi_portal_group *, u32);
+extern int iscsit_ta_t10_pi(struct iscsi_portal_group *, u32);
#endif /* ISCSI_TARGET_TPG_H */
--
1.7.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v1 08/12] Target/sbc: add debug print
2014-02-16 17:38 [PATCH v1 00/12] T10-DIF Initial support for iSER target Sagi Grimberg
` (4 preceding siblings ...)
2014-02-16 17:38 ` [PATCH v1 06/12] Target/iscsi: Add T10-PI indication for iscsi_portal_group Sagi Grimberg
@ 2014-02-16 17:38 ` Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 10/12] IB/isert: Initialize T10-PI resources Sagi Grimberg
` (2 subsequent siblings)
8 siblings, 0 replies; 17+ messages in thread
From: Sagi Grimberg @ 2014-02-16 17:38 UTC (permalink / raw
To: nab; +Cc: target-devel, roland, linux-rdma, ogerlitz, oren
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
drivers/target/target_core_sbc.c | 3 +++
1 files changed, 3 insertions(+), 0 deletions(-)
diff --git a/drivers/target/target_core_sbc.c b/drivers/target/target_core_sbc.c
index 3326acb..fac8c77 100644
--- a/drivers/target/target_core_sbc.c
+++ b/drivers/target/target_core_sbc.c
@@ -673,6 +673,9 @@ sbc_check_prot(struct se_device *dev, struct se_cmd *cmd, unsigned char *cdb,
cmd->prot_type = dev->dev_attrib.pi_prot_type;
cmd->prot_length = dev->prot_length * sectors;
+ pr_debug("%s: prot_type=%d, prot_length=%d prot_op=%d prot_checks=%d\n",
+ __func__, cmd->prot_type, cmd->prot_length,
+ cmd->prot_op, cmd->prot_checks);
return true;
}
--
1.7.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v1 09/12] IB/isert: Introduce isert_map/unmap_data_buf
[not found] ` <1392572297-5333-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (2 preceding siblings ...)
2014-02-16 17:38 ` [PATCH v1 07/12] Target/configfs: Expose iSCSI network portal group T10-PI support Sagi Grimberg
@ 2014-02-16 17:38 ` Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 11/12] IB/isert: Accept RDMA_WRITE completions Sagi Grimberg
4 siblings, 0 replies; 17+ messages in thread
From: Sagi Grimberg @ 2014-02-16 17:38 UTC (permalink / raw
To: nab-IzHhD5pYlfBP7FQvKIMDCQ
Cc: target-devel-u79uwXL29TY76Z2rM5mHXA,
roland-DgEjT+Ai2ygdnm+yROfE0A, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
ogerlitz-VPRAkNaXOzVWk0Htik3J/w, oren-VPRAkNaXOzVWk0Htik3J/w
export map/unmap data buffer to a routine that may
be used in various places in the code and keep the
mapping data in a designated descriptor. Also, let
isert_fast_reg_mr to decide weather to use global
MR or do fast registration.
This commit does not change any functionality.
Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/ulp/isert/ib_isert.c | 238 +++++++++++++++----------------
drivers/infiniband/ulp/isert/ib_isert.h | 14 ++-
2 files changed, 127 insertions(+), 125 deletions(-)
diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 1a88080..299df4d 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -1392,19 +1392,60 @@ isert_rx_completion(struct iser_rx_desc *desc, struct isert_conn *isert_conn,
}
}
+static int
+isert_map_data_buf(struct isert_conn *isert_conn, struct isert_cmd *isert_cmd,
+ struct scatterlist *sg, u32 nents, u32 length, u32 offset,
+ enum iser_ib_op_code op, struct isert_data_buf *data)
+{
+ struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
+
+ data->dma_dir = op == ISER_IB_RDMA_WRITE ?
+ DMA_TO_DEVICE : DMA_FROM_DEVICE;
+
+ data->len = length - offset;
+ data->offset = offset;
+ data->sg_off = data->offset / PAGE_SIZE;
+
+ data->sg = &sg[data->sg_off];
+ data->nents = min_t(unsigned int, nents - data->sg_off,
+ ISCSI_ISER_SG_TABLESIZE);
+ data->len = min_t(unsigned int, data->len, ISCSI_ISER_SG_TABLESIZE *
+ PAGE_SIZE);
+
+ data->dma_nents = ib_dma_map_sg(ib_dev, data->sg, data->nents,
+ data->dma_dir);
+ if (unlikely(!data->dma_nents)) {
+ pr_err("Cmd: unable to dma map SGs %p\n", sg);
+ return -EINVAL;
+ }
+
+ pr_debug("Mapped cmd: %p count: %u sg: %p sg_nents: %u rdma_len %d\n",
+ isert_cmd, data->dma_nents, data->sg, data->nents, data->len);
+
+ return 0;
+}
+
+static void
+isert_unmap_data_buf(struct isert_conn *isert_conn, struct isert_data_buf *data)
+{
+ struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
+
+ ib_dma_unmap_sg(ib_dev, data->sg, data->nents, data->dma_dir);
+ memset(data, 0, sizeof(*data));
+}
+
+
+
static void
isert_unmap_cmd(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn)
{
struct isert_rdma_wr *wr = &isert_cmd->rdma_wr;
- struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
pr_debug("isert_unmap_cmd: %p\n", isert_cmd);
- if (wr->sge) {
+
+ if (wr->data.sg) {
pr_debug("isert_unmap_cmd: %p unmap_sg op\n", isert_cmd);
- ib_dma_unmap_sg(ib_dev, wr->sge, wr->num_sge,
- (wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
- DMA_TO_DEVICE : DMA_FROM_DEVICE);
- wr->sge = NULL;
+ isert_unmap_data_buf(isert_conn, &wr->data);
}
if (wr->send_wr) {
@@ -1424,7 +1465,6 @@ static void
isert_unreg_rdma(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn)
{
struct isert_rdma_wr *wr = &isert_cmd->rdma_wr;
- struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
LIST_HEAD(unmap_list);
pr_debug("unreg_fastreg_cmd: %p\n", isert_cmd);
@@ -1438,12 +1478,9 @@ isert_unreg_rdma(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn)
wr->fr_desc = NULL;
}
- if (wr->sge) {
+ if (wr->data.sg) {
pr_debug("unreg_fastreg_cmd: %p unmap_sg op\n", isert_cmd);
- ib_dma_unmap_sg(ib_dev, wr->sge, wr->num_sge,
- (wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
- DMA_TO_DEVICE : DMA_FROM_DEVICE);
- wr->sge = NULL;
+ isert_unmap_data_buf(isert_conn, &wr->data);
}
wr->ib_sge = NULL;
@@ -1548,7 +1585,7 @@ isert_completion_rdma_read(struct iser_tx_desc *tx_desc,
iscsit_stop_dataout_timer(cmd);
device->unreg_rdma_mem(isert_cmd, isert_conn);
- cmd->write_data_done = wr->cur_rdma_length;
+ cmd->write_data_done = wr->data.len;
pr_debug("Cmd: %p RDMA_READ comp calling execute_cmd\n", isert_cmd);
spin_lock_bh(&cmd->istate_lock);
@@ -2046,54 +2083,39 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
struct se_cmd *se_cmd = &cmd->se_cmd;
struct isert_cmd *isert_cmd = iscsit_priv_cmd(cmd);
struct isert_conn *isert_conn = (struct isert_conn *)conn->context;
- struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
+ struct isert_data_buf *data = &wr->data;
struct ib_send_wr *send_wr;
struct ib_sge *ib_sge;
- struct scatterlist *sg_start;
- u32 sg_off = 0, sg_nents;
- u32 offset = 0, data_len, data_left, rdma_write_max, va_offset = 0;
- int ret = 0, count, i, ib_sge_cnt;
+ u32 offset, data_len, data_left, rdma_write_max, va_offset = 0;
+ int ret = 0, i, ib_sge_cnt;
- if (wr->iser_ib_op == ISER_IB_RDMA_WRITE) {
- data_left = se_cmd->data_length;
- } else {
- sg_off = cmd->write_data_done / PAGE_SIZE;
- data_left = se_cmd->data_length - cmd->write_data_done;
- offset = cmd->write_data_done;
- isert_cmd->tx_desc.isert_cmd = isert_cmd;
- }
+ isert_cmd->tx_desc.isert_cmd = isert_cmd;
- sg_start = &cmd->se_cmd.t_data_sg[sg_off];
- sg_nents = se_cmd->t_data_nents - sg_off;
+ offset = wr->iser_ib_op == ISER_IB_RDMA_READ ? cmd->write_data_done : 0;
+ ret = isert_map_data_buf(isert_conn, isert_cmd, se_cmd->t_data_sg,
+ se_cmd->t_data_nents, se_cmd->data_length,
+ offset, wr->iser_ib_op, &wr->data);
+ if (ret)
+ return ret;
- count = ib_dma_map_sg(ib_dev, sg_start, sg_nents,
- (wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
- DMA_TO_DEVICE : DMA_FROM_DEVICE);
- if (unlikely(!count)) {
- pr_err("Cmd: %p unrable to map SGs\n", isert_cmd);
- return -EINVAL;
- }
- wr->sge = sg_start;
- wr->num_sge = sg_nents;
- wr->cur_rdma_length = data_left;
- pr_debug("Mapped cmd: %p count: %u sg: %p sg_nents: %u rdma_len %d\n",
- isert_cmd, count, sg_start, sg_nents, data_left);
+ data_left = data->len;
+ offset = data->offset;
- ib_sge = kzalloc(sizeof(struct ib_sge) * sg_nents, GFP_KERNEL);
+ ib_sge = kzalloc(sizeof(struct ib_sge) * data->nents, GFP_KERNEL);
if (!ib_sge) {
pr_warn("Unable to allocate ib_sge\n");
ret = -ENOMEM;
- goto unmap_sg;
+ goto unmap_cmd;
}
wr->ib_sge = ib_sge;
- wr->send_wr_num = DIV_ROUND_UP(sg_nents, isert_conn->max_sge);
+ wr->send_wr_num = DIV_ROUND_UP(data->nents, isert_conn->max_sge);
wr->send_wr = kzalloc(sizeof(struct ib_send_wr) * wr->send_wr_num,
GFP_KERNEL);
if (!wr->send_wr) {
pr_debug("Unable to allocate wr->send_wr\n");
ret = -ENOMEM;
- goto unmap_sg;
+ goto unmap_cmd;
}
wr->isert_cmd = isert_cmd;
@@ -2132,10 +2154,9 @@ isert_map_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
}
return 0;
-unmap_sg:
- ib_dma_unmap_sg(ib_dev, sg_start, sg_nents,
- (wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
- DMA_TO_DEVICE : DMA_FROM_DEVICE);
+unmap_cmd:
+ isert_unmap_data_buf(isert_conn, data);
+
return ret;
}
@@ -2179,10 +2200,10 @@ isert_map_fr_pagelist(struct ib_device *ib_dev,
}
static int
-isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
- struct isert_conn *isert_conn, struct scatterlist *sg_start,
- struct ib_sge *ib_sge, u32 sg_nents, u32 offset,
- unsigned int data_len)
+isert_fast_reg_mr(struct isert_conn *isert_conn,
+ struct fast_reg_descriptor *fr_desc,
+ struct isert_data_buf *mem,
+ struct ib_sge *sge)
{
struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
struct ib_send_wr fr_wr, inv_wr;
@@ -2191,13 +2212,19 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
u32 page_off;
u8 key;
- sg_nents = min_t(unsigned int, sg_nents, ISCSI_ISER_SG_TABLESIZE);
- page_off = offset % PAGE_SIZE;
+ if (mem->dma_nents == 1) {
+ sge->lkey = isert_conn->conn_mr->lkey;
+ sge->addr = ib_sg_dma_address(ib_dev, &mem->sg[0]);
+ sge->length = ib_sg_dma_len(ib_dev, &mem->sg[0]);
+ return 0;
+ }
+
+ page_off = mem->offset % PAGE_SIZE;
pr_debug("Use fr_desc %p sg_nents %d offset %u\n",
- fr_desc, sg_nents, offset);
+ fr_desc, mem->nents, mem->offset);
- pagelist_len = isert_map_fr_pagelist(ib_dev, sg_start, sg_nents,
+ pagelist_len = isert_map_fr_pagelist(ib_dev, mem->sg, mem->nents,
&fr_desc->data_frpl->page_list[0]);
if (!fr_desc->valid) {
@@ -2218,7 +2245,7 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
fr_wr.wr.fast_reg.page_list = fr_desc->data_frpl;
fr_wr.wr.fast_reg.page_list_len = pagelist_len;
fr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;
- fr_wr.wr.fast_reg.length = data_len;
+ fr_wr.wr.fast_reg.length = mem->len;
fr_wr.wr.fast_reg.rkey = fr_desc->data_mr->rkey;
fr_wr.wr.fast_reg.access_flags = IB_ACCESS_LOCAL_WRITE;
@@ -2234,12 +2261,12 @@ isert_fast_reg_mr(struct fast_reg_descriptor *fr_desc,
}
fr_desc->valid = false;
- ib_sge->lkey = fr_desc->data_mr->lkey;
- ib_sge->addr = fr_desc->data_frpl->page_list[0] + page_off;
- ib_sge->length = data_len;
+ sge->lkey = fr_desc->data_mr->lkey;
+ sge->addr = fr_desc->data_frpl->page_list[0] + page_off;
+ sge->length = mem->len;
pr_debug("RDMA ib_sge: addr: 0x%16llx length: %u lkey: %08x\n",
- ib_sge->addr, ib_sge->length, ib_sge->lkey);
+ sge->addr, sge->length, sge->lkey);
return ret;
}
@@ -2250,54 +2277,43 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
{
struct se_cmd *se_cmd = &cmd->se_cmd;
struct isert_cmd *isert_cmd = iscsit_priv_cmd(cmd);
- struct isert_conn *isert_conn = (struct isert_conn *)conn->context;
- struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
+ struct isert_conn *isert_conn = conn->context;
struct ib_send_wr *send_wr;
- struct ib_sge *ib_sge;
- struct scatterlist *sg_start;
- struct fast_reg_descriptor *fr_desc;
- u32 sg_off = 0, sg_nents;
- u32 offset = 0, data_len, data_left, rdma_write_max;
- int ret = 0, count;
+ struct fast_reg_descriptor *fr_desc = NULL;
+ u32 offset;
+ int ret = 0;
unsigned long flags;
- if (wr->iser_ib_op == ISER_IB_RDMA_WRITE) {
- data_left = se_cmd->data_length;
- } else {
- offset = cmd->write_data_done;
- sg_off = offset / PAGE_SIZE;
- data_left = se_cmd->data_length - cmd->write_data_done;
- isert_cmd->tx_desc.isert_cmd = isert_cmd;
- }
+ isert_cmd->tx_desc.isert_cmd = isert_cmd;
- sg_start = &cmd->se_cmd.t_data_sg[sg_off];
- sg_nents = se_cmd->t_data_nents - sg_off;
+ offset = wr->iser_ib_op == ISER_IB_RDMA_READ ? cmd->write_data_done : 0;
+ ret = isert_map_data_buf(isert_conn, isert_cmd, se_cmd->t_data_sg,
+ se_cmd->t_data_nents, se_cmd->data_length,
+ offset, wr->iser_ib_op, &wr->data);
+ if (ret)
+ return ret;
- count = ib_dma_map_sg(ib_dev, sg_start, sg_nents,
- (wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
- DMA_TO_DEVICE : DMA_FROM_DEVICE);
- if (unlikely(!count)) {
- pr_err("Cmd: %p unrable to map SGs\n", isert_cmd);
- return -EINVAL;
+ if (wr->data.dma_nents != 1) {
+ spin_lock_irqsave(&isert_conn->conn_lock, flags);
+ fr_desc = list_first_entry(&isert_conn->conn_fr_pool,
+ struct fast_reg_descriptor, list);
+ list_del(&fr_desc->list);
+ spin_unlock_irqrestore(&isert_conn->conn_lock, flags);
+ wr->fr_desc = fr_desc;
}
- wr->sge = sg_start;
- wr->num_sge = sg_nents;
- pr_debug("Mapped cmd: %p count: %u sg: %p sg_nents: %u rdma_len %d\n",
- isert_cmd, count, sg_start, sg_nents, data_left);
- memset(&wr->s_ib_sge, 0, sizeof(*ib_sge));
- ib_sge = &wr->s_ib_sge;
- wr->ib_sge = ib_sge;
+ ret = isert_fast_reg_mr(isert_conn, fr_desc, &wr->data, &wr->s_ib_sge);
+ if (ret)
+ goto unmap_cmd;
+ wr->ib_sge = &wr->s_ib_sge;
wr->send_wr_num = 1;
memset(&wr->s_send_wr, 0, sizeof(*send_wr));
wr->send_wr = &wr->s_send_wr;
-
wr->isert_cmd = isert_cmd;
- rdma_write_max = ISCSI_ISER_SG_TABLESIZE * PAGE_SIZE;
send_wr = &isert_cmd->rdma_wr.s_send_wr;
- send_wr->sg_list = ib_sge;
+ send_wr->sg_list = &wr->s_ib_sge;
send_wr->num_sge = 1;
send_wr->wr_id = (unsigned long)&isert_cmd->tx_desc;
if (wr->iser_ib_op == ISER_IB_RDMA_WRITE) {
@@ -2313,37 +2329,15 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
send_wr->send_flags = IB_SEND_SIGNALED;
}
- data_len = min(data_left, rdma_write_max);
- wr->cur_rdma_length = data_len;
-
- /* if there is a single dma entry, dma mr is sufficient */
- if (count == 1) {
- ib_sge->addr = ib_sg_dma_address(ib_dev, &sg_start[0]);
- ib_sge->length = ib_sg_dma_len(ib_dev, &sg_start[0]);
- ib_sge->lkey = isert_conn->conn_mr->lkey;
- wr->fr_desc = NULL;
- } else {
+ return 0;
+unmap_cmd:
+ if (fr_desc) {
spin_lock_irqsave(&isert_conn->conn_lock, flags);
- fr_desc = list_first_entry(&isert_conn->conn_fr_pool,
- struct fast_reg_descriptor, list);
- list_del(&fr_desc->list);
+ list_add_tail(&fr_desc->list, &isert_conn->conn_fr_pool);
spin_unlock_irqrestore(&isert_conn->conn_lock, flags);
- wr->fr_desc = fr_desc;
-
- ret = isert_fast_reg_mr(fr_desc, isert_conn, sg_start,
- ib_sge, sg_nents, offset, data_len);
- if (ret) {
- list_add_tail(&fr_desc->list, &isert_conn->conn_fr_pool);
- goto unmap_sg;
- }
}
+ isert_unmap_data_buf(isert_conn, &wr->data);
- return 0;
-
-unmap_sg:
- ib_dma_unmap_sg(ib_dev, sg_start, sg_nents,
- (wr->iser_ib_op == ISER_IB_RDMA_WRITE) ?
- DMA_TO_DEVICE : DMA_FROM_DEVICE);
return ret;
}
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index 708a069..18cb2e4 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -55,18 +55,26 @@ struct fast_reg_descriptor {
bool valid;
};
+struct isert_data_buf {
+ struct scatterlist *sg;
+ int nents;
+ u32 sg_off;
+ u32 len; /* cur_rdma_length */
+ u32 offset;
+ unsigned int dma_nents;
+ enum dma_data_direction dma_dir;
+};
+
struct isert_rdma_wr {
struct list_head wr_list;
struct isert_cmd *isert_cmd;
enum iser_ib_op_code iser_ib_op;
struct ib_sge *ib_sge;
struct ib_sge s_ib_sge;
- int num_sge;
- struct scatterlist *sge;
int send_wr_num;
struct ib_send_wr *send_wr;
struct ib_send_wr s_send_wr;
- u32 cur_rdma_length;
+ struct isert_data_buf data;
struct fast_reg_descriptor *fr_desc;
};
--
1.7.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v1 10/12] IB/isert: Initialize T10-PI resources
2014-02-16 17:38 [PATCH v1 00/12] T10-DIF Initial support for iSER target Sagi Grimberg
` (5 preceding siblings ...)
2014-02-16 17:38 ` [PATCH v1 08/12] Target/sbc: add debug print Sagi Grimberg
@ 2014-02-16 17:38 ` Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 12/12] IB/isert: Support T10-PI protected transactions Sagi Grimberg
2014-02-18 20:05 ` [PATCH v1 00/12] T10-DIF Initial support for iSER target Nicholas A. Bellinger
8 siblings, 0 replies; 17+ messages in thread
From: Sagi Grimberg @ 2014-02-16 17:38 UTC (permalink / raw
To: nab; +Cc: target-devel, roland, linux-rdma, ogerlitz, oren
Introduce pi_context to hold relevant RDMA protection resources.
We eliminate data_key_valid boolean and replace it with indicators
container to indicate:
- Is the descriptor protected (registered via signature MR)
- Is the data_mr key valid (can spare LOCAL_INV WR)
- Is the prot_mr key valid (can spare LOCAL_INV WR)
- Is the sig_mr key valid (can spare LOCAL_INV WR)
Upon connection establishment check if network portal is T10-PI
enabled and allocate T10-PI resources if necessary, allocate
signature enabled memory regions and mark connection queue-pair
as signature enabled.
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
drivers/infiniband/ulp/isert/ib_isert.c | 104 +++++++++++++++++++++++++++----
drivers/infiniband/ulp/isert/ib_isert.h | 23 ++++++-
2 files changed, 110 insertions(+), 17 deletions(-)
diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 299df4d..64d5f06 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -87,7 +87,8 @@ isert_query_device(struct ib_device *ib_dev, struct ib_device_attr *devattr)
}
static int
-isert_conn_setup_qp(struct isert_conn *isert_conn, struct rdma_cm_id *cma_id)
+isert_conn_setup_qp(struct isert_conn *isert_conn, struct rdma_cm_id *cma_id,
+ u8 protection)
{
struct isert_device *device = isert_conn->conn_device;
struct ib_qp_init_attr attr;
@@ -119,6 +120,8 @@ isert_conn_setup_qp(struct isert_conn *isert_conn, struct rdma_cm_id *cma_id)
attr.cap.max_recv_sge = 1;
attr.sq_sig_type = IB_SIGNAL_REQ_WR;
attr.qp_type = IB_QPT_RC;
+ if (protection)
+ attr.create_flags |= IB_QP_CREATE_SIGNATURE_EN;
pr_debug("isert_conn_setup_qp cma_id->device: %p\n",
cma_id->device);
@@ -236,13 +239,18 @@ isert_create_device_ib_res(struct isert_device *device)
device->unreg_rdma_mem = isert_unmap_cmd;
}
+ /* Check signature cap */
+ device->pi_capable = dev_attr->device_cap_flags &
+ IB_DEVICE_SIGNATURE_HANDOVER ? true : false;
+
device->cqs_used = min_t(int, num_online_cpus(),
device->ib_device->num_comp_vectors);
device->cqs_used = min(ISERT_MAX_CQ, device->cqs_used);
pr_debug("Using %d CQs, device %s supports %d vectors support "
- "Fast registration %d\n",
+ "Fast registration %d pi_capable %d\n",
device->cqs_used, device->ib_device->name,
- device->ib_device->num_comp_vectors, device->use_fastreg);
+ device->ib_device->num_comp_vectors, device->use_fastreg,
+ device->pi_capable);
device->cq_desc = kzalloc(sizeof(struct isert_cq_desc) *
device->cqs_used, GFP_KERNEL);
if (!device->cq_desc) {
@@ -395,6 +403,12 @@ isert_conn_free_fastreg_pool(struct isert_conn *isert_conn)
list_del(&fr_desc->list);
ib_free_fast_reg_page_list(fr_desc->data_frpl);
ib_dereg_mr(fr_desc->data_mr);
+ if (fr_desc->pi_ctx) {
+ ib_free_fast_reg_page_list(fr_desc->pi_ctx->prot_frpl);
+ ib_dereg_mr(fr_desc->pi_ctx->prot_mr);
+ ib_destroy_mr(fr_desc->pi_ctx->sig_mr);
+ kfree(fr_desc->pi_ctx);
+ }
kfree(fr_desc);
++i;
}
@@ -406,8 +420,10 @@ isert_conn_free_fastreg_pool(struct isert_conn *isert_conn)
static int
isert_create_fr_desc(struct ib_device *ib_device, struct ib_pd *pd,
- struct fast_reg_descriptor *fr_desc)
+ struct fast_reg_descriptor *fr_desc, u8 protection)
{
+ int ret;
+
fr_desc->data_frpl = ib_alloc_fast_reg_page_list(ib_device,
ISCSI_ISER_SG_TABLESIZE);
if (IS_ERR(fr_desc->data_frpl)) {
@@ -420,19 +436,73 @@ isert_create_fr_desc(struct ib_device *ib_device, struct ib_pd *pd,
if (IS_ERR(fr_desc->data_mr)) {
pr_err("Failed to allocate data frmr err=%ld\n",
PTR_ERR(fr_desc->data_mr));
- ib_free_fast_reg_page_list(fr_desc->data_frpl);
- return PTR_ERR(fr_desc->data_mr);
+ ret = PTR_ERR(fr_desc->data_mr);
+ goto err_data_frpl;
}
pr_debug("Create fr_desc %p page_list %p\n",
fr_desc, fr_desc->data_frpl->page_list);
+ fr_desc->ind |= ISERT_DATA_KEY_VALID;
- fr_desc->valid = true;
+ if (protection) {
+ struct ib_mr_init_attr mr_init_attr = {0};
+ struct pi_context *pi_ctx;
+
+ fr_desc->pi_ctx = kzalloc(sizeof(*fr_desc->pi_ctx), GFP_KERNEL);
+ if (!fr_desc->pi_ctx) {
+ pr_err("Failed to allocate pi context\n");
+ ret = -ENOMEM;
+ goto err_data_mr;
+ }
+ pi_ctx = fr_desc->pi_ctx;
+
+ pi_ctx->prot_frpl = ib_alloc_fast_reg_page_list(ib_device,
+ ISCSI_ISER_SG_TABLESIZE);
+ if (IS_ERR(pi_ctx->prot_frpl)) {
+ pr_err("Failed to allocate prot frpl err=%ld\n",
+ PTR_ERR(pi_ctx->prot_frpl));
+ ret = PTR_ERR(pi_ctx->prot_frpl);
+ goto err_pi_ctx;
+ }
+
+ pi_ctx->prot_mr = ib_alloc_fast_reg_mr(pd, ISCSI_ISER_SG_TABLESIZE);
+ if (IS_ERR(pi_ctx->prot_mr)) {
+ pr_err("Failed to allocate prot frmr err=%ld\n",
+ PTR_ERR(pi_ctx->prot_mr));
+ ret = PTR_ERR(pi_ctx->prot_mr);
+ goto err_prot_frpl;
+ }
+ fr_desc->ind |= ISERT_PROT_KEY_VALID;
+
+ mr_init_attr.max_reg_descriptors = 2;
+ mr_init_attr.flags |= IB_MR_SIGNATURE_EN;
+ pi_ctx->sig_mr = ib_create_mr(pd, &mr_init_attr);
+ if (IS_ERR(pi_ctx->sig_mr)) {
+ pr_err("Failed to allocate signature enabled mr err=%ld\n",
+ PTR_ERR(pi_ctx->sig_mr));
+ ret = PTR_ERR(pi_ctx->sig_mr);
+ goto err_prot_mr;
+ }
+ fr_desc->ind |= ISERT_SIG_KEY_VALID;
+ }
+ fr_desc->ind &= ~ISERT_PROTECTED;
return 0;
+err_prot_mr:
+ ib_dereg_mr(fr_desc->pi_ctx->prot_mr);
+err_prot_frpl:
+ ib_free_fast_reg_page_list(fr_desc->pi_ctx->prot_frpl);
+err_pi_ctx:
+ kfree(fr_desc->pi_ctx);
+err_data_mr:
+ ib_dereg_mr(fr_desc->data_mr);
+err_data_frpl:
+ ib_free_fast_reg_page_list(fr_desc->data_frpl);
+
+ return ret;
}
static int
-isert_conn_create_fastreg_pool(struct isert_conn *isert_conn)
+isert_conn_create_fastreg_pool(struct isert_conn *isert_conn, u8 pi_support)
{
struct fast_reg_descriptor *fr_desc;
struct isert_device *device = isert_conn->conn_device;
@@ -449,7 +519,8 @@ isert_conn_create_fastreg_pool(struct isert_conn *isert_conn)
}
ret = isert_create_fr_desc(device->ib_device,
- isert_conn->conn_pd, fr_desc);
+ isert_conn->conn_pd, fr_desc,
+ pi_support);
if (ret) {
pr_err("Failed to create fastreg descriptor err=%d\n",
ret);
@@ -480,6 +551,7 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
struct isert_device *device;
struct ib_device *ib_dev = cma_id->device;
int ret = 0;
+ u8 pi_support = np->tpg_np->tpg->tpg_attrib.t10_pi;
pr_debug("Entering isert_connect_request cma_id: %p, context: %p\n",
cma_id, cma_id->context);
@@ -570,8 +642,14 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
goto out_mr;
}
+ if (pi_support && !device->pi_capable) {
+ pr_err("Protection information requested but not supported\n");
+ ret = -EINVAL;
+ goto out_mr;
+ }
+
if (device->use_fastreg) {
- ret = isert_conn_create_fastreg_pool(isert_conn);
+ ret = isert_conn_create_fastreg_pool(isert_conn, pi_support);
if (ret) {
pr_err("Conn: %p failed to create fastreg pool\n",
isert_conn);
@@ -579,7 +657,7 @@ isert_connect_request(struct rdma_cm_id *cma_id, struct rdma_cm_event *event)
}
}
- ret = isert_conn_setup_qp(isert_conn, cma_id);
+ ret = isert_conn_setup_qp(isert_conn, cma_id, pi_support);
if (ret)
goto out_conn_dev;
@@ -2227,7 +2305,7 @@ isert_fast_reg_mr(struct isert_conn *isert_conn,
pagelist_len = isert_map_fr_pagelist(ib_dev, mem->sg, mem->nents,
&fr_desc->data_frpl->page_list[0]);
- if (!fr_desc->valid) {
+ if (!(fr_desc->ind & ISERT_DATA_KEY_VALID)) {
memset(&inv_wr, 0, sizeof(inv_wr));
inv_wr.opcode = IB_WR_LOCAL_INV;
inv_wr.ex.invalidate_rkey = fr_desc->data_mr->rkey;
@@ -2259,7 +2337,7 @@ isert_fast_reg_mr(struct isert_conn *isert_conn,
pr_err("fast registration failed, ret:%d\n", ret);
return ret;
}
- fr_desc->valid = false;
+ fr_desc->ind &= ~ISERT_DATA_KEY_VALID;
sge->lkey = fr_desc->data_mr->lkey;
sge->addr = fr_desc->data_frpl->page_list[0] + page_off;
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index 18cb2e4..d984172 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -48,11 +48,25 @@ struct iser_tx_desc {
struct ib_send_wr send_wr;
} __packed;
+enum isert_indicator {
+ ISERT_PROTECTED = 1 << 0,
+ ISERT_DATA_KEY_VALID = 1 << 1,
+ ISERT_PROT_KEY_VALID = 1 << 2,
+ ISERT_SIG_KEY_VALID = 1 << 3,
+};
+
+struct pi_context {
+ struct ib_mr *prot_mr;
+ struct ib_fast_reg_page_list *prot_frpl;
+ struct ib_mr *sig_mr;
+};
+
struct fast_reg_descriptor {
- struct list_head list;
- struct ib_mr *data_mr;
- struct ib_fast_reg_page_list *data_frpl;
- bool valid;
+ struct list_head list;
+ struct ib_mr *data_mr;
+ struct ib_fast_reg_page_list *data_frpl;
+ u8 ind;
+ struct pi_context *pi_ctx;
};
struct isert_data_buf {
@@ -148,6 +162,7 @@ struct isert_cq_desc {
struct isert_device {
int use_fastreg;
+ bool pi_capable;
int cqs_used;
int refcount;
int cq_active_qps[ISERT_MAX_CQ];
--
1.7.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v1 11/12] IB/isert: Accept RDMA_WRITE completions
[not found] ` <1392572297-5333-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (3 preceding siblings ...)
2014-02-16 17:38 ` [PATCH v1 09/12] IB/isert: Introduce isert_map/unmap_data_buf Sagi Grimberg
@ 2014-02-16 17:38 ` Sagi Grimberg
4 siblings, 0 replies; 17+ messages in thread
From: Sagi Grimberg @ 2014-02-16 17:38 UTC (permalink / raw
To: nab-IzHhD5pYlfBP7FQvKIMDCQ
Cc: target-devel-u79uwXL29TY76Z2rM5mHXA,
roland-DgEjT+Ai2ygdnm+yROfE0A, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
ogerlitz-VPRAkNaXOzVWk0Htik3J/w, oren-VPRAkNaXOzVWk0Htik3J/w
In case of protected transactions, we will need to check the
protection status of the transaction before sending SCSI response.
So be ready for RDMA_WRITE completions. currently we don't ask
for these completions, but for T10-PI we will.
Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/ulp/isert/ib_isert.c | 19 +++++++++++++++++--
1 files changed, 17 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 64d5f06..50c7948 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -51,6 +51,8 @@ isert_unreg_rdma(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn);
static int
isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
struct isert_rdma_wr *wr);
+static int
+isert_put_response(struct iscsi_conn *conn, struct iscsi_cmd *cmd);
static void
isert_qp_event_callback(struct ib_event *e, void *context)
@@ -1652,6 +1654,18 @@ isert_completion_put(struct iser_tx_desc *tx_desc, struct isert_cmd *isert_cmd,
}
static void
+isert_completion_rdma_write(struct iser_tx_desc *tx_desc,
+ struct isert_cmd *isert_cmd)
+{
+ struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd;
+ struct isert_conn *isert_conn = isert_cmd->conn;
+ struct isert_device *device = isert_conn->conn_device;
+
+ device->unreg_rdma_mem(isert_cmd, isert_conn);
+ isert_put_response(isert_conn->conn, cmd);
+}
+
+static void
isert_completion_rdma_read(struct iser_tx_desc *tx_desc,
struct isert_cmd *isert_cmd)
{
@@ -1771,8 +1785,9 @@ __isert_send_completion(struct iser_tx_desc *tx_desc,
isert_conn, ib_dev);
break;
case ISER_IB_RDMA_WRITE:
- pr_err("isert_send_completion: Got ISER_IB_RDMA_WRITE\n");
- dump_stack();
+ pr_debug("isert_send_completion: Got ISER_IB_RDMA_WRITE\n");
+ atomic_dec(&isert_conn->post_send_buf_count);
+ isert_completion_rdma_write(tx_desc, isert_cmd);
break;
case ISER_IB_RDMA_READ:
pr_debug("isert_send_completion: Got ISER_IB_RDMA_READ:\n");
--
1.7.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v1 12/12] IB/isert: Support T10-PI protected transactions
2014-02-16 17:38 [PATCH v1 00/12] T10-DIF Initial support for iSER target Sagi Grimberg
` (6 preceding siblings ...)
2014-02-16 17:38 ` [PATCH v1 10/12] IB/isert: Initialize T10-PI resources Sagi Grimberg
@ 2014-02-16 17:38 ` Sagi Grimberg
2014-02-18 20:05 ` [PATCH v1 00/12] T10-DIF Initial support for iSER target Nicholas A. Bellinger
8 siblings, 0 replies; 17+ messages in thread
From: Sagi Grimberg @ 2014-02-16 17:38 UTC (permalink / raw
To: nab; +Cc: target-devel, roland, linux-rdma, ogerlitz, oren
In case the Target core passed transport T10 protection
operation:
1. Register data buffer (data memory region)
2. Register protection buffer if exsists (prot memory region)
3. Register signature region (signature memory region)
- use work request IB_WR_REG_SIG_MR
4. Execute RDMA
5. Upon RDMA completion check the signature status
- if succeeded send good SCSI response
- if failed send SCSI bad response with appropriate sense buffer
Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
---
drivers/infiniband/ulp/isert/ib_isert.c | 328 ++++++++++++++++++++++++++++---
drivers/infiniband/ulp/isert/ib_isert.h | 1 +
2 files changed, 299 insertions(+), 30 deletions(-)
diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 50c7948..1eb719a 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -1552,6 +1552,10 @@ isert_unreg_rdma(struct isert_cmd *isert_cmd, struct isert_conn *isert_conn)
if (wr->fr_desc) {
pr_debug("unreg_fastreg_cmd: %p free fr_desc %p\n",
isert_cmd, wr->fr_desc);
+ if (wr->fr_desc->ind & ISERT_PROTECTED) {
+ isert_unmap_data_buf(isert_conn, &wr->prot);
+ wr->fr_desc->ind &= ~ISERT_PROTECTED;
+ }
spin_lock_bh(&isert_conn->conn_lock);
list_add_tail(&wr->fr_desc->list, &isert_conn->conn_fr_pool);
spin_unlock_bh(&isert_conn->conn_lock);
@@ -1657,12 +1661,55 @@ static void
isert_completion_rdma_write(struct iser_tx_desc *tx_desc,
struct isert_cmd *isert_cmd)
{
+ struct isert_rdma_wr *wr = &isert_cmd->rdma_wr;
struct iscsi_cmd *cmd = isert_cmd->iscsi_cmd;
+ struct se_cmd *se_cmd = &cmd->se_cmd;
struct isert_conn *isert_conn = isert_cmd->conn;
struct isert_device *device = isert_conn->conn_device;
+ struct ib_mr_status mr_status;
+ int ret = 0;
+ if (wr->fr_desc && wr->fr_desc->ind & ISERT_PROTECTED) {
+ ret = ib_check_mr_status(wr->fr_desc->pi_ctx->sig_mr,
+ IB_MR_CHECK_SIG_STATUS, &mr_status);
+ if (ret) {
+ pr_err("ib_check_mr_status failed, ret %d\n", ret);
+ goto fail_mr_status;
+ }
+ if (mr_status.fail_status & IB_MR_CHECK_SIG_STATUS) {
+ u32 block_size = se_cmd->se_dev->dev_attrib.block_size;
+
+ switch (mr_status.sig_err.err_type) {
+ case IB_SIG_BAD_GUARD:
+ se_cmd->pi_err = TCM_LOGICAL_BLOCK_GUARD_CHECK_FAILED;
+ break;
+ case IB_SIG_BAD_REFTAG:
+ se_cmd->pi_err = TCM_LOGICAL_BLOCK_REF_TAG_CHECK_FAILED;
+ break;
+ case IB_SIG_BAD_APPTAG:
+ se_cmd->pi_err = TCM_LOGICAL_BLOCK_APP_TAG_CHECK_FAILED;
+ break;
+ }
+ se_cmd->bad_sector =
+ mr_status.sig_err.sig_err_offset / block_size;
+
+ pr_err("isert: PI error found type %d at sector 0x%llx "
+ "expected 0x%x vs actual 0x%x\n",
+ mr_status.sig_err.err_type,
+ (unsigned long long)se_cmd->bad_sector,
+ mr_status.sig_err.expected,
+ mr_status.sig_err.actual);
+ ret = 1;
+ }
+ }
+
+fail_mr_status:
device->unreg_rdma_mem(isert_cmd, isert_conn);
- isert_put_response(isert_conn->conn, cmd);
+ if (ret)
+ transport_send_check_condition_and_sense(se_cmd,
+ se_cmd->pi_err, 0);
+ else
+ isert_put_response(isert_conn->conn, cmd);
}
static void
@@ -1674,7 +1721,43 @@ isert_completion_rdma_read(struct iser_tx_desc *tx_desc,
struct se_cmd *se_cmd = &cmd->se_cmd;
struct isert_conn *isert_conn = isert_cmd->conn;
struct isert_device *device = isert_conn->conn_device;
+ struct ib_mr_status mr_status;
+ int ret;
+ if (wr->fr_desc && wr->fr_desc->ind & ISERT_PROTECTED) {
+ ret = ib_check_mr_status(wr->fr_desc->pi_ctx->sig_mr,
+ IB_MR_CHECK_SIG_STATUS, &mr_status);
+ if (ret) {
+ pr_err("ib_check_mr_status failed, ret %d\n", ret);
+ goto fail_mr_status;
+ }
+ if (mr_status.fail_status & IB_MR_CHECK_SIG_STATUS) {
+ u32 block_size = se_cmd->se_dev->dev_attrib.block_size;
+
+ switch (mr_status.sig_err.err_type) {
+ case IB_SIG_BAD_GUARD:
+ se_cmd->pi_err = TCM_LOGICAL_BLOCK_GUARD_CHECK_FAILED;
+ break;
+ case IB_SIG_BAD_REFTAG:
+ se_cmd->pi_err = TCM_LOGICAL_BLOCK_REF_TAG_CHECK_FAILED;
+ break;
+ case IB_SIG_BAD_APPTAG:
+ se_cmd->pi_err = TCM_LOGICAL_BLOCK_APP_TAG_CHECK_FAILED;
+ break;
+ }
+ se_cmd->bad_sector =
+ mr_status.sig_err.sig_err_offset / block_size;
+
+ pr_err("isert: PI error found type %d at sector 0x%llx "
+ "expected 0x%x vs actual 0x%x\n",
+ mr_status.sig_err.err_type,
+ (unsigned long long)se_cmd->bad_sector,
+ mr_status.sig_err.expected,
+ mr_status.sig_err.actual);
+ }
+ }
+
+fail_mr_status:
iscsit_stop_dataout_timer(cmd);
device->unreg_rdma_mem(isert_cmd, isert_conn);
cmd->write_data_done = wr->data.len;
@@ -2296,9 +2379,12 @@ static int
isert_fast_reg_mr(struct isert_conn *isert_conn,
struct fast_reg_descriptor *fr_desc,
struct isert_data_buf *mem,
+ enum isert_indicator ind,
struct ib_sge *sge)
{
struct ib_device *ib_dev = isert_conn->conn_cm_id->device;
+ struct ib_mr *mr;
+ struct ib_fast_reg_page_list *frpl;
struct ib_send_wr fr_wr, inv_wr;
struct ib_send_wr *bad_wr, *wr = NULL;
int ret, pagelist_len;
@@ -2309,37 +2395,50 @@ isert_fast_reg_mr(struct isert_conn *isert_conn,
sge->lkey = isert_conn->conn_mr->lkey;
sge->addr = ib_sg_dma_address(ib_dev, &mem->sg[0]);
sge->length = ib_sg_dma_len(ib_dev, &mem->sg[0]);
+ pr_debug("%s:%d sge: addr: 0x%llx length: %u lkey: %x\n",
+ __func__, __LINE__, sge->addr, sge->length,
+ sge->lkey);
return 0;
}
+ if (ind == ISERT_DATA_KEY_VALID) {
+ /* Registering data buffer */
+ mr = fr_desc->data_mr;
+ frpl = fr_desc->data_frpl;
+ } else {
+ /* Registering protection buffer */
+ mr = fr_desc->pi_ctx->prot_mr;
+ frpl = fr_desc->pi_ctx->prot_frpl;
+
+ }
+
page_off = mem->offset % PAGE_SIZE;
pr_debug("Use fr_desc %p sg_nents %d offset %u\n",
fr_desc, mem->nents, mem->offset);
pagelist_len = isert_map_fr_pagelist(ib_dev, mem->sg, mem->nents,
- &fr_desc->data_frpl->page_list[0]);
+ &frpl->page_list[0]);
if (!(fr_desc->ind & ISERT_DATA_KEY_VALID)) {
memset(&inv_wr, 0, sizeof(inv_wr));
inv_wr.opcode = IB_WR_LOCAL_INV;
- inv_wr.ex.invalidate_rkey = fr_desc->data_mr->rkey;
+ inv_wr.ex.invalidate_rkey = mr->rkey;
wr = &inv_wr;
/* Bump the key */
- key = (u8)(fr_desc->data_mr->rkey & 0x000000FF);
- ib_update_fast_reg_key(fr_desc->data_mr, ++key);
+ key = (u8)(mr->rkey & 0x000000FF);
+ ib_update_fast_reg_key(mr, ++key);
}
/* Prepare FASTREG WR */
memset(&fr_wr, 0, sizeof(fr_wr));
fr_wr.opcode = IB_WR_FAST_REG_MR;
- fr_wr.wr.fast_reg.iova_start =
- fr_desc->data_frpl->page_list[0] + page_off;
- fr_wr.wr.fast_reg.page_list = fr_desc->data_frpl;
+ fr_wr.wr.fast_reg.iova_start = frpl->page_list[0] + page_off;
+ fr_wr.wr.fast_reg.page_list = frpl;
fr_wr.wr.fast_reg.page_list_len = pagelist_len;
fr_wr.wr.fast_reg.page_shift = PAGE_SHIFT;
fr_wr.wr.fast_reg.length = mem->len;
- fr_wr.wr.fast_reg.rkey = fr_desc->data_mr->rkey;
+ fr_wr.wr.fast_reg.rkey = mr->rkey;
fr_wr.wr.fast_reg.access_flags = IB_ACCESS_LOCAL_WRITE;
if (!wr)
@@ -2352,25 +2451,154 @@ isert_fast_reg_mr(struct isert_conn *isert_conn,
pr_err("fast registration failed, ret:%d\n", ret);
return ret;
}
- fr_desc->ind &= ~ISERT_DATA_KEY_VALID;
+ fr_desc->ind &= ~ind;
- sge->lkey = fr_desc->data_mr->lkey;
- sge->addr = fr_desc->data_frpl->page_list[0] + page_off;
+ sge->lkey = mr->lkey;
+ sge->addr = frpl->page_list[0] + page_off;
sge->length = mem->len;
- pr_debug("RDMA ib_sge: addr: 0x%16llx length: %u lkey: %08x\n",
- sge->addr, sge->length, sge->lkey);
+ pr_debug("%s:%d sge: addr: 0x%llx length: %u lkey: %x\n",
+ __func__, __LINE__, sge->addr, sge->length,
+ sge->lkey);
return ret;
}
static int
+isert_set_sig_attrs(struct se_cmd *se_cmd, struct ib_sig_attrs *sig_attrs)
+{
+ sig_attrs->mem.sig_type = IB_SIG_TYPE_T10_DIF;
+ sig_attrs->wire.sig_type = IB_SIG_TYPE_T10_DIF;
+ sig_attrs->mem.sig.dif.pi_interval =
+ se_cmd->se_dev->dev_attrib.block_size;
+ sig_attrs->wire.sig.dif.pi_interval =
+ se_cmd->se_dev->dev_attrib.block_size;
+
+ switch (se_cmd->prot_op) {
+ case TARGET_PROT_DIN_INSERT:
+ case TARGET_PROT_DOUT_STRIP:
+ sig_attrs->mem.sig.dif.type = IB_T10DIF_NONE;
+ sig_attrs->wire.sig.dif.type = se_cmd->prot_type;
+ sig_attrs->wire.sig.dif.bg_type = IB_T10DIF_CRC;
+ sig_attrs->wire.sig.dif.bg = 0;
+ sig_attrs->wire.sig.dif.app_tag = 0;
+ sig_attrs->wire.sig.dif.ref_tag = se_cmd->reftag_seed;
+ break;
+ case TARGET_PROT_DOUT_INSERT:
+ case TARGET_PROT_DIN_STRIP:
+ sig_attrs->mem.sig.dif.type = se_cmd->prot_type;
+ sig_attrs->mem.sig.dif.bg_type = IB_T10DIF_CRC;
+ sig_attrs->mem.sig.dif.bg = 0;
+ sig_attrs->mem.sig.dif.app_tag = 0;
+ sig_attrs->mem.sig.dif.ref_tag = se_cmd->reftag_seed;
+ sig_attrs->wire.sig.dif.type = IB_T10DIF_NONE;
+ break;
+ case TARGET_PROT_DIN_PASS:
+ case TARGET_PROT_DOUT_PASS:
+ sig_attrs->mem.sig.dif.type = se_cmd->prot_type;
+ sig_attrs->mem.sig.dif.bg_type = IB_T10DIF_CRC;
+ sig_attrs->mem.sig.dif.bg = 0;
+ sig_attrs->mem.sig.dif.app_tag = 0;
+ sig_attrs->mem.sig.dif.ref_tag = se_cmd->reftag_seed;
+ sig_attrs->wire.sig.dif.type = se_cmd->prot_type;
+ sig_attrs->wire.sig.dif.bg_type = IB_T10DIF_CRC;
+ sig_attrs->wire.sig.dif.bg = 0;
+ sig_attrs->wire.sig.dif.app_tag = 0;
+ sig_attrs->wire.sig.dif.ref_tag = se_cmd->reftag_seed;
+ break;
+ default:
+ pr_err("Unsupported PI operation %d\n", se_cmd->prot_op);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static inline u8
+isert_set_prot_checks(u8 prot_checks)
+{
+ return (prot_checks & TARGET_DIF_CHECK_GUARD ? 0xc0 : 0) |
+ (prot_checks & TARGET_DIF_CHECK_REFTAG ? 0x30 : 0) |
+ (prot_checks & TARGET_DIF_CHECK_REFTAG ? 0x0f : 0);
+}
+
+static int
+isert_reg_sig_mr(struct isert_conn *isert_conn, struct se_cmd *se_cmd,
+ struct fast_reg_descriptor *fr_desc,
+ struct ib_sge *data_sge, struct ib_sge *prot_sge,
+ struct ib_sge *sig_sge)
+{
+ struct ib_send_wr sig_wr, inv_wr;
+ struct ib_send_wr *bad_wr, *wr = NULL;
+ struct pi_context *pi_ctx = fr_desc->pi_ctx;
+ struct ib_sig_attrs sig_attrs = {0};
+ int ret;
+ u32 key;
+
+ ret = isert_set_sig_attrs(se_cmd, &sig_attrs);
+ if (ret)
+ goto err;
+
+ sig_attrs.check_mask = isert_set_prot_checks(se_cmd->prot_checks);
+
+ if (!(fr_desc->ind & ISERT_SIG_KEY_VALID)) {
+ memset(&inv_wr, 0, sizeof(inv_wr));
+ inv_wr.opcode = IB_WR_LOCAL_INV;
+ inv_wr.ex.invalidate_rkey = pi_ctx->sig_mr->rkey;
+ wr = &inv_wr;
+ /* Bump the key */
+ key = (u8)(pi_ctx->sig_mr->rkey & 0x000000FF);
+ ib_update_fast_reg_key(pi_ctx->sig_mr, ++key);
+ }
+
+ memset(&sig_wr, 0, sizeof(sig_wr));
+ sig_wr.opcode = IB_WR_REG_SIG_MR;
+ sig_wr.sg_list = data_sge;
+ sig_wr.num_sge = 1;
+ sig_wr.wr.sig_handover.access_flags = IB_ACCESS_LOCAL_WRITE;
+ sig_wr.wr.sig_handover.sig_attrs = &sig_attrs;
+ sig_wr.wr.sig_handover.sig_mr = pi_ctx->sig_mr;
+ if (se_cmd->t_prot_sg)
+ sig_wr.wr.sig_handover.prot = prot_sge;
+
+ if (!wr)
+ wr = &sig_wr;
+ else
+ wr->next = &sig_wr;
+
+ ret = ib_post_send(isert_conn->conn_qp, wr, &bad_wr);
+ if (ret) {
+ pr_err("fast registration failed, ret:%d\n", ret);
+ goto err;
+ }
+ fr_desc->ind &= ~ISERT_SIG_KEY_VALID;
+
+ sig_sge->lkey = pi_ctx->sig_mr->lkey;
+ sig_sge->addr = 0;
+ sig_sge->length = se_cmd->data_length;
+ if (se_cmd->prot_op != TARGET_PROT_DIN_STRIP &&
+ se_cmd->prot_op != TARGET_PROT_DOUT_INSERT)
+ /*
+ * We have protection guards on the wire
+ * so we need to set a larget transfer
+ */
+ sig_sge->length += se_cmd->prot_length;
+
+ pr_debug("sig_sge: addr: 0x%llx length: %u lkey: %x\n",
+ sig_sge->addr, sig_sge->length,
+ sig_sge->lkey);
+err:
+ return ret;
+}
+
+static int
isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
struct isert_rdma_wr *wr)
{
struct se_cmd *se_cmd = &cmd->se_cmd;
struct isert_cmd *isert_cmd = iscsit_priv_cmd(cmd);
struct isert_conn *isert_conn = conn->context;
+ struct ib_sge data_sge;
struct ib_send_wr *send_wr;
struct fast_reg_descriptor *fr_desc = NULL;
u32 offset;
@@ -2386,7 +2614,8 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
if (ret)
return ret;
- if (wr->data.dma_nents != 1) {
+ if (wr->data.dma_nents != 1 ||
+ se_cmd->prot_type != TARGET_PROT_NORMAL) {
spin_lock_irqsave(&isert_conn->conn_lock, flags);
fr_desc = list_first_entry(&isert_conn->conn_fr_pool,
struct fast_reg_descriptor, list);
@@ -2395,10 +2624,39 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
wr->fr_desc = fr_desc;
}
- ret = isert_fast_reg_mr(isert_conn, fr_desc, &wr->data, &wr->s_ib_sge);
+ ret = isert_fast_reg_mr(isert_conn, fr_desc, &wr->data,
+ ISERT_DATA_KEY_VALID, &data_sge);
if (ret)
goto unmap_cmd;
+ if (se_cmd->prot_type != TARGET_PROT_NORMAL) {
+ struct ib_sge prot_sge, sig_sge;
+
+ if (se_cmd->t_prot_sg) {
+ ret = isert_map_data_buf(isert_conn, isert_cmd,
+ se_cmd->t_prot_sg,
+ se_cmd->t_prot_nents,
+ se_cmd->prot_length,
+ 0, wr->iser_ib_op, &wr->prot);
+ if (ret)
+ goto unmap_cmd;
+
+ ret = isert_fast_reg_mr(isert_conn, fr_desc, &wr->prot,
+ ISERT_PROT_KEY_VALID, &prot_sge);
+ if (ret)
+ goto unmap_prot_cmd;
+ }
+
+ ret = isert_reg_sig_mr(isert_conn, se_cmd, fr_desc,
+ &data_sge, &prot_sge, &sig_sge);
+ if (ret)
+ goto unmap_prot_cmd;
+
+ fr_desc->ind |= ISERT_PROTECTED;
+ memcpy(&wr->s_ib_sge, &sig_sge, sizeof(sig_sge));
+ } else
+ memcpy(&wr->s_ib_sge, &data_sge, sizeof(data_sge));
+
wr->ib_sge = &wr->s_ib_sge;
wr->send_wr_num = 1;
memset(&wr->s_send_wr, 0, sizeof(*send_wr));
@@ -2413,8 +2671,8 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
send_wr->opcode = IB_WR_RDMA_WRITE;
send_wr->wr.rdma.remote_addr = isert_cmd->read_va;
send_wr->wr.rdma.rkey = isert_cmd->read_stag;
- send_wr->send_flags = 0;
- send_wr->next = &isert_cmd->tx_desc.send_wr;
+ send_wr->send_flags = se_cmd->prot_type == TARGET_PROT_NORMAL ?
+ 0 : IB_SEND_SIGNALED;
} else {
send_wr->opcode = IB_WR_RDMA_READ;
send_wr->wr.rdma.remote_addr = isert_cmd->write_va;
@@ -2423,6 +2681,9 @@ isert_reg_rdma(struct iscsi_conn *conn, struct iscsi_cmd *cmd,
}
return 0;
+unmap_prot_cmd:
+ if (se_cmd->t_prot_sg)
+ isert_unmap_data_buf(isert_conn, &wr->prot);
unmap_cmd:
if (fr_desc) {
spin_lock_irqsave(&isert_conn->conn_lock, flags);
@@ -2454,26 +2715,33 @@ isert_put_datain(struct iscsi_conn *conn, struct iscsi_cmd *cmd)
return rc;
}
- /*
- * Build isert_conn->tx_desc for iSCSI response PDU and attach
- */
- isert_create_send_desc(isert_conn, isert_cmd, &isert_cmd->tx_desc);
- iscsit_build_rsp_pdu(cmd, conn, true, (struct iscsi_scsi_rsp *)
- &isert_cmd->tx_desc.iscsi_header);
- isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
- isert_init_send_wr(isert_conn, isert_cmd,
- &isert_cmd->tx_desc.send_wr, true);
+ if (se_cmd->prot_type == TARGET_PROT_NORMAL) {
+ /*
+ * Build isert_conn->tx_desc for iSCSI response PDU and attach
+ */
+ isert_create_send_desc(isert_conn, isert_cmd,
+ &isert_cmd->tx_desc);
+ iscsit_build_rsp_pdu(cmd, conn, false, (struct iscsi_scsi_rsp *)
+ &isert_cmd->tx_desc.iscsi_header);
+ isert_init_tx_hdrs(isert_conn, &isert_cmd->tx_desc);
+ isert_init_send_wr(isert_conn, isert_cmd,
+ &isert_cmd->tx_desc.send_wr, true);
+ isert_cmd->rdma_wr.s_send_wr.next = &isert_cmd->tx_desc.send_wr;
+ }
atomic_inc(&isert_conn->post_send_buf_count);
-
rc = ib_post_send(isert_conn->conn_qp, wr->send_wr, &wr_failed);
if (rc) {
pr_warn("ib_post_send() failed for IB_WR_RDMA_WRITE\n");
atomic_dec(&isert_conn->post_send_buf_count);
}
- pr_debug("Cmd: %p posted RDMA_WRITE + Response for iSER Data READ\n",
- isert_cmd);
+ if (se_cmd->prot_type == TARGET_PROT_NORMAL)
+ pr_debug("Cmd: %p posted RDMA_WRITE + Response for iSER Data "
+ "READ\n", isert_cmd);
+ else
+ pr_debug("Cmd: %p posted RDMA_WRITE for iSER Data READ\n",
+ isert_cmd);
return 1;
}
diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h
index d984172..0e2253e 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.h
+++ b/drivers/infiniband/ulp/isert/ib_isert.h
@@ -89,6 +89,7 @@ struct isert_rdma_wr {
struct ib_send_wr *send_wr;
struct ib_send_wr s_send_wr;
struct isert_data_buf data;
+ struct isert_data_buf prot;
struct fast_reg_descriptor *fr_desc;
};
--
1.7.1
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH v1 01/12] Target/transport: Allocate protection sg if needed
[not found] ` <1392572297-5333-2-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2014-02-18 19:49 ` Nicholas A. Bellinger
0 siblings, 0 replies; 17+ messages in thread
From: Nicholas A. Bellinger @ 2014-02-18 19:49 UTC (permalink / raw
To: Sagi Grimberg
Cc: target-devel-u79uwXL29TY76Z2rM5mHXA,
roland-DgEjT+Ai2ygdnm+yROfE0A, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
ogerlitz-VPRAkNaXOzVWk0Htik3J/w, oren-VPRAkNaXOzVWk0Htik3J/w
On Sun, 2014-02-16 at 19:38 +0200, Sagi Grimberg wrote:
> In case protection information is involved, allocate
> protection SG-list for transport.
>
> Signed-off-by: Sagi Grimberg <sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> ---
> drivers/target/target_core_transport.c | 12 ++++++++++++
> 1 files changed, 12 insertions(+), 0 deletions(-)
>
> diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
> index 24b4f65..f029bb7 100644
> --- a/drivers/target/target_core_transport.c
> +++ b/drivers/target/target_core_transport.c
> @@ -2036,6 +2036,10 @@ static inline void transport_free_pages(struct se_cmd *cmd)
> transport_free_sgl(cmd->t_bidi_data_sg, cmd->t_bidi_data_nents);
> cmd->t_bidi_data_sg = NULL;
> cmd->t_bidi_data_nents = 0;
> +
> + transport_free_sgl(cmd->t_prot_sg, cmd->t_prot_nents);
> + cmd->t_prot_sg = NULL;
> + cmd->t_prot_nents = 0;
> }
>
> /**
> @@ -2199,6 +2203,14 @@ transport_generic_new_cmd(struct se_cmd *cmd)
> return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
> }
>
> + if (cmd->prot_type != TARGET_PROT_NORMAL) {
> + ret = target_alloc_sgl(&cmd->t_prot_sg,
> + &cmd->t_prot_nents,
> + cmd->prot_length, zero_flag);
> + if (ret < 0)
> + return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
> + }
> +
> ret = target_alloc_sgl(&cmd->t_data_sg, &cmd->t_data_nents,
> cmd->data_length, zero_flag);
> if (ret < 0)
So the 'zero_flag' for the t_prot_sg allocation should always be true to
avoid garbage in the app-tag field..
--nab
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH v1 02/12] Target/sbc: Set protection operation and relevant checks
2014-02-16 17:38 ` [PATCH v1 02/12] Target/sbc: Set protection operation and relevant checks Sagi Grimberg
@ 2014-02-18 19:53 ` Nicholas A. Bellinger
0 siblings, 0 replies; 17+ messages in thread
From: Nicholas A. Bellinger @ 2014-02-18 19:53 UTC (permalink / raw
To: Sagi Grimberg; +Cc: target-devel, roland, linux-rdma, ogerlitz, oren
On Sun, 2014-02-16 at 19:38 +0200, Sagi Grimberg wrote:
> SBC-3 mandates the protection checks that must be
> performed in the rdprotect/wrprotect field. Use them.
> According to backstore device pi_attributes and
> cdb rdprotect/wrprotect.
>
> Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
> ---
> drivers/target/target_core_sbc.c | 88 ++++++++++++++++++++++++++++++++++---
> include/target/target_core_base.h | 7 +++
> 2 files changed, 88 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/target/target_core_sbc.c b/drivers/target/target_core_sbc.c
> index a448944..dfeb1c2 100644
> --- a/drivers/target/target_core_sbc.c
> +++ b/drivers/target/target_core_sbc.c
> @@ -240,6 +240,11 @@ static inline u32 transport_lba_32(unsigned char *cdb)
> return (cdb[2] << 24) | (cdb[3] << 16) | (cdb[4] << 8) | cdb[5];
> }
>
> +static inline u8 rwprotect(unsigned char *cdb)
> +{
> + return cdb[1] >> 5;
> +}
> +
No need for this one-liner. Go ahead and inline instead..
> static inline unsigned long long transport_lba_64(unsigned char *cdb)
> {
> unsigned int __v1, __v2;
> @@ -569,30 +574,95 @@ sbc_compare_and_write(struct se_cmd *cmd)
> return TCM_NO_SENSE;
> }
>
> +static int
> +sbc_set_prot_op_checks(u8 protect, enum target_prot_type prot_type,
> + u8 op, struct se_cmd *cmd)
> +{
> + switch (op) {
> + case READ_10:
> + case READ_12:
> + case READ_16:
> + cmd->prot_op = protect ? TARGET_PROT_DIN_PASS :
> + TARGET_PROT_DIN_STRIP;
> + switch (protect) {
> + case 0x0:
> + case 0x1:
> + case 0x5:
> + cmd->prot_checks = TARGET_DIF_CHECK_GUARD;
> + if (prot_type == TARGET_DIF_TYPE1_PROT)
> + cmd->prot_checks |= TARGET_DIF_CHECK_REFTAG;
> + break;
> + case 0x2:
> + if (prot_type == TARGET_DIF_TYPE1_PROT)
> + cmd->prot_checks = TARGET_DIF_CHECK_REFTAG;
> + break;
> + case 0x3:
> + cmd->prot_checks = 0;
> + break;
> + case 0x4:
> + cmd->prot_checks = TARGET_DIF_CHECK_GUARD;
> + break;
> + default:
> + pr_err("Unsupported protect field %d\n", protect);
> + return -EINVAL;
> + }
> + break;
> + case WRITE_10:
> + case WRITE_12:
> + case WRITE_16:
> + case WRITE_VERIFY:
> + cmd->prot_op = protect ? TARGET_PROT_DOUT_PASS :
> + TARGET_PROT_DOUT_INSERT;
> + switch (protect) {
> + case 0x0:
> + case 0x3:
> + cmd->prot_checks = 0;
> + case 0x1:
> + case 0x5:
> + cmd->prot_checks = TARGET_DIF_CHECK_GUARD;
> + if (prot_type == TARGET_DIF_TYPE1_PROT)
> + cmd->prot_checks |= TARGET_DIF_CHECK_REFTAG;
> + break;
> + case 0x2:
> + if (prot_type == TARGET_DIF_TYPE1_PROT)
> + cmd->prot_checks = TARGET_DIF_CHECK_REFTAG;
> + break;
> + case 0x4:
> + cmd->prot_checks = TARGET_DIF_CHECK_GUARD;
> + break;
> + default:
> + pr_err("Unsupported protect field %d\n", protect);
> + return -EINVAL;
> + }
> + break;
> + default:
> + pr_err("ERROR: bad opcode %d\n", op);
> + return TCM_INVALID_CDB_FIELD;
> + }
> +
> + return 0;
> +}
> +
sbc_set_prot_op_checks() is probably better served by passing in a bool
to determine read/write, than performing a switch on individual CDB
opcode.
--nab
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH v1 04/12] Target/sbc: don't return from sbc_check for non prot_sg
2014-02-16 17:38 ` [PATCH v1 04/12] Target/sbc: don't return from sbc_check for non prot_sg Sagi Grimberg
@ 2014-02-18 19:55 ` Nicholas A. Bellinger
0 siblings, 0 replies; 17+ messages in thread
From: Nicholas A. Bellinger @ 2014-02-18 19:55 UTC (permalink / raw
To: Sagi Grimberg; +Cc: target-devel, roland, linux-rdma, ogerlitz, oren
On Sun, 2014-02-16 at 19:38 +0200, Sagi Grimberg wrote:
> For transports which use generic new command these
> buffers have yet to be allocated. Instead check afterwards
> if command required prot buffers but none are provided.
>
> Also this way, target may support protection information
> against legacy initiators (writes are inserted and reads
> are stripped).
>
> Signed-off-by: Sagi Grimberg <sagig@mellanox.com>
> ---
> drivers/target/target_core_sbc.c | 3 ---
> drivers/target/target_core_transport.c | 21 +++++++++++++++++----
> 2 files changed, 17 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/target/target_core_sbc.c b/drivers/target/target_core_sbc.c
> index dfeb1c2..3f12726 100644
> --- a/drivers/target/target_core_sbc.c
> +++ b/drivers/target/target_core_sbc.c
> @@ -649,9 +649,6 @@ sbc_check_prot(struct se_device *dev, struct se_cmd *cmd, unsigned char *cdb,
> {
> u8 protect = rwprotect(cdb);
>
> - if (!cmd->t_prot_sg || !cmd->t_prot_nents)
> - return true;
> -
> switch (dev->dev_attrib.pi_prot_type) {
> case TARGET_DIF_TYPE3_PROT:
> cmd->reftag_seed = 0xffffffff;
> diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
> index f029bb7..3a5550a 100644
> --- a/drivers/target/target_core_transport.c
> +++ b/drivers/target/target_core_transport.c
> @@ -1365,6 +1365,13 @@ int target_submit_cmd_map_sgls(struct se_cmd *se_cmd, struct se_session *se_sess
> target_put_sess_cmd(se_sess, se_cmd);
> return 0;
> }
> +
> + rc = target_setup_cmd_from_cdb(se_cmd, cdb);
> + if (rc != 0) {
> + transport_generic_request_failure(se_cmd, rc);
> + return 0;
> + }
> +
> /*
> * Save pointers for SGLs containing protection information,
> * if present.
> @@ -1374,11 +1381,17 @@ int target_submit_cmd_map_sgls(struct se_cmd *se_cmd, struct se_session *se_sess
> se_cmd->t_prot_nents = sgl_prot_count;
> }
>
> - rc = target_setup_cmd_from_cdb(se_cmd, cdb);
> - if (rc != 0) {
> - transport_generic_request_failure(se_cmd, rc);
> - return 0;
> + /*
> + * Fail if protection operation requiers protection
> + * information buffers but None are provided!
> + */
> + if ((!se_cmd->t_prot_sg || !se_cmd->t_prot_nents) &&
> + (se_cmd->prot_op != TARGET_PROT_NORMAL)) {
> + pr_err("ERROR: protection information was requested but "
> + "protection buffers weren't provided.\n");
> + return -EINVAL;
> }
A failure here requires the call to transport_generic_request_failure()
to complete the request with CHECK_CONDITION status, and should return 0
following the failure case for target_setup_cmd_from_cdb() above..
--nab
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH v1 00/12] T10-DIF Initial support for iSER target
2014-02-16 17:38 [PATCH v1 00/12] T10-DIF Initial support for iSER target Sagi Grimberg
` (7 preceding siblings ...)
2014-02-16 17:38 ` [PATCH v1 12/12] IB/isert: Support T10-PI protected transactions Sagi Grimberg
@ 2014-02-18 20:05 ` Nicholas A. Bellinger
8 siblings, 0 replies; 17+ messages in thread
From: Nicholas A. Bellinger @ 2014-02-18 20:05 UTC (permalink / raw
To: Sagi Grimberg; +Cc: target-devel, roland, linux-rdma, ogerlitz, oren
On Sun, 2014-02-16 at 19:38 +0200, Sagi Grimberg wrote:
> Hey Nic and folks,
>
> This patchset introduces target side T10-PI offload support over
> RDMA. Currently the implementation is for iSER transport but can
> be easily extended to SRP (or FCoE in the future).
>
> Should mention that this patchset depends on RDMA signature
> verbs making it for inclusion which will hopefully happen in
> near future.
>
> This code was tested against iSER legacy initiator, and also
> initiator that uses T10-PI offload as well. I'll clean up the
> initiator code in the following days and submit that as well.
>
Looking forward to seeing the initiator code in action.
> This code works under medium loads using backstores:
> - FileIO with DIF emulation.
> - RD with DIF emulation.
> - iBLOCK (scsi_debug with DIF support).
>
> Nic, Can you take this code to rdma_dif branch until RDMA
> Signature verbs set will pass submission? (after review of
> course...).
>
Ok, a few mostly minor comments on this series. Please address these
and repost.
Also, target-pending/rdma-dif has been updated to v3.14-rc2 from the
last RDMA Signature verbs patch series. Please take a moment to verify
that it's update to date.
Thanks!
--nab
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2014-02-18 20:05 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-02-16 17:38 [PATCH v1 00/12] T10-DIF Initial support for iSER target Sagi Grimberg
[not found] ` <1392572297-5333-1-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2014-02-16 17:38 ` [PATCH v1 01/12] Target/transport: Allocate protection sg if needed Sagi Grimberg
[not found] ` <1392572297-5333-2-git-send-email-sagig-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2014-02-18 19:49 ` Nicholas A. Bellinger
2014-02-16 17:38 ` [PATCH v1 02/12] Target/sbc: Set protection operation and relevant checks Sagi Grimberg
2014-02-18 19:53 ` Nicholas A. Bellinger
2014-02-16 17:38 ` [PATCH v1 07/12] Target/configfs: Expose iSCSI network portal group T10-PI support Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 09/12] IB/isert: Introduce isert_map/unmap_data_buf Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 11/12] IB/isert: Accept RDMA_WRITE completions Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 03/12] Target/file: place escape values for protection information format Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 04/12] Target/sbc: don't return from sbc_check for non prot_sg Sagi Grimberg
2014-02-18 19:55 ` Nicholas A. Bellinger
2014-02-16 17:38 ` [PATCH v1 05/12] Target/core: Remove prot_handover use for now Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 06/12] Target/iscsi: Add T10-PI indication for iscsi_portal_group Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 08/12] Target/sbc: add debug print Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 10/12] IB/isert: Initialize T10-PI resources Sagi Grimberg
2014-02-16 17:38 ` [PATCH v1 12/12] IB/isert: Support T10-PI protected transactions Sagi Grimberg
2014-02-18 20:05 ` [PATCH v1 00/12] T10-DIF Initial support for iSER target Nicholas A. Bellinger
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.